Use of scorecards in hospitals
Systematic review
|Updated
Download
Key message
Background
Different tools for performing 'evidence based' leadership has been implemented in hospitals to meet the challenges of modern hospital management. Balanced Scorecard (BSC) and EFQM Excellence-Model (EFQM) are among such tools. The key mission of these is to provide a system for evaluating the organisation from a number of predefined perspectives. The Norwegian Knowledge Centre for the Health Services was asked to conduct a systematic review to assess whether implementation of these tools provide better strategies, better management or better hospital quality.
Methods
A systematic literature review with a defined search strategy, predefined criteria for selecting studies. Quality assessment, data extraction and summary of results were performed by two researchers independent of each other.
Results
We retrieved 639 possibly relevant publications, 71 articles were obtained in full text and five studies were finally included. Four studies reported experiences from implementing BSC and one study reported on EFQM. Included studies described that scorecards may be useful at a local level to define strategic aims, measure quality indicators, and define lower levels for quality action, compare departments over time or to evaluate implementation of new treatment options. We found no evidence that implementation of BSC or EFQM influenced hospital management or quality.
- Three studies reported results after implementing BSC at department level in hospitals in different areas: emergency unit, anaesthesia department and nephrological department. . These studies described development of criteria and indicators for measuring, but are not able to relate results to implementation of BSC or EFQM.
- Two studies reported results on the institutional level. One study used EFQM to evaluate the implementation of evidence based treatment processes in an addiction centre in the Netherlands. Another study compared one hospital in Japan with a Chinese hospital with indicators within the BSC model. Both studies assessed the models as useful for measuring quality.
Conclusion
Research on the usefulness of scorecards is especially challenging due to the fact that scorecards may influence on different levels within a hospital and at different time points.
- There is no evidence that BSC or EFQM influence on hospital performance.
- There are descriptive reports on the usefulness for different purposes in local settings.
- Time series with several pre and post measurements would add to our understanding of the usefulness of scorecards.
- More research and development of suited methodology are needed.
Summary
Background
Hospital management has changed from medical authority to professional management based on "New Public Management" (NPM) –philosophy. The background for this is partly economical and partly ideological. One driving force behind NPM is the notion of public sector as being rigid, ineffective and ruled by professional traditions. The NPM claims to be rational assessments based on documentation of relevant results as well as a clear market orientated organisation. An accelerating growth of hospital expenditure as added to the demand for accountancy of activity, benefit and costs. To meet new challenges of management different models and tools for strategic planning and performance measurement have been developed. One of these tools is Balanced Scorecard (BSC) which implies to continuously evaluate the organisation from four different perspectives:
- The customer perspective
- The business process perspective
- The financial perspective
- The learning and growth perspective
The purpose is to include all these perspectives in the management at different levels in hospitals.
A different model with the same purpose was developed by 'the European Foundation of Quality Management' (EFQM). The model was presented as a framework for assessing organisations for "the European Quality Award". The model has nine criteria, five of which are 'enablers' and four are 'results'. The enablers describe what the organisation does, and the results describe what the organisation achieves. Results are caused by the enablers and the enablers may be improved by feedback of results. The nine criteria are:
Enablers: Leadership, Policy and Strategy, People, Partnership and resources, Processes
Results: Customers, People, Society, Key performance results
The Norwegian Knowledge Centre for the Health Services was asked to conduct a systematic review of literature to find whether it is documented that implementation of these tools provide better strategies, better management or better hospital quality.
Method
We searched the Cochrane Library, EPOC, Medline, Embase, ISI og Cinahl databases by a defined search strategy until February 27th 2007 (see attachment (vedlegg) 1). Studies were included that reported results from using BSC or EFQM in hospitals. Retrieved studies were assessed by two persons, and disagreements were resolved by consensus.
Results
We retrieved 639 possibly relevant publications, 71 articles were obtained in full text and five studies were included in this report.
The included studies described that scorecards may be useful at a local level to define strategic aims by, measure quality indicators, define lower levels for quality action, compare departments over time, or to evaluate implementation of new treatment options. We found no evidence of better management or hospital quality.
Three studies reported results after implementing BSC at department level in hospitals. One study described implementation of BSC in an emergency unit in a Taiwan hospital. Results reported after six months were: improved patient satisfaction, reduced number of complaints, and delay of laboratory tests. Participation in internal education as well as personnel satisfaction improved, and hospital income increased. (All differences p=<0.001) This study was a small uncontrolled before- after study with only one measuring point before and after implementation of BSC. This study concluded that BSC may help structuring quality improvement by providing a framework of four dimensions as guidelines for locally defined indicators.
Another study reported how an anaesthesiology department in a university hospital in Switzerland used the BSC model to develop quality indicators and action criteria for low quality levels. The study did not report follow up results, and thus were not able to assess whether defining quality indicators and responding to low quality improved performance on the BSC dimensions. The authors aimed at comparing results over time as well as comparing them to other departments in the hospital.
A third study described the process of defining local objectives within the four dimensions of BSC in a nephrological department in a public centre in USA. The study had elements of before- and after design. Results were reported qualitatively after six months.
Use of scorecards at institutional level One study used EFQM to evaluate implementation of evidence based treatment processes in an addiction centre in Amsterdam. The evaluating tool was the nine EFQM criteria. The study reported results from 1994 and 2004. Comparison was made on all nine criteria which have remained unchanged in the ten year period. The scoring system however, was changed in 1999. The design was two cross-sectional studies which were adjusted to be comparable.
One study compared one hospital in Japan with a similar hospital in China by developing indicators within the BSC model. The specific indicators were adjusted to what data were available. Significant differences were found between the countries, particularly in the field of internal processes. China used more resources on material investments while Japan used more on human resources (wages). Personnel turnover was 1.3%, in China and 6.9% in Japan: (p=<0.001). Differences were also found in efficiency measured by number of out clinic consultations/ physician/year, number of emergency consultations/ physician. The authors conclude that BSC is a useful framework for comparing hospital across countries.
Comment
The five included studies show that strategic tools like scorecards may be useful by defining areas to be measured. There is a question as to whether such tools should be regarded as methods for defining strategic aims and measure quality, rather than an intervention for quality improvement itself.
- There is no evidence that BSC or EFQM influence on hospital performance.
- There are descriptive reports on the usefulness for different purposes in local settings.
- Time series with several pre and post measurements would add to our understanding of the usefulness of scorecards.
- More research and development of suited methodology are needed.