Performance indicators based on data from national quality registries
Report
|Updated
Key message
Refining and expanding the national performance measurement system to include a set of core measures for public reporting, is one explicit goal of the Norwegian government. Calculation of many such measures may require access to patient-level data from e.g. administrative files, medical records or quality registries.
We performed a model-based evaluation to give advice on whether valid, reliable and evidence-based quality indicators may be developed with data from existing national quality registries. There are approximately 30 such registries in Norway, mainly covering hospital care and a range of clinical areas. Diabetes was chosen as a model for the evaluation. We designed the evaluation process in accordance with acknowledged international methods, in order to demonstrate how quality indicators can be developed and tested with a scientific approach and in a transparent manner. Process and outcome measures for diabetes, chosen in ongoing collaborative projects in which the Nordic countries participate (OECD, Nordic Council of Ministers, WHO), as well as the ones used for public reporting in Denmark and Sweden, formed the main basis for our model.
The indicators for diabetes currently in use in the evaluated measurement systems seem to be based on broad professional consensus, as expressed in evidence-based international guidelines and a systematic review prepared by the Danish indicator project. Norwegian clinical guidelines also give support to the validity of these indicators. The set of measures varied somewhat between the systems, probably due to variations in data availability, but also because of differences in purpose and scope of the reporting systems. We also found that the Norwegian quality registries for adult and childhood diabetes enter pertinent information necessary for calculation of all selected measures. However, data from these registers can not be further validated with regard to precision and minimum bias, until earliest in 2009, when both registers have been in operation for one year.
Partly due to legal requirements to obtain patients' permission, and voluntary cooperation from health personnel to enter data manually, we judge data quality in the registries to be suboptimal for indicator report retrieval, until the registries can supplemented and quality assured by linkage with hospital administrative files. Even a better option will be the development of a web-based information technology for interactive data entry from electronic medical records; however this technology will probably not be available in the near future.
In the meantime, we propose to base further development of existing and new indicators for diabetes, on an alternative dataset extracted for the period 2003-2007 from hospital administrative files, linked and matched with laboratory data and relevant public registries. This project will also have relevance for problems which have to be addressed if other quality registries are to be evaluated for indicator report retrieval.
Summary
Background
Quality indicators may be defined as indirect measures of quality within an area and is one way of measuring and monitoring the delivery and quality of health care services. Various audiences may wish to use them to document the quality of care, make comparisons, make judgements and determine priorities, and to support quality improvement and accountability in health care.
Availability of data will often determine which indicators can be selected for public reporting. Calculation of clinical process and outcome indicators may require access to patient-level data, e.g. from administrative files, medical records or quality registries. There are around 30 national clinical quality registers in Norway; the majority covering hospital care and a range of clinical areas. So far, no systematic and comprehensive public reporting of data from these registers has been carried out in order to inform patients about the quality of care and services.
It is important that quality indicators are meaningful, scientifically sound, generalisable, and interpretable. To achieve this, they should be developed and tested in a systematic way and with scientific rigor.
The Directorate for Health and Social Affairs requested in June 2007 the Norwegian Knowledge Centre for the Health Services to evaluate and give advice on which data in the national quality registries may be suitable for comparative public reporting at the hospital level.
Methods
Due to a limited time frame, we performed a model-based evaluation with basis in one specific clinical condition. We designed an evaluation process in accordance with acknowledged international methods, in order to demonstrate how quality indicators can be developed and tested with a scientific approach and in a transparent manner. Such processes are partly characterized by a stepwise selection in which formalised consensus processes are supposed to make trade-offs between scientific, professional, as well as value based considerations.
Diabetes was chosen as clinical condition, and existing process and outcome measures for diabetes in ongoing collaborative projects in which the Nordic countries participate (OECD, Nordic Council of Ministers, WHO), as well as the ones used for public reporting in Denmark and Sweden, formed the main basis for our assessment.
As a background for the evaluation we also describe how a conceptual framework should form an overarching strategy and define policy-based and scientific criteria for the activities of a national performance measurement and reporting system.
Results
Based on their objectives and the data which are currently collected, a number of the Norwegian quality registries seem to be suited for indicator report retrieval. These registers are to a high degree founded on international standards and aim to contribute to the adoption and adherence of evidence-based guidelines in routine clinical practice.
Partly due to legal requirements to obtain patients' permission and voluntary cooperation from health personnel to enter data manually, many of the registries are vulnerable with regard to data quality. New legislation recently passed will probably contribute to improved quality assurance of the registries by permitting linkage with hospital administrative files, but even this may not solve all problems. We therefore judge data from the registries to be suboptimal for indicator report retrieval until a web-based information technology for interactive data entry from electronic medical records is developed. However, such technology will probably not be available in the near future.
With regard to diabetes, this condition has been prioritised for quality indicator reporting in all measurement systems evaluated. The set of measures varied somewhat between the systems, probably due to variations in data availability, but also because of differences in purpose and scope of the measurement systems. We found that the Norwegian quality registries for adult and childhood diabetes enter pertinent information for calculation of all selected measures in the systems evaluated.
The indicators currently in use in these systems seem to be based on broad professional consensus, as expressed in evidence-based international guidelines and a systematic review prepared by the Danish indicator project. Norwegian clinical guidelines also give support to the validity of these indicators. We therefore conclude that the chosen indicators seem to comply with the evaluation criteria relevance in the context they were developed; as well as scientific soundness, as supported by the literature.
Before indicators are implemented, however, it is important to assure that they also perform well on tests of precision (to determine the reliability of the indicator for distinguishing real differences in provider performance), as well as of minimum bias (to determine the indicators sensitivity to bias, due to differences in patient severity). We have not addressed these questions further; as such tests must be conducted empirically and/or theoretically. Since the Norwegian quality registries for diabetes will not have data representing both adult and childhood diabetes until earliest in 2009 when both registers have been in operation for one year, empirical testing cannot presently be accomplished.
Recommendations
To ensure the legitimacy, comprehensiveness and quality of a performance measurement system in Norway, we recommend that a conceptual framework is designed, articulating guiding principles and priorities. We propose that the extensive work that OECD has put down in their framework form the foundation for this work and that it is tailored to a Norwegian context. Legal requirements, measurement infrastructure, as well as financial and human resources necessary to develop and support such a system are among the important questions to be addressed.
We also recommend that the evaluation performed here is extended to consider which new indicators may be developed for other diseases or conditions and with data from the national quality registries. In our opinion, this work should start to see which data Norway should provide for the quality indicator projects of OECD and the Nordic Council of Ministers. A next step would be to investigate whether these indicators also may be suited for public reporting at the hospital level.
With regard to the diabetes indicators chosen by the systems, we advice that further development is undertaken in a formalised consensus process, in which the evidence- base collected and presented by the systems form a basis for selection of indicators relevant for a Norwegian context.
Owing to the paucity of relevant data that can be extracted from the registries for diabetes indicators until 2009, we propose to base necessary investigations concerning diabetes indicators on an alternative dataset consisting of "routine data" from hospital administrative files. This project can start immediately, since data can be extracted for the period 2003-2007 and be linked and matched with laboratory data and data from relevant public registries. The project will also have relevance for problems which have to be addressed if other quality registries are to be evaluated for indicator report retrieval.