Indicators are one of many sources to knowledge about patient safety. Which indicators we choose to use, determine the picture we are painting. How these choices are made and documented, are important sources of information when the results of the indicators shall be interpreted and comprehended.
Many nations and international organizations use one or another form for formalized consensus processes in the development and choice of quality indicators for the health services. In this report we examine the principle behind the formalized consensus processes and describes one American method and two European tools which have been used for this purpose.
In Norway, there are little or no tradition for conducting formalized consensus processes when choosing indicators within the health services. By building on the knowledge from other land and organizations within this field, we can implant this method by relative simple means. Developing medical quality registries and national indicators can profit from implementing formal consensus processes. There will be an extra cost by introducing this technique, but in the long term, the knowledge base will be sounder and less time used on developing and validating the indicators.
Documentation of consensus processes provide extensive information about each single indicator and how it is considered within the medical environment. Strength and weaknesses for the indicators are documented, and are important background information for interpreting the results when the indicators are used. High cost for data collection may lead to exclusion of indicators, this is important knowledge to get the whole picture of information about sets of indicators. Transparency about indicator selection and the consequences of them are important factors in improving the understanding and use of quality indicators for the health services.
Knowledge about patient safety arises from many sources. Patient safety indicators may provide systematic knowledge about certain aspects of Patient Safety. Before selecting indicators several dimensions should be considered: How to choose indicators? Who will choose the indicators to be implemented? Which criteria for documentation should give directions for the decision-making processes?
This report describes three different consensus processes and discusses what is best suited for choosing patient safety indicators for Norway.
Formalized consensus process
Formalized consensus process can in short be described as a three stage evaluation process. Characteristic of the standardized consensus process is a systematic approach, with a clear description of standards for the documentation of the knowledge base and criteria to for the evaluation, followed by documentation of both processes and outcomes.
First step in a consensus processes is identification and obtaining documentation for the indicator. Professional researchers with a competence in systematic literature review will normally be in charge of providing an overview of existing relevant evidence, methods and standards. Consensus processes will often also include studies that do not fulfil the same strict method criteria as evidence based literature reviews do.
A panel of experts evaluates validity and feasibility of the indicator based on the available documentation. Access to data is considered together with validity and reliability for each indicator. The burden of data collection must be considered alongside the utility value of the indicator. This evaluation is the second stage of the consensus process.
Composition of the expert group is an important factor. The choice of experts can be narrow within one medical speciality or area of health science, or wider, were experts are chosen from different medical specialities and areas of health science.
Expert groups rarely meet at the very first evaluation of the indicators. Documents are distributed to the experts together with a description of the further process for the evaluation of the indicator. The documentation should include a systematic literature review, list of indicators, definition of indicators and instructions. The evaluation is conducted by a survey whereupon indicators are scored based and different measures of validity and access to data specified for the indicator.
The third and last stage in the process is to gather the group of experts to follow through the final evaluation of the indicators. First is a gathering of the professionals with a presentation of the results from the survey, followed by plenary discussions and a new questionnaire evaluation of the indicators. The purpose of the consensus process is not to force a common consensus, but to document the degree of agreement/disagreement regarding the evaluation of the indicators. Indicators that reach a specified level of agreement may be chosen and indicators with a clear degree of disagreement are excluded.
One American method and two European tools
This report provide an introduction to one American method for conducting consensus processes. It is the Rand/UCLA Appropriateness Method which is recognized in work conducted for the Agency for Healthcare Research and Quality in the US, and also is at the base for the formal consensus processes in the National Quality Forum (NQF) in the US. The two European tools are Stepwise Assessment Indicator Framework Approach (SAIFA ) which have been presented and used in a Patient Safety project in Europe, and the other is Appraisal of Indicators through Research and Evaluation (AIRE) developed in the Netherlands. These tools are chosen on the basis that they are specially directed towards the area of indicator selection and evaluation.
Area of interest
Within health services research and administration, formalized consensus processes are conducted when there is a weaker knowledgebase or contradictory information within a field/area. Conducting formalized consensus processes provide a documentation of agreement as well as disagreement. Documentation on the evaluation of data access and validity are valuable information for the future.
The methods imply a professional preparation of a conductor, which administrates and documents the processes. The disadvantage of conducting these processes is that they are relative time and resource demanding. The advantage is the systematic and thorough work as a base for indicators that should provide information about patient safety. The choice is documented and make visible why some aspects are chosen and others not. Because of the relative resource demanding method, it is best suited for selection of indicators on higher levels, national or international.
One solution is to implement simplified versions on a local level. There are available documentation of indicators within most medical specialities or health science areas. By relying solely on already available documentation one can skip the initial phases and move straight to the evaluation of individual indicators. When developing sets of indicators the knowledge base should be properly quality assured.
The method is described for small expert groups, but is in some settings used for large-scale assessment. Use of large-scale assessment may be considered better suited for identifying areas for monitoring, rather than used for evaluating single indicators or sets of indicators.
In Norway, there are little or no tradition for conducting formalized consensus processes when choosing indicators within the health services. By combining knowledge of the methods for consensus processes based on the Rand/UCLA Appropriate method with the tools from the two European initiatives, we have a basis for documenting both development and choice of patient safety indicators for Norway in a proper way.
There is a lot of information about development of indicators, and less of the effect of then in development and implementation strategies for quality improvement. We believe that by a systematic evaluation and quality assurance of the indicators in relation to the context they operate within, they are valuable contributions to quality improvement strategies.
This work is a product of an independent working group and the views presented are the views of this group.
Norwegian Knowledge Centre for the Health Services summarizes and disseminates evidence concerning the effect of treatments, methods, and interventions in health services, in addition to monitoring health service quality. Our goal is to support good decision making in order to provide patients in Norway with the best possible care. The Centre is organized under The Directorate of Health, but is scientifically and professionally independent. The Centre has no authority to develop health policy or responsibility to implement policies.