Improving the use of research evidence in guideline development
Mapping review
|Updated
Summary
In 2005 the World Health Organisation (WHO) asked its Advisory Committee on Health Research (ACHR) for advice on ways in which WHO can improve the use of research evidence in the development of recommendations, including guidelines and policies. The ACHR established the Subcommittee on the Use of Research Evidence (SURE) to collect background documentation and consult widely among WHO staff, international experts and end users of WHO recommendations to inform its advice to WHO. This document summarizes the work that has been prepared as a series of reviews of methods that are used in the development of guidelines. The reviews have been published in Health Research Policy and Systems covering following topics for guideline development: the content of guidelines for guidelines; Priority setting; Group composition and consultation process; Managing conflicts of interest in guidance development; Group processes; Determining which outcomes are important for recommendation; Deciding what evidence to include in guidelines; Synthesis and presentation of evidence for guideline development; Grading evidence and recommendations for guidelines; Integrating values and consumer involvement in guidelines; Incorporating considerations of cost-effectiveness in guidelines; affordability and resource implications of guideline development; Incorporating considerations of equity in guideline development; Adaptation, applicability and transferability of guidelines; reporting guidelines; Disseminating and implementing guidelines; and Guideline evaluation. The following questions and advice resulted from the reviews:
What have other organizations done to develop guidelines for guidelines from which WHO can learn?
Establish a credible, independent committee that evaluates existing methods for developing guidelines or that updates existing ones. Obtain feedback and approval from various stakeholders during the development process of guidelines for guidelines. Develop a detailed source document (manual) that guideline developers can use as reference material.
What should be the key components of WHO guidelines for guidelines?
Guidelines for guidelines should include information and instructions about the following components: 1) Priority setting; 2) Group composition and consultations; 3) Declaration and avoidance of conflicts of interest; 4) Group processes; 5) Identification of important outcomes; 6) Explicit definition of the questions and eligibility criteria ; 7) Type of study designs for different questions; 8) Identification of evidence; 9) Synthesis and presentation of evidence; 10) Specification and integration of values; 11) Making judgments about desirable and undesirable effects; 12) Taking account of equity; 13) Grading evidence and recommendations; 14) Taking account of costs; 15) Adaptation, applicability, transferability of guidelines; 16) Structure of reports; 17) Methods of peer review; 18) Planned methods of dissemination & implementation; 19) Evaluation of the guidelines.
What have other organizations done to implement guidelines for guidelines from which WHO can learn?
Obtain buy-in from regions and country level representatives for guidelines for guidelines before dissemination of a revised version. Disseminate the guidelines for guidelines widely and make them available (e.g. on the Internet).
Develop examples of guidelines that guideline developers can use as models when applying the guidelines for guidelines. Ensure training sessions for those responsible for developing guidelines. Continue to monitor the methodological literature on guideline development.
What criteria should be used to establish priorities for guidelines?
WHO has limited resources and capacity to develop recommendations. It should use these resources where it has the greatest chance of improving health, equity, and efficient use of healthcare resources.
We suggest the following criteria for establishing priorities for developing recommendations based on WHO's aims and strategic advantages:
Problems associated with a high burden of illness in low and middle-income countries, or new and emerging diseases.
No existing recommendations of good quality.
The feasibility of developing recommendations that will improve health outcomes, reduce inequities or reduce unnecessary costs if they are implemented. Implementation is feasible, will not exhaustively use available resources, and barriers to change are not likely to be so high that they cannot be overcome.
Additional priorities for WHO include interventions that will likely require system changes and interventions where there might be a conflict in choices between individual and societal perspectives.
What processes should be used to agree on priorities for guidelines?
The allocation of resources to the development of recommendations should be part of the routine budgeting process rather than a separate exercise. Criteria for establishing priorities should be applied using a systematic and transparent process.
Because data to inform judgments are often lacking, unmeasured factors should also be considered - explicitly and transparently.
The process should include consultation with potential end users and other stakeholders, including the public, using well-constructed questions, and possibly using Delphi-like procedures.
Groups that include stakeholders and people with relevant types of expertise should make decisions. Group processes should ensure full participation by all members of the group.
The process used to select topics should be documented and open to inspection.
Should WHO have a centralised or decentralised process?
Both centralized and decentralized processes should be used. Decentralised processes can be considered as separate "tracks".
Separate tracks should be used for considering issues for specific areas, populations, conditions or concerns. The rationales for designating special tracks should be defined clearly; i.e. why they warrant special consideration.
Updating of guidelines could also be considered as a separate "track", taking account of issues such as the need for corrections and the availability of new evidence.
What should be the composition of a WHO-panel that is set up to develop recommendations?
Groups that develop guidelines or recommendations should be broadly composed and include important stakeholders such as consumers, health professionals that work within the relevant area, and managers or policy makers.
Groups should include or have access to individuals with the necessary technical skills, including information retrieval, systematic reviewing, health economics, group facilitation, project management, writing and editing.
Groups should include or have access to content experts.
To work well a group needs an effective leader, capable of guiding the group in terms of the task and process, and capable of facilitating collaboration and balanced contribution from all of the group members.
Because many group members will not be familiar with the methods and processes that are used in developing recommendations, groups should be offered training and support to help ensure understanding and facilitate active participation.
What groups should be consulted when a panel is being set up?
We did not identify methodological research that addressed this question, but based on logical arguments and the experience of other organisations we recommend that as many relevant stakeholder groups as practical should be consulted to identify suitable candidates with an appropriate mix of perspectives, technical skills and expertise, as well as to obtain a balanced representation with respect to regions and gender.
What methods should WHO use to ensure appropriate consultations?
Identifying and setting priorities for guidelines and recommendations
Commenting on the scope of the guidelines or recommendations
Commenting on the evidence that is used to inform guidelines or recommendations
Commenting on drafts of the guidelines or recommendations
Commenting on plans for disseminating and supporting the adaptation and implementation of the guidelines or recommendations.
Key stakeholder organisations should be contacted directly whenever possible. Consultation processes should be transparent and should encourage feedback from interested parties.
What is the best way to obtain complete and accurate disclosures on financial ties and other competing interests?
Although there is little empirical evidence to guide the development of disclosure forms, minimal or open-ended formats are likely to be uninformative. We recommend the development of specific, detailed, structured forms that solicit as much information as possible about the nature and extent of the competing interests.
How to determine when a disclosed financial tie or other competing interest constitutes a conflict of interest?
There is no empirical evidence to suggest that explicit criteria are preferable to ad hoc committee decisions when deciding if a disclosed financial tie is a conflict of interest.
However, explicit criteria may make decision-making easier.
When a conflict of interest is identified, how should the conflict be managed?
Descriptive studies suggest that appropriate management strategies are best determined on a case-by-case basis. Thus, WHO should use a wide range of management strategies to address disclosed conflicts of interest, with public disclosure of conflicts associated with each meeting as a minimum and recusal of conflicted individuals as the other extreme.
How could conflict of interest policies be enforced?
Although there are no empirical studies of the enforcement of conflict if interest policies, descriptive studies of other organizations and institutions suggest that WHO convene a standing committee to review all financial disclosure statements prior to the commencement of committee meetings/hearings and to make management recommendations when necessary. A standard policy requiring all financial ties to be made public (i.e., recorded into the meeting minutes) should reduce the number of problematic cases. In instances where the conflicts seem intractable, a recommendation of recusal may be necessary to protect the greater interests of WHO and its constituents.
What should WHO do to ensure appropriate group processes?
Various strategies can be adopted to ensure that the group processes in play when panels are developing recommendations are inclusive, so that all voices can be heard and all arguments given fair weight, including the use of formal consensus development methods, such at the Nominal Group Technique or the Delphi method the selection of a group leader who is qualified and responsible for facilitating an appropriate group process.
What types of evidence should be used to address different types of questions?
The most important type of evidence for informing global recommendations is evidence of the effects of the options (interventions or actions) that are considered in a recommendation. This evidence is essential, but not sufficient for making recommendations about what to do. Other types of required evidence are largely context specific.
The study designs to be included in a review should be dictated by the interventions and outcomes being considered. A decision about how broad a range of study designs to consider should be made in relationship to the characteristics of the interventions being considered, what evidence is available, and the time and resources available.
There is uncertainty regarding what study designs to include for some specific types of questions, particularly for questions regarding population interventions, harmful effects and interventions where there is only limited human evidence.
Decisions about the range of study designs to include should be made explicitly.
Great caution should be taken to avoid confusing a lack of evidence with evidence of no effect, and to acknowledge uncertainty.
Expert opinion is not a type of study design and should not be used as evidence. The evidence (experience or observations) that is the basis of expert opinions should be identified and appraised in a systematic and transparent way.
How should existing systematic reviews be critically appraised?
Because preparing systematic reviews can take over a year and require capacity and resources, existing reviews should be used when possible and updated, if needed.
Standard criteria, such as A MeaSurement Tool to Assess Reviews (AMSTAR), should be used to critically appraise existing systematic reviews, together with an assessment of the relevance of the review to the questions being asked.
When and how should WHO undertake or commission new reviews?
Consideration should be given to undertaking or commissioning a new review whenever a relevant, up-to-date review of good quality is not available.
When time or resources are limited it may be necessary to undertake rapid assessments.
The methods that are used to do these assessments should be reported, including important limitations and uncertainties and explicit consideration of the need and urgency of undertaking a full systematic review.
Because WHO has limited capacity for undertaking systematic reviews, reviews will often need to be commissioned when a new review is needed. Consideration should be given to establishing collaborating centres to undertake or support this work, similar to what some national organisations have done.
How should the findings of systematic reviews be summarised and presented to committees responsible for making recommendations?
Concise summaries (evidence tables) of the best available evidence for each important outcome, including benefits, harms and costs, should be presented to the groups responsible for making recommendations. These should include an assessment of the quality of the evidence and a summary of the findings for each outcome.
The full systematic reviews, on which the summaries are based, should also be available to both those making recommendations and users of the recommendations.
What additional information is needed to inform recommendations and how should this information be synthesised with information about effects and presented to committees?
Additional information that is needed to inform recommendations includes factors that might modify the expected effects, need (prevalence, baseline risk or status), values (the relative importance of key outcomes), costs and the availability of resources.
Any assumptions that are made about values or other factors that may vary from setting to setting should be made explicit.
For global guidelines that are intended to inform decisions in different settings, consideration should be given to using a template to assist the synthesis of information specific to a setting with the global evidence of the effects of the relevant interventions.
Should WHO grade the quality of evidence and the strength of recommendations? Users of recommendations need to know how much confidence they can place in the underlying evidence and the recommendations. The degree of confidence depends on a number of factors and requires complex judgments. These judgments should be made explicitly in WHO recommendations. A systematic and explicit approach to making judgments about the quality of evidence and the strength of recommendations can help to prevent errors, facilitate critical appraisal of these judgments, and can help to improve communication of this information.
What criteria should be used to grade evidence and recommendations?
Both the quality of evidence and the strength of recommendations should be graded. The criteria used to grade the strength of recommendations should include the quality of the underlying evidence, but should not be limited to that.
The approach to grading should be one that has wide international support and is suitable for a wide range of different types of recommendations. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach, which is currently suggested in the Guidelines for WHO Guidelines, is being used by an increasing number of other organizations internationally. It should be used more consistently by WHO. Further developments of this approach should ensure its wide applicability.
Should WHO use the same grading system for all of its recommendations?
Although there are arguments for and against using the same grading system across a wide range of different types of recommendations, WHO should use a uniform grading system to prevent confusion for developers and users of recommendations.
Whose values should WHO use when making recommendations?
Values, the relative importance or worth of a state or consequences of a decision (outcomes relating to benefits, harms, burden and costs), play a role in every recommendation. Ethical considerations, concepts that determine what is right, also play a role.
The values used in making recommendations should reflect those of the people affected. Judgements should be explicit and should be informed by input from those affected (including citizens, patients, clinicians and policy makers).
When differences in values may lead to different decisions or there is uncertainty about values, this should also be explicit. If differences in values are likely to affect a decision, such that people in different setting would likely make different choices about interventions or actions based on differences in their values, global recommendations should be explicit in terms of which values were applied and allow for adaptation after incorporating local values.
How should WHO ensure that appropriate values are integrated in recommendations?
All WHO guideline groups should uniformly apply explicit, transparent and clearly described methods for integrating values.
WHO should consider involving relevant stakeholders if this is feasible and efficient.
WHO should develop a checklist for guidelines panels to help them to ensure that ethical considerations relevant to recommendations are addressed explicitly and transparently.
How should users and consumers be involved in generating recommendations?
Including consumers in groups that are making global recommendations presents major challenges with respect to the impossibility of including a representative spectrum of consumers from a variety of cultures and settings. Nonetheless, consideration should be given to including consumers in groups who are able to challenge assumptions that are made about the values used for making recommendations, rather than represent the values of consumers around the world.
WHO should establish a network to facilitate involvement of users.
Draft recommendations should be reviewed by consumers, who should be asked explicitly to consider the values that were used.
How should values be presented in recommendations?
Recommendations should include a description of how decisions were made about the relative importance of the consequences (benefits, harms and costs) of a decision. Values that influence recommendations should be reported along with the research evidence underlying recommendations.
When differences in values would lead to different decisions or there is important uncertainty about values that are critical to a decision, this should be flagged and reflected in the strength of the recommendation.
Adaptable guideline templates that allow for integration of different values should be developed and used when differences in values are likely to be critical to a decision.
When is it important to incorporate cost-effectiveness, resource implications and affordability considerations in WHO guidelines (which topics)?
For cost-effectiveness:
The need for cost/effectiveness information should be dictated by the specific question, of which several may be addressed in a single guideline. It is proposed that the indications for undertaking a cost-effectiveness analysis (CEA) could be a starting point for determining which recommendation(s) in the guideline would benefit from such analysis.
For resource implications/affordability:
The resource implications of each individual recommendation need to be considered when implementation issues are being discussed.
How can cost-effectiveness, resource implications and affordability be explicitly taken into account in WHO guidelines?
For cost-effectiveness:
If data are available, the ideal time to consider cost-effectiveness is during the evidence gathering and synthesizing stage. However, because of the inconsistent availability of CEAs and the procedural difficulty associated with adjusting results from different CEAs to make them comparable, it is also possible for cost-effectiveness to be considered during the stage of developing recommendations.
Depending on the quantity and quality and relevance of the data available, such data can be considered in a qualitative way or in a quantitative way, ranging from a listing of the costs to a modelling exercise. At the very least, a qualitative approach like a commentary outlining the economic issues that need to be considered is necessary. If a quantitative approach is to be used, the full model should be transparent and comprehensive.
For resource implications/affordability:
Resource implications, including health system changes, for each recommendation in a WHO guideline should be explored. At the minimum, a qualitative description that can serve as a gross indicator of the amount of resources needed, relative to current practice, should be provided.
How does one provide guidance in contextualizing guideline recommendations at the country level based on considerations of cost-effectiveness, resource implications and affordability?
All models should be made available and ideally are designed to allow for analysts to make changes in key parameters and reapply results in their own country. In the global guidelines, scenarios and extensive sensitivity/uncertainty analysis can be applied.
Resource implications for WHO
From the above, it is clear that guidelines development groups will need a health economist. There is need to ensure that this is included in the budget for guidelines and that there is in-house support for this as well.
When and how should inequities be addressed in systematic reviews that are used as background documents for guidelines?
The following question should routinely be considered: Are there plausible reasons for anticipating differential relative effects across disadvantaged and advantaged populations?
If there are plausible reasons for anticipating differential effects, additional evidence should be included in a review to inform judgments about the likelihood of differential effects.
What questions about equity should routinely be addressed by those making recommendations on behalf of WHO?
The following additional questions should routinely be considered: How likely is it that the results of available research are applicable to disadvantaged populations and settings?
How likely are differences in baseline risk that would result in differential absolute effects across disadvantaged and advantaged populations?
How likely is it that there are important differences in trade-offs between the expected benefits and harms across disadvantaged and advantaged populations?
Are there different implications for disadvantaged and advantaged populations, or implications for addressing inequities?
What context specific information is needed to inform adaptation and decision making in a specific setting with regard to impacts on equity?
Those making recommendations on behalf of WHO should routinely consider and offer advice about the importance of the following types of context specific data that might be needed to inform adaptation and decision making in a specific setting:
Effect modifiers for disadvantaged populations and for the likelihood of differential effects
Baseline risk in relationship to social and economic status
Utilization and access to care in relationship to social and economic status Costs in relationship to social and economic status
Ethics and laws that may impact on strategies for addressing inequities
Availability of resources to address inequities
What implementation strategies are likely be needed to ensure that recommendations are implemented equitably?
Organisational changes are likely to be important to address inequities. While it may only be possible to consider these in relationship to specific settings, consideration should be given to how best to provide support for identifying and addressing needs for organisational changes. In countries with pervasive inequities institutional, cultural and political changes may first be needed.
Appropriate indicators of social and economic status should be used to monitor the effects of implementing recommendations on disadvantaged populations and on changes in social and economic status.
Should WHO develop international recommendations?
Resources for developing high quality recommendations are limited. Internationally developed recommendations can facilitate access to and pooling of resources, reduce unnecessary duplication, and involve international scientists.
Priority should be given to international health problems and problems that are important in low and middle-income countries, where these advantages are likely to be greatest.
Factors that influence the transferability of recommendations across different settings should be considered systematically and flagged, including modifying factors, important variation in needs, values, costs and the availability of resources.
What should be done centrally and locally?
The preparation of systematic reviews and evidence profiles should be coordinated centrally, in collaboration with organizations that produce systematic reviews. Centrally developed evidence profiles should be adaptable to specific local circumstances.
Consideration should be given to models that involve central coordination with work being undertaken by centres located throughout the world.
While needs, availability of resources, costs, the presence of modifying factors and values need to be assessed locally, support for undertaking these assessments may be needed to make guidelines applicable.
WHO should provide local support for adapting and implementing recommendations by developing tools, building capacity, learning from international experience, and through international networks that support evidence-informed health policies, such as the Evidence-informed Policy Network (EVIPNet).
How should recommendations be adapted?
WHO should provide detailed guidance for adaptation of international recommendations.
Local adaptation processes should be systematic and transparent, they should involve stakeholders, and they should report the key factors that influence decisions, including those flagged in international guidelines, and the reasons for any modifications that are made.
What standard types of recommendations or reports should WHO use?
WHO should develop standard formats for reporting recommendations to facilitate recognition and use by decision makers for whom the recommendations are intended, and to ensure that all the information needed to judge the quality of a guideline, determine its applicability and, if needed, adapt it, is reported.
WHO should develop standard formats for full systematically developed guidelines that are sponsored by WHO, rapid assessments, and guidelines that are endorsed by WHO.
All three formats should include the same information as full guidelines, indicating explicitly what the group preparing the guideline did not do, as well as the methods that were used.
These formats should be used across clinical, public health and health systems recommendations.
How should recommendations be formulated and reported?
Reports should be structured, using headings that correspond to those suggested by the Conference on Guideline Standardization or similar headings.
The quality of evidence and strength of recommendations should be reported explicitly using a standard approach.
The way in which recommendations are formulated should be adapted to the specific characteristics of a specific guideline.
Urgent attention should be given to developing a template that provides decision makers with the relevant global evidence that is needed to inform a decision and offers practical methods for incorporating the context specific evidence and judgements that are needed.
What should WHO do to disseminate and facilitate the uptake of guidelines?
WHO should choose strategies to implement their guidelines from among those which have been evaluated positively in the published literature on implementation research Because the evidence base is weak and modest to moderate effects, at best, can be anticipated, WHO should promote rigorous evaluations of implementation strategies.
What should be done in relation to implementation at headquarters, by regional offices and in countries?
Adaptation and implementation of WHO guidelines should be done locally, at the national or sub-national level.
WHO headquarters and regional offices should support the development and evaluation of implementation strategies by local authorities.
How should the quality of guidelines or recommendations be appraised?
WHO should put into place processes to ensure that both internal and external review of guidelines is undertaken routinely.
A checklist, such as the AGREE instrument, should be used.
The checklist should be adapted and tested to ensure that it is suitable to the broad range of recommendations that WHO produces, including public health and health policy recommendations, and that it includes questions about equity and other items that are particularly important for WHO guidelines.
When should guidelines or recommendations be updated?
Processes should be put into place to ensure that guidelines are monitored routinely to determine if they are in need of updating.
People who are familiar with the topic, such as Cochrane review groups, should do focused, routine searches for new research that would require revision of the guideline.
Periodic review of guidelines by experts not involved in developing the guidelines should also be considered.
Consideration should be given to establishing guideline panels that are ongoing, to facilitate routine updating, with members serving fixed periods with a rotating membership.
How should the impact of guidelines or recommendations be evaluated?
WHO headquarters and regional offices should support member states and those responsible for policy decisions and implementation to evaluate the impact of their decisions and actions by providing advice regarding impact assessment, practical support and coordination of efforts.
Before-after evaluations should be used cautiously and when there are important uncertainties regarding the effects of a policy or its implementation, randomised evaluations should be used when possible.
What responsibility should WHO take for ensuring that important uncertainties are addressed by future research when the evidence needed to inform recommendations is lacking?
Guideline panels should routinely identify important uncertainties and research priorities. This source of potential priorities for research should be used systematically to inform priority-setting processes for global research.