Critical assessment of methodological limitations of included studies
Publisert
You will use your assessments during the GRADE CERQual process to help determine the confidence we have in the QES findings.
About this phase
|
What happens before this step |
Have identified and found all relevant full text articles |
|
Description |
To critically appraise the methodological limitations of the included studies |
|
Why is this step important? |
You will use your assessments during the GRADE CERQual process to help determine the confidence we have in the QES findings |
|
Responsible |
Project leader and at least one other team member |
|
Tasks |
Two people independently or through discussion assess the methodological limitations of the included studies and then come to an agreement where the assessments differ. |
|
Tools |
The recommended tool for NIPH is the adapted Critical Appraisal Skills Program (CASP) Checklist (see questions below)(3-5)
|
|
Deliverable |
A methodological assessment |
There is no consensus on the necessity, merit or appropriate approach to appraising the quality of qualitative research. Many QES authors feel that quality appraisal can assist readers in evaluating the credibility of conclusions and can allow decision makers to understand the transferability of the findings. There are over 100 recognised appraisal tools (6). The CAMELOT project based at the NIPH is attempting to look into how we can solve this problem and best assess the methodological quality of included primary qualitative studies.
We believe that it is important to methodologically assess the primary studies included within a QES. However, the results of these assessments are often determined by how well the study is reported. We do not exclude studies based on methodological assessment, but the assessment can influence the confidence we have in the findings they contribute to during the GRADE CERQual assessments.
The recommended tool at NIPH is the adapted version of the CASP checklist. It uses the following eight questions with possible answer of yes/maybe/no and a comment field to elaborate for each of the questions.
- Are the setting/s and context described adequately?
- Is the sampling strategy described and is this appropriate?
- Is the data collection strategy described and justified?
- Is the data analysis described and is this appropriate?
- Are the claims made/findings supported by sufficient evidence?
- Is there evidence of reflexivity?
- Does the study demonstrate sensitivity to ethical concerns?
- Any other concerns?
The following assessment options are used to describe the methodological limitations of the included studies:
- Minor
- Minor to moderate
- Moderate
- Moderate to severe
- Severe
Relevant resources:
Selecting studies and assessing methodological limitations
Noyes J, Booth A, Flemming K, Garside R, Harden A, Lewin S, Pantoja T, Hannes K, Cargo M, Thomas J, Cochrane Qualitative and Implementation Methods Group Guidance Paper 2: Methods for assessing methodological limitations, data extraction and synthesis, and confidence in synthesized qualitative findings, Journal of Clinical Epidemiology (2018), doi: 10.1016/j.jclinepi.2017.06.020.
Munthe-Kaas HM, Glenton C, Lewin S, Noyes J, Booth A. Systematic mapping of existing tools to appraise methodological strengths and limitations of qualitative research: Stage 1 in the development of the CAMELOT tool. BMC Medical Research Methodology (2019) 19:113