Effectiveness of and experiences with school focused interventions out-side of school settings for children and youth living in child welfare institutions: A mixed methods systematic review
Systematic review
|Published
The purpose of this systematic review is to investigate the effects of and experiences with educational interventions conducted outside of schools for children and youth living in child welfare institutions.
Key message
Children and youth living in child welfare institutions are one of society’s most vulnerable groups, and on average have poorer school outcomes compared with children who are not in institutional care. The purpose of this systematic review is to investigate the effects of and experiences with educational interventions conducted outside of schools for children and youth living in child welfare institutions.
To identify relevant studies, we searched in international databases for studies published between 2010 and June 2021. We used both manual and machine learning processes to screen studies against our inclusion criteria. Two qualitative studies met our inclusion criteria. We extracted data from the included studies and appraised their methodological limitations. The studies were dissimilar and for a number of reasons a synthesis of their findings unwarranted. The studies were from Spain and described the experiences of youth aged 12-17. One study explored a mentoring program while the other used a series of eight workshops to engage youth in science.
There is limited research on the effects of and experiences with educational interventions conducted outside of schools for children and youth living in child welfare institutions. There is a need for more research, using various study designs, in multiple contexts. If interventions are implemented at this point in time, they should be implemented with a robust plan for evaluation so that they can contribute to the limited body of evidence.
Summary
Introduction
Children and youth living in child welfare institutions are one of society’s most vulnerable groups, and on average have poorer educational outcomes compared with children who are not in institutional care. They are less likely to complete primary education, as well as to start or complete secondary or higher education. The Norwegian Directorate for Children, Youth and Family Affairs (Bufdir) therefore needs to know which educational interventions conducted outside of schools for children and youth living in child welfare institutions can improve educational outcomes, as well as children and youth’s experiences of them. With this information, Bufdir can better support children’s educational outcomes and reduce absenteeism and drop-out. This mixed methods systematic review will, together with other evidence on the topic (namely other projects within Bufdir) help to build the foundation for Bufdir’s further work to improve education situations of children and youth living in institutions. The specific focus of this review is educational interventions that have the objective of strengthening school performance and/or reducing absences and dropouts. These interventions should further be delivered outside of school, for example in institutions, activity clubs or by other organisations.
Objective
The purpose of this systematic review is to investigate the effects of and experiences with educational interventions conducted outside of schools for children and youth living in child welfare institutions.
Method
We conducted this systematic review in compliance with NIPH’s methods handbook for evidence synthesis. The project plan is published on the NIPH website (https://www.fhi.no/cristin-prosjekter/aktiv/effekter-av-og-erfaringer-med-eksterne-skolefokuserte-tiltak-for-barn-og-un/). To identify relevant studies, we searched in international literature databases for studies from high-income countries published between 2010 and June 2021 (to capture primary studies) and 2015-2021 (to capture systematic reviews). We had the following inclusion criteria:
|
Study design |
|
|
Population |
Children and youth aged 12-18 who live in child welfare institutions and attend “ordinary” schools |
|
Intervention |
|
|
Comparison |
|
|
Outcomes (quantitative studies) |
Primary outcome:
Secondary outcomes:
- validated and standardised tests (e.g. California Achievement Test, Stanford Achievement Test, Wide-Range Achievement Test, etc.) - results from national or local tests - school grades
- Social skills Intervention or Rating system (SSIS/SSRS) - Child Behavior Checklist (CBCL) – «Total Problems» combined score - Strength and Difficulties Questionnaire (SDQ) – “Total difficulties” combined score |
|
Phenomenon of interest (qualitative studies) (in prioritised order) |
|
We used both manual and machine learning processes to screen studies against our inclusion criteria. Two qualitative studies met our inclusion criteria. We then extracted data from the included studies and assessed their methodological limitations with an adapted Critical Appraisal Skills Programme (CASP) checklist. The studies explored two different interventions, and in one study, children comprised a minority of the sample. We therefore chose to describe each of the studies and their findings instead of conducting a thematic analysis across the studies.
Results
Two qualitative studies met our inclusion criteria. Both were conducted in Spain with youth aged 12-17. One of the studies explored a mentor program. The authors suggest that mentors play a supplemental role towards the academic achievement of children in residential care. The second study used a series of eight scientific workshops to engage youth with science. The authors report that participants showed an improvement in content knowledge, educational aspirations, and scientific vocations. Those experiences modified their own perceptions of their academic and professional expectations, transforming their vision of their future. It is important to note that we assessed both studies as having serious methodological limitations. There was unclear or poor reporting of context, sampling, author reflexivity, and ethical considerations. Because of this, we cannot be sure of how they were conducted or of how the studies’ findings may have been influenced.
Discussion and conclusion
We did not identify any studies that measured the effect of educational interventions implemented outside of schools for children and youth in residential care. We identified and included two studies that explored experiences with this type of intervention. The studies are from Spain and had youth aged 12-17 as participants. One explored a mentor program and the other a series of extracurricular “scientific workshops”. We assessed the studies as having serious methodological limitations which are worth highlighting. There was unclear or poor reporting of context, sampling, author reflexivity, and ethical considerations. For example, neither discussed collecting informed consent from participants, and neither engaged with how the study or researchers themselves may have influenced participants. In qualitative research with vulnerable groups, it is extremely important that researchers reflect on how the study situation and their own actions as researchers can impact the lives of participants, and the consequences of this on study findings. As we did not carry out a thematic analysis to draw out findings across studies, we had no findings to assess our confidence with using the GRADE CERQual approach.
The two studies that we identified do not provide enough evidence to enable to us to answer our research questions. Our review therefore cannot contribute to the development of new educational interventions delivered outside of school for children living in residential care centres in Norway. Our systematic review shows gaps in the research for evidence of both the effect of and experiences with these interventions. More research is needed that explores experiences and perceptions of these interventions (using qualitative study designs) and the effect of the interventions (using quantitative studies with a comparison group) across age groups, countries and contexts. It is important that interventions implemented in the future are accompanied by a solid evaluation plan to build up and strengthen the evidence base on this important topic.
