Evaluating Afterschool Sports Programming: Design and Methods for Outcomes and Mechanisms

Written By: Haley Umans

Project Team: Dr. Tiffany Berry (PI) and Dr. Michelle Sloper (PM), Haley Umans, Yaqiong Wang, Grace Skiles

Evaluation Project: Three-Year Impact Evaluation of LAUSD Beyond the Bell Sports Programs: Linking Quality with Outcomes

Organized sports are the most popular out-of-school (OST) time youth programming.[1] As OST sports programming has grown and gained popularity, the conversation about how to curate quality sports experiences to achieve positive youth development outcomes has been limited.[2] Evaluating OST sports programs creates a unique opportunity to contribute to the conversation by documenting the impact of sports in a local context and providing concrete guidance about how to promote the potential impact of sports to enhance the broader literature base. While evaluation is a powerful tool, engaging in effective and rigorous of evaluations of OST sports programs is challenging. In this blog, I will discuss the design and methods of an evaluation of OST sports conducted by YDEval with the intent to provide inspiration and guidance for your own evaluations of OST sports programs.

YDEval recently completed a multi-year evaluation in collaboration with the LA84 Foundation and Los Angeles Unified School District (LAUSD)’s Beyond the Bell (BTB) afterschool program. We conducted a three-year evaluation of the afterschool sports programming delivered by BTB to over 100 low-resourced middle schools (6th – 8th grade) in Los Angeles, CA. This longitudinal evaluation had two overarching goals: (1) understand how youth benefit from their participation in sports in relation to physical health, social-emotional, and academic outcomes and (2) explore the mechanisms responsible for change in youth sports participants’ outcomes over time. The core findings from the evaluation are discussed in another blog post.

Evaluate Outcomes for Sports Participants

To examine how youth benefit from sports, we employed a longitudinal and quasi-experimental evaluation design. A cohort of 6th grade students from the 100 schools were tracked for three years. Within this cohort, three different groups were created: (1) afterschool sports participants, (2) afterschool program participants who did not participate in sports, and (3) youth who did not participate in the afterschool program. We used these samples to isolate the value of sports, above and beyond the potential value of the afterschool program. Given the scope of these samples, the evaluation capitalized on data collection efforts conducted within the district for several of the evaluation priorities.

Because random selection was not feasible or ethical, these comparison groups were developed using Propensity Score Matching processes in an effort to reduce selection bias between the groups. These groups were created by matching a single person from the larger LAUSD samples to each member of the 6th grade sports participant cohort. The 6th grade sports participants were matched with comparison groups of youth based on gender, ethnicity, poverty index, parent education, English Language Learner status, physical fitness tests, Physical Education (PE) grades, state test scores for Math and ELA, and school attendance; these indicators were assessed during their 5th grade year (one year prior to this study).

Outcomes of Interest & Data Sources

The evaluation examined outcomes related to physical fitness, social-emotional development, and academic development; these outcomes were selected because they represent the core outcomes aligned with the sports program content (as captured in their theory of change). Given the large sample of youth in this study, the evaluation team capitalized on district archival data and surveys administered by the district. This reduced the cost for the evaluation and reduced participant fatigue for participating in data collection. For physical health outcomes, we examined fitness test pass rates (FITNESSGRAM) and PE grades. For social and emotional development, we examined the constructs on the district survey (connectedness, social awareness, self-management, growth mindset, and self-efficacy). For academic outcomes, we examined grades, state test scores, and school day attendance. Gaining access to the data was allowed due to long-standing relationships with the district and program.

Outcome data were analyzed annually for delivery of in an annual impact report. In the final year, longitudinal trends were examined. Consistently, sports participants had greater physical fitness test pass rates, days of school attendance, self-efficacy, and school connectedness. See our blog post linked above for more information about the study findings.

Examine Mechanisms to Outcomes

Access to afterschool sports programs is not sufficient to achieve the potential benefits of sports programming and understand what is leading to youth impact. Research in positive youth development, afterschool programs, and youth sports programming, articulate that programs require high-quality implementation by well-trained coaches[3], consistent attendance[4], and optimal youth experiences to achieve sustained benefits. Because of this, we evaluated if these drivers of outcomes in sports programs were related to our outcomes of interest using a correlational study of a sub-sample of sports participants. We examined these mechanisms in the final two years of the evaluation.

Why were these drivers identified as key mechanisms to promoting intended youth outcomes?

Sufficient attendance was examined because structured afterschool programs cannot transform youth if youth do not fully engage and participate.3 Youth need sufficient exposure to the program’s high-quality offerings on a regular and consistent basis to obtain positive developmental benefits. However, attendance in any afterschool program is not sufficient to foster robust changes in outcomes; youth experiences in the sports program must be high-quality. According to published research, high quality experiences  in sports programming includes positive coaching6, opportunities for skill development6, 11, and opportunities to take on meaningful leadership roles.11 These experiences should embody quality OST programs more generally, which includes creating a safe and supportive environment, offering skill building opportunities, creating opportunities for active and engaged learning, promoting youth voice and leadership, and ensuring diversity, access, and equity.[5] Finally, positive youth experiences in sports programs are driven by confident, qualified, and prepared coaches executing high-quality program activities.3 If coaches are not prepared, they may not be able to implement the program in a way that produces the positive experiences and developmental outcomes of interest.

To examine these mechanisms, 14 of the 100 afterschool sports programs participated in additional intensive evaluation activities. This included five additional data sources: sports participant surveys, sports program attendance, sports program observations, coaches’ surveys, and sport participant focus groups. These data sources were triangulated to form conclusions about critical drivers of youth development in sports (discussed in the findings blog post). Quantitative data collected was used in correlational analyses to examine the relationship between the drivers and outcomes.

This component of the evaluation was particularly important because we must ensure that youth not only receive equitable access to sports, but that sports programs also offer high quality opportunities for youth – such as youth leadership opportunities, positive coaching, and targeted skill development.

In short, the key features of the evaluation can be summarized as such:

  • Extensive mixed methods data collection
  • Employed existing data sources when possible to minimize cost and participant fatigue
  • Relied on research literature to create evaluation tools and identify constructs of interest
  • Triangulated findings across the perspectives of many key stakeholders
  • Supplemented our understand of key processes by evaluating key experiences specific to sports programs (above and beyond OST programs)
  • Explored drivers of outcomes, rather than impact in isolation
  • Employed matched comparison samples to estimate impact when random selection was not ethical

We hope this blog has provided insight and inspiration to carry out your own evaluation of youth sports programs. Please let us know if you have any questions or comments.

References:

[1] Larson, R. W., & Verma, S. (1999). How children and adolescents spend time across the world: work, play, and developmental opportunities. Psychological Bulletin, 125(6), 701.

[2] Lee, O., Park, M., Jang, K., & Park, Y. (2017). Life lessons after classes: investigating the influence of an afterschool sport program on adolescents’ life skills development. International Journal of Qualitative Studies on Health and Well-being, 12(1), 1307060.

[3] Theokas, C., Danish, S., Hodge, K., Heke, I., & Forneris, T. (2008). Enhancing life skills through sport for children and youth. Positive youth development through sport, 6, 71-81.

[4] Coatsworth, J. D., & Conroy, D. E. (2007). Youth sport as a component of organized afterschool programs. New directions for youth development, 2007(115), 57-74.

[5] Eccles, J. S., & Gootman, J. A. (2002). Features of positive developmental settings. Community programs to promote youth development, 86-118.

[6] Fraser-Thomas, J. L., Côté, J., & Deakin, J. (2005). Youth sport programs: An avenue to foster positive youth development. Physical Education & Sport Pedagogy, 10(1), 19-40.

[7] Gould, D., Flett, R., & Lauer, L. (2012). The relationship between psychosocial developmental and the sports climate experienced by underserved youth. Psychology of Sport and Exercise, 13(1), 80-87.

[8] Coatsworth, J. D., & Conroy, D. E. (2007). Youth sport as a component of organized afterschool programs. New Directions for Youth Development, 115, 57-74. DOI: 10.1002/yd

[9] Larson, R. W. (2000). Toward a psychology of positive youth development. American Psychologist, 55, 170–183. doi:10.1037//0003-066X

[10] Petitpas, A. J., Cornelius, A. E., Van Raalte, J. L., & Jones, T. (2005). A framework for planning youth sport programs that foster psychosocial development. The sport psychologist, 19(1), 63-80.

[11] Durlak, J. A., Weissberg, R. P., & Pachan, M. (2010). A meta‐analysis of after‐school programs that seek to promote personal and social skills in children and adolescents. American journal of community psychology, 45(3-4), 294-309.

[12] Ward, S., & Parker, M. (2013). The voice of youth: atmosphere in positive youth development program. Physical Education and Sport Pedagogy, 18(5), 534-548.