Evaluating a Vermont School System’s ‘Mindfulness’ Program
In 2009-2010, the S. Burlington (Vt.) School District piloted a program teaching mindfulness techniques to teachers and students, with a goal of reducing stress among teachers and students, and improving socio-emotional learning in the classroom.
The district sought an evaluation of the year-long program in order to capture effects on self-reported stress and other socio-emotional factors in teachers and students in five classrooms. The school district’s goal was to build the evidence basis for the program, understand the ease and challenges of implementation, and assess the impact it had on participants. Findings would help determine whether the program would continue to receive foundational and internal support.
What We Did
In ongoing consultation, Youth Catalytics:
- Selected an evaluation design and established evaluation questions.
- Researched and selected validated pre- and post-test instruments for teachers.
- Assisted in developing a weekly check-in instrument for adult program participants.
- Created self-report surveys for students.
- Developed weekly questionnaires for teachers in targeted classrooms to track experiences implementing the program and perceptions of impact on students over time.
- Collected data from teachers on a weekly basis and from classrooms on a monthly basis.
- Analyzed data using standard statistical methods.
- Produced a report for the school community, parents and program funder.
The evaluation gathered both quantitative and qualitative data from several groups and subgroups: 41 teachers and staff participating in personal mindfulness training; five teachers implementing mindfulness with students in targeted classrooms; and 84 students in those classrooms. For each group or subgroup, we used more than one instrument to measure change, and asked both numerically-scaled and open-ended questions. Therefore the evaluation, despite its real-world limitations, was as robust as possible; it yielded a range of data that shed substantial light on the impacts of this first-year program on teachers’ personal and professional lives, and on students’ classroom experiences.
Summary findings include:
- A statistically significant decrease in participants’ perception of stress and an increase in mindfulness, as measured by validated tools completed at the beginning and end of the program.
- A significant increase in teachers’ aggregate ‘personal health promotion’ scores, which include perceptions of calmness, happiness and work satisfaction.
- Significant positive effect on students reported by 16% of all teachers and a moderately positive effect reported by another 40% of all teachers.
- Moderate or strong effects on students reported by five teachers using techniques in class over a 71-week period.
- Reports by 84 students in five classes saying the program helped them focus “a little” better in class and made them “a little” more ready to learn.
According to the school district, the evaluation:
- Justified continued support from the School Board. ‘As a result of our strong outcomes, the School Board authorized a line item in the school budget toward the Program, which was previously entirely grant-funded.’
- Strengthened funder reports and led to successfully securing other grant funding.
- Informed school staff and the broader community ‘so that all involved and interested would see that we were holding ourselves accountable to ensure effective program outcomes. We also shared it with participants (including potential participants) to encourage their continued participation.’
- Inspired other educators in the field who are interested in starting similar programs in their schools and communities.
- Was shared at a Center for Mindfulness conference as part of a program overview and outcome presentation, generating interest among participants seeking administration support for similar programs.
- Built a foundation for subsequent evaluations, all of which have demonstrated similar results and ‘are a very useful combined tool for referring anyone who wants to know more about our program and outcomes. These have helped enormously with establishing our program credibility.’
- Initiated a formal data collection process that led a SAMHSA scientific review coordinator to recommend the school pursue a controlled study toward the end of becoming an evidence-based program on the agency registry.
“Working with Melanie Wilson of Youth Catalytics has been one of the most professionally rewarding experiences in my work as a school counselor. Melanie was a model of collaboration and integrity as she carefully guided us through the process of evaluation design and implementation. She seemed to give us 110% of her attention and as a result, we ended up with an extremely valuable program assessment for our purposes. We have been able to build on what we gained from our work and we now have a number of years of data to support the continuation of our program.”
~ Auriel Gray, South Burlington Vermont School Counselor