A REPORT TO THE KISCO FOUNDATION
Laura Detrick, Ph.D.
Ian Martin, Ed.D.
Roxanne Ruzic, Ph.D.
University of San Diego
Center for Education Policy, and Law
The Caster Center for Nonprofit and Philanthropic Research
The Kisco Foundation commissioned researchers at the Center for Education Policy and Law and the Caster Center for Nonprofit and Philanthropic Research at the University of San Diego to work with REBOOT, a nonprofit based in Southern California, to conduct an evaluation of its program, REBOOT Workshops™ (REBOOT). REBOOT is a three-week-long program designed to assist members of the military in making the social and career transition from military service to civilian life by “re-booting” their skills, attitudes, and behaviors. The evaluation detailed in this report was conducted during the summer and fall of 2012, two years after the first REBOOT workshop, which was held in June 2010. The goals of the evaluation were: 1) to work with REBOOT staff to articulate and refine the learning objectives and logic model for the REBOOT program; 2) to evaluate the current REBOOT curriculum and evaluation process in light of the new logic model as well as existing research and best practices in the field; and 3) to provide a baseline program evaluation using data previously collected through the REBOOT program.
Researchers and REBOOT staff collaborated successfully to refine the program’s existing logic model and more clearly articulate program goals and learning objectives for the program. Each of the three weeks of the REBOOT program focuses on a different goal: Week 1 focuses on building self-efficacy, Week 2 focuses on developing a positive outlook on the future, and Week 3 focuses on developing skills necessary to obtain employment and/or pursue further education upon separation from the military. A variety of components, such as a cohort design, experiential learning, instructor coaching, and peer support, are intended to support the research-based curriculum and distinguish the REBOOT experience. A complete description of the program components and logic model are contained in the full report.
After working with REBOOT staff to refine the logic model and articulate program and learning goals, we used these new tools, along with existing best practices and research in the field, to conduct a review of the curriculum currently used for the REBOOT program. This review revealed many strong components of the program, particularly in weeks 1 and 3. The content for weeks 1 and 3, in general, is aligned with well-established theories and constructs, as well as the new program theory and goals for the REBOOT workshops developed through this evaluation. The curriculum for weeks 1 and 3 could be strengthened by making the links between the material presented in the workshop and established, evidence-based practices and scholarly literature more explicit, and by updating or improving some of the resources used. In addition, while the curriculum for weeks 1, 2, and 3 are not currently paired with any outcomes measures to assess learning or behavior change, there are many examples of reliable and valid outcomes measures that could be used to achieve this goal. Samples of potential instruments are provided in the full report.
Based on data collected by REBOOT staff prior to this evaluation, REBOOT participants reflect a range of ages, years of service, and ethnicities. A review of these existing data, supplemented by interviews with selected REBOOT staff and participants, suggest that the program is meeting its objectives and that participants take part fully in the program and find it valuable. Findings from a sample of pre- and post-program survey responses identified substantial improvements in participants’ reports on nine separate measures: 1) having written plans for achieving goals outside the military, 2) having a clear vision of a future outside the military, 3) being connected with a support network outside the military, 4) being prepared to achieve goals outside the military, 5) being able to effectively communicate strengths to potential employers, 6) believing that continual learning has a positive impact on quality of life, 7) understanding how thoughts can lead to actions that affect the transition, 8) placing an increased importance on social life, and 9) feeling confident in the ability to find a job outside the military.
To more effectively capture both short and longer-term outcomes and successes, additional resources are needed to improve the procedures and supports for data collection and analysis. Based on this evaluation, the research team suggests that REBOOT staff take the following actions:
1. Create a participant consent form for evaluation activities – Completed.
2. Create a plan to address curriculum review recommendations – Completed.
3. Select specific instruments to measure learning outcomes – Completed.
4. Revise the pre- and post-program survey instrument – Completed
5. Create platforms so that all assessments may be completed electronically – Completed.
6. Institute the use of unique identifiers so participants’ outcomes can be traced over time – Completed.
7. Revamp the current participant database (Quikbase system) – Completed.
8. Create a plan for ongoing data collection, data analysis, and program evaluation – Completed.
Click here for the full report