Running Head: AN EVALUATION OF BYEP'S MENTORSHIP PROGRA M i Takami Peemoeller University of Colorado Denver School of Public Affairs This client based project is submitted in partial fulfillment of the requirements for the degree of Master of Public Administration in the School of Public Affairs at the University of Colorado, Denver Denver, Colorado Spring 2019
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM i Capstone Project Disclosures This client based project was completed on behalf of Big Sky Youth Empowerment supervised by PUAD 5361 Capstone course instructor Pamela Medina Gutierrez , PhD, and second faculty reader Sandy Zook, PhD . This project does not necessarily reflect the views of the School of Public Affairs or the faculty readers. Raw data were not inclu ded in this document, rather relevant materials were provided directly to the client. Permissions to include this project in the Auraria Library Digital Repository are found as the final Appendix. Questions about this capstone project should be directed to the student author.
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM ii Executive Summary This research study looked at a sample of alumni who participated in the Big Sky Youth Empowerment (BYEP) program, an outdoor adventure mentoring program that targets at risk students. All former alumni with available contact information were solicited, yielding responses from 70 alumni who participated between 2006 2018 and spent between less than one year to four years in the program. This equaled a response rate of 17% and respondents skewed towards younger alumni who had recently graduated the program. Success indicators were measured, and results found that alumni show success in various ways. For example, t he sample graduated high school at a rate of 83%, which is quite close to the national graduation rate of 84% . Around 80% of the sample is employed and 87% have held their longest job for at least 7 months to a year or longer. And w hile 47% of respondents have engaged in risky behavior, 63% of those respondents have sho wn improvement, either by managing their addiction or by not getting arrested since their time at BYEP. When participants were asked their level of agreement to the statement , 96% agreed or strongly a greed, with only 4% indicating that they neither agree nor disagree. To better understand whether success outcomes shift over time or if the results can be replicated, the organization should continue to conduct regular alumni studies. A one year check in is recommended to provide regular results on educational attainment and hiring or shifting responsibilities to include alumni engagement could better assist at risk teenagers in the transition to adulthood.
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM iii Table of Contents Introduction 1 4 ...9 2 2 Conclusion 5 27 Appendices A 0 B 5 C Word Cloud Favorite Memory from BYEP 3 7 D 8
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 1 Introduction Society is quick to establish nonprofit organizations to solve problems, but how often are these organizations evaluated to see whether they answer these problems? Many nonprofits do not stop to ask whether they are solving the problem , or they struggle with i ssues such as having insufficient time or not enough money to conduct evaluations ( Mitchell & Berlan, 2016) . Big Sky Youth Empowerm ent (BYEP) is one such organization tha t is wanting to better understand the effects of their program, and thus is in need of an alumni evaluation . They want to know the answer to the basic question that many nonprofits want to know : did we make a differen ce? If they continue without the answer to this question, they run the risk of not meeting their mission as they could be meeting a problem that no longer exists or that is being solved by a different program. Evaluations also provide legitimacy to organiz ations. This type of important data could help prove a correlation between success and their program, which could attract further support from external stakeholders and strengthen their organization. Additionally, e valuations provide important feedback data from the participants themselves so that organizations can improve the way they operate . Information from this study could be used to further support the good work they do or to highlight changes that they need to make. Fro m a wider public administration perspective, evaluation studies of programs help illuminate trends across programs and support solutions that work. This evaluation study could be used to support wider standards and best practices. Currently, there are not many studies around mentorship programs , and especially few focus on how outdoor mentorship program participants were affected over time. Big Sky Youth Empowerment could use the results of this study to establish themselves as early adopters in this niche field who set the standard for other
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 2 organizations. This would help further establish the organization as leaders and would strengthen organization support. Organizational Information for vulnerable teenagers to experience success and become contributing members of our community was established in 2001 to serve vulnerable teenagers living in Gallatin County, MT. While the program is mainly associated with its use of outdoor activity to build relationships and increase skills, the program heavily focuses on offering workshops for participants and utilizing adult mentors who support and advise teens to help them achieve success. Mentoring and success within the program is often tied to academic and employment outcomes, but there is also a strong focus on behavioral/social growth and changes in at risk behavior. The nonprofit specifically targets youth fr om eighth grade until the end of high school. During the first two years of high school, students in the program engage in weekly workshops, weekend adventures, and community participation. Students discuss topics such as substance abuse, positive coping s trategies, and conflict management that help to directly address at risk behavior. In the second two years of high school, mentors focus on helping students transition to adulthood. Students participate in a similar programming model, with workshops shifti ng to focus on topics such as financial literacy, work placement, and future planning. Regarding evaluation specifically, t he organization conducts pre and post tests upon entering and exiting the program but does not consistently survey alumni . The pre and post tests are mainly focused on behavioral change. Participants fill out a survey that measures behavior components such as self esteem, social skills, and locus of control, and end up with a
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 3 composite score based on their answers. At the end of the program, participants take another survey that measures the same components. Staff then compare change between their ending score and beginning score to see how the program affected them. Measurement of more long term outcomes (such as employment or educa tion attainment) are not built into the survey and is not a consistent part of organizational planning and strategy , more so being conducted when staff have time or funding available. Purpose To help th e organization examine how their program has affected participants after graduation , a n alumni survey focused on their impact will be conducted. Through this process, BYEP will understand whether their outdoor based mentorship program helped students achieve success since leaving the program , as defined by : 1. Educational attainment 2. I ncome and employment 3. Absence of risky behavior Utilizing a mixed methods approach, research will involve distributing an online survey to alumni of the program. The primary research que stion is : Do alumni of the BYEP program experience success, as defined by educational attainment , income and employment, and absence of risky behavior? A secondary question the organization is also interested in is: Do alumni who spent more years in the program experience greater levels of success? Looking at these various will provide a more well rounded look into how part icipants were affected over time. A literature review on mentorship organizations helps to build the reasoning for the study. Methodology and results of the study are presented. Finally, a brief discussion of results and recommendations are provided.
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 4 Lit erature Review Mentorship programs have grown over the years and are now wide ranging , encompassing a variety of different populations and serving them in a multitude of ways (Rhodes, 2002) . Though these programs can vary greatly, they do have the commonality in that they often involve an adult offering support and advice to someone younger. The research around this subject is similarly varied in field, size, populations, etc. As a result, the following literature review does not focus on the exact populations or fields that relate to BYEP, but rather focus es on mentoring outcomes or outdoor youth programs in general. The literature review explores an overview of mentoring theory and relationsh ips, evaluation challenges, how mentorship affects success, and finally, the research around outdoor programming and behavioral outcomes . Mentoring Theory and Relationships compreh transmission of knowledge, social capital, and psychosocial support perceived by the recipient as urther to mention that relevant knowledge, wisdom, or experience (the mentor) and a person who is perceived to have 731). This definition shows a clear link to personal achievement, and this literature review contains studies that support this definition. A reoccurring and important factor that many studies focus on is the relationship between mentors and mentees themselves. An often cited study is Grossman and Rhodes (2002) who
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 5 looked at the depth of the mentor mentee relationship, finding mentor relations hips that lasted at least a year or longer benefited adolescents more than shorter relationships. Grossman and Rhodes (2002) also recommended that volunteer mentors be chosen and matched carefully with participants, as terminating early was found to have a negative effect on youth. Other studies support the benefits of consistent, long term mentorship. In Schwartz, Rhodes, Spencer, and months showed increased effects related to educational attainment, employment, and convictions (p. 169). Meanwhile, those who were not in sustained contact with mentors after 21 months showed no positive effects compared to the control group (Schwartz et al., 2013, p. 167). While these studies are critical to drawing the link between mentor relationships and outcomes, it should be noted that the majority of these studies focus on one to one relationships. There are few studies that explore the type of group mentoring that most directly relates to mentoring relationship of more than two people in which the i nteractions were simultaneous and mentorship formats, categorizing group mentorship into peer groups, many to one mentoring, one to many mentoring, and many to many men toring. In his review, peer group mentoring was the most widely researched and researchers consistently found that it provided personal and professional growth for participants but could easily get off track without proper facilitation (Huizing, 2012, p. 4 4). Many to many mentoring was defined differently, having multiple established mentors within a group of mentees, whereas Peer Group Mentoring (PGM) had shifting mentors. Many to many mentoring showed many of the same positive results as PGM
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 6 but avoided t he problem of lacking strong guidance. Unfortunately, there were fewer research studies for one to many or many to one models. In fact, one to many models found some studies did not support their hypothesis or could not compare their model to others (Huizi ng, 2012, p. 48). Similarly, though many to one found positive results, there were so few studies within this typology that results cannot easily be generalized. While positive outcomes are associated with many of the studies, Huizing (2012) highlights mul tiple other gaps within the literature, including skewed types of studies (too heavily qualitative) or fields of study (only certain populations have been researched). Evaluation Challenges Prior to exploring the relationship between mentoring and success and outdoor programming and its benefits, it is necessary to preface with the largest identified gap within the studies. Despite mentorship programs having existed and grown for a number of years, researchers emphasize that youth mentorship programs have not been sufficiently and empirically evaluated (DuBois, Doolittle, Yates, Silverthorn, & Kraemer Tebes, 2006 ; Zippay, 1995 ; Pedersen, Woolum, Gagne, & Coleman, 2009 ). As DuBois et al. (2006) states, part of the reason may lie in the complexities of mentorship itself . For example, study size has often been limited in mentorship studies due to working with unique populations , which limits the generalizability of many studies . Additionally, DuBo is et al. (2006) points to the lack of longitudinal studies as a weakness within the research. Indeed, far ranging longitudinal studies on mentorship remain a gap in the literature, and there seem to be few to no studies that explore the effects of such pr ograms beyond five or ten years. Pedersen et al. (2009) also agree s that the literature is inconsistent, makes it essential to look at the variety of mentoring relationships and to recognize that
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 7 1312). T hough thousands of mentor based organizations exist and work with thousands more mentors, Pedersen et al. (2009) suggests that there may be no . Each relationship is different and therefore each study a nd the factors affecting success may be difficult to quantify. There are few comprehensive mentoring studies , and studies around youth outdoor mentorship programs and their long term effects are virtually non existent. Given the sparse nature of the litera ture, it is necessary to learn from the research conducted around a variety of mentorship programs. Mentorship and Success Outcomes being in order to set them up for future success. Many mentorship programs are tied with educational or employment goals. For example, Zippay (1995) analyze d the income resources and employment opportunities that mentors provide youth. Utilizing social network theory, Zippay (1995) hypothesized that mentors serve as ties to information and contacts that may not otherwise be e financial mobility (p. 55). Her study, though with a small population, found that the mentor relationship led to mentees enrolling or planning to enroll in post secondary education at a much higher rate than the national average. Other studies seem to su pport this finding. While Lindsay, Hartman, and Fellin (2016) focused on studies around youth with disabilities, they continued to find in their systematic review that there is a possible tie between mentorship programs and school/employment outcomes . Wit hin the realm of educational goals, Herrera, Grossman, Kauh, and McMaken (2011) based mentoring programs, finding that However, these effects di d not
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 8 continue once the mentor mentee relationship stopped, suggesting that having the presence of an adult in their life kept them on track in the short term only. This may be an instance of the diverse findings and variables within this realm of study, a s o ther research has found that mentees are successful even years after the program. For example, Rodriguez Planas (2017) found that the Quantum Opportunity Program (a mentoring program implemented across seven sites in the United States) increased high sc hool graduation by 15%, attending post secondary education by 21%, and completing two years of post secondary education by 32% (p. 166). The study also found that the program affected wages and had impacts even ten years later. Unfortunately, as stated ear lier, longitudinal studies such as this one are rare and therefore hard to find further support on. Further research into the long term effects of mentoring programs are required. Outdoor Programs for Youth Big Sky Youth Empowerment places a distinct emphasis on outdoor programming, thus literature around outdoor or wilderness activities may provide useful for informing the research. As Norton and Watt (2014) show, wilderness expeditions can help improve inte rnal and external assets, showing that outdoor activities increased positive identity and social competencies. Mutz and Muller (2016) also support this, showing increases in life satisfaction and mindfulness among participants of outdoor adventures in Euro pe. This focus on behavioral improvement is also particularly relevant with regard to mediating risky behavior in adolescents. MacDonald, tionally, engaging in risky behaviors related to sex, drugs, and alcohol were also associated with increased violence (MacDonald, Piquero, Valois, & Zullig, 2005). It is
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 9 possible then that participating in wilderness programs can be associated with decreas ing risky behaviors. Research around this subject tends to focus on behavioral change rather than success outcomes related to school graduation or income but can still be useful for making the connection between behavior and success. Regarding risky behav ior specifically, there does activities, academic success, and less risky behavior. Many of the studies focused on pre and post data that were obtained directly before and after major events, so sustained outcomes of these outdoor excursions require further study as it is possible that the events themselves only affected parti cipants in the short term. Th e BYEP study would help establish more of a link between outdoor programming and success, which could be useful to the organization in leading standards for outdoor organizations. The research around mentorship is clearly dive rse and includes a multitude of variables, which can result in challenges with improving mentoring programs. However, the research generally agrees o n the importance of the mentoring relationship, particularly those that last over time . Additionally, studi es around outdoor youth programming show improvements in behavioral changes for adolescents, which could hint towards better future outcomes. T here are not verified connections in the literature between outdoor programming with group mentorship and long term success outcomes , so this study will provide much needed insight for the overall sphere of youth mentorship and for the organization as well . Methodology
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 10 The research study conducted was a n online survey of all alumni to gather further understanding of participant success over time. Success for some may not be related to high school completion and may instead involve having stable income, not engaging in substance abuse, or having fewer run i ns with the law. This survey therefore explore d educational attainment, income/employment, and absence of risky behaviors as indicators of success. Table 1 below outlines the independent and dependent variables for the research, as well as how they are mea sured. Table 1 Research Variables and Measurements Dependent variable: Educational attainment Measured by response to question on highest level of school completed Dependent variable: Income/Employment Measured by questions regarding current employment status, gross annual salary, and annual household income Dependent variable: Risky behavior Measured by response to questions on number of arrests since the program and whether respondent struggles with a drug or alcohol addiction Independent variable: Participation in program Measured by questions regarding last date of participation in program and length of time in program Data Collection The data sample began with a group of 567 past participants. This included all past participants that the organization had information on, even if the only information was a name. A total of 48 participants were initially removed from the study as they had left the program were still attending high school. These participants were removed due to the possibility that students could return to the program while still in high school and since the study was more interested in studying participant lives post high school. This redu ced the sample size to 519. The study attempted to capture a response from each entry
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 11 in the spreadsheet. Unfortunately, for some entries, there was no phone or email contact or presence on social media. Additionally, two contacts stated that the participa nt has either passed away or was accepted into the program but never participated. As a result, these participants were eliminated from the study, reducing the sample size further to 422 alumni. Respondents w ere primarily contacted contact list) to take the survey . Additionally, social media posts encouraging alumni to take the survey were posted on the BYEP Alumni page. Finally, participants were contacted via phone if no email contact or presence on social media was listed . Participants who were contacted via phone w ere given the option to take the survey over the phone or to be emailed a link to the survey. Due to the primary means of collection being electronic, the survey was designed t o be mobile friendly. The bulk of the d ata collection occurred from March 11, 2019 to March 24, 2019. Email reminders and additional social media posts were sent during the second week of data collection to further encourage participation. Participants ran ged in dates of participation from 2002 to 2018 and have participated anywhere from one season to twelve seasons within the program. Data Analysis Respondent data was collected via the online survey platform, Qualtrics. A copy of the survey is included in Appendix A and is broken into the following segments: BYEP Program Questions Experiences as an Adult (included questions on risky behavior) About Your Job About Your Housing Demographic Questions
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 12 Free Response Questions (included ope n ended questions) Respondent d ata was analyzed utilizing excel. Given that length of time spent in the program is an important factor that could indicate success and could hint towards the importance of the program , participants w ere This means that there were five different groups studied: respondents who participated less than one year, one year, two years, three years, and finally, respondents who participated four years or more. From there, d ependent variables were analyzed based o n the responses to questions around educational attainment, income/employment, and absence of risky behavior. The final questions of the survey also included qualitative free response questions, where participants could explain their thoughts on the prog ram in their own words. These responses were manually coded in a separate analysis to further explore what themes arose . Responses were not broken out by group and instead were analyzed as a group to see major trends or concepts. These responses were mainl y used to either reinforce what the organization is doing well or advise on further recommendations for change. Results A total of 8 9 respondents began the survey. Of the 89 , 1 9 opened the survey but did not input any data. This reduced the number of respondents further to 70 . Surveys that were not fully filled out were included in the analysis, so number of respondents for each question varies as participants were able to skip o ver questions or stop the survey at any time. Participants from the 70 surveys ranged in years participated from 2006 to 2018, with zero representatives from 2002 2005 and 2007. 25% of responses were from participants who just left the program in 2018. T he
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 13 sample was roughly equal between male and female participants (32 responses male; 34 responses female ). 8 5 % of responses identified as White/Caucasian, with minority populations identifying as Asian (1.5%) , Hispanic or Latinx (7.6%) , and Native American o r Alaska Native (6%) . Education al Attainment Participants were asked about their highest level of educational attainment. Of the 6 6 respondents who answered this question , 11 ( 17 %) indicated that their highest level of degree 3.3 % for the sample. This have hovered around 85 and 8 6% over the past four yea rs ( Growth and Enhancement of Montana Students, 2018). It is also quite close to the most recently reported U.S. graduation rate of 84 % ( Boyington, 2018) . Figure 1 below provides a breakdown of responses: Figure 1 . Alumni Level of Educational Attainment
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 14 Responses were tabulated to see if more years within the program led to greater educational attainment, but there does not appear to be any pattern within the data. Alumni who ranged in one year to four years of participation were varied in their education al attainment. However, the overall pattern points to a positive relationship between the program and educational attainment. T his population is more at risk for dropping out of high school yet still achieved a high graduation rate that is close to local r ates . Beyond high school graduation, 27% of the sample has received or is working towards a college or trade school degree. As one past participant stated, does not get enough credi t for how much positivity I have in my life Clearly, educational attainment is an important goal for many participants , with 25% of all responses to the question Additionally, age may be a factor affecting the results. Given that the 11 respondents who completed only some high school skew heavily towards recent years ( nine respondents indicated they left the program during 2017 and 2018 ), i t is possible that some participants are still working towards high school equivalency. Future studies may see a rise in higher education attainment. Income/Employment The study included a number of different questions on income and employment to grasp a full picture of income situations and where pe ople were at after the program. The mode for current income is in the $10,000 to $29,999 range , with 74% of participants indicat ing that they make $29,999 or below. Of the remaining responses, 21% make between $30,000 and $49,999, 3% make between $50,000 and $69,999, and 2% make over $90,000. Figure 2 below shows this information arranged by years spent at BYEP.
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 15 Figure 2: Income by Years in BYEP There appears to be a slight relationship between income and years within the program, as the highest earners are also those who were with the program at least three or four years. Those who have spent one or two years in the program tend to congregate towards the lower inco work placement, and future planning helps participants achieve higher incomes. A more comprehensive census would help to explore this relationship further , as the higher income responses are too few to make a strong claim . Again, age may be a factor skewing these results. As people ag e , their incomes tend to increase, and the data reflects this as well, with the higher earners also being in older age groups. Even wi thin the most recent years, the most common
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 16 The study was also interested in employment. A s Figure 3 shows below, more than 80% are employed at least full or part time. 11 % of respondents stated that they were unemployed and seeking work. Figure 3: BYEP Alumni Employment (as of March 2019) It should also be noted that there were responses that fell in multiple categories. These responses indicated situations such as being a freelancer that was neither full nor part time or pursuing school while also working full or part time. As this category is interested in primarily employment, responses were re coded to fall in the appropriate employment categories. Additional questions were asked regarding employment, including the longest the participant has stayed in a job and the number of full time jobs they have held. Of 6 6 responses, only 8 (1 2 %) have indicated that the longest job they held lasted between 2 and 6 months. The rest of the alumni (87.8%) have held their longest job for at least 7 months to one year, or longer .
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 17 T he most common response (36%) have held their jobs for 7 months to 1 yea r , and r oughly half of responses (5 1 %) have held their longest job for a year or more. When asked about the number of full time positions people have held, only 14.9 % have stated that they have not held a full time position in the past five years. The majority of respondents (5 8 %) stated that they have only held one or two full time positions within the past five years. Again, employment was another important goal that alumni mentioned , being the most commonly reported subject that respondents mentione d when asked about their plans for the future. Some alumni clearly connected their educational goals to employment ones: I want to practice, and mainly work with adole scents struggling with depression, anxiety, and personality disorders, as well as those in the juvenile detention system Clearly, the majority of past participants are employed members of society and many are looking to improve their job situation to fur ther their success. Absence of Risky Behavior Absence of risky behavior was measured based on respondent answers to questions around arrests and drug/alcohol addiction. About half of respondents (5 2.9 %) do not exhibit risky behavior within these terms (i.e. they responded that they have never been arrested or struggled with drug/alcohol addiction) , while the other half of respondents indicated that they have been arrested and/or struggled with drug addiction. Figure 4 is a visual representation of arr ests, where 1 6 respondents (2 4 %) indicated that they have been arrested . Figure 4: Alumni Arrests Since BYEP
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 18 Of those who have been arrested, six (8% of alumni) have not been arrested since the program and ten (1 5 % of alumni ) have been arrested anywhere from one to three or more times. Of those ten , six respondents (8% of alumni) have experienced an arrest within the past year. Table 2 below ir most recent arrest was. Table 2 Number of Alumni Sorted by Arrests and Time Frame Not S ince BYEP 1 Arrest 2 Arrests 3 or M ore Arrests Within P ast Y ear 0 4 0 2 3 5 Y ears A go 3 0 1 3 >5 Y ears A go 3 0 0 0 More respondents have struggled with drug or alcohol addiction. Of the 6 8 respondents, 2 4 (35%) said they have struggled with drug/alcohol addiction in their lives. However, 1 6 of those 2 4 ( 67% ) respondents said they no longer battle addiction. This leaves a small subset of
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 19 five respondents (7%) responses that include a text field for people to explain an alternate situation and respondents mentioned being in recovery or not thinking they have a problem (while others think they do). When looking at both variables, 15 participants (22%) specified that they have been arrested since BYEP and/or struggle with drug addiction currently. Though no patterns emerged around years spent within the program, overall responses point to success and improvement on this variable. For example , 67% of respondents who said they have struggled with addiction stated that they no longer struggle with drug/alcohol addiction. 75% of those respondents were under the age of 22. It is possible that simply participating in the program at this critical age can be a factor that helps participants manage their addiction. Additionally, 20 of 32 respondents (63%) who indicated a risky beha vior showed improvement on one or both variables since their time at BYEP. Benefits from the Program Aside from the measured variables, survey respondents were also asked a number of questions on gained skills and how this program contributed to their success. When directly asked about their agreement to , 96% of respondents stated that they agree or strongly agre e with the statement, while only 4% responses. This shows an incredible support for the organization and what it does. Key to many experiences is an overall sense of community provided by the program. As one participant stated,
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 20 Having a positive adult in their life and engaging in and overcoming new activities was also powerful for teens. One participant stated that their favorite memory was their first time skiing: kept saying how scared I was. My mentor kept telling me I could do it, stayed with me the whole way, and was really there for me. It made me realize someone was going to be This feeling of support is exp lored in much of the mentorship literature. As Grossman and Rhodes (2002) pointed out, mentor relationships that lasted longer had better outcomes, and those that terminated earlier adversely affected youth. Aside from the sense of support and community, respondents also gained important skills. T he most common response ( half of all responses ) on learned skills was how to communicate/improve their social skills. Some mentioned specifically that the program helped them with their communication with adults or with public speaking , while others mentioned that the program helped them be more comfortable around their pee rs or helped them express themselves and their emotions in an appropriate way. The second and third most commonly reported skills were teamwork/relationship building and life skills, such as filing taxes, learning how to interview or write a resume, or app lying for college. Other learned aspects that respondents mentioned were having empathy for others, resiliency, critical thinking, patience, commitment, volunteering, confidence, leadership, coping skills, active listening skills, improving their own self worth or happiness, time management skills, and of course, learning how to do certain outdoor activities (such as snowboarding or rock climbing). Of the 6 8
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 21 respondents who answered this question , 11 (1 6 %) indicated that they have not gained any specific sk ills from BYEP. Regarding what respondents wish they could have learned, responses tended to focus on being prepared for adulthood. For example, 18% of alumni mentioned wanting more information on financial education or taxes. Two alumni (4%) mentioned wanting more information on housing, three (5%) wanted more career information, one (2%) wanted more health insurance information, and 21% stated that they wished they learned how to really be an adult. Some respondents mentioned that they did re ceive this type of information but not enough of it , they were too young to be interested in it, or they left the program before they learned this information. of the mo st challenging things, and [it] would be nice to have a program like that for out of high This quote embodies m uch of what the responses hint to that the transition to adulthood was a challenge for them and they did not have the group to fall ba ck on . Despite many reporting that they learned life skills that helped them as an adult in the earlier question, there were also many responses wishing they were more prepared. Other responses for this question included wanting more mental health informat ion, wanting to know how to be better at listening and talking with others, fly fishing, reading/writing, automotive skills, survival skills, horseback riding, and cooking. Respondents were also asked to describe their favorite memory to show what they re call most from the program. snowboarding (often for the first time), meet new friends, do group activities, and have a mentor they could learn from. The most common response to this question involved an outdoor activity,
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 22 but for many doing this activity was not only tied with doing something fun with friends but with learning something new, doing something they would never have been able to afford, or facing their fears. Similarly, when asked about skills they learned or wish they could have learned, the most common responses do not have to do with outdoor activity, showing that the program impacts them beyond si mply doing outdoor activities. This reflects much of the research around outdoor activities that support behavioral outcomes for at risk teenagers. Evidently, even though many years have passed for some of these participants, they still view the program as an important part of their life that taught them skills and helped them feel part of a larger community. Overall, though the study could not determine a strong pattern between time spent in the program and success, it is evident that most participants are doing well after the program. 64% of alumni respondents came from low income families, as evidenced by their answers to having free/reduced lunch while attending school, yet many of those same respondents are now the highest earners in the data set. Ad ditionally, 60% have volunteered since their time in the plans, most mentioned furthering their education, pursuing their career, and being good parents to their child ren. Only four respondents (7%) were not sure about what they would do for the future. This shows a majority of respondents who are future minded and working towards success in life. Discussion and Recommendations Big Sky Youth Empowerment was intereste d in seeing how their program affected students and if length of time spent within the program led to better outcomes. As mentioned
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 23 earlier, there are studies that indicate having a mentor relationship that lasts longer leads to better outcomes (Grossman & Rhodes, 2002; Schwartz, Rhodes, Spencer, & Grossman, 2013). This study found a possible relationship between income and years spent within the program but did not find a relationship with the other variables . Overall, however, multiple variables showed improvement /success and anecdotes supported the importance of the program . Given that Big , group mentoring studies may be more applicable . As Huizing (2012) pointed out, many to many mento ring shows positive results thus far but needs further research . Further studies need to be conducted around the long term results of group mentoring to see whether more time spent within the program is correlated with greater success outcomes. The organiz ation may also want to experiment with other mentoring models to see if another model supports greater success. For example, it may be helpful for some to have one on one opportunities with volunteer mentors. One reoccurring theme throughout the data is while the majority of participants are doing well (i.e. graduating high school, finding employment, not engaging in risky behavior), a small subset of the population continues to struggle or not meet the same level of success as their peers within the stud y (usually around 10 20% of respondents depending on the variable). However, there were no responses that are seemingly behind their peers on every measurement (i.e. stating that they have not finished high school, are unemployed/earning less than $10,000, have been arrested recently/struggle with addiction). This means th at respondents maybe be struggling in one or two aspect of their lives but doing well in others. Given that the age range of this study heavily skews towards younger ages (ages that may still be figuring out the transition to adulthood), further research c ould show whether these responses change and improve over time. Also, given that alumni showed risk factors, it is important to continue to screen potential
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 24 participants and to properly train mentors. DuBois, Portillo, Rhodes, Silverthorn, and Valentine (2 011) emphasize the importance of successful programs being able to appropriately gauge risk profiles of participants, utilize the skill sets mentors have, match mentors and mentees well, and properly set boundaries and expectations for mentors in their res ponsibilities. Some teens may be experiencing severe abuse or trauma, and mentoring programs are not suitable replacements for those teens who are more in need of social work and direct services. The major limitation of the study is its heavy skew towards younger ages (7 5 % of respondents were under the age of 22). This was due to much of the contact info being out of date. Younger participants tended to have more contact information on file or more accurate forms of contact as they may have only recently e xited the program. For older respondents, it was difficult to locate people even with social media as some may have changed their name due to marriage or other circumstances. Often, the only form of contact people had on file was an old phone number of a p arent or guardian that no longer worked. There are multiple ways one could improve connections with alumni, some of which BYEP is already doing. For example, having multiple forms of contact made it easier to find past participants. It is also recommended that further census studies of alumni be conducted at least every three to five years. Not only will this provide details on how younger participants in this study are doing over time, but it will allow another point of contact in order to ens ure that the organization keeps in touch and keeps more up to date contact information. In addition to this, BYEP should conduct a one year follow up with anyone who has left the program. Given the high response rate from people who recently left the progr am, it is likely that the organization can at least collect useful information on educational attainment . BYEP might also consider asking all participants to join the alumni
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 25 group on Facebook upon their leaving of the program in order to stay in touch. BYE P should also be consistent in their updates in order to get regular engagement. Recommendations A major reason for conducting this study is that BYEP did not have the time to do so as an organization. As the organization grows, it might consider shifting responsibilities of positions or adding a position to focus at least partially on alumni engagement. One common issue that respondents mentioned in their questions around skills was feeling like they w ere not ready to be an adult and wishing they still had aspects of the program to fall back on. The program could improve further to cover more life skills, but participants may still feel unprepared due to not being fully receptive towards lessons during their time at BYEP. Having a way for alumni to connect, such as alumni networking or opportunities to engage in alumni workshops, could give past participants the sense of community /support to make the transition to adulthood easier. Additionally , having s omeone focused on alumni engagement would allow the organization to make a concentrated effort towards alumni data collection and attract further stakeholder support . Recommendations for the organization may thus be summarized as below: 1. Continue conducting alumni studies (including a one year check in) to see whether time helps to improve success outcomes , or to see if similar results can be replicated. 2. Shift responsibilities or hire personnel to focus on alumni engagement to better track alumni and assist in the transition to adulthood . Conclusion
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 26 This study was interested in seeing how alumni of the Big Sky Youth Empowerment program achieved success and whether more years within the program increased their success. Findings indicated a slight relationship between years spent in the program and financial success , and overall responses show that this vulnerable population is achieving success similar to national standards and improving on risk variables . Nearly all of the respondents (96%) agreed that the program helped them to achieve success, further demonstrating the importance of this program. However, many responses also mentioned wanting assistance and community further into adulthood . Devoting staff time to alumni engagement could fu rther increase success and assist the organization with truly understanding their impact over time. Additionally, this study continues to add to the unique conversation around mentoring and outdoor programs, showing that it has benefitted vulnerable popula tions over time, but further research is needed to support
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 27 References Big Sky Youth Empowerment. (2018). Big Sky Youth Empowerment organizational summary . Bozeman, MT. Boyington, Briana. (2018). See high school graduation rates by state. U.S. News . Retrieved from https://www.usnews.com/high schools/best high schools/articles/2018 05 18/see high school graduation rates by state Bozeman, B. & Feeney, M.K. (2 007). Toward a useful theory of mentoring: A conceptual analysis and critique. Administration & Society, 39 (6). doi: 10.1177/0095399707304119 DuBois, D.L., Doolittle, F., Yates, B.T., Silverthorn, N., & Kraemer Tebes, J. (2006). Research methodology and youth mentoring. Journal of Community Psychology, 34 (6). doi: 10.1002/jcop.20122 Grossman, J. B., & Rhodes, J. E. (2002). The test of time: Predictors and effects of duration in youth mentoring relationships. American Journal of Community Psychology, 30 ( 2) . https://doi org.aurarialibrary.idm.oclc.org/10.1023/A:1014680827552 Growth and Enhancement of Montana Students. (2018). Summary graduation report [Data file]. Retrieved from https://gems.opi.mt.gov/StudentCharacteristics/Pages/SummaryGraduationRateR eport.as px Herrera, C., Grossman, J.B., Kauh, T.J., & McMaken, J. (2011). Mentoring in schools: An impact study of Big Brothers Big Sisters school based mentoring. Child Development, 82 (1). Retrieved from https://www.jstor.org/stable/29782835 Huizing, R. L. (2012). Mentoring together: A literature review of group mentoring. Mentoring & Tutoring: Partnership in Learning, 20 (1), 27 55. doi: 10.1080/13611267.2012.645599
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 28 Lindsay, S., Hartman, L., & Fellin, M. (2016). A systematic review of mentorship programs to facilitate transition to post secondary education and employment for youth and young adults with disabilities. Disability and Rehabilitation, 38 (14), 1329 1349. doi: 10.3109/09638288.2015.1092174 MacDonald, J.M., Piquero, A.R., Valois, R.F., & Zullig, K.J. (2005). The relationship between life satisfaction, risk taking behaviors, and youth violence. Journal of Interpersonal Violence, 20 (11). doi: 10.1177/0886260505278718 Mitchell, G.E., & Berlan, D. (2016). Evaluation and evaluative rigor in the nonpr ofit sector. Nonprofit Management & Leadership, 27 (2), 237 250. doi: 10.1002/nml.21236 Mutz, M., & Muller, J. (2016). Mental health benefits of outdoor adventures: Results from two pilot studies. Journal of Adolescence, 49 . doi: 10.1016/j.adolescence.2016.03.009 Norton, C.L., & Watt, T.T. (2014). Exploring the impact of a wilderness based positive youth development program for urban youth. Journal of Experiential Education, 37 (4). doi: 10.1177/1053825913503113 Pedersen, P.J., W oolum, S., Gagne, B., & Coleman, M. (2009). Beyond the norm: Extraordinary relationships in youth mentoring. Children and Youth Services Review, 31 . doi: 10.1016/j.childyouth.2009.06.001 Rhodes, J.E. (2002). Stand by me: The risks and rewards of . Cambridge, MA: Harvard University Press. Rodriguez Planas, N. (2017). Schools, drugs, mentoring, and peers: Evidence from a randomized trial in the US. Journal of Economic Behavior & Organization, 139 . doi: 10.1016/j.jebo.2017.0 5.003 Schwartz, S.E., Rhodes, J.E., Spencer, R., & Grossman, J.B. (2013). Youth initiated mentoring:
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 29 Investigating a new approach to working with vulnerable adolescents. American Journal of Community Psychology, 52 . doi: 10.1007/s10464 013 9585 3 Wong, M. D., Strom, D., Guerrero, L.R., Chung, P.J., Lopez, D., Arellano, K., & Dudovitz, R.N. (2017). The role of social emotional and social network factors in the relationship between academic achievement and risky behaviors. Academic Pediatrics, 17 (6). doi: 10 .1016/j.acap.2017.04.009 Zippay, A. (1995). Expanding employment skills and social networks among teen mothers: Case study of a mentor program. Child and Adolescent Social Work Journal, 12 (1). Retrieved from https://link springer com.aurarialibrary.idm. oclc.org/content/pdf/10.1007%2FBF01876139.pdf
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 30 Attachment A: Alumni Survey Welcome to the Big Sky Youth Empowerment Alumni Survey! Thank you for agreeing to take part in this survey. The purpose of this study is to see how participants are doing after they have exited the program. Responses will be used to help the organization understand their impact and help advise the program's future. Your participation in the survey is voluntary and any information you provide will remain completely con fidential. Quotes from the survey may be shared with Big Sky Youth Empowerment, but no personally identifiable information will be shared. This survey should take 5 to 10 minutes to complete. Please answer all of the questions on the following pages to t he best of your ability. Click the right arrow to begin. Experience with BYEP 1. Please input the last year you participated in the Big Sky Youth Empowerment program (BYEP): 2. How long did you participate in the BYEP program? a. <1 year b. 1 year c. 2 years d. 3 years e. 4 years or more 3. Please rate your level of agreement with the following statement: BYEP helped set me on a path toward success in life. a. Strongly Agree b. Agree c. Neither Agree nor Disagree d. Disagree e. Strongly Disagree 4. Have you gained any specific skills from BYE P? If yes, please list below: a. No b. Yes: (please describe) 5. At any point while attending the BYEP program, did you qualify for free/reduced lunch? a. Yes b. No 6. Have you ever volunteered after the BYEP program? Do not count community service that was mandated or c ourt ordered.
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 31 Experiences as an Adult 7. Have you ever been arrested? a. Yes b. No 8. If yes, how many times have you been arrested since your time at BYEP? a. I have not been arrested since my time at BYEP b. 1 time c. 2 times d. 3 or more times 9. If yes, how recent was your most recent arrest? a. Within the past year b. 1 2 years ago c. 3 5 years ago d. More than 5 years ago 10. Have you ever struggled with a drug or alcohol addiction? a. Yes b. No 11. If yes, do you currently struggle with a drug or alcohol addiction? a. Yes b. No c. Other (please specify): Employment Status and History 12. Which of the following best describes your employment status? a. Employed, working full time b. Employed, working part time c. Student d. Stay at home parent e. Unemployed and seeking employment f. Unemployed and not seeking employment g. Disabled and not able to work h. Other (please specify) 13. If employed, what is your gross annual salary? a. Less than $10,000 b. $10,000 to $29,999 c. $30,000 to $49,999 d. $50,000 to $69,999 e. $70,000 to $89,999 f. $90,000 or more
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 32 14. What is your annual household income? Please include income from people who you live with but who file taxes separately. a. Less than $10,000 b. $10,000 to $29,999 c. $30,000 to $49,999 d. $50,000 to $69,999 e. $70,000 to $89,999 f. $90,000 or more 15. Over the course of your job history, what is the average length of time you stay in one job? a. I do not work or have not for some time b. 1 month or less c. 2 to 6 months d. 7 months to 1 year e. 1 to 2 years f. More than 2 years g. Other (please specify) 16. What is the longest amount of time you have sp ent working at any job? a. 1 month or less b. 2 to 6 months c. 7 months to 1 year d. 1 to 2 years e. More than 2 years f. Other (please specify) 17. Between January 2016 and December 2018, how many full time positions have you held? a. 0 b. 1 to 2 c. 3 to 4 d. 5 to 6 e. 7 or more 18. Which statement best describes your tax filing status for the 2015, 2016, and 2017 tax years? a. I did not file taxes for the 2015, 2016, and 2017 tax years. b. I filed taxes for one or two of the years between 2015, 2016, and 2017. c. I filed taxes for the 2015, 2016 an d 2017 tax years. About Your Housing 19. What best describes your current housing situation? a. Homeless or living in car/shelter
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 33 b. Instable housing (staying with friends or family temporarily) c. Living with parents d. Renting alone or with spouse/children e. Renting wi th roommates f. Own house/apartment/condominium/townhouse etc. g. Other housing situation (please specify) 20. Have you ever owned a house/apartment/condominium/townhouse, etc.? a. Yes b. No 21. How many people reside in your household? Include yourself/your dependents and do not count people who live in your house but file taxes separately. a. 1 b. 2 c. 3 5 d. 6 or more Demographic Questions 22. What is your age? a. Under 18 b. 18 22 years old c. 23 27 years old d. 28 32 years old e. 33 37 years old f. Over 37 23. What is your gender? a. Female b. Male c. Other (please specify) d. Rather not say 24. What is your ethnicity? a. African American or Black b. Asian c. Hispanic or Latinx d. Native American or Alaska Native e. Native Hawaiian/Pacific Islander f. White/Caucasian g. Other (please specify) 25. What is your marital status? a. Single (never married) b. Married
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 34 c. In a domestic partnership d. Divorced e. Separated f. Widowed 26. What is the highest degree or level of school you have completed? a. Some high school b. High school diploma c. Hi Set or GED d. Some college but no degree e. f. Bachelor g. h. Doctorate Free Response 27. What was your favorite memory from your time at BYEP? 28. What is something else you wish you could have learned while in BYEP? 29. What are your plans for the future? 30. Finally, BYEP is collecting alumni experi ences to say thank you and farewell to Founder and Executive Director, Pete MacFadyen. Would you be interested in speaking about how BYEP has impacted your life? If so, please list your contact information below, or email email@example.com.
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 35 Appendix B: MPA Competencies The competencies required for this project are listed in the University of Colorado, Below is a listing of each competency and descriptions of the ones that I have demonstrated the most : 1. Lead and manage in public governance a. Though I did not have as much of an opportunity to perform this particular competency, my recommendations reflect an understanding of how an organization can lead and manage participants. 2. Participate in and contribute to th e public policy process 3. Analyze, synthesize, think critically, solve problems and make decisions a. This is perhaps the most obviously shown competency. I had to collect and analyze survey data, synthesize data into results for the organization, and think cri tically about what the results meant and how they applied. I then had to take this information and solve the main question of the organization, which was on how alumni were doing after the program. I also had to make decisions on how the organization shoul d change moving forward. 4. Articulate and apply a public service perspective a. This competency was shown heavily through the literature review and discussion/recommendations. I had to research and articulate different perspectives on public service. I had to u se the recommendations and research that I found in the literature and apply it to this situation as well. 5. Communicate and interact productively with a diverse and changing workforce and citizenry.
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 36 a. This competency is also shown heavily in the capstone as well. As part of the survey, I had to communicate with diverse citizens of various ages, often over the phone. I also had to communicate with multiple stakeholders (including the client and professors) throughout the process.
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 37 Appendix C: Word Cloud Favorite M emory from BYEP
AN EVALUATION OF BYEP'S MENTORSHIP PROGRAM 38 Appendix D: BYEP Alumni Study Infographic
Form Name: capstone repository permission Submission Time: June 1, 2019 2:14 pm Browser: Chrome 74.0.3729.169 / OS X IP Address: 18.104.22.168 Unique ID: 508990480 Location: 45.662399291992, -110.56099700928 Description Area SCHOOL OF PUBLIC AFFAIRS ELECTRONIC CAPSTONE REPOSITORY Description Area Dear Capstone Author and Capstone Client:The Auraria Library Digital Library Program is a nonprofit center responsible for the collection and preservation of digital resources for education.The capstone project, protected by your copyright, and/or created under the supervision of the client has been identified as important to the educational mission of the University of Colorado Denver and Auraria Library.The University of Colorado Denver and Auraria Library respectfully requests non-exclusive rights to digitize the capstone project for Internet distribution in image and text formats for an unlimited term. Digitized versions will be made available via the Internet, for onand off-line educational use, with a statement identifying your rights as copyright holder and the terms of the grant of permissions.Please review, sign and return the follow Grant of Permissions. Please do not hesitate to call me or email your questions.Sincerely,Matthew C. MarinerAuraria LibraryDigital Collections ManagerMatthew.firstname.lastname@example.org Grant of Permissions Description Area In reference to the following title(s): Author (Student Name) Takami Peemoeller Title (Capstone Project Title) An Evaluation of Big Sky Youth Empowerment's Mentorship Program Publication Date 5/28/19 I am the: Client Description Area As client of the copyright holder affirm that the content submitted is identical to that which was originally supervised and that the content is suitable for publication in the Auraria Library Digital Collections.
Description Area This is a non-exclusive grant of permissions for on-line and off-line use for an indefinite term. Off-line uses shall be consistent either for educational uses, with the terms of U.S. copyright legislation's "fair use" provisions or, by the University of Colorado Denver and/or Auraria Library, with the maintenance and preservation of an archival copy. Digitization allows the University of Colorado Denver and/or Auraria Library to generate imageand text-based versions as appropriate and to provide and enhance access using search software. This grant of permissions prohibits use of the digitized versions for commercial use or profit. Signature Your Name Katie DiBerardinis Date 6/1/19 Email Address email@example.com ATTENTION Description Area Grant of Permissions is provided to: Auraria Digital Library Program / Matthew C. MarinerAuraria Library1100 Lawrence | Denver, CO firstname.lastname@example.org
Form Name: capstone repository permission Submission Time: May 29, 2019 2:53 pm Browser: Chrome 74.0.3729.169 / Windows IP Address: 22.214.171.124 Unique ID: 508228753 Location: 45.627998352051, -110.90129852295 Description Area SCHOOL OF PUBLIC AFFAIRS ELECTRONIC CAPSTONE REPOSITORY Description Area Dear Capstone Author and Capstone Client:The Auraria Library Digital Library Program is a nonprofit center responsible for the collection and preservation of digital resources for education.The capstone project, protected by your copyright, and/or created under the supervision of the client has been identified as important to the educational mission of the University of Colorado Denver and Auraria Library.The University of Colorado Denver and Auraria Library respectfully requests non-exclusive rights to digitize the capstone project for Internet distribution in image and text formats for an unlimited term. Digitized versions will be made available via the Internet, for onand off-line educational use, with a statement identifying your rights as copyright holder and the terms of the grant of permissions.Please review, sign and return the follow Grant of Permissions. Please do not hesitate to call me or email your questions.Sincerely,Matthew C. MarinerAuraria LibraryDigital Collections ManagerMatthew.email@example.com Grant of Permissions Description Area In reference to the following title(s): Author (Student Name) Takami Peemoeller Title (Capstone Project Title) An Evaluation of Big Sky Youth Empowerment's Mentorship Program Publication Date 5/29/19 I am the: Author (student) Description Area As copyright holder or licensee with the authority to grant copyright permissions for the aforementioned title(s), I hereby authorize Auraria Library and University of Colorado Denver to digitize, distribute, and archive the title(s) for nonprofit, educational purposes via the Internet or successive technologies.
Description Area This is a non-exclusive grant of permissions for on-line and off-line use for an indefinite term. Off-line uses shall be consistent either for educational uses, with the terms of U.S. copyright legislation's "fair use" provisions or, by the University of Colorado Denver and/or Auraria Library, with the maintenance and preservation of an archival copy. Digitization allows the University of Colorado Denver and/or Auraria Library to generate imageand text-based versions as appropriate and to provide and enhance access using search software. This grant of permissions prohibits use of the digitized versions for commercial use or profit. Signature Your Name Takami Peemoeller Date 5/29/19 Email Address firstname.lastname@example.org ATTENTION Description Area Grant of Permissions is provided to: Auraria Digital Library Program / Matthew C. MarinerAuraria Library1100 Lawrence | Denver, CO email@example.com