In the Cannabis Project, we piloted the online prevention training suite with community members. We recruited 20 individuals who worked with youth in the community and had not been a part of the needs assessment. This group was asked to complete a baseline knowledge survey to measure their initial knowledge of cannabis prevention; watch all four online trainings and complete each associated feedback survey; and complete the knowledge survey again after completion of the trainings. Participants received a $100 e-gift card for their time.

The data collected from the baseline and post-training surveys, as well as the reaction surveys, were summarized and shared with the project team. The baseline and post-training surveys demonstrated the effectiveness of the trainings in increasing knowledge among participants and captured ideas of who may most benefit from the trainings. The reactions surveys confirmed the relevancy of the training content and the accessibility of the training format. They also provided a space for participants to make suggestions for the trainings.

These findings were presented back to community members at an in-person meeting, and a guided discussion about dissemination helped the project team develop community-informed next steps.

• A baseline survey measured participants’ initial knowledge on related topics.
• A post-training survey measured participants’ knowledge after completing the trainings.

Each training had a reaction survey, like this one, that appeared on the screen for participants to complete at the end of the training.

In the Cannabis Project, we collaborated with community champions to identify ways to disseminate the trainings. We held an in-person meeting with community champions, presented the results from the pilot surveys, and facilitated a discussion about the benefits of the trainings and next steps.

We asked community champions to reflect on and share responses to the following questions:

• What would it look like in the community if these trainings were well distributed and adopted?
• Who in the community should engage with these trainings?
• Who can/will reach out to share and encourage these individuals?

Community champions identified potential users for the trainings such as schools (administrators, health teachers, 6-8th grade teachers, counselors), youth mentors, parents, case managers/therapists, health providers, law enforcement, and church-based groups.

After the meeting, we created and shared an action plan document that included the identified individuals in the local community within those user groups. Community champions were encouraged to review the list of individuals and the commitments they had made to contact those individuals. Our team strived to remain in contact with community champions to support carrying out this dissemination plan for the online trainings.

This overview webpage from the Centers for Disease Control and Prevention outlines the goals of program evaluation, the different types of program evaluation, and their uses.

“Using Evaluation to Build Evidence,” created by the HB21-1276 Colorado Prevention Collaborative and The Evaluation Center, is a toolkit to guide innovative or adapted community programs in beginning program evaluation.

SAMHSA’s Strategic Prevention Framework has two appendices dedicated to making the most of evaluation dollars and evaluation design guidance. These resources can help initial planning stages of the evaluation.

In the Cannabis Project, the Kirkpatrick Evaluation Model was used to inform analysis of reaction survey and training registration data. Our evaluation questions guided development of a registration survey that those accessing the trainings completed when creating their user account, a reaction survey that participants were prompted to complete after each training, and a yearly participant survey.

These were our evaluation questions:

Reach:

• Who is accessing the trainings?
• What trainings have been accessed and which online trainings have been revisited?

Reaction:

• How have individuals reacted to the online trainings?

Learning:

• To what extent have knowledge, attitudes, or behavior changes occurred that can be attributed to the trainings?

Behavior:

• To what extent have participants implemented what they have learned?

In the Cannabis Project, we offered systems change coaching as an additional support for community champions working to prevent youth cannabis use. This coaching was focused on preparing participants to implement policy or environmental changes that prevent or delay the initiation of cannabis use by youth.

Through the coaching program we heard a community need for connecting with peers in the field, sharing resources, and furthering learning. The project team partnered with a contractor to offer a free, virtual Community of Practice (CoP) for an hour a month for six months. Community champions and local professionals informed the topics covered. The purpose was to better understand resources, skills, and knowledge that professionals working in youth-serving organizations may need to delay the onset of cannabis use, engage authentically with youth, and support youth to thrive in their communities.

We heard from CoP participants that resources and learnings impacted their work and lives.

“So much information is relatable or in some aspect comparable to some of my counties.”

“The podcast [assigned by facilitators as pre-work] was a great reminder that teens do make healthy choices and take smart risks.”