Project Evaluation

Why evaluate?

Project evaluation is not only a required component of the reporting process of TLEF projects, but it is also an important tool to help you understand what is and isn’t working with your project and which areas need more attention. Ongoing and iterative evaluation will help you to improve project design and implementation, make informed decisions, consider sustainability factors, and ultimately reach your intended goals.

How to evaluate?

Evaluation involves periodically gathering data and reviewing it to determine if stated objectives are being met. CTLT’s evaluation team can help with all aspects of evaluation:

  • Developing a project evaluation plan,
  • Identifying the best method to collect evidence of outcomes,
  • and collecting and analysing data to determine your project’s impact(s).

Here are some resources to help you get started with evaluation:

Sharing your evaluation outcomes

After the funding period ends, you are required to complete a closure report and provide evidence of how changes/impacts were achieved and what evaluation strategies were used. Below are a few examples of clear evaluation summaries:

“In order to assess impact, we engaged in three evaluation strategies, developed for each of the three groups involved … We conducted interviews with 6 faculty partners to assess their overall experience with the project and specifically, the Library’s involvement in OERs. We also asked what they would hope to see for ongoing sustainable support. Several faculty commented that the Library had played an important role in their work in issues such as copyright, CC licensing, publishing and dissemination platforms, and serving as cross-disciplinary connectors with others doing similar work. Most expressed the hope to see some sort of sustained Library support … To assess [student partner] experience, we administered a Qualtrics survey, meant to gauge their learning, work experience, and value of this exposure to open approaches. Students consistently expressed what a valuable learning experience this had been for them.  This was especially true for students involved in creating new content or resources. … To evaluate the impact of the open resources [for students in the classroom] we disseminated a Qualtrics survey to capture student  awareness of OER, their sense of cost savings, and attitudes towards using OER  as alternatives to traditional materials in their classes. Students’ feedback about their OER class experience was very positive.”

Leonora Crema, Library Support for Open Textbook and Open Educational Resource (OER) Creation

 

“We created a beginning-of-term/end-of-term survey for students to probe their knowledge of/understanding of signed languages … [sample survey link and graphs included] … [The response data shows] that the majority of beginning students in Ling 100 thought that yes, ASL is based on English (at least somewhat), which is incorrect. Advanced students at the beginning of a term were much more likely to correctly say “no,” but even then, a handful thought yes. By the end of the term, only one maintained this misconception, and several of the others actually had an even more nuanced view (indicated by the handful of “other” responses). The second set of graphs shows responses to the cultural questions “Is being deaf a medical condition, and should it be considered a disability?” Almost everyone in Ling100 thought the answers to both questions were “yes.” The students in Ling 447 started out with an interesting dichotomy: the majority thought that it is definitely a medical condition, but were split on whether it’s a disability. Meanwhile, a sizeable minority thought it was definitely NEITHER a medical condition NOR a disability. By the end of the term, almost all the students had a much more nuanced view of the situation, with most of them choosing “other” (because the question especially of disability is largely a cultural and personal decision by people who are d/Deaf).”

Kathleen Hall, Integrating Sign Languages into the Linguistics Curriculum

 

“…eChIRP has been offered for free as the primary text resource in CHEM 123 (all sections, all terms), saving students significant money compared to the traditional combination of textbooks ($125) … survey results suggest that we successfully created an easy-to-use resource, with students (n=1102) rating the ease of navigation 75/100 on average (0=very difficult to navigate; 50=neutral; 100=very easy to navigate) and 90% of students rating the ease of navigation above 50 (neutral) … We wrote xAPI statements to track how students use the eChIRP … students were actively engaged with these components because we recorded an average of over 15,000 xAPI statements per day (50 interactions per active student) and up to 90,000 per day before the final exam (see figure 1) … In a controlled study comparing interactive videos to their traditional counterparts.

We found that students who watched the interactive video self-reported that they felt significantly more engaged (p<0.001) compared to students who watched the non-interactive videos (see figure 2). Students also indicated a strong preference for interactive videos over non-interactive videos, as shown in figure 3 below … We have some evidence that the eChIRP has been particularly helpful for students at risk of failing CHEM 123. For example, failure rates in CHEM 123 at Vantage College have dropped since the introduction of the eChIRP. In S2015 (before the introduction of the eChIRP), 9% of students failed. After a small pilot release of the organic in S2016 the failure rate dropped to 6%. In S2017 after the full organic half of the eChIRP was released, the failure rate decreased again to 4%. Finally, after the full pilot release of the eChIRP in S2018 the failure rate was just 1.5%. This trend suggests that having a resource such as the eChIRP maybe helping more students successfully complete CHEM 123.”

Kayli Johnson, Development of an electronic Chemistry Integrated Resource Package for CHEM 123

 

Publications and presentation citations shared in the closure report will be posted on the TLEF website as a means to highlight the scholarly outputs that come as a result of TLEF projects. You are also encouraged to share your findings at CTLT events such as Celebrate Learning Week, or Celebrate SoTL.

Support Available

The CTLT Research and Evaluation team offer a bi-yearly evaluation workshop for TLEF grant holders, as well as workshops related to ethics for teaching and learning projects, survey design and focus groups. You can view a listing of upcoming events which are hosted through the Institute for the Scholarship of Teaching and Learning (ISoTL). 

Do you need more support with evaluating your TLEF project? We can help! Our team of experts is available to help you with the evaluation process through your project’s lifecycle. Please contact Trish Varao-Sousa (trish.varao-sousa@ubc.ca) for evaluation consultation and support.