Part 5 Evaluation: Does ABC LD work?
Part 5. Does ABC LD work (PDF 12 pages)
At first, ABC LD can seem a challenging methodology. The dynamic, group based, rapid development format may feel quite different to existing methods of curriculum design, and even ‘normal’ academic development events. The questions from people unfamiliar with the method are obvious. How will our academic colleagues react? Can such an apparently simple method achieve any useful results? Does it work for all disciplines? Won’t sceptical and resistant colleagues disrupt the workshop? What if I can’t answer the questions that come up? These, and others, are all perfectly valid concerns, and not easy to answer without experiencing the workshop directly. Since launching ABC LD in 2015, the UCL team have therefore made a considerable effort to facilitate ‘train the trainer’ ABC LD workshops in many different institutions, and indeed that was a central component of the Erasmus+ project. The belief remains that by seeing how the format works by participating, attendees can judge for themselves whether it was relevant and useful for themselves and their institutions. In parallel, ABC LD has progressed through several stages of more structured evaluation, from initial ad hoc feedback through qualitative evaluation during a project funded by the Higher Education Funding Council England (Hefce, 2018) to a more quantitative, survey-based approach as part of the Erasmus+ project (references). A few representative participant responses of the Hefce research are as follows.
‘We haven’t had such level of detailed discussion as a team. I think the structure and the materials are facilitated well.’
‘It makes you think about: OK, we are going to use this technique, but where, how, for what and how does it fit with everything else? And this is the way into that, I think.’
‘It was an eye opener. I found it really useful to think about categorising how the learning objectives will be delivered and assessed, and examining the variety of ways that these can be achieved. It made me think more deeply about what skills the students can develop by making them responsible for their learning journey and not simply the content that needs to be delivered to them’.
For the ABC to VLE project evaluation, a questionnaire was agreed by the partners who ran 84 ABC Learning Design workshops in 11 countries, with more than 1035 participants. Feedback was sought from over 60 of the workshops and the project team received 344 participant responses and a further 42 follow up responses about whether they had implemented their plans.
To what extent did you reach the following outcomes by the end of the workshop?
The opportunity, perhaps ‘permission’ to discuss how to design the student journey in a non-judgmental, collegiate atmosphere is always much appreciated. Although only small majority found it had an immediate impact on course redesign, it should be remembered ABC LD is a short-form intervention with academic teams with little or no preparation. Moreover, ‘redesign’ of the course in terms of changes may not be the only desirable outcome. The examination and overt justification of the current course may itself be valuable and this relates to the next point. The finding that participants feel more confident teachers after ABC LD is as unexpected as it is pleasing. We have often seen that the openness of ABC LD can help validate current practices and designs among peers. Teachers can be quite self-critical of their own methods but may find, when discussing with colleagues, that the underlying rationale is quite robust and may require only minor adjustments. Thus, even if few changes are made to the design, the outcome can be satisfying. Not all workshops (at least prior to Covid) focus on implementing educational strategy or policy, so it was almost surprising that over a quarter of respondents recognised a strategic component.
Links and resources
Hefce evaluation report 2018 (extract) link
Associated ABC to VLE case studies
Oxford – Extended interview with an ABC participant (video 7m50)