From Measuring to Learning; how to improve our programs by joint evaluations

We aimed to improve our debate programs and outreach to young people all across Europe by improving our planning, monitoring and evaluation capacities and institutionalise learning from evaluations. IDEA NL and 7 European members, national debate education organisations, have worked with young people for at least a decade and developed basic evaluation capacities, of which little was shared. The partnership aimed to create a cooperation program to develop common indicators for how to measure the impact of debate on young people and how to learn from evaluations. The partners received evaluation and analysis training, tested tools, shared experiences test and analysed the results together. The project had the following objectives: • To exchange good practices on debate evaluations and how to learn from them. To develop one common tool for such evaluations to increase learning from each other • To develop and test one evaluation tool of longer-term high school debate programs to be used in a number of European countries. The evaluation will include active citizenship, self-esteem, school attainments, critical thinking skills and attitudes towards citizenship and politics (based on a previous discussion of our evaluation needs). • To promote learning from evaluation practices and to promote the incorporation of learning and evaluation practices in organisational structures. • To train the participants in developing evaluations, setting up evaluations including baselines, developing innovative ways of learning and evaluating and involving youth in evaluations • To use the evaluation tool to establish the effects of our debate programmes and identify ways in which to improve them • To disseminate the best practices in evaluation and develop guidelines for organisations in IDEA’s network on how to evaluate events, workshops and debate programmes, contributing to further development of the whole network We first focused on understanding the theory and practices of evaluations and shared previous work. We collectively developed a common set of test tools during and between partner meetings in March and May 2015, focused on critical thinking and democratic values. These were tested over summer and collective testing with debaters and a non-debater control group took place in the school year 2015-2016. 1821 Debaters and 670 non-debaters, 109 debate clubs and 140 coaches took part in the evaluation. We developed a set of interacting tools that each looked at different aspects of debate. We used a pre-test/post-test questionnaire and test to look at critical thinking and democratic values like tolerance, political efficacy, civic and political activities and support for democracy. We tested a willingness to change opinion through debate in one-off tests at tournaments. We tracked debate club activities through monitoring and overview surveys and Kirk-Patrick questionnaires about what respondents learned from debate. These and the focus groups taught us about academic achievement and self-esteem, as well as the other topics in more depth. We analysed results and produced reports during fall 2016 and shared the results nationally in training and dissemination events in all countries reaching 425, as well as during a combined expert meeting, dissemination event and partner meeting in February 2017 (reaching 27). Partner meetings were also held in September 2015, February 2016 and April 2016 to analyse the evaluation work and improve on the project. The program has resulted in a solid training and learning by doing in evaluations as well as a thorough analysis of our intervention model in a Theory of Change, a discussion and set of best practices how to use debate as a method to empower young people as active citizens and concrete recommendations about how debate education impacts the academic, social, psychological and political development of young people. Debate trainers can use this information to better understand how to improve the pedagogic and didactic approaches of debate education programs. We are currently devising ways to implement the lessons learned from the project. We have trained youth (work) organisations how to use evaluations as a guide to improving their programs, and how to involve youth in their evaluations. Organisations involved have developed capacity to plan-monitor and evaluate their programs better and have developed proper learning loops to improve their debate programs. We are developing a training method and toolkit to make evaluation methodology even more sustainable and reach out to more organisations. The project created increased cooperation and knowledge sharing on debate effects and evaluation practices in different geo-political contexts across our consortium: we increased access to academic and practical studies and improved cooperation with partners on the subject. We are still sharing the results broadly and widely in the youth and education sector.