Taking a strategic learning approach to evaluation
Taking a strategic learning approach to evaluation
January 2017
Blogs
Nick Wilsdon, Learning & Evaluation Manager at the National Foundation for Youth Music shares his experience of embedding a strategic learning approach to evaluation. He highlights the journey of change in his organisation, and concludes with helpful tips for anyone embarking on a similar journey.
Context
Youth Music is a national charity, investing in music-making projects for children and young people experiencing challenging circumstances. Our projects help young people to develop musically, but they also yield positive personal and social outcomes too. At any one time, we will have between 350 and 400 active projects, working with somewhere in the region of 75,000 children and young people annually.
Evidenced based funding practice sits at the heart of our organisation. Impact is central to our business plan, and evaluation and learning are a crucial to our mission. We are dedicated to measuring the impact of our work, disseminating learning which serves to inform our funding practice and to inform music-making practice in the sector through the generation of evidence based resources and outputs.
From accountability to strategic learning
Prior to the inaugural meeting of the UK Evaluation Roundtable in 2014, the Institute of Voluntary Action Research (IVAR) published a framing paper which posited three main uses for evaluation: accountability, demonstrating impact and strategic learning.
Over Youth Music’s short history, it is possible to trace a trajectory that runs from using evaluation for accountability purposes, through to demonstrating impact and towards strategic learning.
Our first monitoring forms from around the turn of the century sought to identify if funding had been allocated in line with funding agreements – in short, accountability.
By 2008, we had established an internal Research & Evaluation department, who introduced an outcomes approach and published our first annual impact report shortly after. It was clear at this stage that we were using evaluation to demonstrate impact and understand how our funding had made a difference.
By 2012, we had absorbed this ethos into the roles of those managing grants through the creation of the Grants & Learning Officer role during an organisational restructure, embedding learning within the management of grants. We also had established a consistent outcomes framework across our programme. At the time of writing, we are in a position to use evaluation in (nearly!) real time to inform the decisions we make, and we regularly adapt our strategies in response to the changing circumstances around us; strategic learning has become a very real part of how we operate as an organisation.
Shifting mindset
There are numerous practical guides about adopting impact practice (see for example NCVO’s excellent wiki on ‘How to build an impact culture’ or NPC’s equally good ‘Four pillar approach’) and as many organisations will attest, this is not something that happens overnight. Whilst envisioning significant organisational change may appear straightforward on the surface, enacting it involves substantial organisational commitment. Moreover, organisational buy-in is a pre-requisite to successfully building an impact culture, so it is vital to engage senior leaders, trustees and colleagues in the process. Resources like Inspiring Impact’s Measuring Up! tool provide an excellent framework to reflect on your organisations current impact practice with colleagues and can help identify and prioritise areas for development. Highlighting the long-term benefits is essential in order to gain the necessary buy-in to overcome the challenges in the short term.
On a day to day basis, our priorities for handling information are driven by the three guiding principles which are crucial to strategic learning (as highlighted by IVAR):
- Asking the right questions and getting the right data
- Structuring the work to enable regular use of data
- Effectively processing and using the data
Recognising that we are a relatively small charity, with access to a comparatively substantial volume of information from our grant holders alone, foregrounds the necessity for efficiency. Likewise, our portfolio contains many small, grassroots organisations often with limited resources – as such it is essential that we collect quantitative and qualitative data that is focused on the questions that we need to answer in line with our business planning.
Likewise, we seek to make our data as portable as we possibly can. Our quantitative data is published annually and we are currently developing mechanisms to run automated reports which allow for more frequent and detailed analysis. By coding against our outcomes framework in qualitative analysis software, we have created an index and searchable database of the rich range of experiences of all those involved in our projects. This has allowed us to ground everything from internal strategy documents to external guidance and resources in our evidence base, ensuring it is relevant to our stakeholders.
Cycle of learning
An openness to learning helps nurture an organisational ethos that is open to change. By mapping the ebb and flow of knowledge both internally between teams and externally across stakeholders, inefficiencies and missed opportunities can readily be highlighted. For example, we noticed that our Grants & Learning officer’s daily consumption of information through direct contact with our portfolio was sizeable, yet only a fraction of this was formally captured.
By recognising our Grants & Learning Officers as the gatekeepers of this information, and adopting an organisational approach to learning (see Crossan et al.) we devised a light-touch mechanism to support the individual intuition, team interpretation, and organisational integration in a feedforward process. Our Grants & Learning Officers meet on a regular basis to discuss issues of interest to the organisation, distilling key information for dissemination to the wider staff team in all staff sessions. The resultant documentation is indexed for internal use, and our Communications team prepare extracts for external distribution.
This process allows for the transfer of intelligence beyond the individual, ensuring that key information exists beyond the each member of the team.
As Crossan et al. identify, the process works in both directions; the feedforward process supports exploration (i.e. assimilation of new learning) and feedback processes allow exploitation (i.e. making use of what has already been learned). By supporting learning across the organisation, nurturing the tension between exploration and exploitation, it is possible build an impact culture that becomes rewarding and close to self-sustaining. Through this process, organisations can build an openness to change which ultimately supports strategic renewal.
Tips
- Map the flow of knowledge – Who has access to what information? Who does not have access to beneficial information? Who are the gatekeepers of knowledge? How can you easily share that knowledge both internally and externally?
- Engage your senior leadership team and trustees in the process – Demonstrate your assets and highlight the untapped potential
- Reflect on your organisations impact practice – Tools like Inspiring Impact’s Measuring Up! can help identify strengths and highlight areas for improvement
- Optimise periods of change – significant organisational changes can be stressful times, but they can also provide opportunities to lay the foundations for new ways of working.
- Seek out light-touch ways of capturing knowledge – Hold team and all staff sharing sessions and think about the potential audiences for all information to maximise its potential
- Nurture a culture of learning and allow the organisation ownership over it
- Create resources in accessible places and refer people to them at every opportunity. Index your data where possible, and create structure that allows you to cut it in many different ways
- Ensure that you have the appropriate skills within your staff resources to process, interpret and analyse the data you collect
www.youthmusic.org.uk
@youthmusic
@nick_wilsdon
First published on the ACF website