5 types of data for assessing your work: An explainer
I’ve worked with many charities and social enterprises over the years—to help them work out what data they need to understand and improve their work. It can be a confusing question, particularly when people are new to evaluation and research. During our co-design work for the Impact Management Programme we have found that even the word ‘data’ seems to scare some people off.
But recently I have been struck by a simple but very helpful observation: there are basically five distinct types of data to consider when assessing a service. These are as follows:
1)User data
Asks: Is your service effective at reaching the intended target group?
Establishes: The characteristics of your service users.
If your service aims to, for example, reduce loneliness you will need to know that you reach people who are actually experiencing loneliness.
The way to approach this is to go back to your theory of change, which should state clearly who you aim to work with and their particular characteristics. This can then be translated directly into the information you collect from people when they first come to the service. You can do this by just asking your users—for example, ‘How often do you feel lonely?’—or, if applicable, or by finding out from the organisations that refer people to you.
How often to collect it? Routinely. User data is best collected from all your service users during the sign-up stage or shortly afterwards.
2) Engagement data
Asks: How effective is your service at continuing to engage your target service users?
Establishes: The extent to which people use your service and how they use it.
This is about whether or not you effectively deliver the service to your intended users. Familiar terms like ‘activities’ and ‘outputs’ are part of this; but I prefer ‘engagement’ because that’s what actually matters.
Specific questions include:
- How often do people come? For how long?
- How engaged are they?
- What kinds of activities and processes do they engage in?
How often to collect it? Routinely. This information is best collected on an ongoing basis—as and when people use the service—ideally using an electronic case management system with a direct link to user data, so you can look at who engages and who doesn’t.
3) Feedback data
Asks: What do people think about the service?
By getting feedback you will really start to test your theory of change; to establish whether your service gets the reaction you want; and whether it is beginning to work in the way intended. Specific questions might include:
- Do people enjoy the service? Rate it? Find it useful?
- Would they recommend it to someone else?
- What aspects do people rate the best/least?
- What is the quality of relationship you establish with them?
- How could the service be improved?
How often to collect it? Routinely. Keep checking this using both formal methods—such as surveys or simple satisfaction rating systems—and informal methods—like social media, one to one conversations or suggestion boxes. You should aim to collect feedback data from beneficiaries, staff, volunteers—basically anyone involved in the project.
4) Outcomes data
Asks: How have people been influenced or helped by your service in the short-term?
Establishes: The immediate resources, benefits or assets that your users gain from the service.
Key questions might include:
- What is different now? Do people make positive changes in their knowledge, attitudes and behaviours?
- How, if it all, do they think your service has helped?
- Which aspects of the service have helped which types of service users in which circumstances? And which have not?
How often to collect it? Occasionally. Outcome data is best collected by staff or volunteers because they develop the strongest relationships with people. But they should not spend all their time collecting this data, and you may not need to collect it from everyone—you could just collect it from samples of service users or case studies.
5) Impact data
Establishes: The long-term difference achieved for individuals, families, communities.
Asks: Have the outcomes achieved (above) helped people to change their lives for the better?
Your impact is the difference you make. Depending on your service’s theory of change, this might be increased educational attainment, improved physical health, reduced crime, stronger communities, increased social mobility and so on.
How often to collect? Exceptional circumstances. This is the hardest data to collect so should by done through higher-quality evaluations, when enough time has passed and ideally using a comparison group. Many services do not need to collect this data at all, as I have argued in a previous blog.
___
Having defined these categories I now use them every time I’m working on a charity’s data requirements. I find they help us cover all possible data collection needs in a systematic way, and identify any gaps or challenges. They have also been very well received by the organisations we’re working with on the Impact Management Programme—so we wanted to share them more widely.
Do have a go using this in your organisation, and feel free to let me know what you think in the comments.