It’s now six months since I joined the Fulfilling Lives national evaluation team. My background is in community and criminal justice research, so I have a degree of familiarity with the target group. I have evaluated initiatives to help re-engagement on release from prison, to provide pathways to education and ultimately employment, and to help with accommodation and independent living. Offenders often have chaotic lifestyles and multiple needs including experience of homelessness, alcohol and/or drug dependency, and/or mental health issues. You can often find childhood trauma, special educational needs or attachment issues as well. What struck me time and time again when listening to offenders and their workers tell me their stories was the lack of coherent support available to people who are in desperate need of help.
A key feature of Fulfilling Lives is that it is a learning programme; it aims to gather and share learning about what works, what doesn’t and what needs to change. A frequently used method to collect views and experiences from which to develop learning is the survey. In this blog, Kerry Dowding (Research and Evaluation Officer, Fulfilling Lives South East) shares her tricks and tips on how to get the most out of your survey design. Kerry has been creating surveys for the voluntary sector for the last eight years, and received a Masters of Research in 2016.
It’s 4.50pm, and you remember that you promised to send out a few survey questions by the end of the day. You scribble out some questions that roughly fit the bill, and send them off. The responses you get back are less than inspiring. So you end up going with the ideas you originally had for the project. We’ve all been there!
Why bother with survey methods?
A good survey can bring so much more to the table than this, if they are your method of choice (and they don’t suit every situation all of the time). A good survey can provide quality information to help us change systems and organisations, and to move forward in a way which is inclusive and co-produced. When done well, they are the gift that keeps on giving. Looking to get more from your surveys? Here are some hints and tips I’ve picked up along the way.
Before you start
- Why are you doing the survey in the first place? Write one sentence explaining the question you would like your survey to answer (and some smaller aims within that, which is fine – as long as they’re clear). Refer back to this and be strict – leave out anything that doesn’t help you to answer your question.
- How much time do you have to analyse this, really? If you have limited time, write fewer questions. Better to ask five questions that will be well used than twenty which you end up skimming over.
- Do you need to offer incentives? Some groups, like clients or volunteers, might be more likely to do your survey if there is an incentive. This is especially true if the survey is long, or they haven’t been in touch with the organisation for a while.
- How do you want to divide your results? You might like to split your data by area, hostel, gender or shoe size, to compare what different groups say in response to your questions. It’s important you think about this first so you include the questions you need. Just make sure these splits have a clear benefit to your work. For example, asking about shoe size is only really useful if it were clearly linked to something like NDT Which it’s not.
Question types and order
- Keep everyone in the loop. Tell them who you are, why you’re asking them questions, how long it will take, and what you’ll do with it all. It improves your response rate and is generally a good thing to do for people who are sharing their thoughts with you.
- Mix up open and closed questions. People can get bored with all open questions and provide poor quality answers. On the other hand, multi-choice ones don’t give you a lot of information to go on. Try putting open questions towards the end, so people get to them after they’ve been ‘warmed up’ by the easier ones.
- Throw in other types of question. Wake people up a bit by asking them to rank different options, use a slider to rate something from 0% to 100%, or use visual methods. There are lots of different types of question that are more fun to fill in (and analyse) than your regular.
- Don’t scare people off! Keep the sensitive questions for later in the questionnaire. Asking about someone’s criminal convictions or worst fears right off the bat can put people off.
A word on wording
- Be friendly and clear. Use clear, short words that everyone can understand. Think about trying to keep a warm, chatty tone (depending on your audience), and use plenty of examples for any tricky concepts – so there is no ambiguity about what you mean.
- Only one question per item! If you ask 3 things in one question, you don’t know which part of the question the person is answering. Asking ‘Do you feel happy, proud, confident and empowered through your work with Fulfilling Lives?’ in a yes/no question makes it unanswerable. A good rule of thumb is to banish the word ‘and’ from you questions as much as possible, so you can trust the answers you get.
- Keep an open mind. We often hope answers will be positive, but don’t let this optimism seep into your questions and make them look biased. Rather than asking if things are ‘better’ or ‘worse’ than before, just ask about what changes a person has noticed and see what they say.
- Make sure you cover all bases. There should always room for people to give an answer you didn’t expect, say they are unsure or surprise you. Sometimes the best answers refer to ideas you hadn’t even considered.
Getting it out there and making it useful
- When the results are in, get together with others who were involved in the survey to think about what to do next, and when. Set deadlines to act on what you’ve found, so it doesn’t fall down the list.
- Feedback to everyone. Make sure that those who took part know how it went – and any changes you’ve made as a result.
- Improve your method. Take five minutes to talk about how the survey was carried out and what you would do differently next time. This is important so that what you’ve learned isn’t lost.
- Share where you can. Think about if there is anyone else who might like to see the anonymised results/report that comes from the work. Data is an important product these days and being open (as long as it’s safe to do so) can improve relationships in the future.
It might not always be possible to do everything listed here all the time. However, finding ways to improve your survey work can have real and worthwhile benefits: reaching more people, getting better answers, and saving time by keeping focused. We’ve seen the results of this within our own organisation. As a project focused on learning and research, it’s important that we get the right information for ourselves, and to inform how we change systems. Try putting some of these ideas into your work locally, to banish the rushed survey questions forever!
References & further reading
Bryman, A. (2015). Social research methods. Oxford University Press. pp 231-267
Designing a Survey (January 19th, 2018). Retrieved from https://www.sciencebuddies.org/science-fair-projects/references/how-to-design-a-survey
Harrison, Chase. Harvard University Programme on Survey Research: Tip Sheet on Question Wording. (2007, November 17th) Retrieved from https://psr.iq.harvard.edu/files/psr/files/PSRQuestionnaireTipSheet_0.pdf
How to write in plain English. (2018, January 19th). Retrieved from http://www.plainenglish.co.uk/files/howto.pdf
Rea, L. M., & Parker, R. A. (2014). Designing and conducting survey research: A comprehensive guide. John Wiley & Sons.
Thomas, G. (2013). How to do your research project: A guide for students in education and applied social sciences. Sage. pp 207-17
At the centre of the Fulfilling Lives (Multiple Needs) programme is evaluation and here the workforce is on the frontline. For it is they who source, collate and complete the two key measures – the Homelessness Outcomes Star and the NDT Assessment – with the service beneficiaries. Frontline workers say something they enjoy about the programme is the move away from a purely target driven approach. To make the most of this we need to make sure that the essential evaluation is not a chore. Not tasks that are completed as an add-on to a day’s work but ones that are integral to the way of working alongside beneficiaries and peer workers.
Thus, we need to know what it is about the projects that captures the imagination and creativity of practitioners. Ask what is it they like about the work alongside beneficiaries which will secure as full an evaluation picture as possible? It is my contention that it is knowingly being part of a virtuous circle – one which takes theory into practice, evaluates that practice and thereby shapes a better informed and improved theory that can go on to influence wider practice, which is so rewarding. Pride – for both frontline worker and service user – is in curiosity satisfied. In the Fulfilling Lives (Multiple Needs) overall programme then, evaluation of just how each individual beneficiary achieves their aspirations should be accumulated, lead to system change and improve the lives of many people, families and communities with multiple and complex needs. Evaluation, and the curiosity that fuels it, are vital to generating that positive feedback loop on which the projects depend for their results.
In my experience curious practitioners have:
- Clarity of programme and project purpose
- Detailed understanding of their job role and responsibilities
- The skills and training required to make good use of the evaluation tools
- Time to collect the data and information needed
- An ability to make evaluation an engaging part of their relationship with beneficiaries
- Involvement in the analysis and assessment of data and information
- Encouragement to bring their wider knowledge and experience to support and/or challenge data and information
- Support to ask difficult questions about the reliability of data and information
- Knowledge and understanding of the common pitfalls in research and evaluation
- Feedback on what the data and information is revealing across projects
Essentially the outcomes of evaluations should help not hinder, be practical for practitioners and challenge in a way that stimulates further curiosity. Evaluations must be inquisitive about the interventions and approaches. The more we know about what works, and why, the better. In this way the best workforce can be recruited, trained, supported and retained. What follows is not just an effective service but one that is sustainable and replicable because it is founded on research. For, as we all know “research is formalised curiosity. It is poking and prying with a purpose”.
What we need to know from project beneficiaries, frontline workers and their managers is what will assist us all to undertake really effective evaluations? How can we make sure we make the best use of the evaluation tools we have? Do drop me a note at email@example.com or call on my mobile 07947 680 588.
 Zora Neale Hurston
I have blogged previously about purpose and it’s time to move on to the detail – where the devil lies (alongside salvation) . The question is how do we recruit, train, support and retain an effective workforce? A workforce with the knowledge, skills and experience to change lives and systems alongside people with complex needs including homelessness, offending, substance misuse and mental ill-health. One that supports people to overcome barriers to services and engagement. A workforce that is genuine and honest about the collaborative involvement of individuals with lived experience.
It is a big ask to come up with an exhaustive list of what the workforce does and how it goes about its tasks. Anyone that has spent time drawing up job descriptions knows the lengths you can go to trying to capture everything. Equally, they know the struggles of seeking to be both precise and concise. Perhaps that’s why so many job descriptions are rarely used in guiding day-to-day work and are only got out either when there is a vacancy or a workforce problem. In projects like Fulfilling Lives (Multiple Needs) the issues are compounded because many of the roles are evolving and breaking new territory.
Helen Gavaghan, Senior Engagement Worker at Inspiring Change Manchester (ICM), gives an inkling of some of the things workers do to support people to think about change. She says some ‘traditional’ approaches need challenging and staff freed up to be open about working with both their own and their client’s lived experience. She concludes by suggesting learning about what it is that staff are doing is collected and brought together as a tool-kit for other practitioners.
In the Fulfilling Lives (Multiple Needs) projects, when considering the learning that is being fed back overall, I feel there could be more detailed examination of what the frontline project workers do that makes a difference to people’s lives. I believe that there is some valuable learning that could be collected and shared and, if it isn’t, may be lost. General explanations of the approach, such as “building trusting relationships” and “avoidance of judgement” are fine – but I believe are in no way getting down to the specifics and nuances of the great work that is going on.
Vic Citarella delves further into the significance of ‘purpose’ for service users and the workforce in the Big Lottery Fulfilling Lives (Multiple Needs) projects.
Indications are emerging from evaluations that the ‘purpose’ of each Fulfilling Lives project is very important in generating value and ownership. For beneficiaries, engaging in meaningful activities appears to create a vital sense of purpose. Equally for the workforces involved feedback suggests that the ‘purposeful’ nature of job roles generates added value and personal motivation. In my last workforce blog, I suggested that the value stemmed from four sources:
- Meaningful service user engagement
- Concepts of open-endedness and persistence (turned into practice)
- The ideas around psychologically informed environments (PIE) and the like
- Systems Change
In focussing on workforce matters in this series of blogs I have therefore dug a bit deeper to address questions such as: what is the purpose of purpose? How is purpose agreed, described and refreshed? What is it about the four sources which users and workers value that drives them on to achieve their goals?
Sarah Robinson (CFE) and Wayne Richmond and Nicola Plumb (Blackpool Fulfilling Lives) share their reflections on a National Expert Citizens’ Group Workshop
On Monday 28th and Tuesday 29th November 2016 CFE facilitated a two day workshop for the National Expert Citizens’ Group (NECG). The NECG is a group of individuals with lived experience of multiple needs; the group is formed by representatives from each of the 12 funded Fulfilling Lives project areas. The Monday was a training and planning day for the experts by experience. The day focused on training delegates to chair meetings/events and facilitate workshops at events. It was an opportunity to build the confidence and skills of the experts. The activities used to test their skills focused on deciding the direction of the NECG over the next 12 months.
In the first of a series of blogs, Vic Citarella considers the crucial role of the workforce in Fulfilling Lives for people with multiple and complex needs. Vic is keen to start a dialogue with projects on this topic. You can get in touch with him using the details below.
“The CFE and University of Sheffield 2nd annual report into the national evaluation of Fulfilling Lives: Supporting people with multiple needs programme has chapters on ‘interventions and approaches’ and on ‘working the frontline’. The report says it raises as many questions as it answers but without doubt it pinpoints the workforce and what they do as the mission critical factor in the projects. More is promised by way of research and future evaluation. That means, among other things, dialogue with the practitioners, the managers, the stakeholders and the customers of the services.
In the hunt for a bike for my daughter the typing of “kids bikes” in Google returned just shy of 78 million results in less than one second! After years in research I still find it extraordinary the number of sources of information that are returned when performing any internet search (“kids bike bells” a meagre 1.5 million returns in 0.76).
Chatting with friends over the weekend the topic turned to the works of Keanu Reeves (don’t ask) and his big break: the film Bill and Ted’s Excellent Adventure (1989!). We all agreed that the central philosophy of ‘Be excellent to each other’ was a pretty sound one.
I am part of the National Expert Citizens Group (NECG). This is a group which draws people from all twelve Fulfilling Lives areas together to learn and try to influence on issues which are important to us. We all bring our own skills and expertise but the one thing that we have in common is that we all have lived experience of multiple needs.
On Friday 29th January we came together in Stoke-on-Trent to meet with Home Office representatives and Public Health England. On the agenda: The National Drug Strategy Review due to be published in March 2016.
It’s coming up to the end of the year so I thought I would take some time to reflect and take stock. I started my role in January excited. The pioneering nature of the Fulfilling Lives programme interested me but above all was the hope that lives would be transformed both during the life of the project and in a real on-going way into the future. The opportunity to take any part in this, however small, seemed both a privilege and a responsibility.
‘What’s in a name? That which we call a rose, by any other name would smell as sweet.’
Romeo and Juliet (11, ii,1-2)
Shakespeare’s Juliet knew that Romeo’s surname did not take into account all the things she loved about him; he was more than his name. Yet, in order to make sense of our world, we continue to ascribe names and definitions to everything, perhaps forgetting that they can never truly encompass everything that something is.
Events like the National Multiple Needs Summit (22nd April 2015) are always a great opportunity for evaluators like me to have our beloved measures and outcomes illuminated by the personal touch of human experience. Over 300 delegates filed into the circular grand Assembly Hall of Church House in Westminster taking in the impressive glass dome and oak panelled walls. The building was used by the two houses of parliament during the second world war and has echoed to many an historic speech, including Churchill announcing the sinking of the Bismarck, no less.
I am the Communications Lead for the National Expert Citizens Group and the Independent Futures (IF) Group in Bristol – we are the advisory group of people with lived experience of multiple and complex needs. We are equal partners for the Fulfilling Lives project, a Big Lottery funded initiative to see how we can help individuals who keep falling through the gaps in the system. It is evident that many who come under this category – who have experienced at least three out of the four problem areas; mental health, substance misuse, homelessness and offending behavior, are still bouncing from service to service not getting their real issues dealt with properly. Having had hard earned street level experience of accessing these services means that our voice matters. Increasingly we are not only asked to contribute but actions are being taken as a result of what we say.
When we first began working on the Fulfilling Lives: Supporting people with multiple needs evaluation, a question we asked was ‘how many people are affected by multiple and complex needs?’ It seemed like a simple enough starting point. Two years on we’ve learnt very little is simple in trying to address issues that are inter-related and mutually reinforcing, particularly when the service response is too often inflexible and designed to address single issues only. The disconnected nature of support is reflected in the data which can provide an indication of levels of homelessness, offending etc, but not how these commonly related issues overlap. However, a recently published report from LankellyChase Foundation goes some way to addressing this and helping to answer that first, not so simple question.
What’s the collective name for a group of evaluators? A measure of evaluators perhaps? Or how about a puzzling of evaluators? A squabbling of evaluators? Hopefully not the latter.I recently met with fellow evaluators working on strategic investments funded by the Big Lottery Fund. The investments vary from the very young (A Better Start) to old (Ageing Better), from specific needs (NEET young people) to multiple needs (homelessness, substance misuse, offending and mental health). However, there is much common ground in relation to evaluation, with those involved seeking to measure the true impact of those investments and find out ‘what works’, for whom and in what circumstances. Continue reading
This guest blog is from Service User Engagement Co-ordinator Justin Nield. Justin has been working on the Fulfilling Lives: supporting people with Multiple Needs Blackpool Programme funded by the Big Lottery.
I haven’t always been a Programme Co-ordinator and I lived with Multiple and Complex needs for most of my adult life. I spent over 20 years in active addiction, suffered with enduring mental health issues and ended up living on the streets, frightened, confused and vulnerable. Continue reading
I recently spent an informative and enjoyable afternoon at the CLINKS event “Justice Data Lab – one year on”.
The Justice Data Lab is an exciting step forward in making use of government data to better understand what works in reducing reoffending. It is part of a wider, ambitious project led by NPC to open up government data to the not-for-profit sector to help them understand the impact of their work.
A key, but challenging, element of the Fulfilling Lives: Supporting people with multiple needs evaluation is using administrative data, on offending, benefits, use of health services and so on to estimate the cost of supporting people with multiple needs, and evidence the impact of the programme. So we will be keeping a keen eye on how this work develops over the coming months and years and considering ways the evaluation might benefit from it. Continue reading
It is, perhaps, self-evident that people with complex needs frequently require correspondingly multiple and complex responses…. wrote Henwood and Hudson in their 2009 CSCI study Keeping it personal. Now as Carers’ Week passes we have, in the Care Act, the strongest rights yet for carers. When put together with the duty of assessment for young carers, in the Children and Families Act, the legislative framework is suitably reflective of the very complexity identified for policy makers five years ago. It is a challenge for the Fulfilling Lives: supporting people with multiple needs evaluation to explore, understand and share how project investment resolves the problematic issues of real life complexity. Those involved in caring relationships shaped by homelessness, criminal behaviours, substance misuse and fragile mental health are potential benefiting contributors to making the most of that significant investment. The evaluation process has to identify both the benefits and contributions of carers to the success of Fulfilling Lives. Continue reading