Do-It-Yourself Evaluation

held in Birmingham on Wednesday 31 March 2004

This symposium report is divided into the following sub-sections. If you wish to make a shortcut to any particular sub-section then please double-click the relevant heading.

Introduction

Speakers

Presentation 1: An introduction to evaluation. Why do it yourself?

Presentation 2: NIFTY Project: How to undertake small scale evaluation - a Research in Practice development project

Presentation 3: Agency-led workshop session: Service case studies

Conclusion

Key references

Glossary of evaluation terms

introduction

This symposium focused on the long-term benefits of evaluation for social care organisations and the children and families they serve. It began with a discussion of the history, context and role of evaluation in providing good quality social care. This was followed by an overview of the research in practicedevelopment project: NIFTY (Neat, Informative, Feasible, Timely, Yours). The afternoon session was devoted to agency-led workshops facilitated by those who participated in the NIFTY development project. These workshops encouraged reflection on the many issues that practitioners and non-researchers face when undertaking their own small-scale evaluations.

speakers

Celia Atherton is director ofresearch in practice and chaired this symposium.

Mog Ball is the Principal Investigator for the National Evaluation of Sure Start. She is also Honorary Associate Research Fellow at the Institute for the Study of Children, Families and Social Issues, University of London. Mog's presentation outlined the history, context and role of evaluation in providing good quality social care.

Catherine Shaw is Principal Research Officer at the National Children's Bureau and has responsibility for evaluation. Catherine's presentation outlined the scope and content of the NIFTY project.

The agency-led workshops sought to share the learning experiences from the NIFTY development project. See relevant subsection (below) for a summary of the content and facilitator(s) of each session.

presentation 1

an introduction to evaluation. Why do it yourself? by Mog Ball

what is evaluation?

• Evaluation involves systematically collecting information related to the aims and objectives of a service, analysing this information, and then judging the findings. Having made the judgement, a decision has to be made (informed by the evaluation) that may affect service provision.
  
• The idea of evaluation usually comes from reflecting on what you are doing and a desire to improve on this - eg who is using this service? Who is it reaching? etc.

how do you evaluate a service?

• It is important to identify the aims and objectives of a social programme or single service. Only then can you identify ways of measuring its success (ie how far the programme/service satisfies its own aims and objectives).
  
• It is worth looking at past examples of evaluation in the area in which you are interested. These may flag up any potential difficulties as well as provide a useful model of evaluation.
  
• There are toolkits available to support evaluation (see key references, below).
  
• Innovative projects that are based on promising academic research need to have a fixed time limit so that they can be properly evaluated before being rolled out more widely.

how does evaluation 'fit' into service provision?

• Evaluation should be an integral part of good service provision, ie evaluation processes should be part of one's day-to-day work so that you are collecting information whilst doing the work and doing the work whilst collecting the information.
  
• Participation and ownership of the information collected are essential to DIY evaluation.
  
• No outside body can do an evaluation of a service without the cooperation of service providers. Evaluators need to reflect on, and be sensitive to, the impact of evaluation on those involved with service provision.
  
• It is important to recognise that this is where practitioners are in the driving seat - the people who do the work are the only ones who know what is really going on and are therefore best placed to evaluate and improve it.

what are the main benefits of evaluation?

• Evaluation should enhance the work that you do - it should produce learning or service improvement.
  
• None of us can understand what we do unless we occasionally step back to evaluate what we have achieved. This is a natural and proper thing to do.
  
• Most people want to do a good job and get job satisfaction - evaluation can help them to achieve this.
  
• This process can also be applied to oneself: people can learn how they do their work by monitoring and writing things down - this material can then be collated to judge/evaluate their own practice, ie it can help people understand the strengths and weaknesses of their own practice.
  
• A positive experience of self-evaluation can make people realise that monitoring their practice, enables them to reflect on and make informed improvements/judgements about their practice.
  
• Local Sure Start programmes are often nervous about the requirement to evaluate. They feel it will expose them, but actually the requirement is there to encourage reflection and improvement. It entails learning about the service being provided.

what are the main pitfalls of evaluation?

• There are some types of evaluation that you cannot do yourself. Failure to recognise this can lead to unnecessary muddle. For example, the national evaluation of Sure Start cannot be undertaken locally.
  
• It is important to recognise the difference between policy evaluation and service evaluation: central government is responsible for evaluating the long-term impact of its policies whilst local government is responsible for evaluating the effectiveness of its service provision.

presentation 2

NIFTY Project: How to undertake small scale evaluation - a Research in Practice development project by Catherine Shaw

What is NIFTY?

• NIFTY is an 18-month development project run by research in practice in partnership with the National Children's Bureau (NCB)
  
• The acronym stands for: Neat, Informative, Feasible, Timely, Yours
  
• It was a response to a demand from agencies for help with evaluation
  
• It provides tuition, consultancy, mentoring and networking support to a group of Partner agencies that want to develop their skills and confidence in conducting and commissioning evaluations.
  
• To find out more about the aims and objectives of NIFTY as well as the participants and projects see the Development Projects section of the research in practice website or email Sal Lodge.

what do we mean by evaluation?

• Evaluation is derived from the word 'value', ie making quantitative and qualitative judgements of value
  
• Evaluation involves a series of planned activities focusing on a specific project, programme or service
  
• Evaluation entails addressing clearly defined evaluation questions
  
• Evaluation should be systematically conducted (ie not informed by ad hoc or anecdotal evidence)
  
• Evaluation should typically be multi-method - eg statistical data and views/opinions of service users, practitioners, etc
  
• Evaluation, where possible, should have a multiple perspective - ie the project should be seen from a number of different viewpoints, eg service users, staff, etc

what kind of person is an evaluator?

essential qualities:

• Curious
• Open-minded
• Clear thinking
• Sensitive and tactful
• Flexible
• Persuasive
• Creative
• Sense of perspective

desirable qualities:

• Advanced research skills
• In-depth knowledge of the field

what does evaluation involve? A 10-stage cycle

• Firstly, establish a commitment to evaluate (time, resources, etc)
  
• Next, clarify the purpose (who and what is it for?)
  
• Then, set evaluation questions (keep the purpose in mind)
  
• Then, think about the evidence (what kind would be credible?)
  
• Then select methods (identify most appropriate methods)
  
• Next, ensure detailed planning (establish deadlines and milestones)
  
• Then pursue data collection (be systematic and stick to the plan)
  
• Then analyse and interpret the data (NB focus on evaluation questions)
  
• Then report and disseminate findings (identify most appropriate format/method)
  
• Finally, reflect and act on your findings (ie evaluate evidence, make changes to the service if appropriate, then begin again the 10-stage evaluation cycle).

what did NIFTY achieve?

• Fears were overcome and preconceptions overturned
  
• New skills and new systems were put into place
  
• There was a sharper focus on outcomes (ie why are we doing what we are doing?)
  
• There was more effective commissioning
  
• Evaluative thinking became ingrained
  
• There was personal and professional development for individual practitioners doing evaluation, learning new skills and developing fresh ways of thinking about their work
  
• Participants were able to use their evaluation findings to bring about service improvements.

presentation 3

agency-led workshop session: Service case studies

NIFTY Project: Children's Rights Service: Prior to renegotiating the service level agreement for delivering the Children's Rights Service in Lancashire, there was a need to check that the service was effective. Lancashire Social Services therefore wished to ask how satisfied were those young people who had used the service in the previous year. This workshop explored the methods used and how the children's views were used to improve service.

facilitators: Dave McHale, Quality and Review Manager, Peter Francis, Quality Protects manager, Lancashire County Council Social Services.

NIFTY Project: Family Support Services: In order to monitor the progress of the Family Support Best Value Implementation Plan in relation to Family Centres, Medway Health and Community Services wished to ask: does the centre deliver its core family support services? Are services equally provided? What services do commissioners and users perceive as the most helpful? This workshop looked at how data was collected and analysed and how the findings revealed contrast in service user/commissioner perceptions.

facilitator: Caroline Budden, Family Support Manager, Medway Health and Community Services.

NIFTY Project: Family Centre Service: This project entailed evaluating three family centres so as to inform service improvement. Southampton Social Services wished to ask: to what extent do social workers and families agree on reasons for referral to a family centre and what effect does this have on outcomes? This workshop followed the process of collecting data through to a full written report. This work revealed differences between social workers and family views.

facilitators: Hilary Corrick, Acting Head of Services for Children and Families
Trish Roscoe, Team Manager, Southampton Social Services.

NIFTY Project: York House Family Support Service: This workshop looked at a new service where the service manager was keen to build in evaluation from the start. West Berkshire Children and Family Services wished to ask: how successful the service is at achieving outcomes for families and young people? The workshop explored how users defined their own 'outcomes' in agreement with staff, and the extent to which these could be measured.
facilitators: Robert Bradshaw, Family Resource Team Manager, Sue Adamantos, Acting Service Manager, West Berkshire Children and Family Services.

summary overview of workshops: A common theme of these workshops was that the NIFTY project improved the capacity and confidence of practitioners to undertake their own single-service evaluations.

conclusion

The underlying message of this symposium was that evaluation is a positive process that should be an integral part of any professional practice. It encourages reflection and, in the context of single-service evaluation, can either lead to the improvement of existing services or uphold one's confidence in the quality of these services. For this reason, evaluation should be an ongoing and integral part of any good-quality service provision and not something that is 'tacked on' to simply please funders or managers.

key references

Evaluation Toolkit: A practical guide to project evaluation:PDF(new window), Barnardo's
Copies can be bought for £5.00 + p&p from:
Barnardo's Policy, Research and Influencing Unit
Research and Development Section
Tel: 0113 3933210

Evaluation Toolkit: A tailored approach to evaluation for parenting projects:(new window). A joint initiative between the Parenting Education & Support Forum and the National Children's Bureau, with funding from the Home Office, the Tedworth Charitable Trust and the Artemis Charitable Trust,

Robson C (2000) Small Scale Evaluation, Sage

Shaw C, 'A Necessary Evil' Community Care, 22-28 August 2002, pp36-7 - a brief overview of the role and importance of evaluation

Practical Monitoring and Evaluation: A guide for voluntary organisations. Charities Evaluation Services, 4 Coldbath Square, London, EC1R 5HL, enquiries@ces.vol.org.uk

More resources in the back of the NIFTY handbook.

glossary of evaluation terms

baseline - describes the starting point or trend level from which progress towards better outcomes will be measured

benchmarking - a technique developed in the business sector, but now developed more widely, which refers to comparing the performance of similar or otherwise comparable organisations or services

evaluation - a set of systematic, planned activities designed to assess the worth or effectiveness of a service or programme

formative (evaluation) - focuses on processes and, through regular feedback, can assist in the development and improvement of programmes or services. Changes as a result are continually being made as a result of the evaluation

goal or aim - a general description, based on principles and values of what a service (or services) aims to achieve for children

milestone - describes any shorter term targets that help to reach final targets and the desired outcome(s)

monitoring - keeping a regular check on project activities by systematically collecting and recording relevant information

outcome - describes the desired end result and intended improvement, after a specified period, in the well being of children and/or families - in effect what changes it has brought about for its intended target group

outcome measure - describes the measures that will be used to determine whether the desired outcome has been reached

outputs - refers to what is actually produced by a project or service eg a certain number of nursery places, childminders, training courses, publicity leaflets

performance measure - describes the output measures that will monitor service delivery and the efficiency and effectiveness of the agencies concerned

process evaluation - a process evaluation focuses on the actual content and delivery of a service or programme, as opposed to its effects on participants

stakeholder - anyone with an 'interest' in the conduct and finding of the evaluation study

summative (evaluation) - focuses principally on outcomes, rather than (or in addition to) process. It is not expected to draw conclusions until the end of the evaluation period. This will be clearly defined at the outset

target - describes the medium term results that service or services will aim to deliver over a given period of time

value for money measures - describes measurements of whether the service provides a good return for the resources invested in terms of both outputs and eventual outcomes