report on All Agency Audit 2001 evaluating progress

introduction

The All Agency Audit constitutes one element of the research in practice evaluation plan and is designed to monitor the use of research evidence within individual agencies, across the research in practice partnership. This report outlines the results of the second audit, which was conducted in 2001.

A key component of the All Agency Audit is the utilisation of four benchmarks in order to determine how far research in practice's intermediate objectives, which relate to the assimilation of research messages, are being met. These benchmarks are as follows:

• teams have ready access to at least one research-based journal or other substantial sources of research information
  
• the role of evidence based practice is visible in team action plans
  
• aspects of performance management at all levels in the organisation reflect evidence based practice
  
• case reviews, court reports and service planning documents cite research evidence as an aid to decision making

The way in which agencies approach planning and monitoring in relation to three indicators from the Performance Assessment Framework (stability of placements of children looked after; children looked after being absent from school; and relative spending on family support) are also being evaluated.

The findings are presented in line with the benchmarks described above. Thirty agencies responded to the 2001 All Agency Audit, representing a 55% response rate.

access to research information

Nevertheless, some agencies have taken steps to ensure good library usage:

'We have good library facilities which are advertised widely and are now accessed more frequently. Abstracts are available in the library. Relevant journals are purchased and information is widely shared amongst practitioners. 'What works ' is widely distributed.'

When examining the degree to which electronic access to information is available to teams, the picture was less optimistic, with the majority of agencies indicating limited or no use of the internet at team level during work time. Thus 60% of the respondents agreed that only one fifth or less or their teams use the internet.

For some agencies, this low level of internet access is linked to budget constraints:

'Access to information is very limited. We have no internet access currently. Budgetary constraints mean that future developments are likely to be limited. Some staff within central office have internet access but it is impractical for de-centralised staff to access this.'

For others, though, cultural change and training with regard to ICT systems is seen as a necessary requirement for the expansion of the use of the internet:

'Staff now have enough P.Cs. They can access the internet (partially) via the intranet. There is a training need. IT services are liaising with RiP re a training programme to better access the RiP database. Most managers are IT 'scared'.'

Although the level of use of the internet is relatively low, there is nevertheless a slight improvement on the previous year. Some agencies have managed to make good progress in this area and are now utilising the internet to a greater extent. Indeed one tenth of the respondents stated that over 70% of their teams use the internet. In one of the agencies, all teams have access to internet and this has been part of a wider successful strategy based on developing evidence based practice:

'The staff now have a computer ratio of 1:1. At least one member of each team has access to the internet. Strategy & planning regularly disseminate research literature and are developing a resource of evidence-based information.'

While ready access to information will clearly help to promote evidence based practice, the opportunity to discuss, critically appraise and assimilate learning from research will help to promote it further. However, only one quarter of the agencies (8) indicated that such opportunities existed in the form of journal clubs and /or regular meetings where research literature is reviewed and discussed. This reflects little change from the results of the previous year's audit.

team action plans

Furthermore, where plans do exist, half the agencies stated that the majority of these plans are 'evidence based' demonstrating much progress towards evidence based practice.

As was found in 2000, the Quality Protects initiative continues to affect how teams plan:

'All teams work to Departmental Performance Plan and QP MAP. These are based on national / committee objectives which are in turn based on research and make use of management information. Through QP the Division has developed a clearer focus.'

Clearly some agencies still have some progress to make in terms of the development of team action plans. However, many of them recognise the work to be done in this area and, despite barriers, have plans to improve:

'This [team action planning] is very slow to develop/progress due to high staff turnover and fall out. However evidence-based practice is 'happening' and teams are debating practice issues.'

organisational infrastructure for evidence based practice

'We have 17 newly created posts of senior practitioner where evidence-based practice is a key responsibility in the job description.'

This question did however, like the previous year, prove to be a difficult one for many respondents to answer, primarily because getting access to that kind of information is a time consuming and difficult process, which suggests that there is not a standardised policy in relation to this within many of the agencies. Consequently 13 agencies (43%) either did not provide a response to this question or stated that no job descriptions included such a reference:

'I do not have the time to research this section of the audit at present, however my impression is that it will be very few - probably relating to specific project posts only. We are also undertaking a full review of job descriptions.'

As in 2000, the majority of the agencies (67%) indicated that discussion on the use of research was not expressed as a specific function within their supervision policy, although one quarter agreed that research is a specific function of supervision. This proportion remains unchanged from the previous year.

The comments made in relation to supervision raise the question of the extent to which agencies perceive supervision to be an appropriate avenue within the organisation for the promotion of evidence based practice.

Agencies have, however, made further progress in terms of their service action plans/strategy documents, with 70% stating that these documents include the promotion of evidence based practice as an aim, the comparable figure for 2000 being 58%. Agencies have achieved this in a variety of different ways, two of which are listed below:

'A small task group has been established to ensure we benefit from our membership of RiP and to identify ways to promote evidence based practice. In May all managers got together for a day to discuss membership of research in practice and evidence based practice.'

'We have recently established a new 3rd tier post - planning and service development manager. She and her team will have particular responsibility for ensuring that research evidence guides and shapes our planning and service activities.'

Use of evidence in planning care and services
Agencies were asked to indicate the degree to which research evidence is used and/or cited within individual case planning, court reports, and service plans using a scale of 1 - 10.

Sixty-three per cent of the agencies reported limited use/citation of evidence in case planning - the same percentage stated that less than half of their care plans cite research evidence. This represented a slight improvement on the previous year, the corresponding figure being 77%. Indeed only four of the agencies indicated that the majority of their care plans cite research evidence, but in two of these research evidence is always cited, thus representing examples of good practice. The picture was a bit more optimistic in terms of research evidence cited in court reports with one third of the agencies stating that the majority of court reports cite research evidence. Moreover, in four of these agencies research evidence is always cited. This represents some improvement on the previous year when the majority reported limited use/citation of research evidence in court reports. Some agencies did express caution about using research evidence in court reports at all:

'There is legal caution about citing evidence unless the practitioner is a topic expert.'

Use of research evidence in service plans was less widespread, with one third of the agencies stating that the majority of service plans do not cite or use research evidence and only one fifth agreeing that the majority of their service plans use or cite research evidence.

Agencies were asked to provide further details about the use of research evidence in planning care and services but many found this a difficult question. This, to a certain extent, reflects their lack of use of research evidence in these spheres but there was acknowledgement that research in practice can play a role in encouraging the use of research evidence:

'Current practice is relatively limited hence my inability to complete this section. However team leaders are in the process of familiarising themselves with research sources and methods including RiP.'

The role that research in practice can play is emphasised by another agency respondent:

'Currently [use of research evidence] is very patchy. New assessment forms will assist and membership of RiP is an attempt to move in this direction.'

Proportionally more agencies reported a greater degree of use/citation of research evidence in individual care plans than in court reports or service plans, and this represents a change from the previous year when proportionately more agencies were using/citing research evidence in service plans due to government guidance about the composition of such planning documents.

performance assessment framework indicators

N.B Information on this indicator is problematic in its usefulness for comparative purposes because there are issues in relation to differing definitions and interpretations between agencies

conclusion

The findings from the All Agency Audit 2001 illustrate a wide variation across research in practice Partner agencies in relation to their development of evidence based practice. Nevertheless there are some key findings from the audit, which demonstrate that much progress has been made towards evidence based practice within Partner agencies:

• Many teams in agencies have access to research journals and these are often stored in a resource room or library, but little monitoring of the use of these journals takes place and this is something that agencies may want to address during the forthcoming year.
  
• Although electronic access to information is still relatively low within the agencies, it has improved since the last audit, and some agencies have made good progress in this area.
  
• There are few opportunities to discuss and appraise research findings within the agencies and further progress can be made in this direction through the development of journal clubs and regular research meetings.
  
• Team action plans have become more widespread.
  
• Many agencies are encouraging their staff to use research evidence and are including it in job descriptions, and this is reflected by the fact that service action plans and strategy documents often include the promotion of evidence based practice as an aim.
  
• Agencies often use research evidence in individual care plans but are less likely to use it in court reports and service plans and further progress in these two areas can be made.

Thus, despite the variations between agencies and the fact that further progress towards evidence based practice can be made in many of them, it is clear that agencies are taking positive action on many fronts towards the establishment of an evidence based culture. In the current climate of change and uncertainty this must be commended and celebrated. It is essential that Partner agencies and their staff be encouraged and supported to continue to use the audit mechanism both as a tool to benchmark their own progress and to contribute to a more in-depth analysis of developments over time within the research in practice partnership.

Andrea Wigfield
Research Fellow

© research in practice 2002