evaluating research: a beginner's guide

This section is broken down into the following subsections:

• an introductory guide to evaluating research

• a framework for evaluating qualitative research: PDF (new window)

• a framework for evaluating a systematic review or meta-analysis:PDF (new window)

an introductory guide to evaluating research

Evaluating research always sounds rather daunting for the inexperienced but it can be broken down into a number of simple steps.

Firstly, it is worth reminding yourself why it is important to evaluate research.

Perhaps the most important reason is that it enables you to decide the value or worth of a piece of research, given the purposes for which you wish to use it.

If you are going to use research to inform and shape practice or policy then the research needs to be reliable and rigorous.

Having decided that a piece of research needs evaluating, it is worth asking yourself the following questions about the article and its method:

the article

the title:

the authors:

what do you know about the authors? do they have a vested interest in the conclusions of the study?

the abstract:

does it summarise the main points of the study adequately and accurately? be careful as sometimes abstracts promise more than what is written in the rest of the paper.

introduction:

is the problem or purpose of the study clearly stated?

the question:

is/are they stated clearly and concisely? does it/they follow logically from the problems? Is it/are they worth answering? is it/are they answerable?

the literature:

is the background information adequate? does the author appear to know her/his subject? does he/she appraise related research and authoritative statements? Or has he/she strung together citations and quotes which support her/his proposal without consideration of antagonistic arguments? are specific theories used in order to put the study and potentially the findings into context? does this theory seem relevant?

relevance:

aims:

are the aims stated clearly, concisely and precisely? Are they logically related to the original question(s)? how were they formulated eg does evidence from the literature support intuition, instinct and experience? If treatment is being investigated, are the aims related to efficacy and safety?

the method:

design:

is the study descriptive or experimental? is it described adequately? does the chosen design seem appropriate to you? an hypothesis or hypothesis is/are necessary for an experimental design. does it/do they follow logically from the original problem and theories?

assumptions:

are any assumptions being made? is their use explained? are they justifiable and appropriate? was it a pilot study completed eg was a questionnaire or special report pre-tested for validity and/or reliability? were modifications made? what were they and why?

ethical considerations:

has the author considered the ethics of the proposal? is the proposed method ethically acceptable? for example will all service users receive the treatment/intervention they need rather than the treatment needed for the study? will a control group be required to receive a bogus or dummy treatment of dubious efficacy?

participants:

how were people selected? are individuals allocated to alternative treatment /intervention groups? is this ethical? is there an account of how each person was chosen? were specific criteria used to include and/or exclude people in and from the study? are they clearly stated? is the reasoning behind them apparent and sensible?

samples:

was a specific size of sample chosen (eg for statistical purposes)? does it seem adequate to provide utiSusannahble results? if the author aims to make generaSusannahtions to a population on the basis of the findings, who forms this population? is the sample representative of this population?

data collection:

is the method described adequately? could you replicate it from the description? are the reasons for the choice of method stated? if special report forms, assessment forms, questionnaires, or inteview schedules have been used are copies provided with the paper or is an address given for copies?

analysis:

is the method of analysis understandable? have statistical tests been used? are reasons for choice given which explain their appropriateness? do you understand and accept the explanation?

results:

are results intelligible enough for you to interpret them and draw your own conclusions? are they relevant to the stated problem? does your background knowledge and common sense indicate that they are realistic and feasible?

are "raw" data given, or only proportions, percentages, etc after manipulation? are histograms, pie charts and other graphic representations explained? are the tables helpful? if results are based on responses to a questionnaire or interview schedule, what is the response rate? are statistical results included? are they meaningful? is the statistical probability of results by chance included? is it appropriate?

discussion

are the results interpreted in relation to the original question(s)? are the original questions answered? have the aims been fulfilled? does the author discuss any weaknesses in the methodology and factors which may have affected validity or reliability? for example: should selection of sample be discussed?

if criteria of inclusion and exclusion need clarification, is the explanation acceptable? should the advantages and disadvantages of the method of data collection be discussed? are they? have you noticed anything that was omitted? has the author referred to it or ignored it? have the findings been related to the existing body of knowledge and relevant theory? are the clinical implications discussed?

was the project funded? by whom? might the results be biased because of the interests of the financing body?

conclusions

how do they compare (or contrast) with the conclusions you drew from your interpretation of the results? do they relate logically to the results?

recommendations

are the recommended changes self-evident from the reported results? could you attempt to implement them, and should you? is this study an end in itself, or does it suggest further research?

references

is the length of the list more impressive than its quality? are any references conspicuous by their absence?

These questions are very general and could be applied to any piece of written research. If you want more precise guidance, try:

• A framework for evaluating qualitative research: (new window)

• A framework for evaluating a systematic review or meta-analysis: (new window)