The Challenges and Constraints of Evaluations

Download the Guide

Welcome to the fourth blog in our series Demystifying Evaluation from Karl McGrath and Dearbhaile Slane.  

Here we look at some of the common challenges and constraints that can arise during the evaluation process, and some potential strategies you can use to overcome or mitigate them. Understanding the challenges and constraints of an evaluation are crucial factors that feed into its design and implementation.  

The main challenges andconstraints for evaluations 

In their book RealWorld Evaluation: Working Under Budget, Time, Data and Political Constraints, Michael Bamberger and Linda Mabry shine a light on four common constraints for evaluations. These are: 

  1. Budget constraints
  2. Time constraints
  3. Data constraints
  4. Political andorganisational constraints. In CES’s experience, we can certainly testify to the frequency with which these constraints appear in evaluations. And to them, we would also add two more challenges and constraints we often address:
  5. Ethical considerations and constraints
  6. Knowledge and skills constraints. 

1. Budget Constraints 

Evaluations do not have unlimited funding and ‘budget constraints’ are about keeping the costs of the evaluation within its budget.  

Conducting robust evaluations that produce trustworthy findings can be resources intensive. 

Sometimes budget constraints and robustness come into conflict and trade-offs must be made. The budget might be too small to apply the ‘best’ evaluation designs and methods, but the evaluator still needs to produce findings that are meaningful and trustworthy. 

Bamberger and Mabry suggest the following strategies to reduce the cost of an evaluation: 

  • Simplify the evaluation design
  • Clarify the commissioner’s budget and information needs, and prioritise
  • Look for reliable secondary data
  • Reduce the sample size
  • Reduce costs of data collection and analysis
  • Incorporate new information technology. 

2. Time Constraints 

Evaluations will have a definite timeframe and the timing of an evaluation may not always be ideal for answering the questions of interest. ‘Time constraints’ are about conducting the evaluation within its agreed timeframe, or when its timing isnot ideal.  

In CES, there are at least three common time constraints we come across: 

  • The evaluation timeframe is too short to measure the outcomes of interest (e.g. an evaluation is 12 months long, but the intervention is complex and could take upto 5 years to produce certain outcomes)
  • The evaluation is commissioned too late to collect baseline data that will allow are liable comparison before and after an intervention
  • The evaluation timeframe is very short for the amount of data collection and analysis requested

 Some of the strategies suggested by Bamberger and Mabry to manage the time constraints are similar to those for managing budget constraints, and include: 

  • Simplify the evaluation design
  • Clarify the commissioners deadlines and information needs, and prioritise
  • Look for reliable secondary data
  • Reduce the sample size
  • Use more time-efficient data collection and analysis methods and technology
  • Hire more evaluation team staff

To this, we would also add, based on our experience:  

  • To specify the short, medium and long-term outcomes the intervention is intended to achieve, to help set realistic timelines and prioritise what to measure

3. Data Constraints 

Evaluations often have to ‘make do’ with gaps or limitations in the data available to them. ‘Data constraints’ are about conducting an evaluation when critical information needed to address the evaluation questions is missing, difficult to collect or of poor quality.  

Examples CES regularly encounter include: 

  • A lack of baseline data in evaluations which have been commissioned at the end of an intervention
  • Working with small, incomplete or potentially biased samples
  • Trying to encourage the participation of difficult-to-reach groups in an evaluation
  • Some of the strategies suggested by Bamberger and Mabry to reduce data constraints include: 
  • Reconstruct baseline data (e.g. using project administrative data or recall techniques with participants)
  • Adapt methods to facilitate data collection from difficult-to-reach groups (e.g. using participant observation and interviews instead of focus groups, or using snowball sampling techniques where participants can refer others to participate or help you gain access to other participants)
  • Use mixed-methods (e.g. if the most appropriate evaluation design or method is not feasible, collect data from a variety of sources and triangulate to strengthen your conclusions)
  • To this, we would also add, based on our experience:
  • For intervention designers and decision-makers to consider evaluation right from the very beginning so that baseline data can be gathered

It is important to be transparent when describing your findings and conclusions. Data constraints will almost always weaken the certainty of your findings, even if the strategies to reduce data constraints are used. It can be easy, however, to fall into the trap of being overly confident in a set of findings even when data constraints do not support them. As evaluators, it’s important to be clear about the limitations of the evaluation and express an appropriate level of caution about the certainty of your findings.  

4. Political and Organisational Constraints 

Evaluations do not take place in a vacuum. Political and organisational constraints almost always are present in different ways and to different extents. Bamberger and Mabry use the term to refer “not only to pressures from government agencies and politicians but also to include the requirements of funding or regulatory agencies, pressures from stakeholders, and differences of opinion within an evaluation team regarding evaluation approaches or methods”. 

These are not necessarily bad things, though they can sometimes have a negative influence. They can also be seen as a positive indicator of the importance of an evaluation to stakeholders.  

Some political and organisational constraints include:

  • Attempts tointentionally obstruct an evaluation
  • Attempts to underminethe independence of an evaluation
  • Resistance by certain stakeholdergroups to participating in the evaluation 

Managing political and organisational constraints effectively requires good teamwork and communication. It requires the evaluator to demonstrate effective stakeholder management and at times, it will require a firm approach to protect the independence of an evaluation 

At the beginning of the evaluation, it can help to clearly define evaluation boundaries and collaboratively conduct stakeholder analyses. Defining evaluation boundaries with the evaluation commissioners can help to clarify where the evaluation must retain absolute independence and where it can accommodate stakeholder input and preferences.  

A collaborative stakeholder analysis can also help to identify key stakeholders who might be unknown to the evaluator and understand their potential interests and influence on the evaluation. 

5. Ethical Considerations and Constraints 

Ethics are an essential part of our considerations and practice in every evaluation we do at CES. Ethics are about how we should, and do, conduct an evaluation.  

As experienced researchers, all our evaluations follow certain ethical principles:

  • Do no harm
  • Informed consent
  • Voluntary participation
  • Confidentiality andanonymity.  

These are the minimum ethical standards we embed in every evaluation. When deciding which evaluation types, designs and methods are most appropriate for a particular evaluation, we have to be sure that the methods we use will adhere to these principles.  

When working on sensitive topics or with vulnerable populations, there may be additional guidelines to follow. For example, the ethical standards required when working with children and young people may be slightly higher than those for adults. Another example is evaluations of palliative care. Palliative care is an especially sensitive topic with high risk of harm to patients and their caregivers. Specific recommendations on how to conduct ethical palliative care evaluations were published in 2013 by the MORECare expert group. 

6. Knowledge and Skills Constraints 

At CES, we are fortunate to have a team with a wide range of evaluation knowledge and skills. However, there is such an abundance of evaluation types, designs, methods and approaches that no evaluator can be skilled in everything… So, a perhaps obvious question for evaluators to ask themselves when designing an evaluation is ‘what kind of evaluation do we actually have the knowledge and skills to conduct?’.

Related Guides

Related

Work with CES

Get in Touch