CES regularly carries out evaluations for our partners, examining the effectiveness of policies, projects or services to inform future developments or improvements. In this ‘Guide To Evaluation Methods’ we will look at different types of data collections methods we commonly use.
Data collection methods usually fall into two categories: quantitative research methods and qualitative research methods. When methods are combined, this creates a third category known as mixed methods research.
Quantitative research methods use numbers as data. They are often used to identify relationships between different aspects of the data (variables) that can be generalised to a wider population. In evaluations, quantitative research methods work well with questions like ‘does X lead to Y?’ or ‘how much impact did X have on Y?’. Compared to qualitative research methods, quantitative research methods are usually less time-consuming and resource-intensive and lend themselves to larger sample sizes.
Qualitative research methods use words and images as data. They usually seek to understand people’s experiences within their specific contexts. In evaluations, qualitative research methods work well with ‘how’ and ‘why’ type questions. Compared to quantitative research methods, qualitative research methods tend to be more time-consuming and resource-intensive and lend themselves to smaller sample sizes.
Mixed methods research uses either multiple quantitative research methods, multiple qualitative research methods, or some combination of both. Gathering and triangulating evidence from a mix of different sources can strengthen the robustness of your evaluation findings.
Things to consider when selecting and designing data collection methods for your evaluation:
Specific data collection methods
Common data collection methods that we’ll explore in this Guide are:
The first three data collection methods - interviews, focus groups and questionnaires - ask people questions that the evaluator would like to know the answer to.
Interviews can be used to capture in-depth individual perspectives. Interviews tend to produce rich qualitative information and be more adaptable than questionnaires, providing opportunities to clarify or expand on people’s answers. However, interviews can be more time and resource intensive in terms of data collection and analysis.
Focus Groups also produce rich qualitative information. Focus groups can be used to capture a diverse range of perspectives and encourage interaction between participants, or when you want to confirm your analysis with a wide variety of ‘service user’ profiles. Focus groups can be less resource intensive than interviews in terms of data collection and capture a greater number of perspectives. People need to feel safe to share their views and as such, they are not always appropriate for discussing sensitive topics. Focus groups also require a skilled facilitator who can manage group dynamics and ensure that everyone has an equal opportunity to participate.
Questionnaires can be used to understand the general characteristics or opinions of a group of people. It is more common to see questionnaires to collect quantitative data, but they can also be designed to collect qualitative data. Compared to interviews and focus groups, questionnaires are less time and resource intensive. They can also reach a much larger number of people, which is useful if you want to generalise your results to a wider population.
If you want to collect quantitative data, the questionnaire should ask closed questions with a pre-defined set of options that respondents have to choose from as their answer. You can then count the frequency of selection for the various answers.
Sample Question: ‘How did the benefits for you of participating in ‘Programme X’ compare to the drawbacks?’
Answer options:
○ Benefits greatly exceeded the drawbacks
○ Benefits exceeded the drawbacks
○ Benefits and drawbacks were about equal
○ Drawbacks exceeded the benefits
○ Drawbacks greatly exceeded the benefits
When trying to design closed questions, using previously developed standardised measures are a good place to start. These are questionnaires or assessment tools that have already been developed to measure a particular set of behaviours and/or attitudes. CES hosts a database of standardised measures called the Child, Youth and Family (CYF) Interactive Database. This is a free resource which pools detailed information about standardised measurement tools that have been used in Irish research with children, young people and their families and allows you to search for specific tools by theme and age group.
If you want to collect qualitative data, the questionnaire should ask open questions with space for respondents to write their answer freely. For example, an open question with space for a response might look like this:
Question: Please name the benefits and/or drawback you experienced through you participation in ‘Programme X’.
[Space for respondent to write their answer]
Our next set of data collection methods -observations, monitoring systems and document analysis - are (usually) less reliant on asking people questions and gather data in different ways.
Observations involve the researcher carefully ‘seeing’ things (e.g. behaviours, activities, processes, relationships, events, etc.) in specific settings and then recording what they have seen. The observations are usually guided by an observation schedule, which provides guidance on what and how to observe, as well as providing a space for recording notes. In evaluations, it is rare to see observations used as a method on their own. They are more often used as part of a mixture of methods, where the data is then compared and triangulated with that of other sources.
They are many different types of observations, such as structured, unstructured and semi-structured observations, and participant and non-participant observations.
The main strength of using observations is that it gives the researcher a first-hand view of the situation of interest, rather than relying on the views of others, as is the case with interviews, focus groups and questionnaires. However, some of the limitations include:
Monitoring Systems can be used to collect data about a service or intervention in a systematic and continuous way. The data is usually recorded by the service staff and management, rather than the researcher. The kind of data often collected is quantitative data about the amount of activities, processes, outputs and initial outcomes a service produces. By collecting data continuously, monitoring systems provide a useful way to track progress over time.
When developing a monitoring system, some good practices include:
Document Analysis, like observations, is often used in combination with other data collection methods as a way to compare and triangulate findings. It involves systematically reviewing documents that are relevant to your evaluation questions. There are lots of different ways of ‘systematically’ analysing documents, but all document analyses should follow the same basic procedure of finding, selecting, analysing and synthesising the data contained in documents.
In theory, there is no limit to the types of documents that can be included in your document analysis. What matters is that they are relevant to, and help you to answer in some way, your evaluation questions. In evaluations that CES have conducted, document analysis is often used to collect and analyse in-depth qualitative data about, for example, the way an intervention works and the context it is taking place within.
Want to talk to CES about an evaluation project? Book a call with the team here.
