What are evaluation questions?

Evaluation questions guide your evaluation, specify what you want to know about your service and reflect key stakeholder and community interests. They address different concepts dependent on evaluation level. The evaluation questions you develop allow you to add detail and context to the broader questions :

Implementation evaluation: Is the service being delivered as intended?

Before and after evaluation: What changes from the beginning of accessing the service to the end?

Effectiveness evaluation: How strong and in what direction is the effect of the service on the intended outcomes? 

How to develop implementation evaluation questions

With your service design group you will be considering the service Theory of Change and Logic Model (with a focus on the Inputs, Activities and Outputs).

In relation to these elements of your Logic Model, and the mechanisms of change described in your Theory of Change, discuss the following:

• What elements of the service do we think are key to bringing about change for service users/ participants?
• What questions can we answer that will allow us to confirm that these are being delivered as planned, that service users are engaging with them meaningfully at the right time, and that the people being engaged are those we are targeting?
• What do we need to understand about what might be influencing delivery or engagement (e.g. barriers and facilitators)

You can find example implementation evaluation questions in the Evaluation Plan template and examples in our resources section.

Remember that questions should be: 

1. Concise
2. Specific
3. Measurable

Developing Further Evaluation Questions

Before and After questions

If you are planning a Before and After evaluation alongside your Implementation evaluation, you will be focussing on the shorter term Outcomes in your service Logic Model.

You will need to/be planning to implement specific measures or assessments that will allow you to demonstrate change in an outcome for service users from the start to the end of the service. For this reason your questions here will relate specifically to whatever you are measuring. 

In relation to the short term Outcomes in your Theory of Change and Logic Model, discuss the following:

• Do we already have before and after measures in place or have we identified appropriate measures for these outcomes that can be feasibly implemented? 
• What outcomes are we able to measure and what will we expect to have changed from the start to the end of support? E.g. Participants will report an increase in XX, Fewer participants will report XX

You can find examples of before and after evaluation questions in the Evaluation Plan template and examples in our resources section.

Again, questions should be: 

1. Concise
2. Specific
3. Measurable

Effectiveness questions

If you are planning an Effectiveness evaluation, you will already have a good understanding of implementation and potentially changes on short term outcomes.  You will also be developing your evaluation questions in collaboration with an academic research partner.

You will be focussing on the Outcomes and Impact specified in your Theory of Change and service Logic Model. You will also be having discussions around the most appropriate methods, and the availability of data to measure these outcomes,

including the timing of collection, and the timeline required to have data on a sufficient sample of people to be able to complete the appropriate analysis.

Examples of effectiveness questions can be found in our examples.

Once your evaluation questions have been drafted these should be shared with the group, and any other key stakeholders that have not already been included in their development. This should ensure a sense check, and allow agreement on shared priorities. It will also support expectation settings around evaluation.