Evaluation

Evaluation of a service supports decision making around whether it should continue and what (if any) adaptations might be necessary to increase its chances of success. 

But what happens if the evaluation isn’t going to plan?

Even after following all of the guidance, and with a solid evaluation plan, things don’t always happen as expected. 

Risks to evaluation:

• Insufficient or poor-quality data: this can be because service users don’t want to share their data, data systems fail, or data is inconsistently recorded 

Reviewing data quality regularly as part of the monitoring process should identify these issues early so that support can be put in place.  

• Changing circumstances: This is often the result of external factors (Eg., policy changes, pandemics) which disrupt delivery and affect continuity 

These things are usually unavoidable. If delivery continues, it’s necessary to review the service theory of change and logic model to ensure they still reflect reality. If there are changes, these will need to be factored into your evaluation.  

• Service doesn’t match the logic model: This might be because this step of the service design process was rushed, or because of a change of circumstance after the service design process has been completed 

When implementation diverges from the plan, it’s not possible to establish whether intended activities are producing outcomes and there’s no clear basis for what the evaluation is measuring. Again, the logic model should be reviewed and the evaluation plan should be revisited. 

In most situations, there are opportunities to adapt what you’re doing and continue with your evaluation. However, in extreme circumstances it can be necessary to completely rethink your approach or stop. An example of this might be when attempts to integrate a validated measure into a service are unsuccessful and no appropriate alternative can be identified. In this case, you may want to move away from a quantitative evaluation of outcomes and move to a qualitative evaluation. But this can be resource intensive, and funding and/or capacity might mean this is not feasible.  

Service

For a service to be effective, and have a positive impact on outcomes, it needs to engage the right people and keep them engaged long enough to benefit from the support. It also needs to be delivered as intended (with fidelity to the service manual).

Risks to the service:

• Poor recruitment: This might involve the service not receiving sufficient referrals, referrals deciding not to enrol into a service, or enrollees not going on to attend the service
• Poor completion: This might mean lots of people dropping out of the service earlier than anticipated or not attending enough sessions
• Poor implementation/fidelity: This might relate to the service activities not taking place as planned, or poor-quality delivery

Good quality monitoring and a comprehensive implementation evaluation provide an early warning system that identifies problems quickly, allowing timely adjustments to increase chances for success.

Once adaptations are made, continued monitoring helps to determine whether there are improvements, while implementation evaluation distinguishes between adaptations that preserve the core components of the service and ones that undermine the service’s theory of change.

If adaptations do not lead to anticipated improvements, and performance indicators demonstrate continued under performance (i.e., in RED per progression criteria) across multiple reporting quarters, then it is likely the service is not appropriate in the current context.