Levels of Evaluation
Monitoring
Purpose |
Supports regular review of service delivery Relates to the inputs, activities and outputs of the logic model only Provides opportunity to identify challenges around data capture and implementation early, and put support in place |
What it will tell us |
Data quality and suitability Performance against key indicators Potential evaluability (see evaluability checklist) |
What it won’t tell us |
Whether the intervention is effective |
Data |
Basic delivery data |
Method |
Descriptive statistics |
Suggested Outputs & Timing |
Quarterly or termly reporting, but will depend on delivery patterns/length |
What you need |
Consent routinely collected System for data capture Agreed performance indicators |
Implementation
Purpose |
Establishes whether the intervention can be delivered as set out in Service Design Relates to the inputs, activities and outputs of the logic model only Informs future evaluations |
What it will tell us |
Fidelity to the model, which may include recruitment, reach/engagement, satisfaction Possible reasons why the intervention is or is no’t performing as expected What future evaluation might be possible (see evaluability checklist) |
What it won’t tell us |
Whether the intervention is effective |
Data |
Basic delivery data As required: Satisfaction questionnaires Additional questionnaires Interviews / focus groups* Observations* |
Method |
Descriptive statistics *Qualitative analysis |
Suggested Outputs & Timing |
Annual or by contracted periods |
What you need
|
As per monitoring As required: Access to intervention for observations Access to staff and participants for interviews |
Before and After
Purpose |
Establishes whether participants self-report changes in outcomes after participation in a service Relates to short term outcomes of the logic model |
What it will tell us |
An estimate of change in outcome(s) after participation in the service This may provide Early Intervention Foundation (EIF) level 2 evidence |
What it won’t tell us |
Whether the service is effective |
Data |
Standard data set out in data requirements Outcome measures collected at the start and end of participation |
Method |
Basic statistical analyses comparing results before and after the service |
Suggested Outputs & Timing |
Report before the end of the contract period |
What you need |
As per monitoring; Logic model and service design are suitably defined. Use of reliable, valid and relevant outcome measures taken before and after participation in the service |
Effectiveness
Purpose |
Provides evidence of effectiveness of the service Relates to short, medium and long-term outcomes of the logic model |
What it will tell us |
An estimate and interpretation of the strength and direction of the effect(s) of an intervention on an outcome Assessment of certainty of the results This may provide EIF level 3 evidence |
What it won’t tell us |
Effects on those not reached Effects beyond the evaluation period Effects on outcomes not measured |
Data |
Baseline & Outcome data As required: Additional questionnaires Interviews / focus groups* Observations* Comparison group |
Method |
Comparison of outcome(s) between participants and control group; may use randomised control trials or quasi-experimental methods |
Suggested Outputs & Timing |
Depends on scale of intervention, level of participation, and timing of outcome measures |
What you need |
Intervention EIF rating of 2 Sufficient number of participants An identifiable control group |
Background studies
Purpose |
Provide context to evaluations and inform future evaluations |
What it will tell us |
Depends on study aim and design |
What it won’t tell us |
As required: Questionnaires Interviews / focus groups* Observations* Research data |
Data |
|
Method |
Various; depends on study aim and design |
Suggested Outputs & Timing |
Depends on study aim and design |
What you need |
As required: Access to services for observations Access to staff and participants for interviews Willingness of service team to facilitate extra study |