This will require going all the way back to the theory of change level, but the focus will be on updating processes and documentation rather than developing them, and everything should be much quicker:

1. Reconvene the service design group
2. Establish whether the inputs and outputs that are actually happening are
   1. Known and understood
(if not take time to do this)
   2. Can be reasonably expected to achieve the stated outcomes of the service
(if not then the go to Changing or Stopping a Service)
3. Re-write the Theory of Change and Logic Model based on the way the service is being delivered.
4. Follow on from there by checking that documents and processes are still relevant or updating:
   1. Data requirements and systems
   2. Process Flow
   3. Service Design questionnaire/service manual
   4. Performance indicators
   5. Evaluation questions
   6. Evaluation Plan
   7. Consent processes
   8. Data sharing agreements
5. Use your monitoring processes to check and respond quickly if implementation still does not seem to match the logic model

If spotted fast this may be fixable and allow for the evaluation to be completed with minimal changes:

1. Identify possible reasons for missing data collection targets
2. Take steps to address those possible reasons for example:
   1. Change timing or approach to requesting data from service users/partners
   2. Incentivise participation in focus groups/interviews/surveys
   3. Provide access to devices or change collection methods for some data
   4. Train staff to discuss data collection and use with service users
   5. Re-negotiate data sharing agreements with partners
   6. Provide training and dedicated time for staff to complete data entry
   7. Provide staff with additional hardware to enter data

Use monitoring processes to see if this improves the volume of data collected. If it does, then the evaluation may go ahead with a note that data from the low collection period is less complete than ideal.
If data volume does not improve further investigation into the barriers to data collection will be necessary and choice must be made as to whether that can be improved sufficiently or whether it will force you to change or abandon the evaluation.
If data volumes cannot be improved then revisit the evaluation plan and establish what parts of the plan could be completed or whether a different useful piece of evaluation could be done with available data. If changes are made to the evaluation plan, follow that up by updating:

1. Data requirements and systems
2. Process Flow
3. Performance indicators
4. Evaluation questions
5. Consent processes
6. Data sharing agreements

If low data volume is only spotted at the end of the evaluation cycle then the person/group completing the analysis and write up may be limited in the conclusions they can make. It is likely that the primary finding of an evaluation with insufficient data, will be that the service needs to make improvements to data collection.

If this is spotted fast this can be fixed and allow for the evaluation to be completed with minimal changes. Frequently problems with data quality are about inconsistent input of fields like DOB, address, etc that may be entered in multiple ways:

1. Identify what aspect of the data quality is poor – this may require discussion with evaluators, data managers, staff entering data, others.
2. Understand where the poor quality is coming from – collection process, transmission sharing from one place to another, storage.
3. Take steps to address the poor quality:
   1. Dedicate some experienced staff time to cleaning and completing existing data if possible – for example if paper records exist that can checked against or if data exists in multiple databases.
   2. Train staff to input data, share experience, identify areas where data input is proving difficult – it may well be that staff have different understanding of what certain fields require for example.
   3. Improve/simplify data entry systems – tweaks to make a paper form and electronic system match more closely for example may solve some issues or to limit the ways in which data can be entered – such as forcing a consistent date format.
   4. Provide more detailed explanation of field requirements on any forms (digital or physical) being completed by service users or partner organisations
   5. Support service users to enter their own data – staff/trained volunteer completing forms with them, translation provided, transcription provided
   6. Check that the systems used to collect and store data are fit for purpose?
   7. Check that linking data (such as unique identifiers) are consistent and suitable.

If data quality does not improve further investigation into the barriers to data quality will be necessary and choice must be made as to whether that can be improved sufficiently or whether it will force you to change or abandon the evaluation.
If data quality cannot be improved then revisit the evaluation plan and establish what parts of the plan could be completed or whether a different useful piece of evaluation could be done withavailable data. If changes are made to the evaluation plan, follow that up by updating:

1. Data requirements and systems
2. Process Flow
3. Performance indicators
4. Evaluation questions
5. Consent processes
6. Data sharing agreements

If low data quality is only spotted at the end of the evaluation cycle then the person/group completing the analysis and write up may be limited in the conclusions they can make. It may also delay the delivery of an evaluation as it takes longer to make sense of poor-quality data. It is likely that the primary finding of an evaluation with poor data quality, will be that the service needs to make improvements to data systems.

The response to this will depend greatly on which stakeholders find it unacceptable.

1. If necessary for service delivery to continue consider pausing the evaluation while the following steps are completed
2. Identify which stakeholders find it unacceptable
3. Identify aspects or points of the evaluation processes that are unacceptable, through direct conversations with those stakeholders where possible or reports from others if it is not
4. Convene the service group explore whether any of the following may improve acceptability based on what you have learned from stakeholders:
   1. Better communication/information about what the evaluation is and how it works
   2. Training for staff or volunteers (for their own knowledge or their communication about the evaluation to others)
   3. Adjustment of budgets/funding streams
   4. Small changes to the evaluation – eg. The timing or method for collecting some data
   5. Major changes to the evaluation – eg. The type of evaluation that is being done, the type of data collected, to whom data and findings are reported
   6. Other – the specifics of your situation may present ideas or limits.
5. Depending on what changes are required go back through the stages of the framework to adjust processes and documents.

In almost every example of a service finding itself with more opportunity for evaluation the first place to go will be the evaluation plan.

1. Gather key stakeholders to the evaluation plan
2. Explore what there is opportunity to do and what would be most valuable to wider stakeholders and the service itself
3. Ensure that time is spent checking any changes are acceptable to wider stakeholders and particularly to staff and service users
4. Update the evaluation plan to reflect the extension of the evaluation (retaining the original as an archive).
5. From there work both back and forwards ensuring that documents and processes meet the updated needs of the evaluation consider particularly:
   1. Data Requirements and Process flow
   2. Monitoring processes and performance indicators
   3. Consent and Data sharing agreement