Chapter 1 – Planning for Evaluation: Getting Started

Identify Evaluation Objectives – Recommended Actions

1. Determine the SHSP process and performance areas to evaluate.
2. Identify the resources needed to conduct the evaluation.
3. Develop evaluation objectives for assessing the SHSP process.
4. Develop evaluation objectives for assessing SHSP performance (outputs and outcomes).
5. Circulate the objectives and resource requirements among SHSP decision-makers to obtain their support.

Review Methods for Data Collection and Management – Recommended Actions

1. Review the existing data collection and management methods to determine the following:
   1. Quality data are available to track progress and answer the evaluation questions;
   2. Resources are available to collect data not currently available; and
   3. Objectives for which no evaluation data are available are reconsidered or modified.
2. Review the tracking mechanism(s) in place to determine:
   1. Where the information can be obtained and what is captured (e.g., implementation status of SHSP strategies and actions, number and rates of fatalities and serious injuries, etc.).
   2. If information needed for the evaluation is not formally tracked, can a mechanism be put in place or information gathered using an alternative method?

Determine How to Measure Progress – Recommended Actions

1. Review the existing SHSP performance measures and categorize them as (in descending order of importance):
   1. Primary outcome;
   2. Secondary outcome;
   3. Self reporting;
   4. Attitudinal;
   5. Awareness; or
   6. Activity level.

Chapter 2 – Process Evaluation: Getting on the Right Track

Identify Process Evaluation Elements: SHSP Organizational Structure – Recommended Actions

1. Review the SHSP organizational structure to identify and document its format and functions(s).
2. Examine the positions of persons serving on SHSP committees (e.g., steering and executive committees), as well as emphasis area and local/regional/district teams to determine their contribution to the SHSP process and access to leadership and resources.
3. Review the schedule of SHSP leadership and committee meetings to determine if they meet as frequently as planned or needed.
4. Review the SHSP organizational structure to determine the level of support provided to partners in local and regional coalitions.
5. Review the role and function of SHSP committees, teams, and/or groups. Compare these current roles and functions with the expectations set at the beginning of the SHSP process.

Identify Process Evaluation Elements: Multidisciplinary, Multimodal Collaboration – Recommended Actions

1. Review the membership of the various committees, emphasis area action teams, task forces, etc. to assess the degree to which multiple disciplines and modes are represented and actively involved.
2. Determine if mechanisms are in place that facilitate an active, efficient collaborative process, such as MOUs.
3. Review the planning documents of the various agencies and safety partners to discern how well they reflect elements of the SHSP, such as the goals, objectives, and strategies.
4. Determine if the SHSP vision, mission, and goals are clearly and continually communicated to all partners and stakeholders.

Identify Process Evaluation Elements: Goal and Objective Setting Methods – Recommended Actions

1. Review the analysis methods used to set objectives. Are they data driven?
2. Determine if objectives are specific, measurable, time bound, and realistic.

Identify Process Evaluation Elements: Data-Driven and Evidence-Based Emphasis Area Strategies and Actions – Recommended Actions

1. Review methods used for identifying safety problems and selecting emphasis areas. Determine whether safety data analysis was the primary input to problem identification and emphasis area selection.
2. Determine whether the latest safety data and research was used to identify evidence based strategies and actions.
3. Identify strategies in the SHSP that may lack a preponderance of evidence of effectiveness. Determine if there was enough information or evidence (e.g., success in another State) to justify its use, or if an evaluation was conducted to validate the use of the strategy.

Identify Process Evaluation Elements: Aligning Agency Priorities – Recommended Actions

1. Review the HSP, CVSP, HSIP, LRTP, and S/TIPs to determine the degree to which they align with the SHSP.
2. Review other plans, e.g. pedestrian/bicycle plans, corridor plans, local road plans, etc., to determine the degree to which their goals and strategies align with the SHSP.

Process Evaluation Activities – Recommended Actions

1. Review the possible methods to collect process evaluation information. Identify the pros and cons of each method.
2. Determine and document the most appropriate method and the rationale for the selection.

• Include the higher level performance measures (e.g., primary outcomes) in the evaluation if data are available (to answer the evaluation objectives) and consider other, more attainable measures if they are not available.

Identify and Secure Resources – Recommended Actions

1. Identify available resources to support SHSP evaluation.
2. Identify individuals or agencies with the skills to analyze data and other information and provide evaluation support.
3. Explore the availability of universities, professors/graduate students, and others if extra help or expertise is needed.
4. Collect information on current evaluation efforts among the partners, e.g. CVSP, HSP, etc.
5. Determine a timeframe for conducting SHSP evaluation.

Assign Evaluation Roles and Responsibilities – Recommended Actions

1. Determine the agency responsible for coordinating the overall program evaluation effort.
2. Identify the individual(s) who will manage the evaluation effort, collect and analyze data, and report the evaluation results.

Report and Use Evaluation Results – Recommended Actions

1. Identify the parties responsible for generating the evaluation results.
2. Assign a lead person to pull together the results from the various parties and distribute them to the appropriate committees, agencies, etc.
3. Determine who will receive the results of the evaluation, i.e., agency executives, Steering Committee members, all stakeholders, etc.
4. Decide how the results will be formatted, i.e., formal reports, spreadsheets, PowerPoint presentations, etc.
5. Identify how the evaluation results will be used.

Document Evaluation Approach – Recommended Actions

1. Document the evaluation elements (objectives, data needs, resources, roles and responsibilities, application methods, etc.) to formalize the process.

Chapter 3 – Performance Evaluation: Outputs and Outcomes

Performance Evaluation – Recommended Actions

1. Assemble data for assessing output and outcome performance measures.
2. Identify missing data which may prevent assessment of performance measures.
3. Determine if other performance measures can be used to determine progress.
4. Document baseline data.
5. Determine the output and outcome measures for each SHSP emphasis area.
6. Compare output performance measures with baseline data.
7. Compare outcome performance measures with baseline data.
8. Compare observation and/or telephone survey results to measure changes in awareness, attitudes, and behaviors.
9. Collect and review the data available for benefit/cost analyses.
10. Conduct program level benefit/cost analyses where feasible.