Public sector professionals evaluate the impact of public programs

In a workshop with over 100 public service professionals on June 22, Dr. Deborah Stine, the Founder and Chief Instructor of the Science & Technology Policy Academy, showed participants how to design and conduct evaluations of public programs, including how to choose appropriate evaluation methods, collect and analyze data, and communicate findings effectively. Program evaluation is the systematic process of collecting and analyzing information about the effectiveness, efficiency, and impact of a program, intervention, or policy.

Stine provides independent consulting, policy analysis, program evaluation, freelance writing, teaching, and coaching services. Previously, she served in the Obama White House as Executive Director of the President’s Council of Advisors on Science and Technology, a Science & Technology Specialist at the Congressional Research Service, a study director at the National Academies of Sciences, Engineering, and Medicine, and as a professor of the practice in engineering and public policy at Carnegie Mellon University.

To start the workshop, Stine asked participants to think about a program they are familiar with, whether it be something related to their job, or something they participate in themselves. From there, she asked them to identify how well they think the program does its job, what works in the program and what doesn't.

To demonstrate the different elements of program evaluation. Stine used the example of the Special Supplemental Nutrition Program for Women, Infants, and Children, or WIC program, throughout her presentation. Stine utilized ChatGPT to answer the prompt “What about the WIC program works well?” ChatGPT identified three main points:

1. The program improves nutritional status and health outcomes

2. The program provides an educational component about nutrition and health

3. The program provides clear economic benefits

Stine also asked ChatGPT where the program needs to improve, which it identified as limited access and enrollment, restrictive food practices, and technological challenges. Stine used this exercise throughout the workshop to demonstrate how ChatGPT examines the literature available and provides a good starting point for making improvements in a program. 

Why conduct a program evaluation?

Stine explained that program evaluations are essential, especially because in many instances, the theory of action for a program is not clear. She also noted that including program evaluations is essentially the norm in today’s implementation process.

“Many organizations now require program evaluations as a condition of funding,” she said. “So when you’re writing a proposal they ask for a basic logic model as a part of that submission. Sometimes they’ll ask [a program evaluation] at the end [of the program].”

Stine identified 3 main reasons for conducting program evaluations.

1. Accountability: required by the funding organization as a condition of funding

1. Congress, federal agencies, foundations

2. Document goal attainment to justify program existence

2. Developmental: make the case for more funding to potential funders

1. Expand existing services to meet unmet needs

2. Provide new services to the same or similar populations to reach a societal goal

3. Knowledge: review existing programs to identify improvements to increase effectiveness, efficiency, and equity.

“There is not any proposal I have written in the last ten years where I haven’t had to give information on how the program was evaluated to see if it was successful in achieving societal goals and what indicators I would use,” she said.

Stine defined 3 E’s and 3 I’s to assess the past, present, and future of programs:

• Effectiveness in reaching program outcomes and societal impact

• Efficiency 

• Equity

• Indicators of need 

• Implementation and delivery

• Improvements

There are many different kinds of evaluations. These can be conducted internally, to improve an existing program, or externally, where third-party assessment is considered. Program evaluations can incorporate quantitative data, qualitative data, or a mixed-methods approach. Stine encouraged participants to use both quantitative and qualitative methods in their evaluations. For example, she explained how to effectively evaluate an unemployment and job training program intended to get individuals hired. 

“It’s not just enough to say quantitatively how many people were trained in the program and got jobs, but we also want to know what those people thought of those trained workers,” she said.

The CDC has a 6-step program evaluation framework that Stine outlined throughout the workshop. The first 3 steps can occur in the best order for the evaluation context.

1. Engage stakeholders

2. Describe the program

3. Focus evaluation design

4. Gather credible evidence

5. Justify conclusions

6. Ensure the use and share lessons

How do you engage stakeholders?

Engaging stakeholders can often be a challenge when implementing a program.

“Sometimes people are afraid to engage stakeholders,” Stine said. “But we really need to talk to whom our programs are meant to serve.”

Stakeholders can include:

• Implementers

• Decisionmakers

• Those served or affected by the program

• Partners

Stine asked ChatGPT to provide examples of 10 WIC stakeholders to show the scope of involvement certain programs have. She stressed that stakeholders also include potential clients not served or not-fully served by your program.

“ChatGPT is helpful in ensuring you don’t miss someone. Would you have [thought of] the other five [stakeholders]?” she asked.

For a few minutes, Stine allowed participants to think about and identify stakeholders for the program they had in mind at the beginning of the workshop. There are several important questions to pose to stakeholders. These include:

1. Who do you represent and why are you interested in this program?

2. What is important about this program to you?

3. What would you like this program to accomplish?

4. How much progress would you expect this program to have made at this time?

5. What do you see as the critical evaluation questions at this time?

6. How will you use the results of the evaluation?

7. What resources might you contribute to the evaluation effort?

In addition to the 4 E’s and I’s, there are 4 A’s to identify implementation challenges for eligible clients not part of the program.

1. Accessible

2. Awareness of services

3. Availability of services

4. Adaptability of services

“Sometimes people don’t know these services exist and that is often a challenge as well. In the U.S. we have a whole alphabet soup of programs,” she said.

Describing the program

A logic model, Stine explained, is a graphic and text that illustrates the relationship between a program’s activities and its intended outcomes and results. Logic models illustrate the logic or theory of the program or project, focus attention on the most important connections between actions and results, build a common understanding among staff with stakeholders, establish a framework for measurement and evaluation and inform program design, and find gaps in the logic of a program and work to resolve them. 

When creating a logic model it is important to identify the inputs, activities, outputs, customers, outcomes, external influences, and assumptions. However, it is important to note that logic models are not entirely reflective of reality, and can often oversimplify programs. 

“The main thing you can't predict in program evaluation is people’s behavior,” she said. 

Stine used an example from Australia where the program objective was to support Blairtown families and ensure children reached appropriate developmental milestones. In explaining the different elements of the logic model, Stine explained that there should be a direct link between problem statements and outcomes.

To determine the success of a program, one can look at outcome indicators to evaluate how well a program achieved its initial goal. Outcome indicators must:

• Be realistic

• Be specific and measurable

• Be supported by evidence

• Be important

• Include change statements

Once these were identified, Stine gave participants the opportunity to fill out their own logic model with the program they picked at the beginning of the workshop. Logic models, she stressed, need to be developed thoughtfully and intentionally. A logic model can change based on evidence, so it is essential that the model is easy to explain to funders, policymakers, stakeholders, and new staff.

The final steps of evaluation

When focusing your evaluation design you must prioritize users, uses, questions, methods, and agreements. Once this is done you will be able to gather credible evidence and justify the conclusions of your program.

“When you come to your conclusions you want to make sure you aren't just coming to conclusions that are willy-nilly, but ensure that there is a basis to these judgments,” Stine said.

Stine emphasized the importance of sharing the lessons learned from your program evaluation. These lessons should incorporate comments about the design of the program, how the program was prepared, what feedback the program received, what follow-up was needed, and how the program was disseminated.

Personal bias is often a struggle in program evaluation and to avoid this Stine urged participants to understand their outcome indicators fully and to be specific in their expectations before conducting an evaluation. Ethical challenges can also arise in program evaluation, and in these circumstances, evaluators need to ensure they are focusing on all stakeholders, not just the ones that demonstrate the success of the program. Stine also noted the importance of perspective. Different words have unique meanings to people. For example, the word “good” can mean a variety of things. 

As a response to ethical challenges, one must:

• Avoid case focus that focuses on a few clients versus all clients

• Develop precise measures

• Focus on the reality and truth about a program

• Ensure results are based on objective data that can be replicated

• Identify biases early in the process and review them throughout the process

Feedback

In a survey following the workshop, 100% of respondents said they would recommend the workshop to a friend or colleague, and 90% said they would use what they learned in their work. Check out what some of our attendees had to say about the workshop!

“Great content! The session was very detailed and had resources I had not heard of before.” -Colorado attendee, Health and Human Services, Mid-Career

“I've learned practical ways to enhance my work. I liked how Dr. Stine provided prompt examples to use ChatGPT as starting points or as a tool.” -California attendee, Health and Human Services, Advanced Career

“The training was well informed and it provided a sense of beacon of hope.” -New Jersey attendee, Health and Human Service, Advanced Career

 

To watch the workshop, check out the recording here!