Knowing your Linkages program is working requires a clear definition of success, a way to measure that success and a feedback loop to improve Linkages as a result of what the evaluation shows. It starts with agreement on the client-centered results Linkages will help families achieve including a theory of change for getting from point A to point B. To evaluate your Linkages program follow the steps below using the planning tips and links to resources to help guide you along the way.

1. Select Outcomes

Before you begin evaluating, follow the Program Design Road Map to create a clear definition of your Linkages program, who it will serve and what interventions will be used to meet program goals. To figure out whether Linkages is successful, you first need to know what success looks like for your initiative. This means determining which results for families (outcomes) you want Linkages to achieve. In other words, what changes in the lives of Linkages families would you want to see to help you know whether the program has been successful? Refer to How Do I Choose Outcomes to guide the selection of outcomes that best demonstrate the positive impact you expect your Linkages program to accomplish.

Experience tells us…

  • Select a small number of outcomes that are critical to the type of Linkages program you are implementing.

Resources…

2. Customize Your Linkages Logic Model

The next step in evaluation is to refine your hypothesis for why you believe Linkages will work. To do this, review the Linkages Logic Model and customize it to match your specific Linkages initiative. Make your own logic model diagram using the systems perspective version as a template. Review and adjust the Inputs/Resources and Outputs/Activities which will lead to the Outcomes selected as relevant to your Linkages initiative. Also, refine the External Factors that may affect the success of Linkages in your location. Next, work with your planning team to ensure a logical progression from each column of the diagram to the next. Specifically, consider these questions:

  • Resources/Inputs: What are the key ingredients necessary for implementing Linkages? Consider which leadership and staff need to be involved, what training and new procedures are needed, and how large your client base needs to be to make implementation worthwhile. Also consider the data collection and evaluation procedures you need to have in place to evaluate your initiative.
  • Outputs: Outputs generally describe the work that is done through the Linkages program and is characterized by both what gets done and who is reached. What are the activities and processes that define your Linkages initiative? These are the set of actions, interventions, or services that are unique to Linkages. Who is served by Linkages? These are the clients or customers who benefit from Linkages.
  • Outcomes: How will your Linkages initiative benefit families? For most programs, the outcomes answer the question, “Why are we implementing Linkages?” Outcomes are often longer-term results than Outputs and represent the goals of the initiative.
  • External Factors: These are contextual factors that can affect the success of Linkages but are outside the boundaries of the Linkages initiative. Consider how these factors can influence some of the initiative’s inputs or activities. If an external factor is likely to pose a barrier to your program, you can start early on planning around it to the extent possible.

Use your customized logic model to convey a concise summary of your Linkages program to various audiences. It will not only guide evalution, but can serve as a training and communications tool as well.

Experience Tells Us…

  • Make sure your Logic Model is succinct–with enough detail to be useful, but not overly complex.
  • Refer to your Logic Model during implementation to monitor that your Linkages program is on course.
  • The Logic Model can be a living document which can be adjusted as your program evolves.

Resources…

3. Identify the Measures For Your Outcomes

With your customized logic model in hand, the next step is to determine how to measure the outcomes you want to evaluate. This involves knowing who your Linkages families are, translating outcomes into specific data elements and deciding when measurement will occur. Complete this step by engaging your planning team with your county data analysts to answer the following questions:

  • What time period do you want to establish from which to draw the data? For example, you may want to look at data every year on clients who have a Linkages case start date between January 1 and March 30. To measure the outcomes for a particular time period, you may need to create a separate “merged data” file that links selected data from both your Child Welfare (CWS/CMS) and your CalWORKs/TANF data systems.
  • What identifying information is shared commonly between both data systems? Some counties use social security numbers to accurately identify each client. By matching social security numbers of clients in the Child Welfare (CWS/CMS) system with those in the CalWORKs/TANF system, you can pull data for the matched clients into your merged data file.
  • Which clients are your Linkages clients? You will need to make sure that you can identify all of your Linkages clients served during the time period you selected, and include all of them in your merged data file.
  • What information do you need on clients or families to measure each outcome? Go through each of your outcomes and identify the data fields you will need to calculate the outcomes. For example, to measure the outcome “Percent of families that achieve reunification,” you would need:
    • Total number of families with a Linkages case start date during the specified time period.
    • Out of these families, the number of families with at least one child that was removed from the household.
    • Out of these families, the number who were reunified with at least one child.

After answering these questions, you’ll have specific instructions for your county data analysts to proceed with data collection.

Experience tells us…

  • Make sure the data you’ve identified to calculate your outcomes is consistently captured and accessible.
  • Train staff on how to enter the critical data that will help track the success of Linkages.
  • Encourage supervisors to reinforce the value of evaluation with their staff.

Resources…

Guidance on how to “flag” Linkages cases by using Special Project Codes in CWS/CMS

 4. Collect Data

Data collection involves gathering both quantitative data about your Linkages clients and collecting qualitative information directly from staff and clients. Together, this tells you whether Linkages service coordination is effective in changing families’ lives for the better and how staff and clients themselves are experiencing the program.

Quantitative data comes from at least two places: your Child Welfare (CWS/CMS) data system and your CalWORKs/TANF data system. To ensure that you can identify Linkages clients and link each client’s information from both of these systems, be sure to:

  • Label or flag your Linkages clients with a special label or code in your database(s).
  • Alternatively, you could use a common Linkages identifier shared by each data system.
  • Collect and enter data in a consistent way – a process called data validation. For example, if you have a database field called “Linkages start date,” make sure that means the same thing to all of the people who collect and enter the information. Ideally, this type of information should be defined the same way by your Child Welfare program and by your CalWORKs/TANF program.

Qualitative data can be drawn from surveys, interviews or focus groups to reveal the level of satisfaction that staff and clients have about Linkages. See the resource links below for sample satisfaction surveys that can be adapted for use in your location.

Other client related qualitative data can be gleaned from case file reviews. This involves identifying a sample of Linkages case files, then reviewing sources such as the case plan, contact notes and other narrative case documentation for clues about the frequency, depth and quality of service coordination occurring on Linkages cases. This can happen informally as part of supervision or more formally by selecting a random sample of cases from a location where Linkages is fully implemented.

Experience tells us…

  • When an outside evaluator is involved in data collection, protecting client identities is essential
  • Create a unique ID number for each of your Linkages clients. This provide the evaluator with the information they need on each client without providing any personally identifying information like names or social security numbers.
  • A unique ID number should not contain the client’s initials, date of birth, or social security number.

Resources…

 5. Analyze Your Results

This is where you make meaning out of your evaluation data. Invite your Implementation Team, including line staff and supervisors, to help interpret what the results show. In order to have a point of comparison, use outcomes data to examine changes over time. To do this, measure your outcomes for (at least) two different time periods. Compare each outcome to see if there was a change from the earlier to the later time period, and whether that change was positive or negative. Consider the following:

  • Why might the changes you see have occurred? Do they make sense based on what you have observed in your Linkages initiative? In addition to the possibility that Linkages contributed to the changes, what factors outside of Linkages might have contributed?
  • Is there anything different between the two time periods you selected that could cause you to see changes in the outcomes? For example, are there seasonal differences that affect your caseload or clientele?
  • Is the impact of Linkages different for different groups of people? If you have a large enough number of Linkages clients, consider examining the data from different subsets of clients, based on, for example, where in the county they received services, whether the Child Welfare cases were court-ordered or voluntary, or whether clients had CalWORKs/TANF sanctions or not. When comparing the different subgroups, do some show larger changes over time than others? What might that mean?

As your Linkages program continues, you can look at outcomes data from one year to the next to see whether the numbers are improving, getting worse, or staying the same. You may also be able to compare a group of Linkages clients with a group of similar clients that are not in your Linkages initiative to see if Linkages makes a difference. This information is useful in helping you understand areas of success and areas of challenge for your initiative.

Experience tells us…

  • Changes in the evaluation findings for better or worse may not always be due to Linkages–other factors like economic downturns can affect outcomes too.
  • Include those closest to Linkages day-to-day practice to help interpret the evaluation findings.
  • Communicate evaluation results to staff, leadership, stakeholders and other interested parties.

Resources…

6. Adjust Linkages Based on Results

Conclusions drawn from evaluation can suggest changes and refinements necessary to improve or expand your Linkages program. This may include revisions to policies, procedures, staff development or other aspects of Linkages. Gather your implementation team to interpret the findings and identify program improvements; include both line staff and managers in the dialogue. Once the implications of your evaluation results are known, revisit the Implementation Planning Road Map and plan for the various components of Linkages that may need to be adjusted. Be sure to clearly communicate the shifts in your Linkages program to all involved.

Experience tells us…

  • Sustained leadership is necessary to ensure continual evaluation of Linkages programming.
  • Keep key champions informed about program improvements based on evaluation findings. They can help explain to staff why Linkages processes & procedures are being modified.

Resources…

  • Future home for examples of program modifications based on evaluation outcomes.