Publications

The Federal Reserve Bank of Atlanta (Atlanta Fed) developed the nation’s first suite of tools that model public assistance changes within a career planning framework. These tools use an underlying technology called the Career Ladder Identifier and Financial Forecaster (CLIFF). As of August 2022, over 50 partnerships of education, workforce development, and human services organizations are piloting these tools. During 2021 and 2022, pilot participants in three US states conducted external evaluations for their respective pilots on one of the CLIFF tools, the CLIFF Dashboard. The broad goals of these pilots were to evaluate whether the CLIFF Dashboard improves outcomes for workers and provides a valuable resource to career and financial coaches. This Partners Update article summarizes the results of these evaluations and discusses how they may affect the future development of the CLIFF tools.

The Atlanta Fed was not involved in the decision to conduct these evaluations, which were produced by independent third parties. Also, the Bank does not have a financial relationship with the evaluating parties. Sponsors of the pilot program solely selected the evaluators, whose conclusions will help determine the effectiveness of the pilot and the program’s path forward.

Overall, the results were promising and demonstrated a positive impact for using the Dashboard in career planning and coaching settings. In each of the evaluations, 75 percent or more of the clients indicated that the Dashboard was helpful for individual career planning. One client’s reflections on the tool illustrate how it helps clients to understand the long-term benefits of career planning and inform any concerns about the loss of public assistance:

“I was able to gain an understanding of the career I’d like to go into and see how it would affect my benefits. That’s something that’s always intimidating, seeing if the end result will pay off, and it definitely showed me that it will.”

Similarly, each evaluation reported that coaches believed the Dashboard aided their clients with education, training, and career planning. The evaluation results also provided valuable insight on Dashboard enhancements that will increase utility for coaches and clients.

What are the CLIFF tools and how do they benefit workers?
The Atlanta Fed’s Advancing Careers initiative seeks to improve economic mobility and resilience by helping to eliminate barriers to career advancement caused by the loss of means-tested public assistance and financial insecurity. The initiative provides analytical tools and research to assess policy and programmatic solutions; assists partners that adopt the CLIFF tools with training, user support, and evaluation; and builds a constituency of actors who can implement solutions at various scales.

The CLIFF tools were originally designed around a fundamental idea in workforce development—career pathways, or the gradual advancement from entry-level to higher-paying occupations within an industry. Atlanta Fed research has documented that for some workers on public assistance, the higher-paying occupations along a career pathway may not offer a compelling financial gain. The higher pay associated with more advanced occupations may result in benefits cliffs (defined as an income increase that triggers a public assistance loss that makes a worker worse off financially), benefits plateaus (defined as an income increase that triggers a public assistance loss that makes a worker no better off financially), or, more generally, a high effective marginal tax on their increased income. The CLIFF tools are designed to provide coaches and workers a way to forecast these income gains and public assistance losses in the short and long term across various career pathways to support informed career planning. The CLIFF tools account for changes in employment income, household expenses, taxes, and public assistance receipt over time, allowing workers to forecast their future gains from career advancement and changes in public assistance.

The three tools in the CLIFF suite are a simplified calculator or snapshot of changes in net financial resources for different work hours and wages, a Dashboard that models lifetime career choices for hypothetical family types, and a Planner that is a more individualized budgeting resource for career and financial planning.¹ The three tools offer differing degrees of customization and features. Pilot sites, which to date have included state and local coalitions of partners, choose which tool or tools to use depending on their services or their customers’ needs. To support pilot sites, the Atlanta Fed provides online, self-paced interactive training through its Advancing Careers Academy and weekly help desk office hours.²

Each of the three evaluations summarized in this article used the CLIFF Dashboard because of its focus on career choice, simplicity of design, and fewer requirements to enter customer information. The sites determined that the Dashboard was well suited to a pilot study, while leaving open the possibility of expanding usage to the other CLIFF tools after the pilot.

CLIFF Dashboard evaluations in Connecticut, Florida, and Maine
In June 2022, three partnerships released independent, third-party evaluations of their CLIFF Dashboard pilots. In Connecticut, the University of Connecticut evaluated a coordinated pilot implemented across several sites, including an alternative career-focused high school, a family resource and early learning center, and a career center. In Florida, the University of Florida evaluated a pilot implemented at nine regional American Job Centers. In Maine, Maine Applied Research evaluated the statewide pilot that seven diverse nonprofit workforce development organizations had implemented.

The three evaluations featured different research questions even though their research designs and methodologies were similar. In Connecticut, the goal of the evaluation was to assess if an early demonstration of public assistance loss would be helpful to families. In addition, the evaluation sought to determine for which organizational settings and customer populations the Dashboard is best.

The Florida evaluation was designed with a straightforward goal to collect client and staff perspectives on the Dashboard’s user interface and utility. Given the pilot setting of workforce development organizations, the focus of the study was on the Dashboard’s utility for longer-term career planning.

The Maine evaluation investigated three research questions:

• How do participants experience the Dashboard, and how does it impact their understanding of the interplay of benefit changes and career options?
• How can the Dashboard support staff in coaching participants toward their career goals and helping them navigate anticipated benefits cliffs?
• What training and supports can help staff best use the Dashboard?

Despite the differences in research questions, the three evaluations reached many shared conclusions about the Dashboard’s utility for career planning and forecasting public assistance loss, the organizational settings and customer profiles for which the tool has the most value, and recommendations for improvement.³

Clients and coaches found the CLIFF Dashboard to be a useful and user-friendly tool
This section highlights summary findings from each evaluation and is organized by research question.

Connecticut
Is an early demonstration of the benefits cliff helpful to families?

• Looking at the visuals of benefits cliffs and the path to self-sufficiency reduced clients’ stress about losing benefits.
• The CLIFF Dashboard as implemented empowered families to visualize and plan for self-sufficiency.
• The Dashboard was impactful as a career planner by visualizing how different careers compare by lifetime income.

For which organizational settings and customer populations is the Dashboard best?

• The Dashboard was used successfully in job training settings and an alternative high school that focused on career exploration and planning. The Dashboard faced implementation challenges at a two-generation (defined as a focus on parent and child) family resource and an early childhood education center due to staff’s limited access to knowledge about parents’ public assistance receipt, not having a case manager to coordinate resources, and not offering a broad range of training options to parents.

Florida
What are client perspectives on the Dashboard’s user interface and utility?

• 79 percent of client survey respondents reported that the Dashboard impacted their career-related decisions.
• Clients found the Dashboard to be easy to use and understand with the support of pilot site staff.
• The Dashboard appears to be most useful for customers who are exploring career options rather than customers who have already made a training or career decision.

What are staff perspectives on the Dashboard’s user interface and utility?

• Staff reported the Dashboard’s user interface to be easy to use with clients but also offered recommendations to improve usability.
• Staff reported the graphs depicting long-term expenses and earnings projections were particularly useful.
• Staff identified that the clients who gain the most benefit from the Dashboard are those with at least a high school diploma or GED who are interested in a career change.

Maine
How do participants experience the Dashboard and how does it impact their understanding of the interplay of benefit changes and career options?

• Before Dashboard use, 81 percent of participants somewhat or strongly agreed it was possible to become financially self-sufficient within the next five years. After Dashboard use, this increased to 98 percent.
• 83 percent of participants somewhat or strongly agreed that the Dashboard helped them make a career plan.
• All respondents strongly or somewhat agreed that the Dashboard helped them understand how income changes may impact benefits.

How can the Dashboard support staff in coaching participants toward their career goals and helping them navigate anticipated benefits cliffs?

• Organizations involved in the pilot found the Dashboard to be a useful, user-friendly tool for working with individuals making long-term choices about education, training, and careers.
• 96 percent of staff said it helped their client either “a lot” or “somewhat.”

What training and supports can help staff best use the Dashboard?

• Online training was clear, thorough, and gave staff information needed to use the Dashboard effectively with participants.
• One staff member reported the online training to be too slow and recommended a more concise tutorial.

Evaluators recommend more custom options and other enhancements to the CLIFF Dashboard
Each evaluation concluded with recommendations for improving the Dashboard’s functionality to better assist workers and coaches. All three evaluations recommended that the Dashboard include more career options. It is important to note that the list of occupations in the Dashboard was by design limited for these pilots; pilot sites wanted to focus the tool on a prespecified set of desired occupations, rather than showing a list that potentially included jobs that did not offer a promising career path. The CLIFF tools by default now include the full list of nearly 800 occupations available in the US Bureau of Labor Statistics’ Occupational Employment and Wage Statistics. Pilot sites also recommended that the functionality be expanded to better capture training programs such as apprenticeships.

Recommendations also included changes to the inputs and functionality of the Dashboard. For example, the evaluations recommend that the Dashboard be expanded to include additional public assistance programs such as Supplemental Security Income, allow the coach to override the default wage provided, let coaches enter student loans for educational expenses, and add additional income sources beyond employment income from the worker’s primary job.

Additional recommendations focused on the user experience. One evaluation reported that the Dashboard often timed out during a long coaching session, forcing the coach to start over and reenter the information into the Dashboard. The evaluations also recommended that the Dashboard become available in additional languages besides English and rename public assistance programs to correspond to state-specific names rather than general federal names.⁴

What’s next for the CLIFF tools
The evaluation results will inform improvements to the CLIFF tools. Indeed, several of the recommendations for improvement have either already been implemented or are in the process of being added. For example, the ability to enter a custom wage and additional income sources is already available in the CLIFF Planner. The CLIFF tools will also continue to add additional public assistance programs to their calculations and expand the options for household size. A version of the Dashboard will soon be available in Spanish, and the Advancing Careers team and partners are exploring the potential for additional languages. Other changes, such as easier functionality for incorporating apprenticeships, are currently being evaluated.

These evaluation results can benefit new and existing users of the CLIFF tools. In particular, the evaluations can help new pilot sites better understand the organizational settings and target populations (for example, workers currently exploring careers) for which the CLIFF tools offer the most value. The Atlanta Fed will distribute the evaluation results from Connecticut, Florida, and Maine to other CLIFF users to inform their pilot design and coaching practices.

Finally, the Atlanta Fed—in collaboration with other Federal Reserve System colleagues—is evaluating additional CLIFF pilots across the country. The research teams will continue to release more information about the usage and effectiveness of the CLIFF tools as it becomes available.

More information on the Advancing Careers initiative, research, tools, and partnerships can be found at www.atlantafed.org/advancing-careers.

By Brittany Birken and Alexander Ruder, both principal advisers in Community and Economic Development. The views expressed here are the authors’ and not necessarily those of the Federal Reserve Bank of Atlanta or the Federal Reserve System. Any remaining errors are the authors’ responsibility.

References

Maine Applied Research. June 2022. CLIFF Dashboard: Maine Pilot Evaluation. Prepared for the Maine Whole Family Approach to Jobs Team.

Parr, Kathryn E., Harini Buch, Janine McMahon, and Juliany Polar. June 13, 2022. Final Report on the Connecticut Benefit Cliff Pilot. Submitted to the Connecticut 2 Gen Advisory Board.

Schreiber, Maya, Dévonja Daley, and Herman T. Knopf. June 2022. Evaluation of the Career Ladder Identifier and Financial Forecaster (CLIFF) Dashboard: Final Research Report. Submitted to the Florida Division of Early Learning.

_______________________________________

¹ The simplified calculator tool was not available when these pilots began and is currently in beta testing.

² The Advancing Careers Academy was not yet operational for the start of the Connecticut and Florida pilots. In those two pilots, Atlanta Fed staff provided two live training sessions through Zoom.

³ Sample sizes varied across the three evaluations. In Connecticut, 17 staff members participated in a post-use survey and 7 staff members participated in key informant interviews; 48 clients participated in a pre-use survey and 36 participated in a post-use survey, with 7 clients participating in two focus groups. In Florida, 25 staff members participated in the first survey and 17 in the second one, with 32 staff participating in the focus group at the end of the pilot; of the 290 clients who used the tool in the pilot, 52 responded to the post-use survey. In Maine, there were 46 completed staff surveys; 59 clients completed the pre-use survey and 48 completed the post-use survey, with two clients providing feedback in a focus group.

⁴ For example, in Florida the Child Care Development Fund (CCDF) is called the School Readiness Program.