Philanthropic Leaders Gather to Better “Right Fit” Evidence for the Social Sector
February 11, 2019 – The Center hosted more than 30 philanthropic leaders from throughout Los Angeles for an intimate discussion about data collection and the pitfalls of impact evaluation with Mary Kay Gugerty, Nancy Bell Evans Professor of Nonprofit Management at the Daniel J. Evans School of Public Policy and Governance and Faculty Director of the Nancy Bell Evans Center on Nonprofits & Philanthropy at the University of Washington.
Dr. Gugerty shared that the title of her talk and her most recent book, “The Golidilocks Challenge,” which she wrote with co-author Dean Karlan, Professor of Economics and Finance, Kellogg School of Management at Northwestern University, is an analogy that stems from the constant challenge that social sector organizations face collecting and evaluating data. It’s often either too much or too little. She argues that organizations often spend too much time and money on poorly conceived, poorly designed, and poorly implemented impact evaluations and not enough on trying to measure those things that are most closely linked to strategic decision making and organizational performance.
She calls this effort trying to “right-fit” data and evidence. “The idea is basically that we need to do more to encourage organizations to find a balance between how much learning is likely to be generated by evaluating something, whatever that may be, versus what it will cost to do it.”
Dr. Gugerty says that many organizations collect far more data than they actually have the resources to analyze, resulting in wasted time and effort that could be better spent elsewhere. Other organizations collect the wrong data, tracking changes in outcomes over time but not in a way that allows them to know whether the organization or program caused the changes on its own.
Most service delivery organizations, for example, need to know that program beneficiaries are showing up to receive a particular program, and that the organization is at least doing some of the things they said they are going to do if the program is well implemented. Rather than trying to determine the impact that the program has had in a double-blind impact evaluation, the organizations should start by determining whether they are effectively reaching their target populations.
After an organization determines what it is they want to study and their own theory of change, meaning a comprehensive description of how and why a desired change is expected to happen in a particular context, she says that organizations have to determine what data they will collect. She offers the C.A.R.T. principles to help guide organizations. First, is the data Credible, meaning is it valid, reliable and can it be analyzed? Second, is the data Actionable, meaning something that an organization can commit to using? Third, is trying to collect the data Responsible, which raises the question of what it costs to collect compared to its potential benefits? Finally, is the data, Transportable, meaning can the data be used or be useful in a different context?
The presentation garnered numerous questions and dialogue about the challenges for funders in both how they can go about properly evaluating their own work as well as ways that they can better support grantees to evaluate what’s working and what isn’t for them.
The convening was part of The Center’s Leading Practitioner Discussion series, which provides a venue for the local philanthropic community to participate in intimate discussions with notable practitioners who have had a significant impact on the field.
The Center on Philanthropy & Public Policy promotes more effective philanthropy and strengthens the nonprofit sector through research that informs philanthropic decision-making and public policy to advance community problem solving. The Center is a part of the USC Sol Price School of Public Policy, which works to improve the quality of life for people and their communities, here and abroad.
For more information:
Please contact Nicholas Williams, Associate Director, The Center on Philanthropy & Public Policy at [email protected] or (213) 740-8557.