Chapter 3: Measuring Impactful Policies
By Howard W. Buffett
As a policymaker, one of the toughest challenges you will face is how to design policy that delivers the outcomes you desire.
How do you define those outcomes? What are the activities and outputs that will make your policy successful? Answers to these questions rely heavily on how the policy outcomes are monitored, measured, and evaluated.
In this section, we will help you understand why impact measurement is important, how to design an effective measurement system to track a policy’s impact, and we will also provide you with additional resources that can help you get started.
What is Impactful Policy Measurement?
Impactful Policy Measurement is the process of determining:
What the outcomes of a particular policy may be,
How positive or negative those outcomes may be, and
Whether those outcomes are aligned with the intent of the policy.
In short, a system for measuring impact must be designed to capture, quantify, and analyze how effective a policy is at accomplishing its objectives. Without such measurement built in from the start, you can only rely on rough estimations or gut feelings, neither of which are good enough to solve the problems that effective policy is designed to address.
Why is Measurement Important?
One of your main goals is to more effectively improve the social, environmental, or economic conditions of a given population or place. To achieve improvement, you must first define the change that you desire, then analyze how conditions evolve over time towards achieving that desired change. This requires a rigorously designed, and operationally implementable, system of measurement. Simply put, if it cannot be implemented, then the theory will fall short.
How Does Measurement Affect Policy Outcomes?
Generally, activities that are tracked and evaluated will receive more attention and resources from program implementers. Therefore, a well-designed system of measurement can allow for more robust policy creation, program development, and resource deployment. Better measurement systems also enable more in-depth program management, potentially bolstering the overall effectiveness of a policy over time. For the sake of your constituents, you have a duty to consider how the design of policy itself may hamper or improve the delivery of policy outcomes. Impact measurement makes that task simpler, and more efficient, objective, and effective from the start.
What is an Effective Measurement System?
Measurement systems must be designed from the beginning in order to enable and enhance the effectiveness of the policies they evaluate. Capturing data around change over the lifecycle of a policy is easier and more accurate if implemented from the start, rather than attempting to rediscover or recreate data retroactively.
To help enable the overall effectiveness of a policy, such a measurement system must take into account the following principles and address the corresponding guiding questions:
Intentionality
The approach must reflect the intentions of policymakers and their understanding of what change is desired.
Ask: What are the objectives and why are we pursuing them?
Specificity
The approach must integrate discrete goal-setting that defines the policymaker’s intended change over time.
Ask: What results will deliver our objectives and when?
Inclusivity
The approach must incorporate the preferences and priorities of the organizations and communities directly affected by the change.
Ask: Who is involved in defining our objectives and how those objectives are delivered?
Materiality
The approach must capture measures that are significant and necessary for thorough impact assessment and analysis.
Ask: What data is required to understand how well the objectives are delivered?
How Do You Measure for Impact?
In order to know how effective your policy is and if the policy is helping people in the ways you intended, you should consider at least the following:
Type of Impact
This is a measure of how impact is delivered. For example, a homelessness policy may seek to increase affordable housing units, an unemployment policy may seek to increase the number of available jobs, or an education policy may seek to increase student graduation rates.
Magnitude
This is a measure of how much of a given type of impact is delivered by a policy. For example, this could be the total number of affordable housing units developed, the number of jobs created, or how many students have graduated.
Quality
This is a measure of how well a policy delivers its impact and may be comprised of a subset of key performance indicators. For example, an affordable housing policy may seek to reduce housing density and cost burden, in addition to increasing access to public transportation and early childhood education.
Time
This is a measure of how long it will take the policy to deliver its intended outcomes. For example, it may take three years to enact and effectuate a policy resulting in new job growth.
Cost
This is a measure of the financial resources required to deliver a policy’s objectives. For example, a new policy to increase the number of high school graduates may cost a state government $30 million in grants to nonprofit intermediaries.
Calculating each of these aspects is generally straightforward and will provide you with insights on how effective a particular policy may be—or was—at delivering a defined set of objectives. In other words, it will let you know if your policy did what you wanted it to do, and at the scale you wanted to do it. These measures can also be used by partners you are working with to implement your policy, which may streamline operations and coordination.
Where to Look for More?
As you consider options for building an effective approach to impact measurement, there are a number of systems, standards, and models available to guide you. Those that have undergone extensive peer review include:
IRIS
The Impact Reporting and Investment Standards is a catalog of generally accepted metrics for quantifying many types of impact; its successor, IRIS+, categorizes metrics based on impact categories and objectives. More can be found at iris.thegiin.org.
iRR®
Impact Rate of Return ®, developed by this author, is a method for calculating the impact value of assets used to bring about positive impact. It analyzes the aforementioned attributes of impact measurement through a customizable algorithm. More can be found at www.impactrateofreturn.com.
IMP
The Impact Management Project was developed by practitioners seeking a common management framework for impact projects, and is organized around specific dimensions of impact performance (What, Who, How Much, Contribution, and Risk). More can be found at impactmanagementproject.com.
For a more detailed discussion of impact measurement approaches, see: The Global Impact Investing Network: “Getting Started with Impact Measurement & Management” at thegiin.org/imm.
Thumbnail
What You Need to Know
Sound impact measurement systems are a bedrock component of impactful policy design, implementation, and management.
By understanding, from the outset, both the outcomes a policy seeks to achieve, and how you will measure progress towards those outcomes, you can maximize the potential of any policy to generate positive impact.
Impactful policy measurement is the process of determining:
What the outcomes of a particular policy may be,
How positive or negative the outcomes may be, and
Whether those outcomes are aligned with the intent of the policy.
Effective impact measurement requires both an established baseline against which to measure, and attention to the following principles, as well as associated guiding questions:
Intentionality
The approach to measurement must reflect the intent of the policy and an understanding of the problem it seeks to solve.
Specificity
Discrete goals must be set, and correspond to the intended change over time.
Inclusivity
The preferences and priorities of the organizations, communities, and stakeholders directly affected by the change must be incorporated.
Materiality
What is measured must be thorough, and must include measures that are significant and necessary to evaluate the positive (or negative) outcomes of the policy.
At minimum, your impact measurement framework should include the policy’s type of impact, magnitude of impact, quality of impact, cost of impact, and time of impact.
A number of peer-reviewed systems, standards, and models of impact measurement are available to guide you. These include:
IRIS
The Impact Reporting and Investment Standards, and the recently released IRIS+
iRR ®
Impact Rate of Return ®, developed by this author
IMP
The Impact Management Project
© Howard W. Buffett and All Rights Reserved