The Iron Law of Policy Evaluation

We have clear evidence that the vast majority of government policies fail.

The following is an excerpt from my second book Promoting Progress: A Radical New Agenda to Create Abundance for All. You can order e-books at a discounted price at my website, or you can purchase full-price ebooks, paperback, or hardcovers on Amazon.

Other books in my “From Poverty to Progress” book series:

See also my other articles on reforming the policy-making process:

• Why do government policies fail so often?

• The Iron Law of Policy Evaluation (this article)

• The case for randomized trials in public policy

• more

In my previous post, I made the case that government policies typically fail to accomplish their desired results for two key reasons:

1. Our political system has not identified policies that actually work.

2. People involved in politics do not care.

The problem is not bad people. The problem is a bad implementation and evaluation process. Our political system is very poor at implementing solutions that actually work and then iterating based on results. The American people know it and are gradually losing confidence in our governing institutions.

Fortunately, there is another way.

Randomized Controlled Trials

As Jim Manzi pointed out in his book Uncontrolled, a time-tested methodology for determining the most effective policies is Randomized Controlled Trials (RCTs). Widely used in the medical field and rapidly catching on in business, RCTs are an experimental form of impact evaluation in which potential recipients are randomly sorted into two groups: an experimental group, which receives the treatment, and a control group, which does not.

In this book series, I have applied the concept of evolution to a number of areas. I believe that the concept of evolution is extremely useful for understanding complex interactions.

One can think of RCTs as a controlled experiment in evolution. If the government systematically performed RCTs using many different policy options, we would create the variation necessary to fuel evolution. By using clear metrics of the results, we would be measuring outcomes in the same way that evolution “measures” results by the probability that a variation leads to survival and reproduction.

The Iron Law

The dirty little secret of policy evaluation is that the vast majority of government programs that are subjected to rigorous evaluations, including RCTs, show no net positive results. And once one adds in the funding spent on the program, then the net impact is invariably negative. This outcome has been so consistent that policy experts now expect those results.

These results are so consistent that Peter Rossi, an intrepid evaluator of social programs, coined the term the “Iron Law” of policy evaluation. The Iron Law states that “the expected value of any net impact assessment of any large-scale social program is zero.” In other words, the program does not show positive results (Rossi).

Rossi followed up this Iron Law with the even more depressing “Stainless Steel Law.” The Stainless Steel law states: “The better designed the impact assessment of a social program, the more likely is the resulting estimate of net impact to be zero.” In other words, the more rigorous the test, the worse the results.

And Rossi was not some right-winger who hates social spending. He was a left-of-center policy analyst who wanted social policies to work better!

It is astounding to me that the Iron Law is not more widely known. It gets to the very core of the ability of our government to solve problems. Why don’t more people know about the Iron Law of policy evaluation!?!

Moreover, to the best of my knowledge, no public policy expert has refuted the Iron Law. Instead, they just ignore it and move on. But ignoring the results of policies is getting increasingly difficult to do. If anything, as we gradually increase the usage of RCTs and post the results to the internet, more and more evidence for the Iron Law keeps piling up.

Within the policy evaluation field and to a lesser extent among the few political leaders and policy wonks who care about results, the Iron Law is like the body buried in the basement that everyone in the family knows about but does not talk about. And they pretend not to notice that Weird Uncle Joe supposedly went for a walk one day and never came back!

On the rare occasions that the subject of the Iron Law came up in my graduate-level classes, the professor sheepishly mentioned it, and my fellow students got nervous and confused looks on their faces. Then there was an awkward pause, and a brief conversation, before the topic turned to “more important matters.”

Looking back, I am a bit embarrassed and ashamed that I was not willing to confront the significance of these findings. Quite frankly, I was unwilling to question my fundamental assumptions on the effectiveness of government policy at the time. It took me decades for the significance of the Iron Law to sink in.

Virtually everyone involved in politics or policy behaves on an implicit assumption that government programs typically achieve their goals. And those few who do know about the Iron Law, rarely speak up out of fear of starting an uncomfortable conversation with their superiors.

Both the idealists and the machine politicians who dominate politics hate RCTs and the Iron Law because they perceive them as undermining their entire view of the world. So they do not talk about either. To the extent that idealists and machine politicians think about results, they tend to suspect that their favorite policies will show bad results and be eliminated.

But that is exactly why we need RCTs at scale. And in the end, if the long-term decline of American confidence in our governing institutions is not reversed, any political victories scored by idealist and machine politicians will be very short-lived. In the long run, they need the public to be confident in the effectiveness of the government.

Most Good Ideas Fail

If we broaden the perspective, perhaps we should not be so surprised about the Iron Law. The reality is that the vast majority of new ideas fail to show results in the real world. Our world is very complex, and no one human or group of humans fully understands it.

No matter how “great” an idea is, it must also be better than all the other previous “great ideas” of the past that were tried and turned out to actually work. When one thinks of the millions of “great ideas” that billions of our ancestors experimented with, it should not be surprising that the vast majority of new ideas fail. Results are a very high standard that few ideas can meet.

Businesses, scientists, and medical researchers understand that most “good” ideas fail, so they have built methodologies to test them before scaling them up. If the government did the same, we could solve more problems than we can today.