An interesting post on the University of Pennsylvania Reg Blog from Michael Abramowicz, Ian Ayres, and Yair Listokin (AAY) on “Randomizing Regulation,” based upon their piece in the U Penn L. Rev.
If legislators disagree about the efficacy of a proposed policy, why not resolve the disagreement with a bet? One approach would be to impose one policy approach randomly on some members of the population, but not on others, to determine whether the policy meets its goals. This solution would overcome the measurement problems of conventional regression analysis and would provide a useful way to compare regulations and promote bipartisan agreement. Legislators might agree that once such a test is complete, the winning approach would apply to everyone.
For example, regulators could test the Sarbanes-Oxley Act’s most controversial provisions, such as those requiring public companies to institute internal controls and then to have their CEOs and CFOs certify their financial statements, by randomly repealing one or more of those provisions for some corporations for some period of time. Randomization would enable analysts to determine which regulatory regime is optimal by assessing which test-group of corporations has the highest level of success, whether measured by stock price, investor confidence in financial reporting, lack of fraud, or other yardsticks.
Conventional statistical and econometric analytical techniques are often used to measure the efficacy of statutes and regulations, but they face problems that randomized trials would not. Researchers may purposefully or mistakenly omit variables from their regression analyses, leading to incorrect results. Publishers are more likely to feature work that provides statistically significant results, even if those results are not correct, a phenomenon known as publication bias.
No doubt many economists and empiricists are nodding their heads in agreement and drooling at the opportunity to more accurately identify and measure the effects of regulation. Randomization would allow application of techniques far superior to what is typically used. AAY discuss some of the common critiques of randomization in the blog post, and at greater length in the paper. The longer version is worth reading, but here is the short version from the blog post:
Ethical concerns are important, but may not present a significant barrier to using randomized tests. While legal randomized tests would lack the informed consent provided in medical experiments, the government regularly imposes regulations on the public – within constitutional and other legal bounds. Also, randomization sometimes makes the imposition more equal than regulation imposed using predetermined criteria. We tend to think it is worse to impose rules on people because the selected people are unpopular rather than simply because they were selected randomly.How should randomized trials work? The experiments should be large enough to produce meaningful results. The test groups, meanwhile, should be the smallest possible without changing the results outside those test groups. For example, driving speed limits cannot be randomized at the individual level because such a test group size would significantly increase the risk of accidents. However, the test group could be at the county level.Experiments should also be of sufficiently long durations to prevent test subjects from changing their behavior temporarily for the duration of the experiment. For example, if different income tax levels are imposed on different people to see if imposing a higher income tax reduces work output, an experiment of short duration would be more likely to be biased. Workers could wait out a temporary increase in income tax level by temporarily working less, and plan to work more once their income tax level decreases.There is no problem, under current standards of judicial review, with administrative agencies testing out different regulations on their own. Agencies could put their proposed experimental regulations through the regular notice and comment process. After running the experiment, the agencies could provide a randomization impact statement explaining why the agency decided to test regulations through that process, describing the experiment, and providing its results. Because randomization provides for more objective analysis of policy results, courts should be more deferential in conducting hard look review to agencies that have selected policies through this approach.