Archives For

Do biased stats provide bogus economics? A primer on publication bias and power

Eric Fruits —  29 March 2018 — 1 Comment

If you do research involving statistical analysis, you’ve heard of John Ioannidis. If you haven’t heard of him, you will. He’s gone after the fields of medicine, psychology, and economics. He may be coming for your field next.

Ioannidis is after bias in research. He is perhaps best known for a 2005 paper “Why Most Published Research Findings Are False.” A professor at Stanford, he has built a career in the field of meta-research and may be one of the most highly cited researchers alive.

In 2017, he published “The Power of Bias in Economics Research.” He recently talked to Russ Roberts on the EconTalk podcast about his research and what it means for economics.

He focuses on two factors that contribute to bias in economics research: publication bias and low power. These are complicated topics. This post hopes to provide a simplified explanation of these issues and why bias and power matters.

What is bias?

We frequently hear the word bias. “Fake news” is biased news. For dinner, I am biased toward steak over chicken. That’s different from statistical bias.

In statistics, bias means that a researcher’s estimate of a variable or effect is different from the “true” value or effect. The “true” probability of getting heads from tossing a fair coin is 50 percent. Let’s say that no matter how many times I toss a particular coin, I find that I’m getting heads about 75 percent of the time. My instrument, the coin, may be biased. I may be the most honest coin flipper, but my experiment has biased results. In other words, biased results do not imply biased research or biased researchers.

Publication bias

Publication bias occurs because peer-reviewed publications tend to favor publishing positive, statistically significant results and to reject insignificant results. Informally, this is known as the “file drawer” problem. Nonsignificant results remain unsubmitted in the researcher’s file drawer or, if submitted, remain in limbo in an editor’s file drawer.

Studies are more likely to be published in peer-reviewed publications if they have statistically significant findings, build on previous published research, and can potentially garner citations for the journal with sensational findings. Studies that don’t have statistically significant findings or don’t build on previous research are less likely to be published.

The importance of “sensational” findings means that ho-hum findings—even if statistically significant—are less likely to be published. For example, research finding that a 10 percent increase in the minimum wage is associated with a one-tenth of 1 percent reduction in employment (i.e., an elasticity of 0.01) would be less likely to be published than a study finding a 3 percent reduction in employment (i.e., elasticity of –0.3).

“Man bites dog” findings—those that are counterintuitive or contradict previously published research—may be less likely to be published. A study finding an upward sloping demand curve is likely to be rejected because economists “know” demand curves slope downward.

On the other hand, man bites dog findings may also be more likely to be published. Card and Krueger’s 1994 study finding that a minimum wage hike was associated with an increase in low-wage workers was published in the top-tier American Economic Review. Had the study been conducted by lesser-known economists, it’s much less likely it would have been accepted for publication. The results were sensational, judging from the attention the article got from the New York Times, the Wall Street Journal, and even the Clinton administration. Sometimes a man does bite a dog.

Low power

A study with low statistical power has a reduced chance of detecting a true effect.

Consider our criminal legal system. We seek to find criminals guilty, while ensuring the innocent go free. Using the language of statistical testing, the presumption of innocence is our null hypothesis. We set a high threshold for our test: Innocent until proven guilty, beyond a reasonable doubt. We hypothesize innocence and only after overcoming our reasonable doubt do we reject that hypothesis.

Type1-Type2-Error

An innocent person found guilty is considered a serious error—a “miscarriage of justice.” The presumption of innocence (null hypothesis) combined with a high burden of proof (beyond a reasonable doubt) are designed to reduce these errors. In statistics, this is known as “Type I” error, or “false positive.” The probability of a Type I error is called alpha, which is set to some arbitrarily low number, like 10 percent, 5 percent, or 1 percent.

Failing to convict a known criminal is also a serious error, but generally agreed it’s less serious than a wrongful conviction. Statistically speaking, this is a “Type II” error or “false negative” and the probability of making a Type II error is beta.

By now, it should be clear there’s a relationship between Type I and Type II errors. If we reduce the chance of a wrongful conviction, we are going to increase the chance of letting some criminals go free. It can be mathematically shown (not here), that a reduction in the probability of Type I error is associated with an increase in Type II error.

Consider O.J. Simpson. Simpson was not found guilty in his criminal trial for murder, but was found liable for the deaths of Nicole Simpson and Ron Goldman in a civil trial. One reason for these different outcomes is the higher burden of proof for a criminal conviction (“beyond a reasonable doubt,” alpha = 1 percent) than for a finding of civil liability (“preponderance of evidence,” alpha = 50 percent). If O.J. truly is guilty of the murders, the criminal trial would have been less likely to find guilt than the civil trial would.

In econometrics, we construct the null hypothesis to be the opposite of what we hypothesize to be the relationship. For example, if we hypothesize that an increase in the minimum wage decreases employment, the null hypothesis would be: “A change in the minimum wage has no impact on employment.” If the research involves regression analysis, the null hypothesis would be: “The estimated coefficient on the elasticity of employment with respect to the minimum wage would be zero.” If we set the probability of Type I error to 5 percent, then regression results with a p-value of less than 0.05 would be sufficient to reject the null hypothesis of no relationship. If we increase the probability of Type I error, we increase the likelihood of finding a relationship, but we also increase the chance of finding a relationship when none exists.

Now, we’re getting to power.

Power is the chance of detecting a true effect. In the legal system, it would be the probability that a truly guilty person is found guilty.

By definition, a low power study has a small chance of discovering a relationship that truly exists. Low power studies produce more false negative than high power studies. If a set of studies have a power of 20 percent, then if we know that there are 100 actual effects, the studies will find only 20 of them. In other words, out of 100 truly guilty suspects, a legal system with a power of 20 percent will find only 20 of them guilty.

Suppose we expect 25 percent of those accused of a crime are truly guilty of the crime. Thus the odds of guilt are R = 0.25 / 0.75 = 0.33. Assume we set alpha to 0.05, and conclude the accused is guilty if our test statistic provides p < 0.05. Using Ioannidis’ formula for positive predictive value, we find:

  • If the power of the test is 20 percent, the probability that a “guilty” verdict reflects true guilt is 57 percent.
  • If the power of the test is 80 percent, the probability that a “guilty” verdict reflects true guilt is 84 percent.

In other words, a low power test is more likely to convict the innocent than a high power test.

In our minimum wage example, a low power study is more likely find a relationship between a change in the minimum wage and employment when no relationship truly exists. By extension, even if a relationship truly exists, a low power study would be more likely to find a bigger impact than a high power study. The figure below demonstrates this phenomenon.

MinimumWageResearchFunnelGraph

Across the 1,424 studies surveyed, the average elasticity with respect to the minimum wage is –0.190 (i.e., a 10 percent increase in the minimum wage would be associated with a 1.9 percent decrease in employment). When adjusted for the studies’ precision, the weighted average elasticity is –0.054. By this simple analysis, the unadjusted average is 3.5 times bigger than the adjusted average. Ioannidis and his coauthors estimate among the 60 studies with “adequate” power, the weighted average elasticity is –0.011.

(By the way, my own unpublished studies of minimum wage impacts at the state level had an estimated short-run elasticity of –0.03 and “precision” of 122 for Oregon and short-run elasticity of –0.048 and “precision” of 259 for Colorado. These results are in line with the more precise studies in the figure above.)

Is economics bogus?

It’s tempting to walk away from this discussion thinking all of econometrics is bogus. Ioannidis himself responds to this temptation:

Although the discipline has gotten a bad rap, economics can be quite reliable and trustworthy. Where evidence is deemed unreliable, we need more investment in the science of economics, not less.

For policymakers, the reliance on economic evidence is even more important, according to Ioannidis:

[P]oliticians rarely use economic science to make decisions and set new laws. Indeed, it is scary how little science informs political choices on a global scale. Those who decide the world’s economic fate typically have a weak scientific background or none at all.

Ioannidis and his colleagues identify several way to address the reliability problems in economics and other fields—social psychology is one of the worst. However these are longer term solutions.

In the short term, researchers and policymakers should view sensational finding with skepticism, especially if those sensational findings support their own biases. That skepticism should begin with one simple question: “What’s the confidence interval?”

 

In behavioral economics, truth on the market , , ,
permalink

An Apollo 13 approach to Obamacare

Eric Fruits —  7 July 2017

“Houston, we have a problem.” It’s the most famous line from Apollo 13 and perhaps how most Republicans are feeling about their plans to repeal and replace Obamacare.

As repeal and replace has given way to tinker and punt, Congress should take a lesson from one of my favorite scenes from Apollo 13.

“We gotta find a way to make this, fit into the hole for this, using nothing but that.”

Let’s look at a way Congress can get rid of the individual mandate, lower prices, cover pre-existing conditions, and provide universal coverage, using the box of tools that we already have on the table.

Some ground rules

First ground rule: (Near) universal access to health insurance. It’s pretty clear that many, if not most Americans, believe that everyone should have health insurance. Some go so far as to call it a “basic human right.” This may be one of the biggest shifts in U.S. public opinion over time.

Second ground rule: Everything has a price, there’s no free lunch. If you want to add another essential benefit, premiums will go up. If you want community rating, young healthy people are going to subsidize older sicker people. If you want a lower deductible, you’ll pay a higher premium, as shown in the figure below all the plans available on Oregon’s ACA exchange in 2017. It shows that a $1,000 decrease in deductible is associated with almost $500 a year in additional premium payments. There’s no free lunch.

ACA-Oregon-Exchange-2017

Third ground rule: No new programs, no radical departures. Maybe Singapore has a better health insurance system. Maybe Canada’s is better. Switching to either system would be a radical departure from the tools we have to work with. This is America. This is Apollo 13. We gotta find a way to make this, fit into the hole for this, using nothing but that.

Private insurance

Employer and individual mandates: Gone. This would be a substantial change from the ACA, but is written into the Senate health insurance bill. The individual mandate is perhaps the most hated part of the ACA, but it was also the most important part Obamacare. Without the coverage mandate, much of the ACA falls apart, as we are seeing now.

Community rating, mandated benefits (aka “minimum essential benefit”), and pre-existing conditions. Sen. Ted Cruz has a brilliantly simple idea: As long as a health plan offers at least one ACA-compliant plan in a state, the plan would also be allowed to offer non-Obamacare-compliant plans in that state. In other words, every state would have at least one plan that checks all the Obamacare boxes of community rating, minimum essential benefits, and pre-existing conditions. If you like Obamacare, you can keep Obamacare. In addition, there could be hundreds of other plans for which consumers can pick each person’s unique situation of age, health status, and ability/willingness to pay. A single healthy 27-year-old would likely choose a plan that’s very different from a plan chosen by a family of four with 40-something parents and school aged children.

Allow—but don’t require—insurance to be bought and sold across state lines. I don’t know if this a big deal or not. Some folks on the right think this could be a panacea. Some folks on the left think this is terrible and would never work. Let’s find out. Some say insurance companies don’t want to sell policies across state lines. Some will, some won’t. Let’s find out, but it shouldn’t be illegal. No one is worse off by loosening a constraint.

Tax deduction for insurance premiums. Keep insurance premiums as a deductible expense for business: No change from current law. In addition, make insurance premiums deductible on individual taxes. This is a not-so-radical change from current law that allows deductions for medical expenses. If someone has employer-provided insurance, the business would be able deduct the share the company pays and the worker would be able to deduct the employee share of the premium from his or her personal taxes. Sure the deduction will reduce tax revenues, but the increase in private insurance coverage would reduce the costs of Medicaid and charity care.

These straightforward changes would preserve one or more ACA-compliant plan for those who want to pay Obamacare’s “silver prices,” allow for consumer choice across other plans, and result in premiums that more closely aligned with benefits chosen by consumers. Allowing individuals to deduct health insurance premiums is also a crucial step in fostering insurance portability.

Medicaid

Even with the changes in the private market, some consumers will find that they can’t afford or don’t want to pay the market price for private insurance. These people would automatically get moved into Medicaid. Those in poverty (or some X% of the poverty rate) would pay nothing and everyone else would be charged a “premium” based on ability to pay. A single mother in poverty would pay nothing for Medicaid coverage, but Elon Musk (if he chose this option) would pay the full price. A middle class family would pay something in between free and full-price. Yes, this is a pretty wide divergence from the original intent of Medicaid, but it’s a relatively modest change from the ACA’s expansion.

While the individual mandate goes away, anyone who does not buy insurance in the private market or is not covered by Medicare will be “mandated” to have Medicaid coverage. At the same time, it preserves consumer choice. That is, consumers have a choice of buying an ACA compliant plan, one of the hundreds of other private plans offered throughout the states, or enrolling in Medicaid.

Would the Medicaid rolls explode? Who knows?

The Census Bureau reports that 15 percent of adults and 40 percent of children currently are enrolled in Medicaid. Research published in the New England Journal of Medicine finds that 44 percent of people who were enrolled in the Medicaid under Obamacare qualified for Medicaid before the ACA.

With low cost private insurance alternatives to Medicaid, some consumers would likely choose the private plans over Medicaid coverage. Also, if Medicaid premiums increased with incomes, able-bodied and working adults would likely shift out of Medicaid to private coverage as the government plan loses its cost-competitiveness.

The cost sharing of income-based premiums means that Medicaid would become partially self supporting.

Opponents of Medicaid expansion claim that the program provides inferior service: fewer providers, lower quality, worse outcomes. If that’s true, then that’s a feature, not a bug. If consumers have to pay for their government insurance and that coverage is inferior, then consumers have an incentive to exit the Medicaid market and enter the private market. Medicaid becomes the insurer of last resort that it was intended to be.

A win-win

The coverage problem is solved. Every American would have health insurance.

Consumer choice is expanded. By allowing non-ACA-compliant plans, consumers can choose the insurance that fits their unique situation.

The individual mandate penalty is gone. Those who choose not to buy insurance would get placed into Medicaid. Higher income individuals would pay a portion of the Medicaid costs, but this isn’t a penalty for having no insurance, it’s the price of having insurance.

The pre-existing conditions problem is solved. Americans with pre-existing conditions would have a choice of at least two insurance options: At least one ACA-compliant plan in the private market and Medicaid.

This isn’t a perfect solution, it may not even be a good solution, but it’s a solution that’s better than what we’ve got and better than what Congress has come up with so far. And, it works with the box of tools that’s already been dumped on the table.

In Affordable Care Act, health care reform debate, Labor Law, taxes
permalink

Unsurprising evidence that hiking the minimum wage hurts low wage workers

Eric Fruits —  1 July 2017

On July 1, the minimum wage will spike in several cities and states across the country. Portland, Oregon’s minimum wage will rise by $1.50 to $11.25 an hour. Los Angeles will also hike its minimum wage by $1.50 to $12 an hour. Recent research shows that these hikes will make low wage workers poorer.

A study supported and funded in part by the Seattle city government, was released this week, along with an NBER paper evaluating Seattle’s minimum wage increase to $13 an hour. The papers find that the increase to $13 an hour had significant negative impacts on employment and led to lower incomes for minimum wage workers.

The study is the first study of a very high minimum wage for a city. During the study period, Seattle’s minimum wage increased from what had been the nation’s highest state minimum wage to an even higher level. It is also unique in its use of administrative data that has much more detail than is usually available to economics researchers.

Conclusions from the research focusing on Seattle’s increase to $13 an hour are clear: The policy harms those it was designed to help.

  • A loss of more than 5,000 jobs and a 9 percent reduction in hours worked by those who retained their jobs.
  • Low-wage workers lost an average of $125 per month. The minimum wage has always been a terrible way to reduce poverty. In 2015 and 2016, I presented analysis to the Oregon Legislature indicating that incomes would decline with a steep increase in the minimum wage. The Seattle study provides evidence backing up that forecast.
  • Minimum wage supporters point to research from the 1990s that made headlines with its claims that minimum wage increases had no impact on restaurant employment. The authors of the Seattle study were able to replicate the results of these papers by using their own data and imposing the same limitations that the earlier researchers had faced. The Seattle study shows that those earlier papers’ findings were likely driven by their approach and data limitations. This is a big deal, and a novel research approach that gives strength to the Seattle study’s results.

Some inside baseball.

The Seattle Minimum Wage Study was supported and funded in part by the Seattle city government. It’s rare that policy makers go through any effort to measure the effectiveness of their policies, so Seattle should get some points for transparency.

Or not so transparent: The mayor of Seattle commissioned another study, by an advocacy group at Berkeley whose previous work on the minimum wage is uniformly in favor of hiking the minimum wage (they testified before the Oregon Legislature to cheerlead the state’s minimum wage increase). It should come as no surprise that the Berkeley group released its report several days before the city’s “official” study came out.

You might think to yourself, “OK, that’s Seattle. Seattle is different.”

But, maybe Seattle is not that different. In fact, maybe the negative impacts of high minimum wages are universal, as seen in another study that came out this week, this time from Denmark.

In Denmark the minimum wage jumps up by 40 percent when a worker turns 18. The Danish researchers found that this steep increase was associated with employment dropping by one-third, as seen in the chart below from the paper.

3564_KREINER-Fig1

Let’s look at what’s going to happen in Oregon. The state’s employment department estimates that about 301,000 jobs will be affected by the rate increase. With employment of almost 1.8 million, that means one in six workers will be affected by the steep hikes going into effect on July 1. That’s a big piece of the work force. By way of comparison, in the past when the minimum wage would increase by five or ten cents a year, only about six percent of the workforce was affected.

This is going to disproportionately affect youth employment. As noted in my testimony to the legislature, unemployment for Oregonians age 16 to 19 is 8.5 percentage points higher than the national average. This was not always the case. In the early 1990s, Oregon’s youth had roughly the same rate of unemployment as the U.S. as a whole. Then, as Oregon’s minimum wage rose relative to the federal minimum wage, Oregon’s youth unemployment worsened. Just this week, Multnomah County made a desperate plea for businesses to hire more youth as summer interns.

It has been suggested Oregon youth have traded education for work experience—in essence, they have opted to stay in high school or enroll in higher education instead of entering the workforce. The figure below shows, however, that youth unemployment has increased for both those enrolled in school and those who are not enrolled in school. The figure debunks the notion that education and employment are substitutes. In fact, the large number of students seeking work demonstrates many youth want employment while they further their education.

OregonYouthUnemployment

None of these results should be surprising. Minimum wage research is more than a hundred years old. Aside from the “mans bites dog” research from the 1990s, economists were broadly in agreement that higher minimum wages would be associated with reduced employment, especially among youth. The research published this week is groundbreaking in its data and methodology. At the same time, the results are unsurprising to anyone with any understanding of economics or experience running a business.

In economics, Labor Law, markets, regulation , ,
permalink