Connecting the Dots

March 22, 2018
Posted by Jay Livingston


Brilliance in science is sometimes a matter of simplifying – paring away complicated scientific techniques and seeing what non-scientists would see if they looked in the right place. That’s what Richard Feynman did when he dropped a rubber ring into a glass of ice water – a flash of brilliance that allowed everyone to understand what caused the space shuttle Challenger disaster.

Andrew Gelman isn’t Richard Feynman, but he did something similar in his blog post about an article that’s been getting much buzz, including at Buzzfeed, since it was posted at SSRN two weeks ago. The article is about Naloxone, the drug administered to people who have overdosed on heroin or opoiods. It keeps them from dying.

The authors of the article, Jennifer Doleac and Anita Mukherjee, argue that while the drug may save lives in the immediate situation, it does not reduce overall drug deaths. Worse, the unintended consequences of the drug outweigh its short-run benefits. Those whose lives are saved go back to using drugs, committing crimes, and winding up in emergency rooms. In addition, a drug that will prevent overdoes death “[makes] riskier opioid use more appealing.” 

The title is “The Moral Hazard of Lifesaving Innovations: Naloxone Access, Opioid Abuse, and Crime.” (A moral hazard is something that encourages people to do bad things by protecting them from negative consequences.)

Naloxone didn’t happen all at once. In 2013 fewer than ten states allowed it; the next year the number had doubled. In 2015, only nine states still did not allow its use. Doleac and Mukherjee used these time differences to look at bad outcomes (theft, death, ER admissions) before and after the introduction Naloxone in the different states.  Here are some of their graphs.

(Click on an image for a larger view.)


They conclude that “broadening Naloxone access led to more opioid-related ER visits.” As for deaths, “in some areas, particularly the Midwest, expanding Naloxone access has increased opioid-related mortality.”

There are reasons to be skeptical of the data, but let’s assume that the numbers – the points in the graph – are accurate. Even so, says Andrew Gelman (here), there’s still the question of how to interpret that array of points. Doleac and Mukherjee add lines and what I assume are confidence bands to clarify the trends. But do these added techniques clarify, or do they create a picture that is different from the underlying reality? Here’s Gelman:

The weird curvy lines are clearly the result of overfitting some sort of non-regularized curves. More to the point, if you take away the lines and the gray bands, I don’t see any patterns at all! Figure 4 just looks like a general positive trend, and figure 8 doesn’t look like anything at all. The discontinuity in the midwest is the big thing—this is the 14% increase mentioned in the abstract to the paper—but, just looking at the dots, I don’t see it.


Are these graphs really an optical illusion, with the lines and shadings getting me to see something that isn’t really there? My powers of visualization are not so acute, so to see what Gelman meant about looking only at the dots, I erased the added lines and bands. Here is what the graphs looked like.


Like Gelman, I can’t see any clear patterns showing the effect of Naloxone. And as I read the reactions to the paper, I sense that its results are ambiguous enough to provide rich material for motivated perception. Conservatives and libertarians often start from the assumption that government attempts to help people only make things worse. The unintended-consequences crowd – Megan McCardle, for example (here) – take the paper at face value. Liberals Richard G. Frank, Keith Humphreys, and Harold A. Pollack (here), who have done their own research on Naloxone – are more skeptical about the accuracy of the data.*

----------------

* This reminded me of a post I did in the first year of this blog.  It was about an editorial in the WSJ that included an utterly dishonest, ideologically motivated connect-the-dots line imposed on an array of points. The post is here.

No comments: