[Epistemic status: I think I probably wrung the right conclusions out of this evidence, but this isn’t the only line of evidence bearing on the broader gun control issue and all I can say is what it’s consistent with. Content warning for discussion of suicide, murder, and race]

I.

From a Vox article on America’s Gun Problem, Explained: “On Wednesday, it happened again: There was a mass shooting — this time, in San Bernardino, California. And once again on Sunday, President Barack Obama called for measures that make it harder for would-be shooters to buy deadly firearms.”

Then it goes on to say that “more guns mean more gun deaths, period. The research on this is overwhelmingly clear. No matter how you look at the data, more guns mean more gun deaths.” It cites the following chart:

…then uses the graph as a lead in to talk about active shooter situations, gun-homicide relationships, and outrage over gun massacres.

Did you notice that the axis of this graph says “gun deaths”, and that this is a totally different thing from gun murders?

(this isn’t an isolated incident: Vox does the same thing here and here)

Gun deaths are a combined measure of gun homicides and gun suicides. Here is a graph of guns vs. gun homicides:

And here is a graph of guns vs. gun suicides:

The relationship between gun ownership and homicide is weak (and appears negative), the relationship between gun ownership and suicide is strong and positive. The entire effect Vox highlights in their graph is due to gun suicides, but they are using it to imply conclusions about gun homicides. This is why you shouldn’t make a category combining two unlike things.

II.

I am not the first person to notice this. The Washington Examiner makes the same criticism of Vox’s statistics that I do. And Robert VerBruggen of National Review does the same analysis decomposing gun deaths into suicides and homicides, and like me finds no correlation with homicides.

German Lopez of Vox responds here. He argues that VerBruggen can’t just do a raw uncontrolled correlation of state gun ownership with state murder rates without adjusting for confounders. This is true, although given that Vox has done this time and time again for months on end and all VerBruggen is doing is correctly pointing out a flaw in their methods, it feels kind of like an isolated demand for rigor.

So let’s look at the more-carefully-controlled studies. Lopez suggests the ones at the Harvard Injury Control Research Center, which has done several statistical analyses of gun violence. They list two such analyses comparing gun ownership versus homicide rates across US states: Miller Azrael & Hemenway (2002), and Miller Azrael & Hemenway (2007).

(does it count as nominative determinism when someone named Azrael goes into homicide research?)

We start with MA&H 2002. This study does indeed conclude that higher gun ownership rates are correlated with higher murder rates after adjusting for confounders. But suspiciously, it in fact finds that higher gun ownership rates are correlated with higher murder rates even before adjusting for confounders, something that we already found wasn’t true! Furthermore, even after adjusting for confounders it finds in several age categories that higher gun ownership rates are correlated with higher non-gun homicide rates (eg the rates at which people are murdered by knives or crowbars or whatever) at p less than 0.001. This is really suspicious! Unless guns are exerting some kind of malign pro-murder influence that makes people commit more knife murders, some sort of confounding influence has remained. Let’s look closer.

The study gets its murder rate numbers from the National Center for Health Statistics, which seems like a trustworthy source. It gets its gun ownership numbers from…oh, that’s interesting, it doesn’t actually have any gun ownership numbers. It says that there is no way to figure out what percent of people in a given state own guns, so as a proxy for gun ownership numbers, it will use a measure called FS/S, ie the number of firearm suicides in a state divided by the total number of suicides.

This makes some intuitive sense. Among people who want to commit suicide, suppose a fixed percent prefer to use guns compared to other methods. In that case, the determining factor for whether or not they use a gun will be whether or not they have a gun. Hospitals diligently record statistics about suicide victims including method of suicide, so if our assumption holds this should be a decent proxy for gun ownership within a state.

There’s only one problem – I checked this against an actual measure of gun ownership per state that came out after this study was published – the CDC asking 200,000 people how many guns they had as part of the Behavioral Risk Factor Surveillance System Survey – and the FS/S measure fails. When I repeat all of their analyses with their own FS/S measure, I get all of their same positive correlations, including the ones with non-gun homicides. When I repeat it with the real gun ownership data, all of these positive correlations disappear. When I look at exactly why this happens, it’s because FS/S is much more biased towards Southern states than actual gun ownership is. Real gun ownership correlates very modestly – 0.25 – with 538’s ranking of the Southern-ness of states. FS/S correlates at a fantastically high 0.62. For some reason, suicidal Southerners are much more likely to kill themselves with guns than suicidal people from the rest of the States, even when you control for whether they have a gun or not. That means that MA&H 2002 thought it was measuring gun ownership, but was actually measuring Southern-ness. This is why they found higher homicide rates, including higher rates of non-gun homicide.

So we move on to MA&H 2007. This study was published after the CDC’s risk survey, so they have access to the same superior gun ownership numbers I used to pick apart their last study. They also have wised up to the fact that Southern-ness is important, and they include a dummy variable for it in their calculations. They also control for non-gun crime rate, Gini coefficient, income, and alcohol use. They do not control for urbanization level or race, but when I re-analyze their data including these factors doesn’t change anything, likely because they are already baked in to the crime rate.

They find that even after controlling for all of this stuff, there is still a significant correlation between gun ownership level and gun homicide rate. Further, this time they are using good statistics, and there is not a significant correlation between gun ownership and non-gun-homicide rate. Further, there is a correlation between gun ownership and total homicide rate, suggesting that the gun-gun-homicide correlation was not just an artifact of people switching from inferior weapons to guns while still committing the same number of murders. Further, this is robust to a lot of different decisions about what to control or not to control, and what to include or not to include.

I repeated all of their analyses using two different sources of gun ownership data, a couple different sources of homicide and crime rate data, and a bunch of different plausible and implausible confounders – thanks a lot to Tumblr user su3su2u1 for walking me through some of the harder analyses. I was able to replicate their results. Pro-gun researcher John Lott had many complaints about this study, including that it was insensitive to including DC and that it was based entirely on the questionable choice of controlling for robbery rate – but I was unable to replicate his concerns and found that the guns-homicide correlation remained even after DC was included and even when I chose a group of confounders not including robbery rate. I was unable to use their methodology to replicate the effect in places where it shouldn’t replicate (I tried to convince it to tell me tractors caused homicide, since I was suspicious that it was just picking up an urban/rural thing, but it very appropriately refused to fall for it). Overall I am about as sure of this study as I have ever been of any social science study, ie somewhat.

This study doesn’t prove causation; while one interpretation is that guns cause homicide, another is that homicide causes guns – for example, by making people feel unsafe so they buy guns to protect themselves. However, I doubt the reverse causation aspect in this case. The study controlled for robbery rate; ie it was looking at whether guns predicted homicides above and beyond those that could be expected given the level of non-homicide crime. My guess is that people feeling unsafe is based more on the general crime rate than on the homicide rate per se, which would make it hard for the homicide rate to cause increased gun ownership independently of the crime rate.

If guns are in fact correlated with more homicide, how come me and VerBruggen found the opposite in our simpler scatterplot analysis? This is complicated, but I think the biggest part of the answer is the urban/rural divide. Rural people have more guns. Murder rates are higher in urban areas. Race also plays a part: whites have more guns, but black areas have higher murder rates. Finally, the North and West seem to have more guns, but murder rates are highest in the South (which is what produced the bogus effect on the last study). All of these differences are large enough to cancel out the gun/no-gun difference and make the raw scatterplot look like nothing. This study didn’t address all those things directly, but its decision to control for non-gun crime rate and poverty took care of them nevertheless. As the old saying goes, guns don’t kill people; guns controlled for robbery rate, alcoholism, income, a dummy variable for Southernness, and a combined measure of social deprivation kill people.

If this is all true, how come I spent so much time yelling at that first study with worse data? Because I worry that if people only see the good studies, they’ll get complacent. Vox posted these two studies as proof that there was a state-level gun-murder correlation. The first one was deeply flawed, but the second one turned out to be okay. Do you think Vox realized this? Do you think they would have written that article any differently in a world where both studies were flawed? As long as you trust every scientific paper you see – let alone every scientific paper you see on your side in a highly politicized field – even when you’re right it will often just be by luck.

III.

Vox also voxsplains to us about America’s unusually high gun homicide rate.

Having presented this graph, they say that “To understand why that is, there’s another important statistic: The US has by far the highest number of privately owned guns in the world.”

Even granting, as we saw above, that gun ownership does indeed increase homicide rates, this is not the most important factor in explaining America’s higher homicide rate, or even close to the most important factor. Let me give a few arguments for why this must be the case:

1. The United States’ homicide rate of 3.8 is clearly higher than that of eg France (1.0), Germany (0.8), Australia (1.1), or Canada (1.4). However, as per the FBI, only 11,208 of our 16,121 murders were committed with firearms, eg 69%. By my calculations, that means our nonfirearm murder rate is 1.2. In other words, our non-firearm homicide rate alone is higher than France, Germany, and Australia’s total homicide rate. Nor does this mean that if we banned all guns we would go down to 1.2 – there is likely a substitution effect where some murderers are intent on murdering and would prefer to use convenient firearms but will switch to other methods if they have to. 1.2 should be considered an absolute lower bound. And it is still higher than the countries we want to compare ourselves to.

2. There are many US states that combine very high firearm ownership with very low murder rates. The highest gun-ownership state in the nation is Wyoming, where 59.7% of households have a gun (really!). But Wyoming has a murder rate of only 1.4 – the same as right across the border in more gun-controlled Canada, and only about a third of that of the nation as a whole. It seems likely that the same factors giving Canada a low murder rate give Wyoming a low murder rate, and that the factors differentiating the rest of America from Wyoming are the same factors that differentiate the rest of America from Canada (and Germany, and France…). But this does not include lower gun ownership.

3. There are many US states that combine very low firearm ownership with very high murder rates. The highest murder rate in the country is that of Washington, DC, which has a murder rate of 21.8, more than twenty times that of most European countries. But DC also has the strictest gun bans and the lowest gun ownership rate in the country, with gun ownership numbers less than in many European states! It seems likely that the factors making DC so deadly are part of the story of why America as a whole is so deadly, but these cannot include high gun ownership.

If not gun ownership, what is the factor making America so much more deadly than Europe and other First World countries? The traditional answer I always heard to this question was that America had a “culture of violence”. I always hated this answer, because it seemed so vague and meaningless as to be untestable by design. If the NRA waves their hands and says “eh, culture of violence”, how are you going to tell them they’re wrong?

But we can work with this if we assume the culture of violence (or, if you want to be official about it, “honor culture”) is more common in some populations and areas than others. Some of the groups most frequently talked about during these lines are Southerners and various nonwhite minorities. This provides a testable theory: if we compare American non-Southern whites to European countries mostly made up of non-Southern whites, we’ll find similar murder rates. But first, some scatter plots:

This is murder rate by state, correlated with perceived Southernness of that state as per 538’s poll. I’ve removed DC as an outlier on all of the following.

Who'd have thought populating half a country with the descendants of a group of people called "Border Reavers" would cause so much trouble?

— Scott Alexander (@slatestarcodex) December 3, 2015

And this is murder rate by state correlated with percent black population:

This would seem to support the “culture of violence” theory.

Can we adjust for this and see what the murder rate is for non-Southern whites? Sort of. The Economist gives a white-only murder rate of 2.5 (this is based on white victims, whereas we probably want white perpetrators, but the vast majority of murders are within-race so it doesn’t make much difference). And Audacious Epigone has put together a collection of white murder rates by state. I can’t find anything on non-Southern white murder rates per se, but one hack would be to take the white murder rate in non-Southern states and assume there aren’t any Southerners there.

Our main confounder will be urbanization. Western Europe is about 80% urban, so let’s look at states at a similar level. The four northern states that are closest to 80% urban are Colorado, Oregon, Washington, and Connecticut. I’m throwing out Colorado because it has a large Latino population who can’t be statistically differentiated from whites. That leaves, Washington (2.4), Connecticut (2.0), and Oregon (2.0). So possibly adjusting out Southerners brings us down from 2.5 (all whites) to 2.1 or so (non-Southern whites)? Again, compare to Germany at 0.8, Canada at 1.4, and America at 3.8.

There’s one more factor that needs to be considered:

This is a plot of the gun death rate vs. the robbery rate. There’s a strong correlation (r = 0.78). Robbery is heavily correlated with percent black, percent Southern, and urbanization, so it’s probably coming from the same place. Nevertheless, it seems to correlate with murder better than any of them alone, maybe because it’s combining all three measures together. I was able to make a linear model using those three measures that correlated at r = 0.79 with murder, about the same amount that robbery does. I should also mention that robbery correlates negatively with gun ownership at r = – 0.52, but this disappeared when controlled for urbanization.

So my very tentative conclusion is that although the US murder rate is much higher than that of other First World countries, this is partly due to the existence of various cultural factors not present in those other nations. When we adjust those away, America’s murder rate falls from 3.8 to 2.1. Which is still higher than Germany’s 0.8 or Canada’s 1.4.

Is that extra due to guns?

IV.

According to MA&H 2007, each absolute percentage point in gun ownership was related to a 2.2 relative percentage point difference in homicide. This part of the study was beyond my ability to check, and I’m not sure why they switched from absolute to relative percents there, but suppose we take it seriously.

America has a gun ownership rate of 32%, so if we somehow decreased that to zero, we would naively expect about a 70% decrease in homicides. Unfortunately, only 67% of American homicides involve guns, so we’re back to pretending that eliminating guns will not only have zero substitution effect but also magically prevent non-gun homicides. This shows the dangers of extrapolating a figure determined by small local differences all the way to the edge of the graph (I’M TALKING TO YOU, RAY KURZWEIL).

Maybe we can be more modest? Canada has a gun ownership rate of aboot 26%, so…

…wait a second. I thought we’ve been told that the US has a gun ownership rate seven zillion times that of any other country in the world, and that is why we are so completely unique in our level of gun crime? And now they’re telling us that Canada has 26% compared to our 32%? What?

Don’t trust me too much here, because I’ve never seen anyone else analyze this and it seems like the sort of thing there should be loads of analyses of if it’s true, but I think the difference is between percent of households with guns vs. guns per capita. US and Canada don’t differ very much in percent of households with guns, but America has about four times as many guns per capita. Why? I have no idea, but the obvious implication is that Canadians mostly stop at one gun, whereas Americans with guns buy lots and lots of them. In retrospect this makes sense; I am looking at gun enthusiast bulletin boards, and they’re advising other gun enthusiasts that six guns is really the bare minimum it’s possible to get by with (see also “How many guns can you have before it’s okay to call your collection an ‘arsenal’?”, which I have to admit is not a question that I as a boring coastal liberal have ever considered). So if the guy asking that question decides he needs 100 guns before he gets his arsenal merit badge, that’s a lot more guns per capita without increasing percent household gun ownership. This should actually be another argument that guns are not a major factor in differentiating US vs. Canadian murder rates, since unless you’re going on a mass shooting (WHICH IS REALLY RARE) you wouldn’t expect more murders from any gun in a household beyond the first. That means that the small difference between US and Canadian household percent gun ownership rates (32% vs. 26%) would have to drive the large difference between US and Canadian murder rates (1.4 vs. 3.8), which just isn’t believable.

…okay, sorry, where were we? Canada has a gun ownership rate of about 26%, so if America were to get its gun ownership as low as Canada, that would be -6 absolute percentage points = a 13% relative decrease in murder rate = the murder rate going from 3.8 to 3.3 = a 0.5 point decrease in the murder rate. That’s pretty close to the difference between our 2.1 US-sans-culture-of-violence estimate and the 1.4 Canadian rate – so maybe beyond the cultures of violence, the rest of the US/Canada difference really is due to guns?

(I’m not sure whether I should be subtracting 13% from 2.1 rather than 3.8 here)

In Germany, 9% of households own firearms (wait, really? European gun control is less strict than I thought!) Using MA&H’s equation, we predict that if the US had the same gun ownership rate as Germany, its murder rate would drop 50%, eg from 3.8 to 1.9. Adjust out the culture of violence, and we’re actually pretty close to real Germany’s murder rate of 0.8.

How much would gun control actually cut US gun ownership? That obviously depends on the gun control, but a lot of people talk about Australia’s gun buyback program as a model to be emulated. These people say it decreased gun ownership from 7% of people to 5% of people (why is this number so much lower than Canada and Germany? I think because it’s people rather than households – if a gun owner is married to a non-gun-owner, they count as one gun-owner and one non-owner, as opposed to a single gun-owning household. The Australian household number seems to be 19% or so). So the gun buyback program in Australia decreased gun ownership by (relative) 30% or so. If a similar program decreased gun ownership in America by (relative) 30%, it would decrease it by (absolute) 10% and decrease the homicide rate by (absolute) 22%. Since there are about 13000 homicides in the US per year, that would save about 3000 lives – or avert about one 9/11 worth of deaths per year.

(note that our murder rate would still be 3.0, compared to Germany’s 0.8 and Canada’s 1.4. Seriously, I’m telling you, the murder rate difference is not primarily driven by guns!)

Is that worth it? That obviously depends on how much you like being able to have guns. But let me try to put this number into perspective in a couple of different ways:

Last time anyone checked, which was 1995, about 618,000 people died young (ie before age 65) in the US per year. Suppose that the vast majority of homicides are of people below 65. That means that instituting gun control would decrease the number of premature deaths to about 615,000 – in other words, by about half a percentage point. I’m having to borrow this data from the UK, but if it carries over, the average person my age (early 30s) has a 1/1850 chance of death each year. Gun control would decrease that to about 1/1860. I’m very very unsure about the exact numbers, but it seems like the magnitude is very low.

On the other hand, lives are very valuable. In fact, the statistical value of a human life in the First World – ie the value that groups use to decide whether various life-saving interventions are worth it or not – is $7.4 million. That means that gun control would “save” $22 billion dollars a year. Americans buy about 20 million guns per year (really)! If we were to tax guns to cover the “externality” of gun homicides preventable by Australia-level gun control, we would have to slap a $1000 tax on each gun sold. While I have no doubt that some people, probably including our arsenal collector above, would be willing to pay that, my guess is that most people would not. This suggests that most people probably do not enjoy guns enough to justify keeping them around despite their costs.

Or if all gun enthusiasts wanted to band together for some grand Coasian bargain to buy off the potential victims of gun violence, each would have to contribute $220/year to the group effort – not totally impossible, but also not something I can really see happening.

This is very, very, very, very very tentative, but based on this line of reasoning alone, without looking into the experimental studies or anything else, it appears that Australia-style gun control would probably be worth it, if it were possible.

(I didn’t price in the advantages of guns in terms of preventing state tyranny and protecing freedom, which might be worth subsidizing, but my guess is that if 32% gun ownership is enough to maintain freedom, 22% gun ownership is as well)

V.

In summary, with my personal confidence levels:

1. Scatterplots showing raw correlations between gun ownership and “gun deaths” are entirely driven by suicide, and therefore dishonest to use to prove that guns cause murder (~100% confidence)

2. But if you adjust for all relevant confounders, there is a positive correlation between gun ownership and homicide rates (~90% confidence). This relationship is likely causal (~66% confidence).

3. The majority of the difference between America’s murder rate and that of other First World countries is not because of easier access to guns in America (~90% confidence).

4. But some of it is due to easier access to guns. This is probably about 0.5 murders/100K/year.

5. An Australian-style gun control program that worked and had no side effects would probably prevent about 2,000 murders in the US. It would also prevent a much larger number of suicides. I am otherwise ignoring suicides in this piece because discussing them would make me too angry.

6. Probably the amount of lost gun-related enjoyment an Australian-style gun control program would cause do not outweigh the benefits.

7. This is not really enough analysis to make me have a strong opinion about gun control, since this just looks at the correlational evidence and doesn’t really investigate the experimental evidence. Contrary to what everyone always tells you, experimental evidence doesn’t always trump correlational – there are cases where each has its strengths – but it wouldn’t be responsible to have a real opinion on this until I look into that too. Nevertheless, these data are at least highly consistent with Australia-style gun control being a good idea for the US.

If you want to look into this more, here is a CSV version of all the relevant data.