Bias in Studies of the Efficacy of Firearms and Bear Spray

Attempting to quantify the effectiveness of defensive methods for use against bears suffers from significant selection bias. It is impractical to enlist large numbers of volunteers, randomly select half, give half a method of protection, leave the others as a control group and place both in equal chances of being attacked. Instead we rely on gathering data from people who were attacked, who had various types of defensive weapons. Obtaining a valid sample is nearly impossible, because successful defenses have a strong bias of not being reported. The more spectacularly a defense fails, the more likely it is to come to the attention of the media and to make headlines. It will be collected when people search for “bear attacks”. Researchers collecting data may exhibit bias in their selection of what incidents to include.

In the most famous of the firearms and bear studies, “Efficacy of Firearms for Bear Deterrence in Alaska” (1883 to 2009) the purpose was not to determine how effective firearms are in defense against bears. It was to find ways in which firearms did not work in defense against bears. From an interview of Tom Smith by Wes Siler in Outside online:

The point of “Efficacy of Firearms” wasn’t to arrive at a conclusion on whether or not firearms work but, rather, to analyze the reasons why they didn’t—“poor aim, no time to use them, jammed, etc.,” elaborates Smith.

When you are looking for ways in which firearms do not work, you will tend to find more of them. In “Efficacy of Firearms” Tom Smith and Stephen Herrero openly admit their selection bias. From Efficacy of Firearms” p. 5:

Therefore, even if more incidents had been made available through the Alaska DLP database, we anticipate that these would have contributed few, if any, additional human injuries. Second, including more DLP records would have increased the number of bears killed by firearms. Finally, additional records would have likely improved firearm success rates from those reported here, but to what extent is unknown.

From the above, it is clear “Efficacy of Firearms” selected incidents to include as many incidents with human injury as possible. The authors considered the inclusion of incidents which would have improved firearm success rates, to be unimportant. From wikipedia:

Selection bias is the bias introduced by the selection of individuals, groups, or data for analysis in such a way that proper randomization is not achieved, thereby failing to ensure that the sample obtained is representative of the population intended to be analyzed.^[1] It is sometimes referred to as the selection effect. The phrase “selection bias” most often refers to the distortion of a statistical analysis, resulting from the method of collecting samples.

Such bias is not always conscious. Tom Smith loves bears. From Tom Smith:

You can take an entire wilderness with no bears and there’s just something fundamentally different about it. We love these animals; they’re wonderful.

Tom Smith is an ardent bear spray proponent. Published by the official Interagency Grizzly Bear Committee:

 “Smith is an ardent advocate of using bear spray over firearms.”

One of the main purposes of bear spray is to prevent people from killing bears. Emotional attachments tend to bias a persons judgement in ways which are difficult to guard against.

Researchers also need to guard against Confirmation bias. Here is the definition:

confirmation bias, people’s tendency to process information by looking for, or interpreting, information that is consistent with their existing beliefs.

In our ongoing study of the efficacy of handguns for defense against bears, I and colleagues did not start with an assumption of how effective handguns were. We were looking for cases where handguns failed. If handguns were ineffective, there should be many documented cases of handgun failures when fired in defense against bears. This sounds similar to what Tom Smith wrote. The difference is, guard against selection bias, it was decided to include all cases in which a handgun was fired in defense against a bear.  This eliminated subjective bias as to whether a handgun in a backpack or cabin, which was never fired, constituted a handgun failure.

Failure to draw and use a handgun or to draw and use bear spray is a user failure, not a weapons failure. It can be addressed with training, practice, and technique. We used a criteria similar to Smith & Herrero in “Efficacy of Bear Deterrent Spray in Alaska“. In their bear spray papers they only included cases where bear spray was sprayed.  They excluded cases where it was not sprayed. One critique of the Smith & Herrero firearms study was it included cases where the firearms were not fired. Of the 71 incidents in the bear spray study, only 9 involved charging grizzly bears. Bear spray worked to stop three of those charges.

Smith & Herrero limited their efficacy studies to Alaska. Choosing to limit the data pool is another way selection bias may come into play.

The failure of a weapon system, whether bear spray or firearms, results in much more news coverage than a successful defensive use. Studies which rely on public information sources are necessarily biased toward a lower  success rate.  When researchers deliberately work to find failures over successes, selection bias is unavoidable.

An important way to check for selection bias is to look at the raw data. If raw data is not available to the public, the research cannot be checked.  What one person might have concluded was a success, another might conclude was a failure.

In our studies of the efficacy of handguns used as a defense against bears, all the incidents and sources are available to the public. This is not the case in the Smith & Herrero studies. It would be illuminating to compare incidents in our different approaches, to see if they were graded differently. Someday, the raw data for the Smith & Herrero studies may be released.

An important part of research is the ability to falsify a hypothesis. The hypothesis that handguns are effective as a defense against bears is easily falsifiable. The way to do this is to document cases where a handgun was fired in defense against a bear, and the bear attack was not stopped, because the bear was not killed or did not leave the scene in response to the gunfire. In the last seven years we have documented three failures out of 143 cases where handguns were fired in defense against bears. It is a success rate of 98%.  In the soon to be published update, a fourth failure has been found. The success rate continues at 98%.

In 1930, it was plausible a person could use a pistol against a bear, be killed, and never be found. By 1960, it became highly unlikely. The wide availability of aircraft, motorized transport, expansion of radio, television and print media had changed the landscape.  76% of documented cases of handguns fired in defense against bears have happened since 2000. In our soon to be published update, 96% occurred after 1960. People killed or mauled by bears are news. They make headlines around the world. Successful defenses against bears are not news. They tend to not be reported, be relegated to official reports, brief mentions in a local media outlet, a diary, or briefly mentioned online.  Of documented cases, only one failure involved a fatality. One person was killed when a .22 handgun was fired in defense against a polar bear, In 1995.

Nine people have been killed where bear spray has been sprayed in defense against a bear, in eight incidents from 2003 to 2023. Because of the geographic limitation and timing of the Smith and Herrero bear spray studies, none of those eight incidents are included in their statistics. The original “Efficacy of Bear Deterrence Spray” study involved data collected from 1985 to 2006. In the followup study for both bear spray and firearms “Human-bear conflict in Alaska: 1880-2015: Alaska Human-Bear Conflict“, published in 2018, four additional bear spray incidents were added from 2006 to 2015, increasing the original 71 incidents to 75. Of the 75 incidents, at least 21 of them were by bear management personnel to harass bears. 70 of the incidents were deemed successful, or 93%.  In the 75 incidents, 4 people were injured. The number of firearms incidents increased from 269 to 328, an addition of 57. The number of handgun incidents increased by 6, from 37 to 43. The total number of incidents is given as 682. The number of incidents where firearms were not present or bear spray was not sprayed appears to be 279. The criteria used does not appear to have changed from previous studies. Tom Smith confirmed the data from previous studies was included in the numbers for the latest paper.

Readers are urged to send documented cases of handgun failures to AmmoLand. Any documented case where a handgun is fired in defense against a bear is welcome as an addition to our growing database.  So far, successful defensive uses of handguns outnumber failures by roughly 50 to 1.

©2024 by Dean Weingarten: Permission to share is granted when this notice and link are included.

Gun Watch