This article is an expanded excerpt from my book, Make Rules Without Making Enemies. The chapter from which it is excerpted deals with counting, trending, and analyzing terrible events. Terrible events are, by definition, rare, because, if they were common, we would not survive to count, trend, and analyze them. The example given here is an OSHA recordable incident, but you can substitute any event that threatens the survival of a company, a person, or even a civilization. These won’t happen often, but they will receive a lot of attention and resources when they do, so how to distribute those resources will always be an important question.

You may also recognize the die-rolling mechanic from my article, Why is Prediction so Hard?. Including the book and a speech given in 2018, this is my third telling of this story, each from a different angle. It’s that important.

You are in Charge

Let’s look at five facilities in an imaginary company you have been assigned to oversee. We’ll name these facilities Kish, Eridu, Mari, Uruk, and The Pit of Death. The Pit of Death is TWICE as dangerous as any of the other four, but you don’t know that. Let’s count your recordable incidents and try to figure it out.

What if we don’t? The naive strategy

I’m going to go over three strategies for inferring an answer from limited data. I’ve named these strategies “naive”, “smart”, and “hurts”. The naive strategy is to ignore the data and partition all support evenly. For comparison, we’ll look at that one first.

No surprises there. With five facilities, everyone gets 20% of the support. Let’s move on to same data science!

The smart strategy: target your resources

If only one facility is your most dangerous, why are you “wasting” your limited support on the other four? Certainly, a targeted approach is better. For that, you’ll need data. Let’s simulate some data for your simulated company.

We’ll simulate incident distribution with a six-sided die representing your five facilities. The names of Kish, Eridu, Mari, and Uruk, your four safer facilities, are each painted on one side of the die. Pit of Death is painted on both of the remaining sides. This means that, on average, every third roll will land on Pit of Death. This is an absurd tilt. Pit of Death has worse odds than any game in Vegas, Monte Carlo, Macau, or Canto Bight. Pit of Death is the reason you pay the license for Power BI®. Pit of Death is the subject of your next quarterly review. Pit of Death is a sure bet to lose. So let’s roll …

There’s some math here, but I’m going to do it for you. We’ll throw three such dice, each signifiying a recordable incident. That’s three recordable incidents across five facilities. Depending on your hours worked at each facility, you may be in serious trouble. Your imaginary company may already be out of business, but you have data!

There are 216 ways three dice can land (many equivalent). If you saw every roll, you would see 516 candidates for “facility you should target.” Sometimes three candidates in one roll (all dice land on different sides), sometimes two candidates (two dice the same, one different), and occasionally only one (all dice the same). By “the same,” I mean the same facility, even if that facility is painted on two sides. There are a lot of things you could see, but I’ll print this next part in bold text, because it’s easy to forget:

We don’t get to see all the rolls. Just one.

The challenge is to look at one roll and make a data-driven decision. Let’s play analyst for a moment and define what Hold ‘Em players would call the “Nash Equilibrium” for this challenge.

In Texas Hold ‘Em, “Nash Equilibrium” refers to a state in which no player can improve their outcome by changing their strategy, given the strategies employed by the other players—In other words, perfect play. In our case, the only other players are risk and luck, so our perfect play is simple.

• 3: If any side shows three times, wager that is your dangerous facility.
• 2|1: If any side shows twice, wager the side showing twice is your dangerous facility.
• 1|1|1: If all dice land on different sides, assume your dangerous facility is one of them and pick randomly.

Following this …

In the universe of all possible rolls, you’ve nearly doubled support to your target facility, and at a comparatively small cost to your other facilities.

Yay, data science?

The hurts strategy

You knew there was a catch, didn’t you?

With the “naive” strategy, your target facility was guaranteed 20% of your support. With the “smart” strategy, your target facility is more than likely going to get NOTHING. The “smart” strategy has an almost 2:1 chance to miss entirely, meaning you will have done NOTHING about your most dangerous facility.

We don’t get to see all the rolls. Just one.

The way to fix this is straightforward, but not easy. Fortunately, we’ve been doing a bit of it already. If you were reading very closely, you might have picked up on the fact that 33% of die rolls land on Pit of Death. Given your goal of identifying your unsafe facility, that gives you a win rate of 33%. But the “smart” strategy won 37% of the time. How is this possible?

We beat 33% by ignoring a piece of data. When two of the three dice landed on one facility, we ignored the third die. That dramatically raised the win rate of 2|1 (one double, one single) rolls. Here are the numbers for our “Nash Equilibrium” strategy:

These are the win rates of our three roll types. Shaded areas are a hit; unshaded areas are a miss.

• 1|1|1 means three different facilities.
• 2|1 means two dice on one facility, one die on another.
• 3 means all three dice on the same facility.

You will also notice that the bars are different widths. Each bar width is scaled to the chance of seeing that roll type on the one roll you get to see. This is unfortunate, because only the thin 3 bar has a good chance to lead us to the target facility.

Our “ignore the single on 2|1 rolls” strategy improved our win rate on 2|1 rolls, but 2|1 rolls are still a loser. On a 2|1 roll, you have a 56% chance to miss your target. 1|1|1 rolls are worse, with a 75% chance to miss. Only 3 rolls are likely to win, and even then only 67% of the time.

This would be a different kind of game if we were only giving out support, but we’re giving out intervention and correction too. Missing your target facility 2 out of 3 times doesn’t just mean the “wrong” facilities get support most of the time, it means the wrong facilities get disrupted and penalized most of the time. Unfortunately, some of the best people I know have been devoured by this phenomenon. Heroes one quarter, villains the next.

The “hurts” strategy hurts, because it requires us to be more data-driven in a manner that appears less data-driven. The “hurts” strategy is to take a target action—maybe—if you see a 3 roll (1:17 chance in this model) and to act naive in other cases. Worse, as a rulemaker, you’ll have to encourage others to act naive.

I said in an earlier chapter that, as rulemakers, if we want to save lives, we have to regulate in a way that is consistent with the continued employment of the regulated, because we don’t get to make rules for the unemployed. The same goes for our own employment. Be gentle when pushing back against the data absolutism our industry helped create.

But what if you waited for JUST ONE MORE data point?

With more dice (more recordable incidents), you will have a bit more data and a bit better chance to identify your most dangerous facility.

Even with a tilt like this one (one facility twice as dangerous as any other), you’d have to wait for (do nothing until) nine incidents across five facilities to have a better than even chance to correctly identify your most dangerous facility.

If you waited for twenty-nine incidents, you’d reach a 75% chance, meaning you’d only have a 1:3 chance of supporting the wrong facility or firing the wrong facility manager.

Waiting works on paper, but your facilities aren’t waiting with you. Every new employee, new process, or new tool is another side on your die, and the weak signal from your rare-event data disappears into more and more noise. Besides, do you want to be the one to tell the CEO to “wait” after the twenty-eighth incident?

If, when reading this article, you recoiled at the idea of “supporting the wrong facility” or “wasting your support on good facilities,” believing all support it good, no matter where it goes … then you’re really on to something. Supporting the organization as a whole is always a sound investment, and isn’t just for people who are bad at math.