8 Simple Rules for Avoiding Being Hoodwinked by Specious “Analytics”

• Tweet
• Pin It

If you are like me, a social media-using Kings fan, you are likely inundated with a steady diet of “analytics” graphics and numbers to support this or that cause/conclusion. Much of what is shared is silly. This article will represent a simple handbook for understanding why and identifying future silliness.

Note: this is meant with no ill-will towards anyone in particular. If you like sharing the latest xG% numbers just to troll, good for you. If the various player cards are just that sweet-sweet hit of content to satisfy insatiable cravings, keep at it. If you find objections to be obnoxious and feel that standards of evidence on the site formerly known as twitterdotcom need not exceed that of sports talk radio, who am I to tell you what to enjoy?

But for everyone else…those who actually want to understand what is happening in the NHL…here are eight rules (well…not really rules…more like heuristics, truisms…aphorisms?!) to use whenever digesting or discussing hockey analytics (here used interchangeably with on-ice xG stats, not to be confused with other interesting stuff like optimal timing of goalie pulls and statistical analysis of whether full prostration is the most effective 2on1 defensive strategy).

Rule 1: “Small sample size” is not just a throat-clearing

Everyone knows small sample size is a problem. That’s why responsible analysts preface early season or heavily filtered analysis with a disclaimer of “small sample size, but…”

This isn’t good enough. Small sample size makes most arguments useless. There are two reasons for this: the reason most people understand and the more fundamental reason.

The first reason is noise. People get this. If a measure has a lot of variability, then a random sample might not reflect the general population.

Imagine two bottomless urns filled with xG% marbles. One urn is named Cody Ceci and one is named Cale Makar. Each time you draw a marble, you see a single game xG%. How many marbles do you have to draw to know that the *true* average xG% of Cody Ceci’s unchanging and infinite urn is less than that of Cale Makar?

There are standard statistical approaches to problems like this. The problem is posed as a question of statistical power (i.e. given observed variance, how many observations would you need to reach a conclusion with a certain level of confidence). Most people learn the question the opposite way: how likely is it that we would see this set of observations by chance?

Well…Ceci and Makar have each played 11 games this season. If we treat each game as a separate observation, then their average per-game xG% is 39% and 62%, respectively. We can run a simple test using each player’s variance and calculate that we would expect to observe a gap at least this big 0.64% of the time if Ceci’s *true* xG% is not less than Makar’s…sheesh, that’s a rough start; Ceci ruined my argument! Get better, Ceci!

I could quibble with this sterile hypothesis-testing approach (e.g. players are not unchanging!, these numbers are conditional upon all sorts of things), but this would get too boring. Let’s just focus on real examples to show how much things change.

Cale Makar has a reputation for being really-really good. Jack Johnson (fairly or not) has a reputation for being not so good. In 2021-22, they played on the same Cup-winning Avalanche squad. Makar won the Norris Trophy and Johnson had a third pair role through the playoffs. Here’s how MoneyPuck charts each defenseman’s 5-on-5 xG% numbers throughout that season.

Look how noisy these numbers are! They shift wildly from game to game! Johnson, clearly the inferior defender, still has the superior numbers in 35% of games!

That’s not the only example. Here’s Drew Doughty and Andreus Englund…would anyone looking at the game-by-game numbers have the slightest idea who had the reputation as the superior player?

What about early season commentary? Is that reliable?

Here’s a scatterplot showing cumulative xG% (per NaturalStatTrick) for Kings defensemen over the last three full seasons. X-axis shows the numbers for the each season’s first 10 games; Y-axis shows the full season. There is no relationship!

Let’s sidetrack to baseball. I heard a discussion between two of my friends near the very start of this season that went something like this:

Deranged Lunatic: I’m telling you…Teoscar Hernandez…this guy is something! I think he’s going to win MVP!

Voice of Reason: Really? I like the optimism. Why do you think that?

DL: First two games: 3-7, 1HR, 4RBIs, 2BBs…if he keeps it up all year, it’s a lock.

VoR: *sigh* It’s two games. One hit in either direction would totally change how you feel. Even bad players have two game stretches like this.

DL: Are you saying he is a bad player?!

VoR: Hardly. We just can’t use early season numbers to justify our opinions. Do you know the difference between an all-star level .300 hitter and a replacement-level .200 hitter?

DL: What?

VoR: About two hits a week. That’s it. The cumulative effect of lining up a bunch of .300 vs. .200 hitters is huge, but you can’t know what you have until the numbers accumulate.

Most intelligent people understand Voice of Reason is correct here, especially in baseball. Two games in an MLB season is not enough data to separate signal from noise…the noise here being that bad players have good games all the time and good players have bad games all the time.

Sadly, humbled by having his reasoning so easily picked-apart, Deranged Lunatic went to the internet to try to learn more about how to think about performance in sports. He got lost and ended up mainly in forums dedicated to hockey. Here’s the conversation that ensued a few weeks later.

Voice of Reason: Why so glum?

Deranged Lunatic: It’s Teo…he really had a nightmarish game last night. I don’t know how he can expect to stay on the field with results like this.

VoR: No doubt it was a rough game for the team last night, losing 9-2, but I hardly think Teo was a noteworthy part of it. 1-4 with 1 walk is not too bad.

DL: So you’re saying he had no noteworthy offensive contributions AND he was putrid defensively?

VoR: “Putrid”? Did I miss something? He had no errors. Do you recall some play he could have made in right field.

DL: Poor Voice…alas this time it is I with the upper hand in our great intellectual joust! You have not been schooled in the techniques of modern analytical methods like I have!

VoR: Go on…

DL: The man was on the field for zero runs for and nine against, for an on-field run share percentage of 0%! That’s awful!

VoR: …

DL: Don’t even throw something at me like “runs are low frequency events!” On an Expected Run basis he did just as bad.

VoR: *stunned silence*…so your view is…fielders are responsible for runs even if they had nothing to do with the play and you can’t pinpoint exactly what they are doing wrong?

DL: Get out of the Stone Age! We now have data and are able to surpass the biases that once bound us to poor analysis.

VoR: *walks away shaking head*

When I say there is a more fundamental error in using small sample on-ice numbers, Deranged Lunatic is demonstrating it here.

We don’t use on-field team performance metrics in baseball because baseball is a low event game. A watcher can pinpoint everything that happens and why it happens. Hockey is a high-event/low-scoring game. A thousand different things are happening with and without the puck over the course of the game and we can’t even begin to measure it all. Clears/box-outs/shots/regroups/cycles/screens/puck battles/zone coverage/transition coverage/face punching/checking…which of these matter and how much? The initial promise of the on/off ice stuff (Corsi%, Fenwick%, xG%) was that we can get an intermediate measure between the high-event/hard to measure stuff and the low event thing we care about: goals.

The problem is over a small sample of games, these numbers aren’t even a measure of performance. A single player can literally do everything right, all things equal, and still have negative on-ice xG%. You just don’t see the counterfactual: an even more negative xG%.

Rule 2: The coach is not rolling dice

Suppose you want to know the effect of X on Y. The first thing you do is to get some data and look to see how Y changes when X changes.

That’s not everything though. Ice cream sales don’t cause shark attacks, Tylenol doesn’t cause fevers, and your gas pedal does cause your car to accelerate. Sometimes X and Y both move because of some third factor Z, sometimes Y causes X, and sometimes X moves only to stabilize Y. To really understand the effect of X on Y, you have to understand why X changed in the first place.

Economists call this the search for “exogeneity”. Exogenous changes are ones where the source of change in X was outside the system. Social science papers are filled with things like random weather events, changes based on lottery outcome, and strange series discontinuities for this reason.

The below chart is plausible evidence of exogenous impact. There is a clear break in Matt Roy’s cumulative xG% in the 2022-23 season right when his partner switches from Durzi to Gavrikov. It’s a strong visual along with a narrative that clearly outlines what was happening at the time. This would be evidence that Gavrikov is really good.

On the other hand, I often see statistics like the below displayed online. This shows (NST) on-ice xG% for Joel Edmundson and Brandt Clarke in 2024-25 when they played together versus when they played apart.

The conclusions are clear as day for some: Edmundson is bringing down Clarke and Clarke is raising Edmundson, just the opposite of the protective relationship the coaches and media evoke!

This is where Rule 2 comes into play. What is the source of variation in pairs? Weren’t they attached at the hip for most of the year? Did the coach roll dice to make sure we have clean measurements of defensive pair movements?

This movement in xG% without the standard partner has to happen with some other partner (maybe?…more on that later). Where does Clarke find his improvement and where does Edmundson find his decline? Below are the distinct (meaning none of the listed combinations will overlap with each other) combinations for Clarke and Edmundson:

Strange. Most of the improvement for Clarke doesn’t come from the highly regarded Anderson and Gavrikov, but from this big “Others” category (i.e. not Gavrikov, Anderson, or Moverare) and it is most pronounced when the team is trailing. In fact, Edmundson is associated with some of the best outcomes for Clarke when the team is leading. For Edmundson, much of the downward movement is happening in “Others” as well, while the standard righties of Spence and Doughty are associated with better outcomes than Clarke when the score is tied.

What is going on here? This is the soup of…stuff. Those posters, rather than capturing a clear effect of what happens to player performance when a partner changes, are capturing a great soup of causes and effects going in the background of a game. Is it neutral gameplay mid-line-change? Is the first shift following the power play equally advantageous as the second (more on this in Rule 3)? Do teams hold back when protecting a lead?

Another example to elaborate the point. Most of us thought of Anderson-Doughty as an inseparable defensive pair before Doughty’s 2024 ankle injury. If we apply the same exercise on the data from 2021-22 to 2023-24, it looks like Anderson is dragging Doughty down, with Anderson’s non-Doughty minutes declining to 47% xG%. And yet, we can remember times during that period when Anderson got hurt and it was difficult for the team to overcome the injury.

Here’s a table to illustrate. Anderson’s negative effect only appears when both players play. When Anderson plays without Doughty because Doughty is injured, he has as positive of results as Doughty does when he plays without Anderson because Anderson is injured. The coach didn’t roll dice for us; don’t measure the soup.

Rule 3: Little things mean a lot

Like most hockey fans, I used to pay attention to the official +/- as a stat and think it must have some value. Then in January 2022-23 I noticed something strange. Matt Roy and Sean Durzi were defensive partners for almost the whole season and their +/- ratings were wildly different. How could that be? Shouldn’t they be on the ice for almost identical outcomes?

The answer is basically shorthanded goals and deployment. Durzi played powerplay minutes; shorthanded goals-against hurt him. Roy played more penalty kill minutes; shorthanded goals-for helped him. Through the end of January, the asymmetrical bookkeeping effect of how power play versus shorthanded goals are counted resulted in +6 swing to Roy relative to Durzi, which was large given their numbers at the time. Add in a three-goal difference between the two in 3on3 overtime (-2 vs +1), another extra goal against Durzi with the goalie pulled, and a little bit of noise, and you end up with a -14 for Durzi and -3 for Roy.

I bring this up as an introduction to Rule 3: don’t be surprised when some small issue ends up totally swinging the results of some statistic.

Let’s go back to the Edmundson/Clarke example. Who were the “Others” that drove up Clarke’s non-Edmundson 5on5 minutes?

These defensemen fall into different categories: fringe NHLers like Jones and Englund and the Kings’ more skilled offensive defensemen like Doughty and Spence.

Clarke-Jones was a full-time pair for two games early in the season when Edmundson was out for personal reasons (fatherhood). In the first game, the Maple Leafs jumped out to a 6-0 lead and Clarke-Jones put up a 75% xG% over the course of the 6-2 loss. Then the pair had a 65% xG% share against the Canadiens in a 4-1 win. Meanwhile, Clarke-Englund’s strong numbers stem entirely from a January 10^th game against the Jets where the pair posted a gaudy 0.7/0.22 xGF/xGA in under six minutes of ice time, though this xGF% was not wildly out of line with the team’s overall numbers in that game.

I go into this detail just to dismiss it with reference to Rule 1, while also noting these minutes are likely highly protected. Now let’s focus on Spence and Doughty.

One could just cite Rule 1 and move on, but there is something deeper here. There is definitely a score effect, as Spence and Doughty’s xG% with Clarke is an astronomical 81%/89% when trailing across the last two seasons. There is definitely something being captured in terms of how line changes are made, as Spence’s numbers are consistently strong even through the pair’s 75 appearances, even though the majority of their games were under 30 seconds of ice-time together and these <=30 second game results accumulated an 80% xG%.

There’s something else though. Across the 2024-25 and 2025-26 seasons (though 10/29/2025), Doughty and Clarke have played around 21 minutes together at 5on5, with a dominant xGF% of 78.5%. During the same period, the pair played 13 minutes together on the power play…a very high ratio of power play to 5-on-5 time.

Here’s the question: does a power-play really *end* when the clock strikes zero? How many goals can you recall where they weren’t “power play” goals because the penalty ended shortly beforehand. How long does it take, on average, for gameplay to return to a neutral state?

The Doughty-Clarke combination makes for an easy opportunity to spot this effect because it is a rare two-defenseman power play combination, but what if this is a systematic bias in the data? I don’t have the underlying shift data to analyze specifically what is happening in those moments, but the below scatter of NHL defensemen from the 2024-25 season is suggestive. The x-axis is what I’ll call Game State weighting. It’s a weighted average of game state divided between 4-5 PK (represented as -1), 5-5 (represented as 0), and 5-4 (represented as 1). The y-axis is each player’s relative 5-on-5 Corsi % (i.e. player Corsi% minus team Corsi%; I’m using Corsi because xG is more noisy and its calculations change depending on game state).

Min 700 minutes.

One could imagine lots of other factors that might contribute to the above correlation, but it should serve as a reminder that some effect might be a result of a small detail. The game is littered with them. Things like faceoff location, timing of line changes (my favorite example, after noting that Cale Makar’s non-Mackinnon minutes were unimpressive, JFresh looked into it and found that most of the difference had to do with how Mackinnon changed on the way back), and general circumstances of the event you measure can all lead to varying levels of distortions.

Rule 4: All relative numbers are relative to *something*

Sometimes relative stats will be used as a proxy for player impact on team performance. For example, Joel Edmundson had a perfectly respectable 50.6% xG% in 2024-25, but the team’s xG% was 55.6% in the games he played, giving Edmundson a relative xG% of -4.8%. So the logic goes…his presence on the ice reduced the teams xG% by nearly five points.

Rule 4 simply makes a basic logical observation: relative numbers are zero sum. They must sum to zero.

Even if one disregarded all the other rules so eloquently expounded here and took that 4.8 point negative value as literal measure of Edmundson’s impact, just remember it can only be impact relative to teammates. Edmundson might be worse than Anderson, who is worse than Clarke, who is worse than Spence, who is worse than Gavrikov, but it won’t tell us anything about whether any of those guys are good or bad relative to the league. They can’t all have positive relative numbers.

Rule 5: 5-on-5 on-ice numbers must be shared among five teammates

Rule 5 is the inverse of Rule 4 in some ways. “Sure,” says my opponent in this discussion, “they can’t all have positive relative numbers, but we can look to see who is an outlier.”

Lower your expectations for this approach. At the NHL level, there is rarely an apple so rotten that it ruins the bunch…at least not in an obvious way. On the other hand, there are exceptional players (e.g. McDavid, Mackinnon) that are a vortex of positive on-ice results for anyone playing with them. If you see someone sharing information that indicates some player is extremely good or bad, the first reaction should be, “who has he been playing with? Are their numbers equally good/bad? How do you know this player was the source of the result?”

Rule 6: The average of the individuals must average to the average of the whole

Rules 4-6 have overlap, but there are different themes. As with rule 4, here we reference the zero-sum nature of a lot of measures, but in this case we emphasize how the whole is made.

Are players ingredients in the recipe? Can we swap out the bad flavors for the better ones and make a better dish?

Or is the team a pie? You can slice it differently, but it doesn’t change what it is.

In 2024-25, the Kings finished with the sixth best record in the league, 4^th best xG%, 2^nd best GF%. Is it possible that some players were dragging the team down in spite of those outcomes and the team could have done even better? Sure, it’s possible. But it’s also true that there has to be some bad minutes out there for every team. If you measure a team that excels, how do we know that the player isn’t just being given the bad slice of pie?

For example, what predicts xG% performance of new players: the players’ old xG% or the team xG% of the new team? The answer is the latter:

Rule 7: Predictive measures aren’t outcomes

Only goals matter. Expected Goals only matter to the extent they tell us something about goals. Be careful when any commentor declares outcomes as bad on the basis of Expected Goals.

Rule 8: Someone’s blackbox won’t make rules 1-7 go away

Rules 1-7 are true. Just because someone has fancy software or graphics, it doesn’t make these issues go away. If you must choose an online expert, choose one who is careful.

• Tweet
• Pin It