An All About the Jersey Hockey Stat Primer: Three Common Biases in Stats

So far in past weeks, I have written out primers for hockey statistics that have become increasingly popular and advanced knowledge of the game. They are meant to help introduce, explain, and detail the meaning for each. There have been some real impacts from these public developments in addition to fans being increasing their understanding of the game. However, they are not perfect. No one ever said they were perfect. I even wrote about some of the drawbacks within those primers. But in order to gain a fuller understanding of hockey stats there are three big general drawbacks that need to at least be acknowledged. There are more, but there are three I think should be highlighted over all else as they directly tie into the stats we use - “basic” or otherwise.

But, First, Let’s Make a Sandwich

We typically have posts go up at around 11 AM ET and my posts are usually lengthy. You may want to get a sandwich together since it is close to lunchtime. You may feel like making a sub. Like a turkey sub. It is pretty easy to make:

1. Gather the ingredients you want on the sub. For example: a sub roll, slices of turkey breast, tomato, lettuce, onion, vinegar, oil, salt, and pepper. Feel free to get different ones for your sub.
2. Slice up the tomato and onion into slices.
3. Shred some lettuce leaves into small pieces.
4. If it is not pre-cut, cut the sub in half length-wise.
5. On one half, lay down the slices of turkey, rolling them up as needed.
6. Place the tomato slices, the onion slices, and then the shredded lettuce.
7. Sprinkle oregano, salt, and pepper on top as needed. Squirt oil and vinegar on top of that if desired.
8. Place the other half on top.
9. Eat the sub.

The benefit of this kind of sandwich. is that you can modify the ingredients in many ways. If you want turkey and roast beef on it, then go for it. If you do not want salt, then by all means, forget the salt. It is your sandwich. I just listed out what I would do to make my preferred sub. The process for making it is about the same.

Keep this in mind for the rest of this post. Especially for the third drawback.

The Streetlight Effect or We See Only What We Can See

The streetlight effect is a kind of observational bias that is rife in hockey and sports statistics as a whole. This is usually explained with the following anecdote. It is nighttime on a street and a man loses their keys. The man is found under a streetlight looking for the keys. A second man comes by and asks what the first man is doing. The first man says, “I’m looking for my keys.” The second man responds, “OK, where did you lose them?” The first man responds, “I don’t know. That’s why I’m looking” The second man follows-up, “If you do not see them under the light, then why are you looking here?” The first man says, “Because this is where the light is.” There are variations on the anecdote but the main point remains: The first man is only searching in the light because he can actually see his keys - not because that is where he thinks he lost them.

This is a kind of observational bias and this one is specifically highlighted because there is only so much information that we can get from hockey. The game of hockey is fast, frantic, and difficult to fully observe. Even if you train yourself to look for nuances in a player’s game, you are only going to be able to watch one player or one game at a time. Watching all 82 games for all 31 teams is near impossible; much less remembering all of those observations and doing something with them. All of the stat sites from Behind the Net and Time on Ice back in the day to Natural Stat Trick and Evolving-Hockey today is all driven by the data recorded by the NHL. If the data is not recorded by the NHL, then it is not at those sites and there is not much we can do with them.

As a result, we can only determine if players or teams are good based on what we have available. Legitimate questions about a performance may not be able to be fully answered because we do not have the right data. Statistical models are limited to what data is available, which may miss data that would strengthen the model’s validity.

One thing that can be done to combat this effort is to go explore the darkness. That is, put in the work and track what is not being tracked. This is what was initially done for scoring chances. More recently, the effort has come through for passes and zone transitions. They have been primarily tracked by Corey Sznajder (link goes to his Patreon explaining what he does). His data has been very useful to better identify who is actually driving puck movement forward - an actual playmaker - beyond what their Corsi or on-ice expected goals value may be. His data has also been very useful to support the idea that carrying in the puck on offense is superior to dumping it in, and that denying carry-ins speaks well to the defensive effort. Sznajder is someone who did not stay under the proverbial streetlight; he went into the dark to look for keys. Of course, this (and other efforts like this) is an incredibly tall order. If Sznajder wakes up tomorrow and decides he does not want to do it, then, well, we have to hope someone else is willing to put in the effort at all - much less whether they are any good at it. Looking for keys outside in the dark is not easy.

We can only make judgments on what we know. Unless we make the effort to find out what we do not know, we are all currently under a streetlight hoping that the keys we are looking for is under our lights.

Success Bias or We Only See Who Made It

All NHL players at one point or another have to prove that they belong on a NHL team at some point in their careers. For a lot of prospects and veteran minor leaguers, they are working hard and honing their craft in the hopes of getting an opportunity. Whether it is in training camp, preseason, or as a call-up for performance or injury-related reasons, the goal is the same: make a good enough impression that the team wants to keep you in the NHL. If not now, then for the future.

The problem is that we primarily see those who successfully made it. We do not always look at who did not.

This is especially true with goaltenders. There are 31 NHL teams and so there are 62 spots for goaltenders assuming teams carry only two goaltenders on their active roster. One of those 62 goalies may have earned that opportunity by being a legitimately good enough to play at the NHL level. Or they got it by being really good for a short period of time and the team hopes or expects that would continue. Or they got it because the team had few other options. We tend to not consider the possibility that a player who failed to make the most of their chance just had a bad day and would otherwise be good.

A great example from a few years back is Scott Wedgewood. He is now back in the organization he started with. The Devils drafted him in the third round of the 2010 season. He posted overall save percentages of 89.9% and 90.3% in each of his first two full seasons for Albany after turning pro in 2013-14. In the 2015-16 season, Wedgewood played way above that level and posted an overall save percentage of 93.3% in 22 games with Albany. Wedgewood was called up to New Jersey for four games in the 2015-16 season. He was absolutely fantastic. He gave up just five goals out of 116 total shots, with only two allowed in his first three NHL games. Between his season in Albany and this call-up, it looked like Wedgewood could be Wedge-wall. But he was kept in Albany for 2016-17 and so he decided to hit the market for the following season. until he was dealt to Arizona.

Arizona retained him for the following season and he started in the NHL. Both were supported by his brief success in New Jersey. But that success was limited to those four games. With Arizona, Wedgewood did not stick in the NHL. He appeared in 20 games for the Coyotes in 2017-18, put up an overall save percentage of 89.3% (which is not good), and ended up back in the minors.

Fans, bettors, players, coaches, and teams are limited in the amount of information they can work with. A decision may need to be made well before we could get enough information to be confident if a performance is legitimate. Arizona bet that Wedgewood could play closer to what he did with New Jersey. That brief success gave him a shot. The bet did not work out. But success bias played a role in that signing happening at all. Had Wedgewood not played out of his mind in those four games in New Jersey, then he may have stayed in New Jersey’s system or would have been signed by another team but kept solely in the AHL.

The thing is that we have to keep in mind, especially for young players and goaltenders, we are looking at the players who made the most of their chances. The players that do not may not necessarily get another one - and it may be a result of just a bad night as opposed to the player being truly bad. Some will get more shots than others. Positions of need, being a former first rounder (e.g. Michael McLeod) tends to get more chances than a former sixth rounder, and call-ups due to injuries are all legitimate. And there is certainly a sunk cost concern in giving more opportunities to players who do not make the most of them. But the bias inherent is that we are looking at those who did.

This has to be at least recognized, especially when looking at developing players. Not everyone “blossoms” at the right way. Jesper Bratt broke out way earlier than expected. Nikita Gusev is a great example of why you should not give up on a player after a crummy month. Blake Coleman made it to the NHL after he was 25. But they are the successes and sometimes we need to at least consider the “failures” to get the knowledge we need. This should be especially heeded by team personnel who work with younger players.

Scorer Bias or We Use Data That May Not Be 100% Accurate

In every NHL game, a scorer records the events in the game. This events include shot attempts, shots on net, goals, hits, blocks, misses, giveaways, takeaways, penalties (who took it, the type of penalty, who drew it), faceoff wins, and stoppages in play. For almost all of these events, the scorer also records who was on the ice. While it is not in the play by play log that gets posted for every game on NHL.com, the metadata includes the location of the event. It is through this data that we can sum up a player’s point totals, count up their Corsi, and provide the input for statistical models like expected goals or Goals Above Replacement among other tasks.

This also means that if the scorer does not record data accurately, then we will not have accurate results. And, unfortunately, scorer bias is a thing.

While all scorers record data into the same system, not all scorers judge events or record data in the same way. As with any manual process, human error is a factor. Mistakes can be made. Interpretations of what is a hit or a giveaway or even a shot on net vary from scorer to scorer or, to put it another way, arena to arena. I’ve written about this before as far back as in 2010 and more recently in detail in 2018 and in 2019. What this means is that certain stats are over or undercounted in the arenas for various teams.

Let us look at 2019-20 briefly (Aside: NHL.com now has this data under Miscellaneous). Did you know the Devils led the league in blocked shots on the road last season? I learned that while writing this post; they blocked 579 in 35 road games last season. That is not really a good thing, by the way. Anyway, this is also surprising because the Devils were credited for 454 blocks in 34 home games - the 18th most among home teams. What is more likely? The Devils played so differently away from home that they blocked an extra three shots per game away from Newark? Or that the scorer in Newark undercounted the number of blocks they made? I would bet on the latter since the Devils, as bad as they were last season, really did not play dramatically different on the road. This is a good example of scorer bias.

The issue of it is rampant in other areas and other stats. Madison Square Garden has been historically inaccurate with shot location, which is a critical part of expected goals models. (Aside: Meghan Hall has this more recent post using 2019-20 data about shot location and features a tableau to play around with.) Whoever is recorded giveaways for the Isles’ home games last season decided to be really generous since the Isles were credited for 2.3 times more giveaways than the scorers did when they were visitors. Las Vegas led the league in team hits last season when they were at home and finished just 19th in hits in their road games. Some of it is not that extreme, but the larger doubt remains: Are the scorers who over/undercount it actually correct and the rest of the league is wrong, or do most scorers get it right and the over/undercounters are mistaken? The truth: We do not know.

There are so many workarounds this drawback. One popular one was to use road data only. This would minimize the impact of one scorer in a given stat. The problem is that it cuts the larger population size in half, it means we need to wait longer to start drawing meaningful conclusions about a team or player’s performance - if we can at all. Remember the Law of Large Numbers? If 82 games may not be enough to be confident in a conclusion, then 41 games will not. Another one is to adjust the stats accordingly. Some sites like Natural Stat Trick have score and venue adjustments to modify stats based on the state of the score of the game for the event (tied, up one, down one, etc.) and the venue. Over a lot of games, though, those adjustments do not change that much. Since the adjustments may not significantly change a stat, another strategy is just to accept it and deal with it.

A proper solution may be coming, though. The NHL has been working with SportsMEDIA Technology to develop a Puck and Player Tracking system. This system was originally planned to be utilized in full for the 2020 Stanley Cup Playoffs, but that was pushed back between the pandemic and the Return to Play format. I doubt it will be fully implemented for 2021 since I would think having a season at all is the larger priority. But this system could be the method that future game event data is recorded. It will allow for tracking of all kinds of data we did not have before, such as player location for an event (e.g. is a player screening the goalie, where is a deflection taking place, where is the defense), puck movement before a shot attempt, and player positioning. Depending on how much data is made available to the public, it could lead to some big developments in hockey analytics and take it much further than what we currently have. Getting back to this subject, having the same system set up for each team means we do not need to have the data come from a scorer with their interpretation of what is happening on the ice and their ability to record it accurately. It could really solve the issue of scorer bias, which will help put the hockey stats we do have on a sturdier foundation.

Until then, the stats we do have are built on a sandy foundation.

Let’s Go Back to that Sandwich

While I used hockey examples where relevant, that turkey sub sandwich I detailed is an easier way to show off the practical impact of all three of these biases.

For the streetlight effect, in making this sandwich, I could not go outside. Let’s say a snowstorm happened). I had to use whatever I had in my kitchen. I did not have all of the intended ingredients and had to make do with what food I did have. While I may have made a sandwich, it differed from the planned turkey sub I wanted. But I was limited to what I could access.

For the success bias, I only had confidence that I could make this sandwich because I made it once before. If I failed to make a sandwich the first time - it tasted bad, it made me sick, whatever - then I may not have even tried to do it again. Instead of trying to learn from why I failed, I relied only on past success to give me the belief to make a turkey sub. Likewise, if I was making the sub for one of you - one of the People Who Matter - then you may be more willing to let me make you something that I did before instead of something for the first time or something I failed at making the first time.

For the scorer bias, imagine if the ingredients were questionable. From my eyes, I could see that the bread was moldy, the turkey expired a day ago, the tomato was starting to get overripe, and the lettuce was turning brown. I could make a sandwich with all of that. It might be good. But it might not be all that good - and I may not want to eat it, much less make one for someone else with those ingredients.

The larger point in all three of those cases is that the actual process for making the sub did not change. The actual method of putting food on one piece of bread before putting the other one on top is still the method. It is still a viable method for making a sandwich. Regardless of whether I had the right ingredients available, I successfully did it before or not, and especially whether the ingredients were still good to eat, the recipe is still intact. It is still intact if we bring up other kinds of potential issues with sandwich making (read: other kinds of observational or selection biases).

Concluding Thoughts Back to Hockey Stats

I bring all of this up as to highlight two points. First, yes, there are legitimate drawbacks with hockey stats (and sports stats). There are biases that we may be able to mitigate but we currently do not at this time. If your main criticism of hockey stats is that it is not accurate, it is not perfect, and that bad data may yield bad results, then I am here to say: you’re not wrong.

The second point is that these drawbacks and biases do not mean we should ignore them all and consider the whole process to be a waste of time. Again, the process itself may be logically sound even if the data has problems. A sandwich recipe with bad ingredients or limited ingredients or successful sandwich maker only does not make the recipe invalid.

If anything, efforts can (and are) be made to address them - which makes the stats better. Expected goal models may suffer from inaccurate shot locations and the conclusions we get from the models may suffer from the reality that we’re looking only at successful players and in certain areas of the game. But if we improve shot location data, obtain more data from other leagues to validate what shots are valuable, add additional data that factors into a shot’s quality, and make it clear what the model does and does not tell us, then the expected goal model has more value. The answer to a drawback or a statistical bias is not to throw the model away. It is to mitigate the risks; clearly define what the stat does or does not tell us; and improve the data or the model as needed.

This is not to sat that models and stats cannot be criticized. But by acknowledging these biases and other issues, we can focus on more constructive criticism. Through this, hockey analytics can continue to get better and make gains in the public sphere. (I presume that teams with resources dedicated to analysis have done much more to address these concerns and may even have methods that we are not aware of yet.)

Your Turn

This is a bit of a tricky subject as it is not so much about hockey stats but issues with them as a whole. Again, the idea is not to state that these biases exist so hockey stats can/should be ignored. It is tempting to do so. Who wants to spend time on flawed results? However, recognizing an elephant in the room does not mean the room is not worth being in. Even just understanding these bias can help us make better evaluations, judgments, and even better stats through trying to correct the issues. It can even improve how we discuss and think about players - even if there is not much we can do about the biases.

As a final point, just because something is flawed does not mean it has no value. There is no perfect metric for hockey performances. No one ever said one - Corsi, expected goals, whatever - was perfect. We just need to be aware of what the stat or model represents and, ideally, aware of their drawbacks from biases to interpretation.

With a potential 2021 season coming up, this primer series may be ending soon. Maybe there will be one or two more posts. When it ends, I will put them all into a section so you can view them all in one spot on the site together. What I would like to know from you, the People Who Matter, is what I should focus on next? Passing? All-encompassing models like GAR? The impacts of analytics (why this all matters)? Please let me know in the comments. Thank you for reading. And I hope your sandwich is good if you made one.