Analytics Say Sami Vatanen is Likely Not The Devils Best Defenceman (ft. Interview with EvolvingWild)

Here’s the narrative that most people would likely tout. The Devils were a team in 2017 that wasn’t supposed to do anything due to a depressingly barren roster — particularly at defense. They overperformed due to an MVP season from Taylor Hall, surprising seasons from first-year players Hischier, Bratt, Coleman, and Gibbons, and a trade by Shero that landed us the #1 defender we needed to shore up the blue line. Is that what happened?

The most I could possibly see someone saying about the team is that we clearly improved in overall shot ratio — which is good! But the team’s expected goal numbers improved only marginally in 5v5 and not at all in all situations. Furthermore, the actual goal results got worse when he joined the team and, unsurprisingly, the wins followed suit — we got 12% less of the available points when compared with pre-trade.

Now some of this is surely because of goaltending/luck — perhaps most of it. But it’s not immediately obvious from these metrics that Sami Vatanen has completely revitalized the blueline. At least not to me. Is that surprising? Surely, even if the team hasn’t seemed vastly improved, he’s not to fault. He’s been excellent.

... or has he?

According to Manny Perry’s WAR model, Sami Vatanen had a negative WAR in 2017-2018. He likely was very adversely impacted by a pretty terrible start to the season with Anaheim, but that was only about 20% of his total minutes so he couldn’t have been THAT good with the Devils. In aggregate, the only less valuable defender on the Devils roster was Steve Santini. So... yeah. Not great. Luke and Josh Younggren (@EvolvingWild) have their WAR results posted on their excellent website, Evolving-Hockey, where they not only split up WAR by team, but they update it during the season. He had a GAR of -3.1 with Anaheim and a +4.5 last year with the Devils. So far, he’s a -0.2 this year. His GAR rate was 4th among Devils defenders last year (Butcher, Lovejoy, and Mueller were clearly higher) and 4th again this year (Greene, Severson, Butcher). Worth noting that not only is it very early in the year, but EW suggest 3 years or ~3000 minutes before using these metrics confidently.

Despite those results, when I recently asked Devils fans on Twitter, it was clear that he was unanimously the best defender on the Devils. I’m sure there are other places where you could find similar conclusions online. Who is right? Are the computer boys missing something important that we fans are seeing? Or are we biased by the eye test and the narrative of the defensive savior? I asked EvolvingWild a few questions about these metrics to see if we could get some enlightenment on the disconnect.

Interview with WAR-modelers, Luke and Josh Younggren, of Evolving-Hockey

My goal here was chiefly to figure out if there could be something in Vatanen’s game, specifically his role — the coaching staff asks A LOT of him — that the GAR models could be overlooking. I’ll ask about accounting for zone starts, shift location, opponents, etc.

CJ: Sami Vatanen is not viewed particularly favorably on your model, and last year Manny’s model hated him. Devils fans generally believe he is our best defender. Many fans say he’s got a much tougher job than other defenders and if your model prefers players like Butcher/Lovejoy last year or Greene/Severson this year, it must be missing some aspect of that. How would you respond?

EW: Our GAR model accounts for zone starts and TOI% - at least the best we can with this type of model. I would check against the RAPMs (Regularized adjusted plus minus) on our site for more of an “underlying numbers picture”. Specifically, the expected goals and shot rates at even strength. The offensive components of our GAR are based on a long-term goal RAPM so it should be pretty close to the single season GF RAPM metric — looks pretty average to me.

CJ: With regards to QoC (quality of competition), is there something that could be getting missed. For instance, does it account for the time in the shift that they enter the game -- a la Dellow and Matt Cane?

EW: I can tell you that shift start time relative to their competition doesn’t actually matter in a shift regression like this to a significant degree. The model’s predictivity would see negligible if any improvement with their inclusion.

CJ: So if opponents are sufficiently adjusted for, maybe their spot on the ice isn’t being considered fully enough. You account for the zone of faceoffs, but are on-the-fly shifts all treated equally? Are distinctions made about shift-starts and mid-shift faceoffs? I know, for instance Natural Stat Trick now designates Zone Starts as a subset of Zone Faceoffs.

EW: We don’t have the full shift data so we just use faceoffs. I’m not sure how much it matters because we’ve never tested it. So it might not be the best method for adjusting for zones, but I would imagine indicating on-the-fly shifts in a regression like this might be problematic as basically every shift without a faceoff, it seems, would be an OTF shift I think.

CJ: So in terms of “top 2/4 etc.” defender, what would your appraisal of him be in general? What about specifically for this team — is he the Devils best defender?

EW: Probably a 3rd/4th D is apt - maybe 2nd or 3rd but I’d have to look more into the spread of D GAR. As to who is the best defender, based on what we’ve seen from our GAR model and the RAPMs during this timeframe, I’d say that title would go to Will Butcher.

CJ: One last question, since I know that will strike a chord with some fans given the perceived difference in their respective roles. If Vatanen and a “less sheltered” defender like Will Butcher were to switch roles. In what way, if at all, do you think that would impact their GAR totals/rates?

EW: That’s a really hard question to answer. Ideally they should be isolated from those factors, but it’s a really hard thing to measure (given we don’t know what their actual role is from a data perspective). I would like to test that, but we would need to have historical announces lines for each game going back X number of years... which we don’t have.

Commentary on Interview

So for anyone who doesn’t speak analyticese, I’ll boil down the results. WAR does make adjustments for usage based on their competition (measured by TOI) and usage (measured by zone starts — not including on-the-fly shifts). When you account for all players on the ice, Vatanen’s impacts, while not that bad, are not particularly noteworthy. Unlike, say, Will Butcher’s. He’s likely not a top pairing guy, nor the most valuable defenceman on the team, if we are to believe these metrics.

I do think that Vatanen is viewed too favorably by the Devils faithful due to an appeal to authority — “Hynes plays him 24 minutes a game so he must be really great.” However, I’ve seen Greene be given new life since being taken off that top pairing that Hynes buries. Same happened to Lovejoy after being taken out of that role and put with Butcher last year — winning “Best Comeback” in the AAtJ awards in the process. I can’t help but think the same would happen if Vatanen were to be freed from the spot as well.

I think that it’s likely the specific usage of players, specifically defenders, has been insufficiently explored in extreme cases — Hynes is well-known for giving historically difficult usage to guys top pairing defenders. My hunch is that there is something we’re missing with these analytics. I’m not smart enough to be more precise, unfortunately. I’ve read over Manny’s notebook, and EvolvingWild’s RITSAC presentation multiple times, but I can’t figure out the exact anatomy of the “magic” component I believe they’re missing. And perhaps it’s nothing. Perhaps I’m being biased by what has happened on my specific team and what my eye test tells me. All I can tell you is that I see what Vatanen does for the team, and the hockey nerd in me is always waiting for the impact to show up in the analytics, and is always leaving disappointed.

What do you guys think? Is Sami the best defender on the Devils? Or is it someone else? What do you think of the WAR models’ depiction of him on this squad. Are they missing something? If so what?

Thanks for reading, and leave your thoughts below!