The Playoffs Are a Crapshoot, Part 4: Luck vs. Skill

In case you missed the first 3 parts of “Playoffs are a Crapshoot”, here they are:

In the previous installment, we estimated the Bradley-Terry rating for the playoff teams.  While these are the best estimates we can make using only the 2019 head-to-head results, they overstate the strength of the strongest teams.  This is because of the phenomenon (often discussed here with respect to player results) of regression to the mean.

Baseball win records are the result of skill and luck.  Before we can intelligently discuss regression to the mean, we need to be somewhat more careful about what we mean by this. 

Many years ago, I was hired by the government to help them quantify luck versus skill in online poker.  The research in that case was interesting, but even more interesting was just asking people I knew what they thought about the question.  Almost uniformly, when asked, people thought that poker was 80-90 percent skill, although they weren’t able to really justify this feeling in any obvious way. 

Justice Scalia, in my favorite dissenting Supreme Court opinion of all time, PGA Tour v. Martin, expressed the distinction of luck vs. skill in professional golf this way:

In …[the majority decision], the Court first finds that the effects of the change are “mitigated” by the fact that in the game of golf, weather, a “lucky bounce,” and “pure chance” provide different conditions for each competitor and individual ability may not “be the sole determinant of the outcome.”  I guess that is why those who follow professional golfing consider Jack Nicklaus the luckiest golfer of all time, only to be challenged of late by the phenomenal luck of Tiger Woods. The Court’s empiricism is unpersuasive.

It may seem counterintuitive that there is anything such as luck at all.  What does it even mean?  The easiest way (for me) to think about it is as follows. Imagine we cloned the Braves (or any other team) 29 times.  Now have those 30 teams play a 162 game schedule against each other.  This is baseball, so in every game one of these teams will win.  Because the teams are equally skillful by construction, the winner must be determined by something else.  This is what we called luck, and we don’t care what it actually is.  But in every game, there is an even chance that either of the team wins because of it.  (I’m ignoring home field advantage here… just imagine that we flip a coin before the game to decide who the home team is.) When we do this, we certainly don’t expect each team to go exactly 81-81, even though no team is better than any other team.  When we flip a coin 162 times, sometimes we’ll get 90 heads.  Sometimes we’ll get 75.  We can in fact quantify exactly how likely each level of wins is.  The graph of those wins looks like this:

About 19 percent of teams will win 80-82 games, and about 5 percent of teams will win 70 or less, with an equal number winning 92 or more.  The fundamental insight is this: if actual records exhibited this distribution, they would be indistinguishable from a situation in which every game was random – we could just as well flip a coin as play the game for all that “skill” means.

Let’s be very clear, though about what this doesn’t mean.  It doesn’t mean that baseball players aren’t highly skilled.  It simply means that they are pretty equally skilled.  This is the big lesson I learned in the online poker exercise.  People want to win when they play online poker and there are various rooms with people at various skill levels.  Players sort themselves into these rooms:  if someone has too much success, people move to other rooms and they shift around until they are playing people of roughly the same skill level as them.  At that point, the observed results are essentially random. 

Chess is thought of as a game with no luck components at all.  But what if players of equal skill play one another?  Who wins?  Do they draw every game?  No!  I am a terrible chess player, but I can get my winning percentage up to 50 percent by finding someone as bad as I am.  But there are limits to this self selection.  At the highest level, some chess players are more skillful than everyone else.  That doesn’t mean they win every game.  (The reasons for this are not philosophically clear, but a simple one is that skill is variable, depending on rest, health, concentration level and any number of other factors.)

At the highest level, better teams win more games than the coin flip model, and worse teams lose more.  This is really not saying anything more that there are games in which the probability of a particular team winning is more than 50% and that this amount is due to some underlying talent of the team (as opposed to something like home team advantage.)  The higher this probability, and the more often it happens, the less and less the overall distribution will look like the coin-flip distribution above. 

As it turns out we can measure the actual dispersion of MLB results.  Last year, the distribution of wins was considerably flatter than the random distribution shown above.  This is because some teams are considerably better than others.  And the difference in the shapes allows us to actually quantify the relative contributions of skill and luck.

If we use the last 4 years as indicative of the overall dispersion of wins, we find that the standard deviation of overall team performance is 13.2 games.  (This is actually higher than the dispersion observed in the last twenty years.)  This means, among other things, that about 68 percent of teams in any given year will win between 68 and 94 games, and 95 percent of teams will win between 56 and 107 games.  If it weren’t for last year’s results, in which 4 teams had over 100 wins and 4 teams had over 100 losses, this figure would have been considerably lower. 

In any case, we know that over a 162 season, the variance induced by pure luck (the coin-flip variation) is about 6.3 games.  So that suggests that the variance induced by skill is 13.2-6.3, or 6.9 games.  Thus, in the last four years, the outcome of the baseball season has been slightly more skill than luck.  (This assumes luck is independent of skill.)

Now that we have decomposed baseball into luck and skill, we are ready to adjust for regression to the mean.  That’s in the next installment. 

Author: JonathanF

Alive since 1956. Braves fan since 1966. The first ten years were pretty much wasted. Exiled to Yankees/Mets territory in 1974 --- bearable only with TBS followed by MLB.TV.

18 thoughts on “The Playoffs Are a Crapshoot, Part 4: Luck vs. Skill”

  1. Asdrubal Cabrera to the Nats, 1/2.5MM. This combined with Castro and Harris signing might put them out on Donaldson.

    I talked about Asdrubal on our @3FlagsFlying podcast a few weeks back. Once again, he gets less money than he deserves. Has been worth over 200MM for his career and has only been paid 60MM. Was a 2 WAR player last year and just got 2.5MM.

  2. Great work again, JonathanF. You’ve demonstrated quite clearly the flaw in Scalia’s attempt to be clever. By his lights, if Nicklaus was the most skillful golfer in the world, which he was for several years, he should have won every tournament. But as you show, a lesser skilled opponent will win some percentage of the time against the more skilled. Of course over a large enough sample the more skilled wins more often (as Nicklaus did).
    I hope I’m not crossing the no politics line in critiquing Scalia. But remember that Justice Roberts says the Supreme Court is not political—they are just umpires calling balls and strikes. So what could be more appropriate for a baseball blog than the Supreme Court!

  3. JonathanF, I love this installment. In fact, it may be my all time favorite article on this site, and that is a high bar.

    Thank you.

  4. The Cabrera signing is the shape of things to come for MLB veteran free agents. A couple of months ago I mentioned that it looked to me that free agent signings were now being negotiated not based on $9M/WAR but instead on some curved $value/WAR, where larger WAR #’s equated to larger $/WAR amounts. This theory helped explain the “middle-class free agent malaise” seen over the last few seasons.
    I wanted to write a piece on the topic but unfortunately the data is very murky and doesn’t really yield definite results other than to say that while some teams (including the Braves) have obviously shifted to a newer compensation model, other teams (Detroit, Cincinnati and Anaheim being the worst offenders) have not.
    Which brings us to Asdrubal. Given that his defense is now unacceptably bad at both short and second, this leaves him as a bad defensive 3rd baseman who will likely also see time at 1st and corner outfield. His 1.6 WAR projection in 450 plate appearances is dead average. 5 years ago he would have landed a 2 year deal at about $6M per yet today he gets much less. The reason is that thanks to the new $curve the pay rate on those first 2 WAR for a positional regular at a non-premium position is now close to league minimum. I suspect he may have gotten a slightly better deal elsewhere, but he may just like Washington.

  5. Gosh, coop, thanks.
    Tfloyd, yep, sorta. The Casey Martin case didn’t bring out the best in either side’s reasoning faculties, though. But Scalia’s snark was truly epic.

  6. @4 That absolutely makes sense. Just like the tax code is progressive, the value of a single WAR should be progressive. Players exist on a bell curve, too. There are lots of 1-2 WAR players, fewer 3 WAR players, and very much fewer 4+ WAR players. That means that that 4th WAR point is much more valuable than the 3rd and so forth. I’ve wondered for a long time why this didn’t seem to be intuitively included in the salary structure. In fact, I would be surprised if the dollar value increase of WAR beginning at 3 was linear and should be at least geometric if not exponential.

    I would not consider this to be a “middle class FA malaise” at all, but a natural progression of value in performance.

  7. I like where this is going with Donaldson. The gNats are on Plan B (including haggling with the Cubs over Bryant) and the Braves are on to filling out the Gwinnett roster like I said by signing Pete Kozma. If Donaldson does sign this week, it will be time to all hail AA’s genius for how he’s set this team up to win this year while allowing for more improvement in subsequent years.

  8. I think Scalia’s remarks are really on point. With regards to the discussion of luck, there’s always the clichaic fallback of “luck being the residue of design”. When was it, way back when, the Giants won the NFL Championship in the snow not because they were the best team but because they brought the correct shoes. You can prepare for bad luck and be ready for it then, after your preparation has paid off, be hailed for how lucky you were. If Ender gets injured, you bring up Riley; if Markakis gets injured, you bring up Duvall; if Dansby gets injured, you sign Hech; Wright/Wilson fall flat and you sign Keuchel. Without these moves, do the Braves win the East? Probably not. But people would say the Braves were awfully lucky that they had Riley and Duvall and were lucky that Keuchel and Hech were available when they were.

    I also think spreading out the luck over a larger sample size can mitigate it. Nicklaus won more tournaments by being the most skilled. Bad luck would largely apply to individual shots more than a whole tournaments worth of shots and, over 18 holes on each of four days would be equaled out by both good luck for him and bad luck for other players. Also, minimizing the number of tournaments he won (i.e. saying he should have won them all) also belies the fact that he finished top 5 in a lot more tournaments.

  9. A few separate reports, completely unofficial yet have broken news several times before, are saying Braves are signing Donaldson officially on Monday.

    Also, if that happens, I think there are going to be more moves and not just roster filler.

  10. agreed, even if the reports of Donaldson are true, I think a few more moves will be needed. TDA and Smith will improve the team, and I like hamels more than keuchel but adding a big OF bat or another TOR starter would make a huge difference over the course of the year and into the playoffs.

  11. Braves signed LHP Chris Nunn to a MILB deal. He dominated AA then struggled at AAA, but throws 100 and is a great gamble by Anthopoulos.

  12. The combination of rumors of a pending deal with the Braves, the Nats acquiring infielders, and reports that the Twins are pessimistic have me feeling pretty good that Donaldson is getting done on Monday.

  13. After Jonathan’s series, we are going to start on a new series reflecting on the 2010s. I just completed the 2011 season and had such a good time going back reading Mac’s work.

  14. These chemicals might cause medical difficulties for them since they grow older.
    Be it contemporary, traditional, chic, modern, and hi-fashion you’ll have all the baby
    clothing you desire. 1- You should always purchase clothes that can be simple
    to wear and take off.

Leave a Reply

Your email address will not be published. Required fields are marked *