It is apparent to many observers of the NFL that the current overtime system (coin toss to determine which team kicks off, first points thereafter decide the game) is flawed. In a large number of cases, the team who receives the opening kick scores on this first possession, ending the game before the other team has had a chance at an offensive possession. Overall, teams winning the toss won 60% of all OT games between 2000 and 2007 (http://www.advancednflstats.com/2008/10/how-important-is-coin-flip-in-ot.html) and 59% of all OT games from 1994-2003 (http://www.footballcommentary.com/otauctions.htm). This means that a team will gain or lose 10% of win probability depending on a completely random event, clearly an unpalatable state of affairs. But it's worth examining the issue in more detail, as it is not as easy a case to resolve decisively as it may appear at first, and it gives rise to some interesting considerations along the way.
1) Proponents of the current system often attempt to entirely dismiss the primary objection - that the game can be decided without both teams having possession - by saying that the losing team in this scenario should have "played some defense." This is line of reasoning is obviously inadequate. It rests on the fallacy - still all too common in sports discussions - of treating game events as arising deterministically, rather than probabilistically, from the actions of the players. The fact that a team allows a score on a given possession does not necessarily imply that their performance was in any way sub-par; it is perfectly possible to perform at an average level - or even above this - and still suffer a reversal due to the random breaks of the game. (Note that these are not random in the strict quantum mechanical sense - but are determined by margins that are too fine to be precisely correlated with physical and/or mental actions of the players.) A given level of performance implies a probabilistic distribution of game outcomes, and vice versa. In the absence of further information, the end result of one possession is much too small a sample to infer anything about how well either team played. In general, NFL teams score on around 30% of their possessions. This does not mean that teams "played some defense" 70% of the time, and failed to do so on the other 30%. A good defensive performance on a possession may reduce the average probability of a score to 25% or 20%, but it does not reduce it to anywhere near 0% - that's just not how the world works. So even if the defensive team does all that can be asked of them on the first overtime possession, they have still been exposed to a substantial risk of losing immediately, while their opponents have faced only the comparatively trivial risk of losing on a turnover retured for a score.
2) Opponents of the current system often introduce a red herring into the discussion by calling it "unfair." As those who support the system tend to gleefully point out in response, this is not correct - both teams have an equal shot at the advantage resulting from the coin toss, so no fairness criteria are violated. A more precise way to state the objection is that the result of the game is highly influenced by an exogenous factor. After all, a system that determined the result of a game that was tied at the end regulation entirely by a coin toss (winner of the toss wins the game, no OT play at all) would be fair to both teams - but it would also be pretty stupid. One of the most important goals in constructing rules for a game should be to ensure that the greatest predictor of success is performance across a set of related skills. Some element of physical randomness is unavoidable (see #1 above) and even desirable. But introducing external random factors - such as coin tosses - into situations where they have a large impact on the outcome unnecessarily weakens the correspondence between performance and result.* If one were to graph the win probability of both teams over the course of the game, one of the largest single-event changes would result from the OT coin toss - it would rank alongside the game's biggest plays, and those in the "clutchiest" of situations, as one of the most significant events in determining a winner. When considered in these terms, the objection to the coin-toss/sudden death system should be obvious.
3) Even with the validity of the objection established, commonly-suggested alternatives often introduce flaws of their own. The most obvious possibility - stipulating an equal number of possessions for each team - risks making the game last substantially longer, exposing the players to greater risk of injury and (silly as this consideration sounds) interfering with TV scheduling. It also increases the possibility of the game ending in a tie, which many NFL fans seem to find an insufferable proposition. On the other hand, resorting to alternating possessions from the 25-yard line, as in NCAA football, offends football purists who deplore such gimmickry.
4) As a result, it may seem that we are trapped in a "de gustabus non disputandum est" situation, where every possible solution has its own strengths and weaknesses, and it is a matter of taste whether one prefers the current system or one of the alternatives. However, we could circumvent this problem if we were able to state the major objectives of an optimal overtime system, and present solutions that dominate the current system by performing equal or better in all categories. It seems to me that the main criteria are:
1) Minimize the probability of a tie
2) Minimize the amount of time it takes to reach a result
3) Minimize the effect of exogenous factors on the determination of the result
4) Maximize the similarity between OT and regulation play (i.e. avoid gimmicks)
5) Maximize "excitement" (the subjectivity of this criterion means that it can probably serve as a catch-all for all other specific criteria that one might think of)
Clearly, the current system does pretty well at 1, 2, 4 and 5. Solutions involving equal possessions or a fixed-time OT period improve on 3, but at the cost of a decline in 1 and 2. So whether one prefers these alternatives to the current system depends on how much weight one assigns to 1-3 on the list above. This makes it hard for supporters of the respective solutions to make any progress in persuading the other.
But there seem to be good candidates for solutions that dominate across all five criteria:
1) Simply extend the fourth quarter. When time runs out in the fourth quarter with the game still tied, add 15 minutes to the clock, and continue play as before, with the next points deciding the game. This still gives one team the advantage of having the first opportunity to score, but the difference here is that there is no exogenous factor - the advantage is built-in to the game state at the end of the fourth quarter, such that both teams have ample time with which to factor it into their plans earlier in the game. Not only is a coin flip not responsible for a sudden spike in win probability - there is no spike in probability at all. (To confirm this, consider that the last moments of regulation are de facto sudden death, as there will be no time left for a score in response. So if a team keeps possession across the nominal end of the fourth quarter and into overtime, their win probability will not change, except to increase as their distance to field goal range decreases.)
So this possibility obviously does at least as well as the current system in categories 1, 2, and 3. In my opinion, it gets a particular added boost from its performance in category 5. Under the current system, a team driving down the field with little time left, trailing by 3 or 7 points, will often play for a tie, taking their 50% chance of winning in overtime minus whatever chance there is that their opponent scores in regulation. Under the extended fourth quarter overtime scenario, teams in this situation would know that the opponent would in effect be guaranteed to win the overtime coin toss, giving them greater incentive to play for an outright win in normal time. Teams trailing by 3 with a minute remaining might forego a 30-yard field goal on 4th and 2, and play for a touchdown instead, and teams who score a late touchdown when trailing by 7 may choose to attempt a two-point conversion. The decisions made by coaches in such situations are often excessively conservative, failing to maximize win probability (as well as being relatively boring.) It is therefore a beneficial secondary consequence of this overtime system that it would give them a nudge in the right direction. Besides being more exciting and representing less frustratingly bad decision-making, this incentive to play for a win also feeds back into the second criterion, making it more likely that the game will end sooner, without any extra time played at all.
Several people have objected that this system would take away the two-minute drill at the end of a game, as teams now know that with the ability to continue a possession beyond the end of the quarter, they now have no reason to hurry. But this only applies to situations where the game is tied - teams trailing by anywhere from one to eight points would still have to run their two-minute offense. And when the game is tied, the two-minute offense only has to reach the opposing team's 30-yard line in order to have a decent shot at a winning field goal. Is this really the most exciting thing in the world? (Perhaps Patriots fans have reason to be a little biased on this question, recalling SB XXXVI). In any case, this consideration is also counterbalanced by the fact that it avoids a situation where a team is pinned deep in their own territory late in the game decides to run out the clock and play for overtime rather than attempt to do anything with the final possession of regulation (what John Madden thought that the Patriots should have done.)
2) The key feature of the extended fourth quarter system is that both teams know which team will have the first chance to score in the sudden death period, before this period starts. This allows them to spread the win probability deficit out throughout many game events throughout regulation, and allows them to increase their chance of overcoming it entirely through the decisions they make. Therefore, overtime systems that make use of this principle allow us to keep the sudden death feature (which cuts down on time and on ties) without the ugly spike in win probability that occurs when one team suddenly finds out that they are the first to have their heads on the chop block (no pun.) There are other ways to accomplish this: giving the first overtime possession to the home/road team, performing the OT coin flip at half-time, and having the opening coin flip count as the OT coin flip are the most obvious. In both cases, the team who will be at a sudden-death disadvantage may have slight incentive throughout the game (or the second half in the case of the half-time flip) to unbalance the score by attempting two-point conversions in situations where they otherwise would not. This could be kind of cool, though I still prefer to give the incentive to unbalance the game to the team who trails last. Retaining the coin flip, either at the outset of the game or at halftime, allows the overtime period to begin with a kickoff, which might be construed as superior performance in the fourth criterion, and takes care of the two-minute drill problem (though it allows teams in some cases to sit on the ball and force overtime, as in the current system.) Personally, the extended fourth quarter remains my favorite of these possibilities.
3) Another solution entirely is to keep the sudden death element of overtime, but to determine possession by bidding on field position. For a full discussion of this, see http://www.footballcommentary.com/otauctions.htm. It could be maintained that this is subpar according to the fourth criterion, as it involves a coaching decision the like of which never appears at any other point in a football game. But it fulfills the basic requirement of dominating across the first three categories without resorting to anything excessively foolish, and therefore would be a better choice than the current system.
Thursday, November 26, 2009
Saturday, November 21, 2009
Why do people overrate the punt?
My last post tried to show that the criticism of Belichick's fourth-down decision is largely misguided - that a very realistic value of the probability of the Colts scoring from a short field causes the decision to break even, even if all other assumptions are made so as to favor the punt. If I am correct, this would fit the general trend in NFL practice and commentary: most coaches choose to punt far more often than a probabalistic analysis indicates that they ought to, and most pundits compound the situation by criticizing coaches on the rare occasions where they correctly forego a punt (but almost never vice versa.) So perhaps we should follow up by looking at the meta-question: why are people so eager to believe in the punt? After all, if people are going to go around spouting irritating nonsense about sports, we can at least use this as a natural sociological experiment to identify the sorts of conceptual errors to which human beings are particularly prone. Here is a letter I sent to Boston Metro on Monday morning following the Colts game, in which I offer a possible answer for this in the case of punting:
--------------------------------
The mad scramble by fans and pundits alike to condemn Belichick's 4th-and-2 gamble on Sunday (believe it or not, the numbers suggest that he probably made the right decision) is evidence of a more general issue worthy of discussion: the punt is currently the most overrated strategy in American sports. Contrary to what the "Trust your defense!" crowd seems to believe, punting does not send the ball across a magical Rubicon that ensures that the other team will fail to score. It adds some distance - generally around 40 yards - to how far they will have to go, making it somewhat less likely that they will succeed. This can make it a valuable tactical option in certain game scenarios. But in a great many cases, it is merely a foolish waste of an opportunity to do something productive with an offensive possession, whether scoring points, or, in this case, running out the clock without the other team having a chance to touch the ball. The fact that NFL teams punt far more often than they ought to has been mathematically demonstrated several times: the ground-breaking analysis was done by David Romer of UC Berkley in a paper called Do Firms Maximize? Evidence From Professional Football, and espn commentator Gregg Easterbrook (aka TMQ) routinely exposes the folly of the automatic fourth-down punt in his columns.
So why is it so firmly ensconced in conventional wisdom that punting (in all but a tiny minority of fourth down situations) represents "playing the percentages," when in fact the opposite is often true? The answer seems to lie in a phenomenon known by economists as "risk aversion" - we are hard-wired to prefer the certainty of some fixed amount of most things (food, money, etc) rather than an uncertain outcome where we will either end up with a great deal, or none at all. This is true even if the weighted average of the uncertain outcome is higher than the certain one: when signing a ten-year contract, most of us would choose a guaranteed income of $50,000 rather than a possibility of either $5000 or $100,000, to be decided by a coin flip. (Risk aversion is one of the major reasons why people purchase insurance.) This is a sensible response to uncertainty when making economic decisions: personal well-being, not money, is the ultimate currency here, and for most people, being rich is not that much better than being comfortable, while poverty can be ruinous. But risk aversion is not rational when the uncertain quantity is win probability in sports, as this probability itself is the ultimate standard by which the decision is to be evaluated. (As such, it is not subject to decreasing marginal utility - a strictly linear relationship is maintained at all values.) Here, ten times out of ten, we should prefer a decision that will leave us with either a coin-flip 5% or 100% of winning to one that will guarantee us a 50% chance of winning. While the specific numbers differ widely from case to case, this general template underlies the various possibilities arising from a fourth down decision. A punt will guarantee a relatively negligible impact on win probability - we don't score, and we lower the probability that the other team scores. Nothing much has changed. An attempted fourth-down conversion, on the other hand, will have a much greater impact on win probability in one direction or the other: either we keep the ball, with a good chance to put points on the board, or we give the other team the ball with good field position. We instinctively balk at this latter scenario, treating a coach who chooses it as though he were an investment manager exposing his client's 401k to ruin in pursuit of a big payoff. In fact, we should be celebrating coaches like Belichick who are bold enough to look past the irrationality of the conventional wisdom, and make decisions that put their team in the best position to win.
--------------------------------
The mad scramble by fans and pundits alike to condemn Belichick's 4th-and-2 gamble on Sunday (believe it or not, the numbers suggest that he probably made the right decision) is evidence of a more general issue worthy of discussion: the punt is currently the most overrated strategy in American sports. Contrary to what the "Trust your defense!" crowd seems to believe, punting does not send the ball across a magical Rubicon that ensures that the other team will fail to score. It adds some distance - generally around 40 yards - to how far they will have to go, making it somewhat less likely that they will succeed. This can make it a valuable tactical option in certain game scenarios. But in a great many cases, it is merely a foolish waste of an opportunity to do something productive with an offensive possession, whether scoring points, or, in this case, running out the clock without the other team having a chance to touch the ball. The fact that NFL teams punt far more often than they ought to has been mathematically demonstrated several times: the ground-breaking analysis was done by David Romer of UC Berkley in a paper called Do Firms Maximize? Evidence From Professional Football, and espn commentator Gregg Easterbrook (aka TMQ) routinely exposes the folly of the automatic fourth-down punt in his columns.
So why is it so firmly ensconced in conventional wisdom that punting (in all but a tiny minority of fourth down situations) represents "playing the percentages," when in fact the opposite is often true? The answer seems to lie in a phenomenon known by economists as "risk aversion" - we are hard-wired to prefer the certainty of some fixed amount of most things (food, money, etc) rather than an uncertain outcome where we will either end up with a great deal, or none at all. This is true even if the weighted average of the uncertain outcome is higher than the certain one: when signing a ten-year contract, most of us would choose a guaranteed income of $50,000 rather than a possibility of either $5000 or $100,000, to be decided by a coin flip. (Risk aversion is one of the major reasons why people purchase insurance.) This is a sensible response to uncertainty when making economic decisions: personal well-being, not money, is the ultimate currency here, and for most people, being rich is not that much better than being comfortable, while poverty can be ruinous. But risk aversion is not rational when the uncertain quantity is win probability in sports, as this probability itself is the ultimate standard by which the decision is to be evaluated. (As such, it is not subject to decreasing marginal utility - a strictly linear relationship is maintained at all values.) Here, ten times out of ten, we should prefer a decision that will leave us with either a coin-flip 5% or 100% of winning to one that will guarantee us a 50% chance of winning. While the specific numbers differ widely from case to case, this general template underlies the various possibilities arising from a fourth down decision. A punt will guarantee a relatively negligible impact on win probability - we don't score, and we lower the probability that the other team scores. Nothing much has changed. An attempted fourth-down conversion, on the other hand, will have a much greater impact on win probability in one direction or the other: either we keep the ball, with a good chance to put points on the board, or we give the other team the ball with good field position. We instinctively balk at this latter scenario, treating a coach who chooses it as though he were an investment manager exposing his client's 401k to ruin in pursuit of a big payoff. In fact, we should be celebrating coaches like Belichick who are bold enough to look past the irrationality of the conventional wisdom, and make decisions that put their team in the best position to win.
Comments
on anything I say here (or anything in general, for that matter) can be e-mailed to dajepson at gmail dot com.
Brief addendum on P(S)
Re-reading my previous post on the 4th-and-2 question, I realize it may seem a bit disappointing that I failed to offer a stronger conclusion on whether or not Belichick's decision was ultimately correct. I thought it important to err on the side of understatement - to elucidate the reasons why the punt tends to be overrated in such situations without using any over-optimistic estimates that might encourage a skeptic to dismiss the thought process entirely. However, I think it is possible to make a stronger attempt at showing why 0.77 is such a realistic value for P(S), without jeopardizing the tightness of the case:
2) The probability of a TD is higher than that suggested by the season yards-per-drive averages of the two teams. We must give P(S) a small boost due to the fact that Indianapolis is playing at home, on their favored artificial turf, in front of a crowd who will not disrupt their snap counts. More importantly, we must give P(S) a substantial increase because the Colts are playing with all four downs in this situation - there will be no punts or field goals. Most drives in the season-long data set would have only used three downs to try to pick up a first down, with a kick of some sort occurring on fourth down if there was no success.
For these reasons, I fail to see how anyone can confidently maintain that P(S) was below 0.77.
Why Belichick's 4th-and-2 decision was almost certainly correct
There are many decisions from last Sunday's game vs. Indianapolis for which Bill Belichick could reasonably be second-guessed. The unconventional decision to attempt to convert a 4th and 2 from his own 28-yard line, leading by six points as the two-minute warning approached, was not one of them. Note that this is not the same as saying that the decision was necessarily correct; this assessment depends on the exact values of the various probabilities involved. No one knows these for sure. However, I will show that a) even with very conservative estimates of the probabilities, the 4th down conversion comes out looking very reasonable, and b) the particular assumption made by many of Belichick's detractors - that the Colts were all but guaranteed to score following a failed 4th down play - actually confirms that his decision was correct. Paraxodical as it seems, only by assuming that the Patriots had a good chance of stopping the Colts from the short field can one maintain a case that the punt was the better choice. This will be explained in more detail below.
We start by assigning variables to the relevant probabilities. Let's call the probability of the Patriots picking up the first down P(1), the probability of a Colts TD following a failed fourth-down play P(S), and the probability of a Colts TD following a punt P(L). Then our equation for finding the break-even point between the punt and the attempt to pick up the first down is:
P(1) + [1-P(1)] x [1-P(S)] = 1-P(L)
The left side of the equation contains the probabilities that of the Patriots winning the game following their fourth-down play: either by successfully picking up the first down [P(1)], or by failing to pick up the first down [1-P(1)] but preventing the Colts from scoring anyway [1-P(S)]. The break-even point occurs when this sum is equal to the probability of winning the game following a punt, which is essentially equal to the probability that the Colts fail to score from a longer field [1-P(L)].
Procedural note: we find the break-even point because it is easier to deal with equations than with inequalities. Once we have found the break-even point, it will be clear that deviations in one direction favor one decision, and vice versa.
This equation contains two assumptions which need to be addressed:
1) The Patriots are guaranteed to win the game if they pick up the first down. This is not quite true, but it is very close. With one time-out, Indianapolis's best-case scenario, barring a very improbable event, is getting the ball back with about 30 seconds and no time-outs, needing to drive for a TD. Following a normal punt, the chances of this - even for Peyton Manning - are truly miniscule. Their only significant prospects come from a fumble on a Patriots' running play, a blocked punt, or a long punt return. These are all very improbable events which do little to raise their win probability - especially when multiplied by the probability of converting such an event into a TD, which is still not a guarantee. Other analyses have used 92% as the Patriots' win probability figure here; this is derived from data in all similar score/down/distance situations, regardless of time-outs - a factor which makes a world of difference here. Using the 92% estimate errs on the side of caution, which is generally laudable. But here it seems like a truly gratuitous concession to the case for punting.
2) No outcomes need to be considered other than those considered in our equation. The main omission here is the chance that the Patriots win with a field goal following a Colts touchdown. This probability adds to the expected value of the fourth-down attempt, so the omission is generous to the case for punting (almost certianly by a larger margin than the 100% win probability estimate following a successful conversion helps the case for fourth-down attempt.)
To solve the equation, we need to supply values for P(1), P(S), and P(L). Since we can never be sure of the exact values, it is important to select very conservative estimates from the point of view of our ultimate conclusion. Many statistical analyses of the decision have used fourth-down conversion data to estimate that the Patriots had a 55-60% chance of success, and used this to show that the fourth down attempt had a higher expected win probability; while this may be a good estimate, it is susceptible to the following skeptical response: "I don't believe that the probability was really this high - the Patriots offense was looking shaky, the Indy defense was fired up, the five-receiver set was a poor choice, etc. etc.....therefore I reject the conclusion reached by your analysis."
Bill Simmons (The Sports Guy) provided a perfect example of this thought process in his espn.com article on the subject. He suggests that the probability was more like 1 in 3 - the rate at which offenses have successfully made two-point conversions with passes on the road in the last three years. This is obviously a ridiculously low estimate - he would have us believe that an offense featuring Tom Brady, Wes Welker and Randy Moss has 12% less chance of picking two yards on a single snap than the average NFL team does when they attempt a two-point conversion (a situation where the defense also has less field area to defend, further raising the offense's degree of difficulty.) Nonetheless, we will use his figure as our estimate of P(1), confident that none but the wilfully obstinate can now object that we are overestimating the Patriots' chance of success. This means that our equation is now:
.33 + .67 (1-P(S)) = 1-P(L)
Now we come to the crux of the entire issue: the underlying reason why so many people have misjudged this decision so badly (and why hardly an NFL game goes by where a coach or a commentator does not badly misjudge a punting decision.) How much does the Colts' probability of scoring change depending on whether they take over following a failed fourth down or following a punt? Unfortunately, there is an enduring irrationality baked into the manner in which conventional wisdom evaluates such questions, which can loosely be characterized as follows:
"Not punting is a huge risk. The other team may end up with great field position, and will probably score. Punting gives the defense a great chance to stop them."
No element of this statement is categorically false, but every element involves a stretching of the truth to the point that the ultimate conclusion is badly distorted from what is suggested by a dispassionate account of the probabilities involved. (Why have people been so inclined to accept stretching of the truth on this particular issue? I will address this in another post.) In fact, while punting does reduce the probability that the opponent will score, it does so by a much smaller margin than is commonly believed, particularly when dealing with a powerful opposing offense such as that of the Colts.
To see why, consider a league of hypothetical football, where every first down is spotted ten yards from the previous first down - i.e. if you complete a 20 yard pass on 3rd and 2, you only get to move two yards down the field. Let's stipulate also that we know empirically in this league that first downs are equally easy to convert at any point on the field - i.e. 1st and 10 on your own 30 is no more or less difficult to convert than a 1st and 10 in the red zone. So in this league, the number of first downs you need to achieve in order to score a touchdown is precisely determined by the number of yards away at the beginning of the drive. And since the probability of a first down does not change over the course of the drive, if we have an estimate of the probability that a given offense will pick up a first down against a given defense (let's call it p) then the probability of scoring from the other team's 30 is p^3, the probability of scoring from your own 30 is p^7. This means that the probability of scoring from two given points on a field is given by an exponential relationship determined by the ratio of the two distances to the end zone. The probability of scoring from your own 30 is equal to the probability of scoring from the other team's 30 raised to the power of 7/3.
What does this have to do with real football? The point is that every adjustment that we have to make in order to conform with actual football works in the direction of making a score from a longer distance MORE likely (in actual football) than predicted by the exponential model derived in hypothetical football. The primary factors are that 1) in real football, it is harder to move the ball in the red zone, as the opposing defense has less area to defend, and 2) the greater field distance affords a greater opportunity for the offense to pick up "residual yards" with each first down, thus reducing the number of first downs required for a score. (We also may need to adjust the implicit assumption in the hypothetical model that first-and-goal success probabilities conform to fractional exponents. But this adjustment - even if it works against the probability of a long-field score, which is by no means apparent - should be dwarfed in scope by the two primary factors listed above.) Therefore, the exponential approximation is a lower bound estimate for expressing the probability of a long-field score in terms of the probability of a short-field score.
In the present case, we can use this lower bound as another conservative assumption from the point of view of the case for the attempt to pick up the first down. We can also - again generously to the punting case - assume that all fourth down failures would leave the ball at the Patriots' 28-yard line, and that Chris Hanson would average a 42-yard net on a punt. This means that we now estimate that P(L) = P(S)^2.5, thereby giving us both of the Colts' scoring probabilities once we make a single judgment about how easily they would be able to move the ball against the Patriots' defense on this final drive. (This method depends on the assumption that the clock was not a significant factor, which seems to be unobjectionable. A drive requiring 70 yards in two minutes [with one time-out] for a team that is very comfortable with a no-huddle offense is hardly likely to fail because the clock hits triple zero.)
Our equation for the break-even point now becomes:
.33 + .67[1-P(S)] = [1-P(S)]^2.5
All we have done to get here is to 1) implement some assumptions that clearly err on the side of the case for punting, and 2) make use of a little mathematical manipulation in order to express the equation in terms of a single variable.
So now the million-dollar question: what is P(S)? Again, no one knows for sure - we can look at all the data we want for how often teams have scored a touchdown from 28 yards out in this situation.....but none of those teams were QBed by Peyton Manning at the peak of his powers, playing against an undermanned and worn-out defense, who clearly had no adjustments left to offer against the Colts' hurry-up offense. And the main point is this: the vast majority of those who rushed to criticize Belichick's decision almost certainly did so based on the assumption that a 4th-down failure would be a terrible outcome, leaving the Colts in great position to score. In fact, as can be easily verified by plugging values into the equation, the higher the probability that the Colts would score from a short field, the WORSE the punting option becomes.
How can this possibly be the case? The answer is that if it was going to be such an easy proposition for the Colts to move the ball 30 yards down the field, it was not going to be that much harder, following a punt, for them to move 30 yards, then another 30, then another 10. Only if there was a significant probability that the defense could stop them within any given 30 yard chunk of field would the exponential relationship add enough equity to the punt to make it the superior option. This is a classic case of the punting fallacy: that an 40 extra yards of field position must somehow make an enormous difference to the opponent's prospects of scoring.
Solving the equation gives a break-even point when P(S) = 0.77. Only values lower than this favor the punt - and remember that we made a whole host of punt-friendly assumptions even to get here. I don't pretend to know for sure that P(S) was equal to or greater than 0.77 - if you believe that P(S) was below this threshold, and that the decision was suboptimal as a result, then more power to you. We should remember, however, that Bill Belichick was in a better position than anyone to evaluate how likely the Colts were to score from the 28 against his defense. I maintain that in any case, 0.77 is a realistic enough threshold that the decision becomes immune from rational second-guessing - even if we believe that P(S) is, say, 0.6, how can we be confident that our estimate is superior to his? Further, I would be willing to bet that 99.9999% of those who condemn the decision believe that P(S) was considerably higher than 0.77. If this is the case, there is no mathematical substance to their argument - they are merely falling back on the extremely foolish conventional wisdom that the punt is always the "safe" choice.
We start by assigning variables to the relevant probabilities. Let's call the probability of the Patriots picking up the first down P(1), the probability of a Colts TD following a failed fourth-down play P(S), and the probability of a Colts TD following a punt P(L). Then our equation for finding the break-even point between the punt and the attempt to pick up the first down is:
P(1) + [1-P(1)] x [1-P(S)] = 1-P(L)
The left side of the equation contains the probabilities that of the Patriots winning the game following their fourth-down play: either by successfully picking up the first down [P(1)], or by failing to pick up the first down [1-P(1)] but preventing the Colts from scoring anyway [1-P(S)]. The break-even point occurs when this sum is equal to the probability of winning the game following a punt, which is essentially equal to the probability that the Colts fail to score from a longer field [1-P(L)].
Procedural note: we find the break-even point because it is easier to deal with equations than with inequalities. Once we have found the break-even point, it will be clear that deviations in one direction favor one decision, and vice versa.
This equation contains two assumptions which need to be addressed:
1) The Patriots are guaranteed to win the game if they pick up the first down. This is not quite true, but it is very close. With one time-out, Indianapolis's best-case scenario, barring a very improbable event, is getting the ball back with about 30 seconds and no time-outs, needing to drive for a TD. Following a normal punt, the chances of this - even for Peyton Manning - are truly miniscule. Their only significant prospects come from a fumble on a Patriots' running play, a blocked punt, or a long punt return. These are all very improbable events which do little to raise their win probability - especially when multiplied by the probability of converting such an event into a TD, which is still not a guarantee. Other analyses have used 92% as the Patriots' win probability figure here; this is derived from data in all similar score/down/distance situations, regardless of time-outs - a factor which makes a world of difference here. Using the 92% estimate errs on the side of caution, which is generally laudable. But here it seems like a truly gratuitous concession to the case for punting.
2) No outcomes need to be considered other than those considered in our equation. The main omission here is the chance that the Patriots win with a field goal following a Colts touchdown. This probability adds to the expected value of the fourth-down attempt, so the omission is generous to the case for punting (almost certianly by a larger margin than the 100% win probability estimate following a successful conversion helps the case for fourth-down attempt.)
To solve the equation, we need to supply values for P(1), P(S), and P(L). Since we can never be sure of the exact values, it is important to select very conservative estimates from the point of view of our ultimate conclusion. Many statistical analyses of the decision have used fourth-down conversion data to estimate that the Patriots had a 55-60% chance of success, and used this to show that the fourth down attempt had a higher expected win probability; while this may be a good estimate, it is susceptible to the following skeptical response: "I don't believe that the probability was really this high - the Patriots offense was looking shaky, the Indy defense was fired up, the five-receiver set was a poor choice, etc. etc.....therefore I reject the conclusion reached by your analysis."
Bill Simmons (The Sports Guy) provided a perfect example of this thought process in his espn.com article on the subject. He suggests that the probability was more like 1 in 3 - the rate at which offenses have successfully made two-point conversions with passes on the road in the last three years. This is obviously a ridiculously low estimate - he would have us believe that an offense featuring Tom Brady, Wes Welker and Randy Moss has 12% less chance of picking two yards on a single snap than the average NFL team does when they attempt a two-point conversion (a situation where the defense also has less field area to defend, further raising the offense's degree of difficulty.) Nonetheless, we will use his figure as our estimate of P(1), confident that none but the wilfully obstinate can now object that we are overestimating the Patriots' chance of success. This means that our equation is now:
.33 + .67 (1-P(S)) = 1-P(L)
Now we come to the crux of the entire issue: the underlying reason why so many people have misjudged this decision so badly (and why hardly an NFL game goes by where a coach or a commentator does not badly misjudge a punting decision.) How much does the Colts' probability of scoring change depending on whether they take over following a failed fourth down or following a punt? Unfortunately, there is an enduring irrationality baked into the manner in which conventional wisdom evaluates such questions, which can loosely be characterized as follows:
"Not punting is a huge risk. The other team may end up with great field position, and will probably score. Punting gives the defense a great chance to stop them."
No element of this statement is categorically false, but every element involves a stretching of the truth to the point that the ultimate conclusion is badly distorted from what is suggested by a dispassionate account of the probabilities involved. (Why have people been so inclined to accept stretching of the truth on this particular issue? I will address this in another post.) In fact, while punting does reduce the probability that the opponent will score, it does so by a much smaller margin than is commonly believed, particularly when dealing with a powerful opposing offense such as that of the Colts.
To see why, consider a league of hypothetical football, where every first down is spotted ten yards from the previous first down - i.e. if you complete a 20 yard pass on 3rd and 2, you only get to move two yards down the field. Let's stipulate also that we know empirically in this league that first downs are equally easy to convert at any point on the field - i.e. 1st and 10 on your own 30 is no more or less difficult to convert than a 1st and 10 in the red zone. So in this league, the number of first downs you need to achieve in order to score a touchdown is precisely determined by the number of yards away at the beginning of the drive. And since the probability of a first down does not change over the course of the drive, if we have an estimate of the probability that a given offense will pick up a first down against a given defense (let's call it p) then the probability of scoring from the other team's 30 is p^3, the probability of scoring from your own 30 is p^7. This means that the probability of scoring from two given points on a field is given by an exponential relationship determined by the ratio of the two distances to the end zone. The probability of scoring from your own 30 is equal to the probability of scoring from the other team's 30 raised to the power of 7/3.
What does this have to do with real football? The point is that every adjustment that we have to make in order to conform with actual football works in the direction of making a score from a longer distance MORE likely (in actual football) than predicted by the exponential model derived in hypothetical football. The primary factors are that 1) in real football, it is harder to move the ball in the red zone, as the opposing defense has less area to defend, and 2) the greater field distance affords a greater opportunity for the offense to pick up "residual yards" with each first down, thus reducing the number of first downs required for a score. (We also may need to adjust the implicit assumption in the hypothetical model that first-and-goal success probabilities conform to fractional exponents. But this adjustment - even if it works against the probability of a long-field score, which is by no means apparent - should be dwarfed in scope by the two primary factors listed above.) Therefore, the exponential approximation is a lower bound estimate for expressing the probability of a long-field score in terms of the probability of a short-field score.
In the present case, we can use this lower bound as another conservative assumption from the point of view of the case for the attempt to pick up the first down. We can also - again generously to the punting case - assume that all fourth down failures would leave the ball at the Patriots' 28-yard line, and that Chris Hanson would average a 42-yard net on a punt. This means that we now estimate that P(L) = P(S)^2.5, thereby giving us both of the Colts' scoring probabilities once we make a single judgment about how easily they would be able to move the ball against the Patriots' defense on this final drive. (This method depends on the assumption that the clock was not a significant factor, which seems to be unobjectionable. A drive requiring 70 yards in two minutes [with one time-out] for a team that is very comfortable with a no-huddle offense is hardly likely to fail because the clock hits triple zero.)
Our equation for the break-even point now becomes:
.33 + .67[1-P(S)] = [1-P(S)]^2.5
All we have done to get here is to 1) implement some assumptions that clearly err on the side of the case for punting, and 2) make use of a little mathematical manipulation in order to express the equation in terms of a single variable.
So now the million-dollar question: what is P(S)? Again, no one knows for sure - we can look at all the data we want for how often teams have scored a touchdown from 28 yards out in this situation.....but none of those teams were QBed by Peyton Manning at the peak of his powers, playing against an undermanned and worn-out defense, who clearly had no adjustments left to offer against the Colts' hurry-up offense. And the main point is this: the vast majority of those who rushed to criticize Belichick's decision almost certainly did so based on the assumption that a 4th-down failure would be a terrible outcome, leaving the Colts in great position to score. In fact, as can be easily verified by plugging values into the equation, the higher the probability that the Colts would score from a short field, the WORSE the punting option becomes.
How can this possibly be the case? The answer is that if it was going to be such an easy proposition for the Colts to move the ball 30 yards down the field, it was not going to be that much harder, following a punt, for them to move 30 yards, then another 30, then another 10. Only if there was a significant probability that the defense could stop them within any given 30 yard chunk of field would the exponential relationship add enough equity to the punt to make it the superior option. This is a classic case of the punting fallacy: that an 40 extra yards of field position must somehow make an enormous difference to the opponent's prospects of scoring.
Solving the equation gives a break-even point when P(S) = 0.77. Only values lower than this favor the punt - and remember that we made a whole host of punt-friendly assumptions even to get here. I don't pretend to know for sure that P(S) was equal to or greater than 0.77 - if you believe that P(S) was below this threshold, and that the decision was suboptimal as a result, then more power to you. We should remember, however, that Bill Belichick was in a better position than anyone to evaluate how likely the Colts were to score from the 28 against his defense. I maintain that in any case, 0.77 is a realistic enough threshold that the decision becomes immune from rational second-guessing - even if we believe that P(S) is, say, 0.6, how can we be confident that our estimate is superior to his? Further, I would be willing to bet that 99.9999% of those who condemn the decision believe that P(S) was considerably higher than 0.77. If this is the case, there is no mathematical substance to their argument - they are merely falling back on the extremely foolish conventional wisdom that the punt is always the "safe" choice.
Subscribe to:
Posts (Atom)