For the past thirteen seasons, we have been producing a Critical Call Index (“CCI”) to evaluate the cost of sub-optimal decision-making among NFL play-callers. The ZEUS simulation model is used to evaluate the relative merits of PAT, Kickoff and 4th down decisions and the subsequent loss of Game-Winning Chance (“GWC”) as a result of inferior choices. In each of these categories the model essentially treats the decision as a binary choice: Extra point vs two point conversion, regular vs on side kick, field goal vs first down attempt, punt vs first down attempt. With respect to first down attempts the model can choose an appropriate play for the required yardage (ie run, short pass, medium pass, long pass).
For each decision, the ZEUS model plays the game to completion 100,000 times beginning with each of the choices in question. Each of those simulated games is a unique game log and incorporates the customized characteristics of the opposing teams. The model is refined each year to accurately represent the current NFL rules, accommodate all game circumstances, and incorporate statistical distributions of actual NFL teams. In order to ensure the accuracy of the settings, the model is continually calibrated against empirical NFL data.
In the accompanying charts, there is both an actual (Graph #1) and normalized value for the cost of sub-optimal decisions. Actual costs can be heavily influenced by the competitiveness of the games evaluated. For instance, a poor choice on 4th down can have a far greater impact on GWC during the 4th quarter of a close game than a situation where a team is trailing by four touchdowns in the first half. By normalizing the conditions of each decision and accounting for the number of occurrences, we can get a somewhat fairer representation of the decision-making abilities of NFL coaches (Graph #2).
As in so many previous years, there is a great deal of parity among NFL play-callers. The annual cost in cumulative GWC among all 32 teams this past season varied between 0.34 and 0.74 games for the 16 game regular season with a mean cost of 0.53 games (Graph #1). The mean cost in 2013 was approximately 0.63 games with a range between 0.3 and 1.58 games. Those that recall some of our earlier CCI reports may recognize these figures as being a bit lower than in the past. This effect is due to a change in the calculation method, which previously incorporated pass vs run costs on 4th down. It should also be noted that situations where a team intentionally attempted to draw the opponent off sides on 4th and short and were assessed a delay of game penalty were not included in the final results.
While the statistical evidence supporting more aggressive actions on fourth down has been available for many years now, and despite the talk of advanced analytics emerging in the NFL, teams are still performing very poorly with respect to critical calls, regardless of overall records (Graph #2). During the past four seasons, the number of total fourth down attempts in the NFL is down about 10% from the prior four years (Graph #3) and the cost of sub-optimal decisions on all critical calls has stayed relatively constant.
No one would argue the New England Patriots are a very well coached team as evidenced by their 6th Super Bowl appearance and 4th win this past season. However, when it comes to critical calls, Bill Belichick finishes near the bottom of the NFL ranks (Graph #1 and #2). In each of the last two seasons, the Patriots squandered approximately 3/4 of a game in cumulative GWC during the regular season. They also ranked in the middle of the pack for aggressive (run or pass) fourth down actions (Graph #4). Again, as revealed in Oakland’s top ranking (Graph #1), the cost of inferior critical calls can be greatly influenced by a team’s frequency of close games.
The most important take-away from this study is the fact that all NFL teams are still costing themselves significantly by a general risk aversion on fourth downs. This aversion is also generally present on PATs and especially kickoffs where on-side attempts are not exercised as often as recommended. However, the costs of poor decision-making in these categories, pales in comparison to the cost of “taking the points” by kicking field goals in the red zone – a practice that has changed little during the past 13 years.