The Hidden Game of Football: A Revolutionary Approach to the Game and Its Statistics

By Bob Carroll, Pete Palmer, John Thorn and Aaron Schatz

The 1988 cult classic behind football’s data analytics revolution, now back in print with a new foreword and preface.

Data analytics have revolutionized football. With play sheets informed by advanced statistical analysis, today’s coaches pass more, kick less, and go for more two-point or fourth-down conversions than ever before. In 1988, sportswriters Bob Carroll, Pete Palmer, and John Thorn proposed just this style of play in The Hidden Game of Football, but at the time baffled readers scoffed at such a heartless approach to the game. Football was the ultimate team sport and unlike baseball could not be reduced to pure probabilities. Nevertheless, the book developed a cult following among analysts who, inspired by its unorthodox methods, went on to develop the core metrics of football analytics used today: win probability, expected points, QBR, and more. With a new preface by Thorn and Palmer and a new foreword by Football Outsiders’s Aaron Schatz, The Hidden Game of Football remains an essential resource for armchair coaches, fantasy managers, and fans of all stripes.

• Language: English
• Publisher: University of Chicago Press
• Release date: Feb 27, 2023
• ISBN: 9780226825878

Book preview

PART ONE

1

A Flying Wedge into the Future

At first we thought someone was kidding us, but the voice at the other end of the telephone insisted politely that it really belonged to a newspaper reporter calling from Denmark. We recognized an accent of some sort, but with our iron ear for nuances, it could just as easily have been Doughnut as Danish.

Your name was given to me by someone at the American Embassy as someone who can tell me about American football, the caller explained. Now it made sense. Not that we’re used to being recommended by the U.S. government for much other than filing our tax returns on time, but we do have a friend who does something or other for the State Department. The last time we’d talked to him, he was on his way to Europe.

What do you want to know? we asked.

Everything, he said.

He meant it. His editor had assigned him to write an article on American football, and he candidly admitted that about all he knew was the term Super Bowl. What he really needed was a brief but accurate history of professional football from its origins up to yesterday.

So we started with Walter Camp.

About the time we got to the 1950s in our telephone lecture, we were struck by a sense of irony. For nearly a year, we’d been spending most of our thinking moments at a keyboard trying to explain The Hidden Game of Football to what we expected would be rather football-sophisticated readers. Now, we were trying to come up with the words that would interpret a hundred years of football to an obviously bright but completely uninitiated listener.

As we rattled into the 1960s, panic replaced the irony. Had we gauged our readers correctly? Our listener on the telephone could ask for further explanation when we hadn’t been clear enough, but what could a reader do, stop in the middle of a chapter and send a telegram?

Ah, yes, the Super Bowl! said the Dane. We’d finally rung a familiar bell.

We rushed through the ’70s and ’80s, anxious now to get back to the keyboard.

And what about the strike? he asked.

It had been a disaster for the players, we said. And, although the fans came back to the stadiums and their TVs readily, the failure of the strike would put the players into the courts, which might ultimately be disastrous for the owners. And, we added, it’s had quite a strong effect on our own work. We didn’t explain that part to the Dane, but we’ll tell you about it in a moment.

After the Dane thanked us and hung up to write his story, we did a lot of thinking about you, the reader. Here’s what we decided:

You already know a great deal about football. You may have played it in high school, college, or professionally. Perhaps you’re a coach. Perhaps you’re a bettor. Certainly you’ve been watching games for years. You’re an informed fan.

So why are you reading this book? To find out morel

We think we can help you. We think that by showing you the statistical side of football, including some stats you’ve never seen before and some familiar stats used in new ways, we can increase your football fun. Stats can be a lot of fun. They give us a basis for evaluation and comparison of teams, players, and coaches. They may start more arguments than they settle, but that’s part of their appeal, too. We can’t teach you how to block or tackle, and we won’t diagram any plays, but by the time you finish this book, you should have a greater understanding and a deeper appreciation of what goes on between the goalposts.

The ideas we’re going to investigate should have some application to high school and college football, but we’re going to focus on the pros. That’s partly by choice; we’re pro fans. And it’s partly by necessity; pro football, despite some serious gaps, has a more complete statistical history. We’ll be traveling through sixty-seven years of pro football statistics here, but our primary target will be the last ten years. The game changes. New strategies, new concepts, and new rules make the averages and ratios of today very different from those of a couple of decades ago. What’s happening now?

To come up with our answers, we’ve used the statistical footprints of literally thousands of pro football games. We’ve calculated more than five hundred NFL games on a play-by-play basis. That’s more than two seasons’ worth. For specific reference, we’ll use all of the games played in the 1986 season.

Why not 1987? You’ve probably figured that out already. The strike. As we told the Dane, it affected our work. The use of replacement players for one-fifth of the season threw everything out of kilter. Some teams completely altered their personalities—and, of course, their statistical tendencies—while the replacements were on the field. And once the regulars returned, it still took some of the teams a couple of weeks to get back into character. The result is that we just don’t trust the numbers from 1987.

It may sound contradictory, but we think the 1986 stats are more predictive of 1988 than the closer year. The bright side is that we had more time to delve into the 1986 figures. And that extra time allowed us to go deeper in some areas than we originally planned.

We’re going to cover a lot of ground here.

Part One of this book deals with history and general concepts. That sounds a little dry, but we’re betting you’ll find it exciting.

In the next chapter, How Football Got That Way, we’ll give you History of Football 101, an off-tackle smash through the whole history of the game, starting from before it was football and ending with the 1980s. We’ll explain why the pro game depends predominantly on passing, how the two-platoon system changed the size of players, explore the effects of some significant rule changes, and generally prepare you for what comes next.

What’s Wrong with Traditional Pro Football Statistics, the third chapter, lets us cheer for the numbers that are available and complain about the ones that aren’t. Fortunately, we’ve been able to come up with some of the missing pieces. We’ll also begin our main mission: to make you a better interpreter of what you see.

Our handy-dandy computer has been essential in creating this book, and in Punch, Zip, Whirr . . . Touchdown, we’ll show you how the face of football has been reshaped by bits and bytes. As much as some old-timers long for the good old days when they made decisions by the seat of their pants, those days are as long gone as moleskin leggings.

The fifth chapter, Looking for a Winning Situation, is a lot of fun. We’ll take a play-by-play account of a real game—there’s an official record for every NFL game—and show how such accounts could create new and more meaningful statistics for individual players. For example, it’s quite possible to have numbers that actually measure a runner’s effectiveness in clutch situations.

In Driving for Glory, we’ll look at some real drive charts and show how you can estimate Win Probability (just what it sounds like—the probability of winning the game) by correlating the score, time left, and field position. Obviously, this is at the root of all strategic decisions, but that doesn’t mean every coach understands it. We’ll even look at a couple of extreme examples, a blowout and a game where the wrong team won.

With "You Can Get There from Here," the seventh and last chapter in Part One, we introduce the concept of point potential related to field position. We’ve examined hundreds of drive charts to arrive at some surprising answers. For example, we’ll show you why a team may be better off going for that first down than punting—even if it’s deep in its own territory. And did you know a successful field goal may actually cost the kicking team points?

In Part Two of the book, we’ll take the game apart and see how the various parts can be measured—and how much they contribute to the final score.

For example, in The Joy of Sacks, and Other Mistakes, we’ll calibrate the effects of fumbles, interceptions, and, naturally, sacks. We think they may be a bit less important than those glitzy TV graphics would have you think, but judge for yourself.

In Special Teams, we measure actual kickoffs (now where else can you find that?), punts, and returns. There are some simple things teams could do with these to improve their Win Probability. We’ll tell them about it.

Kicking Up a Storm gives us the opportunity to return to one of our favorite subjects: the importance of not kicking that field goal. Here we can go after it in depth. And did you ever wonder what happened to the quick kick?

The longest chapter in the book is The Glory, the Blame, and the Ratings, but that’s not our fault. You see, before we could show you how passers should be rated, we had to explain how they are rated. Joe Montana says he doesn’t understand the NFL’s Passer Rating System, but by the time you get through this, you will. And you’ll know how it can be fixed.

Ever hear of YPSG? It’s an easy way of looking at the records of running backs that tells you more than traditional totals. You’ll read all about it in The Running Game.

Of course, no one runs or passes very well without a good, strong—but always anonymous—line in front of him. In Opus for the Unsung, we’ll show you the good, the bad, and the ugly among NFL lines.

Everybody has heard of the two-minute drill. We’ll explain why it’s really a minute longer, and how teams could better use that time, in Three Minutes to Fat Lady.

We end Part Two with Profile of a Winner, where we put all the parts back together and see if it’ll fly.

The third part of the book takes on some of the more exciting and controversial side issues. These are the kinds of things fans argue about until the early morning hours, and we’ll give them plenty of ammunition.

The Draft Dodge tells why the event that’s been described as a new national holiday is really a rip-off, raising false hopes of positive change while preserving the status quo. Can it be improved? Yes. Will it be? Probably not.

Pro football games are America’s most popular wagering events. From friendly backyard bets among buddies to serious bettors, everybody’s looking for an edge. In Wanna Bet? we’ll give you a primer on how it works, tell you what to look for, and suggest some ways you can improve your odds of winning.

Debates over who was the greatest never lose their fascination, but unless you know how to interpret the numbers, you won’t win many scuffles. Players from ten, twenty, and thirty years ago can be compared to today’s stars, but not by simply putting one number next to another. Jim Brown, Don Hutson, Sammy Baugh, and others who thrilled the fan of yesterday will always lose in that kind of confrontation.

In Great Performances, we’ll look at some of the best (and a couple of the worst) efforts ever made on a football field. And in Rumblings in the Pantheon, we’ll see how some players who were the very best of their time have been ignored by the Hall of Fame and name a few inductees and candidates who have questionable credentials for enshrinement.

And finally, in The Tables, we’ll give you a season-by-season review, along with the most significant numbers for each year.

As you read The Hidden Game of Football, you will find many of your suspicions confirmed and many of your confirmed beliefs challenged. That’s what makes football the great game it is. Long after the final gun, we can investigate, analyze, and just plain enjoy our experience. Yogi was wrong; the game is never over.

2

How Football Got That Way

There’s probably no chance at all that they’d consider changing their name to the National Armball League, but that would be a lot more descriptive of the NFL’s version of America’s Game than the present foot-ball misnomer. The colleges play foot ball, using the run as their regular modus operandi. The pros live and die by the pass. And while a foot is a necessary item for any passer, neither the left nor right is the featured part of his anatomy. The last NFL quarterback to win a job with his feet was the Bears’ Bobby Douglass, and that mistake happened in seasons they’d just as soon forget in Chicago. As far as the pros are concerned, the kick and the run—two efforts showcasing the feet—are definitely secondary to what happens in the defensive secondary.

Cynics think the NFL emphasizes passing only to improve its TV ratings. They insist the pros are afraid to run the ball as God intended because viewers will get bored and switch to Cagney & Lacey. Throwing—the football version of the cop-show car chase—is more exciting to watch than running and keeps the viewers’ hands off the dials. The cynics are wrong, of course. The pros throw because they have to.

Any pro team that can’t put the ball in the air with a high percentage of satisfactory results can count on an early position come next Draft Day. The same can be said for a team that treats defending against the pass as a lost art. Purists sneer that Rozelleland is beginning to look like a basketball league, but don’t hold your breath awaiting a change unless you look good in blue. It’s going to be armball for the NFL until Punky Brewster gets her pension.

Just how pro football changed from a game based on what players did with their feet to today’s celebration of the fly pattern is worth looking at.

A SIMPLE GAME

Actually, the object of the game is very simple: move a previously agreed upon item (a football) a previously decided distance (one hundred yards) within a previously set time period (sixty minutes) more often than your opponent. One side tries to move the item; the other tries to stop them. Pro footballers move their oddly shaped item best by throwing it; college teams excel at running with it. If someone demonstrates that the ideal way to get a football from here to there is by mail, all the coaches will buy stamps the next day. Move the item. In its basics, football is no different from basketball, hockey, water polo, or several other games.

A game is played within a prescribed space—a basketball court, a hockey rink, a swimming pool, or a football field. The first three come in preferred sizes with some variations; a football field these days is always 100 yards long and 53.33 yards wide. There are always the same number of players on each side, depending on the sport—five in basketball, six in hockey, seven in water polo, and eleven in football—according to what works best. Put twenty-two men on a basketball court and they’d keep bouncing dribbles off other guys’ toes; put that many in a swimming pool and somebody would