I had the pleasure of spending a few minutes with Aaron Schatz, editor-in-chief of Football Outsiders, one of the best independent football sites on the internet. In addition to ground-breaking statistical analyses, Football Outsiders also publishes great hardcore football articles, breaking down the intricacies of play design, offensive line work, and more. If you like the technical side of football, you need to be reading their website. For an example of some of their technical football analysis, scroll down to the bottom of this article from January of this year, where Mike Tanier breaks down part of the Saints offense.
You can read an excerpt from my interview with Aaron after the jump. To hear the entire, unedited interview, listen to this week's Who Dat Report.
Stuart: For those who aren't familiar with it, your main statistic is called DVOA. What is DVOA?
Aaron: DVOA stands for defense adjusted value over average, although it's not defense adjusted when we're measuring defenses or special teams. It's sort of a leftover from the beginnings of the site, but that's what we call it, and that's what we've called it for seven years. The basic idea here is that it takes every play of the NFL season, and compares it to an average play based on situation and opponent. So that is the nutshell explanation. We give every play a success value that's based not only on how many yards it gets, but how many yards it gets towards a first down. Because, for example, a three yard play is a good play on a third and two, and it's not a good play on a third and ten.
Aaron: Then we compare that to how other plays have done in that situation, that place on the field, adjusting for the score. Obviously at the last two minutes of the first half, defense will let the offense get more yardage, because it doesn't matter; last few minutes of the game, depending on what the score is. And then we adjust for opponents as well. Special teams are done a little differently, but we end up with a percentage that is how much more efficient each team is compared to the NFL average. And it's a good power rating, but it may not be the best overall power rating out there, to be honest. One interesting thing is a lot of the sort of advanced power ratings get similar results. But what makes DVOA special is that because it is done a per play basis, it can be broken down per play in any split you want. So it's not just a power rating that ranks teams from one to 32. It's more useful as a way to look at match ups, because football is a game of match ups. And you can look at when teams succeed as far as down and distance; when teams succeed as far as area on the field; are they better in the red zone; which players are doing better; and so forth. So that's the main stat on the site.
Stuart: And so, DVOA considers each individual play, not the drives. Can you sort of use it to predict game outcomes as well, or is it not so good for that?
Aaron: Yeah, I mean we have some drive stats on our site as well. And we do use the DVOA and the other stats to predict games. As part of our premium content, we pick games against the spread each week. That equation is a little more complicated than just comparing the two teams' ratings. It uses a couple specifics, like Splits, and obviously it take things like home field into account, and takes the fact that the domed teams and the warm weather teams have a harder time on the road in December. But, in general, it does a good job of figuring out which teams have sort of been screwed by the luck fairy, and are better than their record might otherwise indicate, and which teams have been dancing with the luck fairy and are not as good as the records might otherwise indicate, for example the New York Jets.
Aaron: Who are a good team. I'm not saying they're not a good team, just going into last week, it was pretty clear they were not as good as the Patriots, and obviously I did not expect to be shown that on the field to quite the same extent that it was, but it was pretty clear the Patriots had played better this year.
Stuart: Yeah. Now this is a family (website) so I don't want to go into too much detail, but frankly, getting screwed by the luck fairy sounds OK to me.
Aaron: Well that's the bad luck fairy. I refer to it that way.
Stuart: She looks a little different than maybe the good luck fairy does.
Aaron: Yes. I'm sure the Saints fan know about years of being screwed by the luck fairy, although things have gone pretty well the last two seasons.
Stuart: Yeah, we do. And we'll get into Saints specifically in just a minute. I do have one more question about DVOA, generally. And so when you were picking, how did you decide what variables to include in sort of this idea of success rate? Did you just pick that out of a hat? How did you go about deciding?
Aaron: It starts out with the book "Hidden Game of Football", by Pete Palmer, Bob Carroll, and John Thorn. It came out like 20 years ago at this point. And they have the success points formula in the book. It's been adjusted. I've adjusted it since then to try and maximize how well we measure winning in the modern game, but that's where it starts out with. The idea of one point for a successful play being 40% of the yards on first down, 60% on second down, 100% on third down, extra points for longer, negative points for turnovers ...
It’s not question of variables because here's the thing: DVOA is not a statistic that takes a number of other statistics, puts them into a big sausage making formula, and spits out a number. What DVOA looks like is a big spreadsheet where every one is a different play, and it looks up an average for that play based on a number of different variables, like game situation, location on field and so forth. But it's not like you know, in baseball you've got OPS. OPS takes one stat, which is slugging percentage, and another stat which is on base percentage, and those two are figured out in a certain way. And then you add them together and that gives you OPS. You're not looking at individual at bats one by one when you figure that out.
Aaron: So DVOA is not like that. Instead, it's looking at individual plays one by one.
Stuart: Right, so that's a little different than say like what they, the generic win probability of AdvancedNFLStats.com or something...
Aaron: Right, or what the simple rating system does at (Pro-Football-Reference.com).
Now, here's the thing. When you look at the overall power rating I think that simple rating system and (Advanced NFL Stats’) thing, and Sagarin and us, you're gonna end for the most part with the same teams coming out as overrated and underrated. My guess is that all of these sites say that Jacksonville is a little overrated, or you know, Kansas City is not as good as their record because of their schedule. And probably all of these sites feel that San Diego is better than their record. The advantage of DVOA is that you can look at splits of you know, by down, or zone, or whatever, to look at match ups in future games...
Stuart: Let's jump into some of the specifics. Talking about the Saints then, overall the Saints have a good record this year, but there've been some close games. How does DVOA rate the Saints this year?
Aaron: We have them ninth as of this week. They've been rising steadily throughout the year. One of the things that we have is a stat called "weighted DVOA", which drops the importance of earlier games, and makes the more recent games stronger. And the Saints are a team that does better in that stat than it does in you know, total for the season because they've played better in the second half of the year. They've benefitted from one of the easiest schedules in the league, by our account only St. Louis has had an easier schedule so far. But obviously they've played really well against that schedule. We have them seventh on offense right now, eleventh on defense, and 26th on special teams.
Stuart: So how do the Saints stack up against the rest of the NFC then, because you know, that's more important than the AFC really. How are they looking compared to say the Falcons and Eagles, and those other teams.
Aaron: We have them fifth in the conference. Now the Falcons I know are 10-2 and obviously have the inside track to the number one seed. And right now they are the best chance to win the Super Bowl out of any of the NFC teams because of the value of home field in the playoffs. But we actually have the Eagles, Packers, and Giants all higher than the Falcons, although those teams are close together. And then we have the Saints a little bit behind the Falcons. Then there's a big gap where the Saints are ninth, and the next NFC team in our ratings is Tampa Bay, which is seventeenth.
Stuart: And probably falling fast.
Aaron: So, this has been the situation for a couple of years. And it really is true this year, which is that the AFC is a conference where there's much more quality between teams, even though the Patriots and Steelers have been very dominant this year. The AFC has more kind of good teams. The NFC is very stratified between really good teams and bad teams. And so the way we have our numbers this year, there are really five good teams in the NFC. I mean Chicago has worked their way to some wins and their rating is probably a little lower than it should be because of the one game where Todd Collins and Caleb Hanie had to play quarterback. But in general the good teams are the Eagles, the Packers, the Giants, the Falcons, and the Saints. And unfortunately one of those teams will very likely miss the playoffs because of the stupid NFC west.
Stuart: Yeah. And with that…Let’s actually talk about that, the playoff odds. You also do this cool playoff odds simulation which I read week to week and follow as if it were the stock market. How do you do these playoff odds simulations and how are they looking for the Saints?
Aaron: It’s an equation that takes what the current ratings are. Early in the season we use a rating that combined our pre-season projection with the early games. By the time you hit this point in the season we’re using the weighted DVOA so that we’re dropping the importance of the earlier games and mostly looking at the last ten weeks of the tournament. Then it figures out the likelihood that the teams will win based on the ratings of the two teams and whose at home and the December thing for the warm dome teams etc. It played out the season 10,000 times assigning wins and losses based on those odds. Right now we have the Saints with the 5th highest playoff odds in the NFL at 89.6%. That’s because the way that the records are, plus the quality of the teams, it’s virtually assured that the two teams from the NFC South are going to make the playoffs.
Aaron: Just like it is pretty much virtually assured that the two teams from the AFC East and the two teams from the AFC North are going to make the playoffs. However, the Saints have a very small chance of winning the division. We have them only with 12% chance of winning the division.
Stuart: Yeah. What about the Saints (offensive) line, actually? Subjectively, they’ve seem not so good this year. A lot of penalties. Now penalties are up NFL-wide. That might be because of the umpire moving into the backfield…
Aaron: Right. Offensive holding calls are up. I happen to think…I don’t have the penalty numbers in front of me…But by the other numbers we use the Saints may be the best line in the league. We have a stat called Adjusted Line Yards that looks at runs but cuts off yards when you get to a certain length. Because when you get past a certain amount, it’s more about Reggie Bush and Chris Ivory in the open field, not the blocking. When you look at that stat, the Saints are seventh.
(We also) look at runs and short yardage, when offensive line is most important, and they're ninth. When you look at sack rate, adjusted for situation / opponent, they’re 5th. Some of that is Drew Brees, and his ability to get the ball away, and he doesn't take sacks. But some of it's the line. So they were better than last year even in all those stats. So the Saints definitely when you look at things -- let's not just look at one year, but let's look at, you know, how they've been over the last two or three years. They're one of the top three or four lines.
Stuart: What can you help us tease out with the defense, because they started out strong and they've really been struggling lately it seems. Has there been a change in the defense, or is it just sort of fluctuation? How have they looked?
Aaron: I don't know. We have them roughly the same. I mean, I don't think it's really changed that drastically on defense. You know, weeks 11 and 13 they didn't play that well, but they were pretty good in 8 and 9. (Scanning notes)…Pretty good in 6 and 5...
Stuart: One of the great things you do is help people remember that you know, there are all of the things you see and hear talked about, like sports talk radio, that go on in a football game; but there are a lot of hidden factors that affect a team's success. Things like injuries, schedule strength, random variation, and all of that. And I think the Football Outsiders stuff really helps people to remember that and see what's going on there.
Aaron: That's a big goal. I mean a lot of the goal of what we're trying to do is to sort of filter out the effect of context from the numbers. People often get it wrong about "stats guys", they think we're trying to surpass scouting or replace scouting. That's not true. Everybody who looks at football looks at both numbers and what they see with their eyes. Numbers are stats. What you see with your eyes is called scouting, whether you're doing it on a bar stool or you're an advanced scout or head coach. The problem is that the conventional stats suck. Okay? So we are here to improve that half of the discussion. But you know obviously, scouting things are still important.
Football is a very different game. Baseball is where the stats thing really came about. Now it's really spreading into football and basketball. But both me and the basketball guys have to deal with the issue that in baseball you can drop anybody into a rotation and drop anybody into a lineup, and that'll hit the same, pitch the same, park factor is really the only thing that matters. With football and basketball you have got to consider the style of the team. And in football that's even bigger than in basketball. You know, Jabari Greer is a great example, as is Jason David. Jason David, good corner in Cover-2 systems, horrifying playing man. Jabari Greer, not that good in Cover-2, great playing man.
Aaron: And you have to know these things when you're moving players around. That's why you can't you know, come up with some sort of uber-statistic that tells you the value of a player or you know, the dollar value of a player on any team, because the dollar value of a player will change you know, based on what team they're on.
And I also happen to believe that things like chemistry, and leadership, and motivation do exist. We don't write about them, we don't try to come up with stats about them because they're intangibles, that's why they're called intangibles, because they're not tangible. So, we just assume they're out there and we measure what we can measure.
And I'll also point out, not only do we do all the stats on Outsiders, but we have a lot of articles that break plays down. Today, for example, we have a big thing breaking down some Sam Bradford film that Doug Farrar wrote. We do college stats now. We're really expanding in that area. And I've gotta admit that the NFC South may be the worst represented conference, the worst represented division on our message boards. We don't have a lot of Bucs fan, Falcons fans, Carolina fans…we have more Steelers, Patriots, and Eagles fans than you can shake a stick at, but...
Stuart: Well, Aaron, we do appreciate your (being interviewed). And folks, if you're not reading Football Outsiders, at least checking in on the stats but also these other articles, then you're missing out. This year they got a lineman to come and write articles on line play that are great. This article, Walkthrough, every week where Mike Tanier…
Aaron: Mike Tanier is the funniest man writing about football.
Stuart: That is true, and also one of the smartest. His play (breakdown) stuff in addition to his funny stuff is as good as it gets.
Aaron: You just wait for this week. If you guys are looking for a way to make fun of Atlanta before week 16, Walkthrough is going to be awesome for you this week…three words, baby -- Real Housewives of Atlanta. That's four words, but one is a really small word.
Stuart: Well, I will wait for that with bated breath.