How Good Are the Saints?

Ranking NFL teams is folly. Since each team doesn't play each other team, we have to use guesswork, assumptions, and a bit of voodoo to try to figure out who is better than whom. However, we're headed toward the playoffs, and I thought it’d be worthwhile to try to figure out just how good the Saints are. My gut tells me they’re in the top handful of teams in the NFC, and somewhere in the second tier of teams in the NFL, but let’s see if we can move beyond gut feelings after the jump.

There are many ways to evaluate a team’s strength. One way is to look at their record, compare it to other teams records, and leave it at that. That’s the Bill Parcells "you are what your record says you are" approach.

Looking at records, the Saints are tied for 2nd in the NFC (with the Bears, behind the Falcons) and tied for 3rd in the NFL.

While the Parcells approach is satisfying (and ultimately correct in the sense that a 7-win team does have 7 wins), it’s not a particularly good way of comparing teams. After all, an 8-8 team with an easy schedule is not the same as an 8-8 team who has played the hardest schedule in the league. If we want to really compare teams, I think we need to move beyond record and consider the context in which that record was achieved.

The same goes for statistics. There are some great statistics that measure the quality of NFL teams. I’m partial to Yards Per Attempt (after I make my adjustments for sacks and interceptions), but there are plenty of others that are even better (though harder to calculate and understand). However statistics are of limited value if you don't take context into account.

My contention, and the basis for this article, is that statistics, wins, losses, etc., in a vacuum are enlightening, but only tell part of the story. By looking at the conditions under which those statistics, wins, losses, etc., occurred, you can increase your predictive power.

In other words: wins, losses, and stats without context do a great job of describing how a team has done in the past. However, if you are trying to predict how a team will do in the future, or against other teams, then it helps to go beyond wins, losses, and basic statistics. You need to look at the context in which those wins, losses, and basic statistics occurred.

The problem is that it's difficult to account for context. Fortunately, there are a few websites out there that do this for us. So, we can look to those websites, combine them with our own subjective judgment, and use that to get a sense of how good the Saints are. That’s what I plan to do here.

One other disclaimer: I'm aware that these statistical measures aren't perfect. Statistical analysis in the NFL will never catch up to baseball: the game is too complex. However, I think that looking at these sorts of analyses gives you a much better idea of what's going on than does looking at the various "power rankings" at sites like SI.com, ESPN.com, etc. If you disagree, that's fine, you can let me know about it in the comment thread. However, you should also let me know where you rank the Saints among NFL teams (and especially NFC teams) and why. I'll publish the best comments in this week's 4th and Geaux.

Football Outsiders

The first site I look to when judging teams is Football Outsiders , which is one of the more popular "advanced statistics" sites in the NFL. While they aren’t perfect, they are about as good as it gets for overall NFL analysis.

FO’s primary way of measuring team quality is through their Defense-Adjusted Value Over Average (DVOA). Without getting too bogged down in the details, DVOA basically assigns a value to each play run and compares the success of that play to how the league has performed in similar situations. This value is then adjusted for the quality of defense faced.

While no statistic is perfect (that’s why they play the game, right?) DVOA is a pretty good stat that correlates well with winning. In fact, if you use DVOA to pick games against the spread (after making a slight adjustment for home field, weather, etc.), you’ve done quite well for yourself over the last several years. This year, the Football Outsiders premium picks are 106-80-6 vs. the spread. That would place the Football Outsiders premium picks in the top 17% of pickers in the famed Las Vegas Hilton Supercontest with one important difference: the DVOA picks are for each game, where the entrants in the Supercontest get to choose their favorite 5 picks each week.  (NOTE: You really shouldn't bet on football, even with DVOA or other advanced metrics.). DVOA is also highly correlated with wins, much moreso than the adjusted Yards Per Attempt that I talk about in 4th & Geaux.

According to DVOA, the Saints are the 5th best team in the NFC and 9^th in the NFL*. FO’s stats have the Saints offense ranked 7^th in the NFL, defense ranked 11^th, and special teams ranked 26^th.

*Interesting note: if the Saints were in the AFC, they'd be ranked 5th in the AFC, as well.

That offensive rank struck me as a bit low, but then I remembered that it's a season-long measure, and takes quality of opponent into account. While the Saints' offense has been on a tear lately, they weren't particularly good at the start of the season*. Their offense is improving: when I wrote the first draft of this article earlier this week, the Saints' offense was ranked 14th in the league! Hopefully Reginald will round into form and the offense will continue to improve as the playoffs approach.

*That's a critical point to remember when looking at some of these computer-generated rankings: they take the entire year into account, not just how a team is doing currently. The advantage of doing so is that the rankings are less likely to get screwed up by the week-to-week swings that happen in any NFL season. The disadvantage is that the rankings may be a bit late to pick up on trends. In other words: it's good to ignore the week-to-week fluctuations, unless the week-to-week fluctuations are a sign of genuine improvement. The hard part is knowing which is which.

Ignoring the AFC (the comparisons become too hard), who does Football Outsiders have ranked higher than the Saints in the NFC? In order, it’s Philadelphia, Green Bay, NY Giants, and Atlanta. Football Outsiders has Philadelphia ranked as the best team of the NFC by a pretty good distance, Green Bay and the Giants almost identical, and Atlanta and New Orleans almost identical. The next teams in the NFC are way back in quality.

Now, whether or not you agree with these rankings, take a second to understand the logic behind them. The Falcons and Saints are ranked behind the other teams for two primary reason: quality of victory (in the case of the Falcons and Saints, who have had a lot of close victories…particularly the Falcons) and quality of schedule. According to FO, the Saints have had the second easiest schedule in the NFL, where as Philly’s schedule is ranked 19^th, Green Bay’s 21^st, New York’s 27^th, and Atlanta’s 10^th. The Saints’ opponents have been much easier than the other teams’, yet they still haven’t looked great against many of those opponents.

Subjectively, these rankings strike me as approximately right, although I don’t know what to make of the Giants. They don’t seem all that great to me, but I might be wrong. 

Advanced NFL Stats

AdvancedNFLStats.com is an interesting site run by an interesting dude. It’s by far the most statistically complex of the NFL sites I’m aware of, in that it uses advanced, technical techniques (regression, simulation, tests of significance, etc.) to analyze the NFL. As someone who has taken way to many statistics in graduate school, Advanced NFL Stats gets my nerd parts tingling.

The basic stat that Advanced NFL Stats looks at is called Generic Win Probability (GWP), which is essentially the probability that an NFL team would beat an "average" NFL team on a neutral field. Brian Burke (former Navy pilot and Advanced NFL Stats Guy) calculates this based on a variety of efficiency stats, including offensive pass and run efficiency, defensive pass and run efficiency, interception rates, fumble rates, and penalty rates. He then makes opponent adjustments (the context that I’m looking for!) and derives GWP from a bunch of mathematical voodoo. Overall, Burke claims that his model correctly picks about 69-70% of games. Not bad: GWP does a better job of choosing (straight-up, not against the spread) winners than just picking the favorite does. It also does a better job than most people do in "pick em" pools.

Through week 13, Advanced NFL Stats has the Saints ranked 5^th in the NFC and 15^th in the NFL. That’s about what I expected in the NFC, but a bit lower than I’d expect in the NFL.

Let’s look at the NFC teams. GWP ranks Green Bay as the best team in the NFC by a pretty good margin, followed by the Giants, Philadelphia, Minnesota, and Chicago. The Saints are barely ahead of Dallas (which might explain why the Thanksgiving game was so close) and are significantly ahead of Atlanta and Tampa.

These rankings are slightly different from my subjective feelings. Subjectively, I think that we’re significantly better than Minnesota and pretty close to Atlanta. I have no idea about Chicago and the Giants (they both seem to be so hot and cold), and think we’re worse than Green Bay and Philadelphia.

The model used by Advanced NFL Stats indicates that the AFC is a LOT better than the NFC. I don’t know for sure, but my gut feeling is that the stats are right, based on the games I've seen by the top teams in each conference. However, you could convince me otherwise (reader HB-NOLA points out that the AFC/NFC matchups have been very even this year).

Cold Hard Football Facts

ColdHardFootballFacts.com is the most simplistic of the "advanced" NFL stat sites that I know of. It has a metric poop ton of stats, but few of them account for any sort of context. However, some of the stats are so fun that I think they’re worth looking at, anyway.

First are the "Quality Wins". CHFF calls a quality win any win against a team that has currently has a winning record. For example, the Saints' victory over Tampa Bay in week 6 counts as a Quality Win because Tampa has a winning record. However, if Tampa loses out and finishes 8-8 or 7-9, then that victory will no longer count as a Quality Win.

I mentioned above that I think a team’s record is a flawed way of viewing a team, but it’s still better than nothing. Quality wins is a limited measure of how good a team is*, but it's a fun statistic that (unlike GWP, DVOA, etc.) is easy to understand and can be a pretty good measure of a team's strength

*Some of the problems with Quality Wins: An 8-8 team and a 7-9 9-7 team might not be all that different, but one of them counts as a Quality Win and the other doesn't. Additionally, teams might be significantly different early in the year than later (for example, due to injury), but Quality Wins doesn't take that into account. Also, there’s some evidence that it’s more important how you play against bad teams than how you play against good teams

The Saints are 2-1 (67% winning percentage) against teams with winning records, which is second in the NFC, behind Atlanta (5-2, 71%), tied with Chicago and the Giants (2-1, 67%) and ahead of Philadelphia (3-2, 60%) and Green Bay (2-2, 50%). If you look at the AFC teams, the Saints would be in second place, behind the Patriots who are 3-1 against teams with winning records.

Two things are striking about that statistic: (1) the Saints have done better than I thought against teams with winning records, and (2) the Saints haven’t played anybody with winning records, but neither has any other NFC team!

Peeking back at the schedule, their victories are against the Steelers (which I think of as a very good team), and the Bucs (which I think of as a pretty average team), and their loss is against the Falcons (which I think of as a good, though overrated team). So, while I’d rather have a winning record against teams with winning records than not, all this stat really tells me is that we’ve beaten the Steelers. That’s good, but not particularly revealing.

So, I’m buoyed by the Quality Standings, but what they really tell me is that the NFC has a lot of fair-to-middling teams, or at least very few dominant teams.

Let’s look at some of the other CHFF stats. Now, these are stats you have to take with a grain or two of salt, because they don’t even pretend to take quality of opponent into account. However, they do teach us a bit about the Saints, so we’ll go through them in rapid-fire fashion to see if we can find a trend.

Bendability Index: The Bendability Index is simply Yards Allowed/Total Points. This statistic gives the number of yards per point allowed is attempt to measure how good teams are at preventing other teams from scoring. The higher the number, the harder it is for opponents to score on a team.

The Saints are ranked 9^th in the NFL in "Bendability", with an average of 17.13 yards per point allowed. That’s good for 4^th in the NFC, behind Green Bay, Chicago, and Atlanta.

Scorability Index: The Scorability Index is similar to the Bendability Index, but it measures how efficiently a team scores points. To calculate, just divide Offensive Yards by Total Points. In other words, you’re measuring yards per point scored. The lower the number, the more efficiently a team scores points.

The Saints are ranked 23^rd in "Scorability", requiring an average of 15.99 yards per point scored. That’s 9^th in the NFC.

Passer Rating Differential: Passer rating differential is simply the difference between a team’s offensive passer rating and defensive passer rating (that is, the passer rating that the defense allows).

The Saints offensive passer rating is 92.74 (9^th in the NFL), their defensive passer rating is 79.26 (9^th in the NFL), and their differential is +13.48, which is 8^th in the NFL and 4^th in the NFC.

Back to the original question: How good are the Saints?

Before this article, my gut feeling told me that the Saints were about the 10^th-best team in the NFL, 4th or 5th in the NFC. Looking at the statistics, it looks like I was in the right neighborhood. Most of the advanced analyses that I've read point to the Saints being one of the best teams in the middle of the pack, league-wide and about the 4^th or 5^th best team in the NFC. That’s significantly worse than their record would indicate, but I don’t think record is a good predictor of future success compared to some of these other metrics.

For the fun of it, here's a quick list of the power rankings from Football Outsiders, Advanced NFL Stats, and a few popular sports websites:

• Football Outsiders: 5th in NFC, 9th in NFL
• Advanced NFL Stats: 5th in NFC, 15th in NFL
• Cold, Hard, Football Facts Quality Standings: 2nd in NFC, 3rd in NFL
• ESPN.com: 2nd NFC, 5th NFL
• SI.com (Don Banks): 2nd NFC, 5th NFL
• National Football Post: 2nd NFC, 4th NFL
• CBSSports.com: 4th NFC, 7th NFL
• FoxSports.com: 2nd NFC, 3rd NFL

While there is a lot of variation the Saints NFL-wide rankings, the NFL-wide rankings aren't that important. After all, the Saints play the Ravens in a couple of weeks, but won't meet another AFC team until the Super Bowl, should we get there. Focusing on the more important NFC rankings, we can argue for hours over whether the Saints are the 2nd-best team, 4th-best team, or whatever, but it doesn't really matter. The consensus seems to be that the Saints are in the top handful of NFC teams.

After reading the sites, the rankings, and the articles, and combining it with my subjective judgement based on watching the games, talking to people for my podcast, and writing my weekly columns, I think the Saints are probably 4th or 5th in the NFC. That would make them a decent-to-good team, but not a very-good-to-great team.

So, what does that mean? It means the Saints are definitely Super Bowl contenders, but will need to beat some teams that are probably better than they are in order to get to the championship. Given the way the Saints have played this year, I don’t have a lot of faith that they’ll do so. However, I’ve been wrong before (back in the early 1980s…long story), and I hope that I am again in this case. The Saints certainly have been playing better on offense over the last several weeks, but I'm concerned that the defense and special teams are hamstringing us.

That kick that Garret Hartley missed against Atlanta looms large over our season. It would be a lot easier to beat teams that are better than us if we played them at home, which we likely won’t have the luxury of doing.

If I had to turn this into a prediction so that you can ridicule me when it’s wrong, I’d say the Saints look like they’ll get to the second round of the playoffs (after all, their first opponent will probably be the NFC West champion) and struggle to move beyond there. If they continue to play like they’ve played so far, I don’t think they’ll make it to the NFC Championship game. If they get on a hot streak (or if Reginald and PT's return make a big difference), though, the sky is the limit. This team definitely can repeat, but at this point, I don't think that they will.

(You see, that way I can say I’m right now matter what happens. I'm not stupid!)

You can listen to Stuart each week on The Who Dat Report and The NFC South Report or follow him on Twitter. An earlier version of this article appeared at WhoDatReport.com, but this version is better. Trust me: I wrote them both.