All opinions are my own and do not necessarily reflect those of Novo Nordisk.

The NFL draft is happening as I am writing this post. And of the many draft-related pieces I’ve read in the past few days, one from Vox.com particularly stood out. The article, by Joseph Stromberg, describes research by Cade Massey and Richard Thaler (here and here) about the skewed and irrational choices often made by teams during trades of draft picks. In essence, teams are likely to pursue a strategy in trading up that suggests they believe they have a much greater ability to forecast the future performance of a given player than is actually the case. Put another way, rather than following a strategy of diversified risk, teams commit to a specific player that they feel they need to get, rather than simply seeing who’s available when they are scheduled to pick and choosing the best player on their draft board.

Historical analysis shows that the difference between various players drafted at the same position is often negligible; on top of that teams who aggressively trade down and gather more picks in the lower rounds generally do better in terms of the value they receive for the money they spend in salaries. One might argue this is an artifact in part of the NFL Rookie salary structure, but even without that, players taken in later rounds will always command smaller salaries. Getting similar value for less money is generally a good thing.

If you’ve read posts from this blog before you know where I’m going. Drafting NFL rookies sounds a lot like developing drugs.

When I read the article in Vox (and I also recommend an article on a related theme by Neil Paine at fivethirtyeight.com), I was reminded of a post at the blog Drug Baron which divided drug development strategies, roughly, into what the blog called “pick the winners” versus “kill the losers.” In the former strategy, a company decides early in the development cycle that a given target is a winner. The company then places large amounts of resources and emphasis on a few targets. The latter strategy instead suggests taking an extremely broad initial approach, but being ruthless in pruning the pipeline as potential therapeutics progress.

In statistical terms the first strategy lends itself to Type 1, or false positive errors. The second, to Type 2, or false negative errors. In the first strategy one can find companies reaching a later point in the development cycle and realizing the winner really isn’t whereas the second means some real, viable drug candidates are discarded early because the company sets a high bar for progression.

So, the NFL draft. The work of Massey and Thaler suggests many NFL teams follow a “pick the winners” strategy in that they decide specific players are what that team needs to make a substantial improvement.* This leads to rather lopsided draft trades, such as the one made by the Washington Redskins to ensure they could pick Robert Griffith III in 2012 (three first round and one second round pick to the Rams). RG III is an exciting, talented player. He could yet lead Washington to the Super Bowl. And he cost the team an exorbitant sum. The question is not whether RG III is talented. The question is whether teams can accurately forecast the future impact of any player, and if not, if trading up makes sense from the standpoint of trying to find undervalued assets.

Here’s where the article in fivethirtyeight.com provides some additional perspective. That analysis shows that drafting is a crap-shoot. Front offices do not generally outperform random chance in terms of the value returned by their picks. Below I am reproducing one of the key graphs from Neil Paine’s piece. If there was an ability of at least some teams to outperform their peers, one would see some sign of a trend.

 

This makes a lot of sense when one considers the vast world of things, outside of physical ability and other elements which an NFL team can measure leading up to the draft, that could affect eventual value of a player. Injuries. Distractions. Money. There’s an element of luck. Not the Andrew kind, the rolling the dice kind.

In Stromberg’s piece, he makes an argument that a “kill the losers” strategy is ultimately the better strategy for NFL teams. With so many players of high caliber coming into the draft, narrowing down too quickly to a specific player can be detrimental. Rather, gather as many good players as possible and see how they shake out once you get them to the preseason workouts. Or, in draft parlance, trade down and accumulate picks.

Interestingly, this seems to be the strategy aggressively being pursued by the Seattle Seahawks (disclaimer, I live in Seattle, take anything I say herein with a grain of salt). A recent Sports Illustrated note by Doug Farrar describes how Pete Carroll has sent a message to agents of players who are likely to go undrafted. His letter describes how every Seahawk player has to earn his playing time, and that anyone who comes to Seahawks camp will be in the competition. There’s a lot of proof that Carroll really believes this, including his choice of Russell Wilson to be his quarterback at the beginning of the 2012 season over the much more experienced Matt Flynn. Carroll’s letter is both a sales pitch and an expression of the “kill the losers” strategy in which pedigree and previous knowledge are deemed not nearly as important as performance on the field.

By the way, the Seahawks just traded down, exchanging their 2014 first round pick to Minnesota for Minnesota’s second and fourth round picks. More of the “kill the losers,” large number of candidates, approach.

Now then. Two ways in which this applies to drug development: First, to make a tortured analogy, players in training camp are like drugs entering Phase I. This is where everything really shakes out, and many of the experiments performed up to that point to shore up confidence are moot. You see safety and signs of efficacy in Phase I or you don’t. So maybe it’s better to just get as many things as possible through development to the point where you can enter trials with a potential therapeutic that is very likely to be safe. Efficacy, who knows?**

Second, both articles on the draft discuss the concept of the Overconfidence effect. In this cognitive bias, the more information a person or an organization gathers about a specific item, person or topic, the more confident that person is in choosing that item, person or topic even when the additional data does not substantially improve predictive ability.*** This can lead to rosy projections for a potential therapeutic which may not be warranted from an objective standpoint.

I’ve been mulling over how the recent GSK-Novartis asset swap might be classified along this axis. While one could view it as a “pick the winners” strategy in that it reduces the diversity of indications each company is pursuing, I actually think it more accurately fits in the “kill the losers” category since it allows each company to have a greater diversity of assets within their respective chosen areas (oncology for Novartis, vaccines for GSK).

In any case, it will be interesting to see how these strategies play out in drug development. Especially as the demand for truly better efficacy in new drugs gets greater, and as the industry demonstrates how difficult it is right now to rationally, objectively find those better drugs, it may be time for some humility and a strategic approach that diversifies risk and lets empiricism, rather than opinion, decide which drugs move ahead.

*I confess. I do this all the time when planning my fantasy baseball draft. This year, for example, I decided I was going to get Troy Tulowitzki NO MATTER WHAT COST. This is bad. The fact that Tulowitzki is currently the most valuable player in baseball so far this season is immaterial. I used bad process.

**I realize there are competing issues in Phase I, including the really huge issue of getting patients to enroll in an experimental therapy. Realistically, without a  lot of additional experimental data produced during drug development, companies may find it impossible to run a large number of Phase I trials even if they want to.

***Parenthetically, this is actually troubling for me from a philosophical standpoint, since  I’m a genomics researcher and make my livelihood by generating ever more data. Hmmmm.