Should Basic Lab Experiments Be Blinded to Chip Away at the Reproducibility Problem?

An earlier version of this piece appeared on the Timmerman Report.

Note added 23Feb2016: Also realized that I was highly influenced by Regina Nuzzo’s piece on biases in scientific research (and solutions) in Nature, which has been nicely translated to comic form here.

Some people believe biology is facing a “Reproducibility Crisis.” Reports out of industry and academia have pointed to difficulty in replicating published experiments, and scholars of science have even suggested it may be expected that a majority of published studies might not be true. Even if you don’t think the lack of study replication has risen to the crisis point, what is clear is that lots of experiments and analyses in the literature are hard or sometimes impossible to repeat. I tend to take the view that in general people try their best and that biology is just inherently messy, with lots of variables we can’t control for because we don’t even know they exist. Or, we perform experiments that have been so carefully calibrated for a specific environment that they’re successful only in that time and place, and sometimes even just with that set of hands. Not to mention, on top of that, possible holes in how we train scientists, external pressures to publish or perish, and ever-changing technology.

Still, to keep biomedical research pushing ahead, we need to think about how to bring greater experimental consistency and rigor to the scientific enterprise. A number of people have made thoughtful proposals. Some have called for a clearer and much more rewarding pathway for reporting negative results. Others have created replication consortia to attempt confirmation of key experiments in an orderly and efficient way. I’m impressed by the folks at Retraction Watch and PubPeer who, respectively, call attention to retracted work, and provide a forum for commenting on published work. That encourages rigorous, continual review of the published literature. The idea that publication doesn’t immunize research from further scrutiny appeals to me. Still others have called for teaching scientists how to use statistics with greater skill and appropriateness and nuance. To paraphrase Inigo Montoya in The Princess Bride, “You keep using a p-value cutoff of 0.05. I do not think it means what you think it means.”

To these ideas, I’d like to throw out another thought rooted in behavioral economics and our growing understanding of cognitive biases. Would it help basic research take a lesson from clinical trials and introduce blinding in our experiments? Continue reading

Could pro sports lead us to wellness?

Comment From Bill
St. Louis is being hindered in the stretch drive by some kind of GI bug passing through (so to speak) the team. Reports have as many as 15 guys down with it at once. That seems a lot, but given the way a baseball clubhouse works, my question is why don’t we see more of that? Answering that baseball players are fanatically interested in sanitation and hygiene ain’t gonna cut it, I don’t think…

12:10
Dave Cameron: They have access to a lot of drugs.

–comment from a chat at Fangraphs, September 24, 2014

So this comment caught my eye. Ever since I began following sites like BaseballProspectus.com and Fangraphs.com, and reading things like Moneyball, I’ve found myself thinking about efficiency and unappreciated or unexplored resources in different situations.

I realize this was a throwaway line in a baseball chat. But it piqued my interest because it seems to point out something that’s maybe underappreciated and understudied about how sports teams go about their business–specifically, the kinds of things they do to keep their athletes healthy.

My question is, does this represent a potential source of “Found Research” data that could help the rest of us reach wellness? Continue reading

Trying to figure the way through a 401(k) life

Thomas Friedman’s thoughts on how we’re becoming a 401(k) nation have been kicking around the back of my mind for about a year. His Op-Ed piece described the shift in how retirement plans in the US have largely shifted from pensions to 401(k)s and used that transition to make a point about the changing nature of work.

In a pension plan, a defined input (so many year of work) leads to a defined output (a regular payment that starts upon retirement until the day you die) with the risk assumed by the employer. In a 401(k), there’s still a defined input (regular deposits into a managed investment account) but how much a person gets at the end carries no guarantees and the risk sits squarely with the employee.

Friedman’s insight was that work itself is following that same path. Where once the defined achievements of education and learned skills were enough to guarantee continued employment and a good, middle-class career (at least), that’s not really the case anymore.

The news a some months back that Amgen will be closing its Seattle site this year really drove that point home. It was a reminder that biopharma, like so many other industries, isn’t immune to the implications  of the 401(k) life.

Continue reading

The power law relationship in drug development

All opinions are my own and do not necessarily reflect those of Novo Nordisk.

A few weeks ago a friend and I had the great opportunity to go see Nate Silver speak at the University of Washington. He’s a funny, engaging speaker, and for someone like me who makes his living generating and analyzing data, Silver’s work in sports, politics and other fields has been inspirational.  Much of his talk covered elements of his book, The Signal and the Noise, which I read over a year ago. It was good to get a refresher. One of the elements that particularly struck me this time around, to the point that I took a picture of his slide, was the concept of the power law and its empirical relationship to so many of the phenomena we deal with in life.

Nate Silver graph small

Figure 1: Slide from Nate Silver’s talk demonstrating the power law relationship in business–how often the last 20% of accuracy (or quality or sales or…) comes from the last 80% of effort.

Because I spend way too much time thinking about the business of drug development, I started thinking of how this concept applies to our industry and specifically the problem the industry is facing with creating innovative medicines.

Continue reading

Drug development and the NFL draft

All opinions are my own and do not necessarily reflect those of Novo Nordisk.

The NFL draft is happening as I am writing this post. And of the many draft-related pieces I’ve read in the past few days, one from Vox.com particularly stood out. The article, by Joseph Stromberg, describes research by Cade Massey and Richard Thaler (here and here) about the skewed and irrational choices often made by teams during trades of draft picks. In essence, teams are likely to pursue a strategy in trading up that suggests they believe they have a much greater ability to forecast the future performance of a given player than is actually the case. Put another way, rather than following a strategy of diversified risk, teams commit to a specific player that they feel they need to get, rather than simply seeing who’s available when they are scheduled to pick and choosing the best player on their draft board.

Historical analysis shows that the difference between various players drafted at the same position is often negligible; on top of that teams who aggressively trade down and gather more picks in the lower rounds generally do better in terms of the value they receive for the money they spend in salaries. One might argue this is an artifact in part of the NFL Rookie salary structure, but even without that, players taken in later rounds will always command smaller salaries. Getting similar value for less money is generally a good thing.

If you’ve read posts from this blog before you know where I’m going. Drafting NFL rookies sounds a lot like developing drugs. Continue reading