One way to improve clinical trial reporting: a Yelp-style rating system

This piece originally appeared in the Timmerman Report.

STAT recently published an in-depth report about the many research centers that don’t bother to publicly disclose the results of their clinical trials, even though they are required to do so. This follows on a New England Journal of Medicine article back in March that had a similar analysis of the lack of reporting and publication of clinical trial data to

Most observers of biomedical research would agree that getting clinical trial data out about what happened in a trial is pretty important, whether the trial succeeded or failed. After all, biomedical translational research is most meaningful when done on human subjects and negative information can be quite informative and useful. Animal models are nice, but translation of results from animals to humans is a spotty proposition at best. We need to know what’s working, and what’s not, to know how to best allocate our research resources and how to treat patients.

The lack of reporting is an embarrassment for research. It’s also understandable, because so far the FDA hasn’t used its authority to punish anyone for delayed reporting. Nobody appears to have lost any research funding because they failed to post trial results in a timely manner. Universities told STAT their researchers were “too busy,” given other constraints on their time, to report their results. So what really seems to be going on is that reporting is prioritized below most other activities in clinical research.

It was interesting and eye-opening that industry fared better than academia in both the STAT story and the NEJM article with respect to how many studies have been reported. Having seen the industry process first-hand, I’d speculate that (at least for positive trials) there’s a much stronger incentive to get data out in public. Successful trial results can create buzz among clinicians and patients, revving up trial enrollment which can then help get a new drug on the market faster, and convince people to use it when it’s available. It may be that in academia the effort of getting trial results in the required format for is perceived as too much work, relative to the rewards. Academics are naturally going to spend more energy on directly rewarded activities like writing grant proposals and writing peer-reviewed scientific publications that help them win even more grants, promotions, and other accolades. Well okay. If this is the case, then figuring out new incentives may be key.

So what would work? Anyone who participates in a clinical trial is providing time, may be subject to risks and often is asked to provide samples that are biobanked to support future exploratory and translational research. It’s like when people donate to food banks. I’m pretty sure they mean that food to be eaten and not to sit on a shelf. These participants in clinical trials deserve to have their volunteerism rewarded.

This got me thinking about how to empower patients to get more of what they want. Patient-centered research is a buzzword these days, and for good reason. Patients have at times been an afterthought in the biomedical research enterprise. I thought of services like Yelp and Uber and Angie’s List and other peer-to-peer systems that allow users to get information, provide feedback and give ratings to specific providers. And I wondered: could this be a way to apply pressure to clinical trial researchers to improve their reporting? Continue reading

No, CRISPR-Cas won’t save the day for ag biotech

You want to know how to drive a scientist crazy? Insist that you believe something that’s not supported by current scientific evidence. Tell her vaccines cause autism, or creationism is just as valid a theory as evolution, or that climate change isn’t really happening, I mean, after all, a monster blizzard hit Washington DC this January! Global warming, pssh…

There’s an old episode of Friends that did a good job of showing how this kind of conversation goes. Phoebe professes not to believe in evolution and Ross, a paleontologist, keeps trying to convince her that evolution is real using scientific evidence and logic. He grows increasingly frustrated and insistent as she continues to deny the basis of his life’s work, finally losing it when she goads him into admitting (like a good scientist) that even theories like evolution are not immune from questioning and testing.

We train scientists to carefully generate, weigh and use evidence. To no one’s surprise, this leads many scientists to generalize and think that in all matters having to do with the physical world we all should and of course will follow the evidence. Yes, sometimes that leads to unpopular ideas, and sometimes the ideas change as the weight of evidence changes. This training can make scientists kind of boring at cocktail parties. Still, the overall scientific process keeps moving forward and it’s because of this reliance on evidence.

But many people (including, at times, even some scientists) don’t always think the same way about things in the physical world. And that’s why I’m pessimistic that CRISPR-Cas technology will peacefully resolve the Genetically Modified Organism (GMO) debate. Continue reading

Should Basic Lab Experiments Be Blinded to Chip Away at the Reproducibility Problem?

An earlier version of this piece appeared on the Timmerman Report.

Note added 23Feb2016: Also realized that I was highly influenced by Regina Nuzzo’s piece on biases in scientific research (and solutions) in Nature, which has been nicely translated to comic form here.

Some people believe biology is facing a “Reproducibility Crisis.” Reports out of industry and academia have pointed to difficulty in replicating published experiments, and scholars of science have even suggested it may be expected that a majority of published studies might not be true. Even if you don’t think the lack of study replication has risen to the crisis point, what is clear is that lots of experiments and analyses in the literature are hard or sometimes impossible to repeat. I tend to take the view that in general people try their best and that biology is just inherently messy, with lots of variables we can’t control for because we don’t even know they exist. Or, we perform experiments that have been so carefully calibrated for a specific environment that they’re successful only in that time and place, and sometimes even just with that set of hands. Not to mention, on top of that, possible holes in how we train scientists, external pressures to publish or perish, and ever-changing technology.

Still, to keep biomedical research pushing ahead, we need to think about how to bring greater experimental consistency and rigor to the scientific enterprise. A number of people have made thoughtful proposals. Some have called for a clearer and much more rewarding pathway for reporting negative results. Others have created replication consortia to attempt confirmation of key experiments in an orderly and efficient way. I’m impressed by the folks at Retraction Watch and PubPeer who, respectively, call attention to retracted work, and provide a forum for commenting on published work. That encourages rigorous, continual review of the published literature. The idea that publication doesn’t immunize research from further scrutiny appeals to me. Still others have called for teaching scientists how to use statistics with greater skill and appropriateness and nuance. To paraphrase Inigo Montoya in The Princess Bride, “You keep using a p-value cutoff of 0.05. I do not think it means what you think it means.”

To these ideas, I’d like to throw out another thought rooted in behavioral economics and our growing understanding of cognitive biases. Would it help basic research take a lesson from clinical trials and introduce blinding in our experiments? Continue reading

Google Knows What’s in Your Inbox, But It Shouldn’t Get Your Genome Without Consent

Originally posted in the Timmerman Report

I once had an idea for a science fiction story where everyone was paranoid about their genetic information getting out because of a misguided belief that genes equal destiny and that the burden of privacy is all on the individual. People would wear protective suits and carefully guard against leaving any iota of tissue out in public—not a single follicle or skin flake. All to prevent anyone else—potential employers, rivals, even potential lovers—finding out information about their genes.

I planned the story as a satire, taking our current world where Precision Medicine and cheap genome sequencing and not-quite-as-cheap genome interpretation are real things, and extrapolating to an absurdity. I wanted to highlight the kinds of more realistic challenges we might face as we learn more about our genes and face increasing questions about privacy and access to health care services. Of course, I thought this was completely speculative; I’d just be building a straw man story to make a point. I knew something this extreme would never really happen.

But maybe I was wrong. Continue reading

The immune system as a metaphor for the Turing Pharma mess

H/T to @DerekLowe for the initial metaphor of Turing as a virus, and to the many, many commenters and journalists who’ve covered this topic.

A few weeks ago when Turing Pharmaceuticals’ plan to jack up the price of Daraprim by 5000% hit the news, I found myself glued to Twitter and other journalistic sources as each day brought a new wave of insults, reactions, condemnations, harangues, plans and concessions. I’ve frankly never seen any piece of biopharma industry news evoke this kind of response. When even U.S. Presidential candidates—well, the Democratic ones, anyway—are compelled to offer their takes and plans for reform, it seems that something fundamental may have changed.1

The question is, what should the biopharma industry take away from this? While a few CEOs such as Biogen Idec’s George Scangos and Novartis’ Joseph Jimenez have spoken out against the price hike, and both PhRMA and BIO have taken actions to repudiate Turing Pharma’s actions, there are still demands that biopharma come out with a clearer, more unified position—or risk being labeled as hypocritical. But that doesn’t seem to be happening. Just recently Ian Read, the CEO of Pfizer, told Evercore ISI that he wasn’t worried about Democratic presidential candidate Hillary Clinton’s proposals to control drug prices. Not the most conciliatory or self-aware comment to make, especially for a CEO who went on the record last year saying Biopharma has a reputation problem. So how to influence the industry to take this problem seriously?

Behavioral economics tells us that framing can be a key component influencing decision making. Among other things, a framing approach makes choices and consequences as real as possible to people through the use of analogy and filters. So I thought to myself, we all work in biology. Maybe a biological analogy might help drive home what’s at stake. And given that immuno-oncology is so hot these days it makes plasma look cool, I thought an immune related frame might be helpful. So let me present the recent pricing fiasco through the lens of the immune system.2

An extended analogy between an immune system response and the reaction to Turing Pharma
Immune system element Analog in the recent Turing Pharma debate
Priming antigens Recent discussions about the price of Sovaldi, and several other new drugs in indications like Hepatitis C, cardiovascular disease, and oncology primed the public to discussions of drug prices. Also, the Affordable Care Act’s influence on employer-provided health plans has led to rising out of pocket costs and therefore higher visibility for prices for medications.
Subthreshold immune response The lack of response to earlier instances of companies hiking the prices of generic drugs for companies like Questcor, Valeant and URL Pharma.
Virulent, triggering pathogenic infection Turing Pharmaceuticals raising the price of Daraprim by 5000%.
Immunization adjuvant Turing CEO Martin Shkreli’s amazingly annoying and provocative internet persona and behavior.
Innate immune response Social media and other responses to the price increase. Phrases like “drug price hikes,” “vulnerable populations,” and “maximizing shareholder value” that serve as pattern recognition motifs that trigger a speedy response like a macrophage engulfing a Staphylococcus bacterium.
Adaptive immune response The various regulatory plans being proposed–price caps, single payer negotiations, transparency initiatives, etc.–to stem drug price increases long-term.
Commensal bacteria Biopharma


The response to Turing Pharma didn’t come out of nowhere; there had already been several different factors that led the public to be primed for an affront like the Turing Pharma price hike. Over the last few years several high profile, high priced drugs have entered the market and raised real fears about the affordability of new medicines for conditions like Hepatitis C and high cholesterol that affect a substantial proportion of the US population. In addition, the Cadillac tax and other requirements of the Affordable Care Act appear to be leading to higher out-of-pocket expenses for patients (which is a feature, not a bug), further sensitizing them to drug prices. These developments have created social sensitivity to drug price changes, just as exposure to potential antigens primes the immune system for a stronger later response.

So why didn’t earlier, similar actions by other biopharma companies raise the same response? I think there are two main factors: first, randomness. We live in a world where a multitude of stimuli are clamoring for our attention. In that kind of environment there’s only enough oxygen for a few things to rise to wide-spread attention. I suspect the earlier price increases, some of which were in scale similar to what Turing did, just didn’t break the surface tension of the internet. Which brings me to the other factor: Turing’s CEO.

In vaccination, there’s a continual effort to find new adjuvants. These are substances one adds to a vaccine to really jack up the immune response. Martin Shkreli’s behavior served as a kind of adjuvant, exacerbating the effect of the initial insult of Turing’s pricing change. And that led to the initial, wild, Internet-wide response.

The concept of social shaming was recently examined in Jon Ronson’s book “So you’ve been publicly shamed.” Among the many things the Internet has enabled has been the ability to rapidly pivot on perceived malfeasants and nail them to the wall with a response not unlike that big rolling stone ball that almost squashes Indiana Jones at the opening of “Raiders of the Lost Ark.” Once a negative event does enter the attention economy, vituperation is swift and merciless. However, this innate immune response predictably fades over time. The real, lingering effect is the adaptive immune system of society—regulatory policy, which we see being proposed and debated as the US Presidential election cycle proceeds. The adaptive immune system has memory for certain pathogens, and remains ready to strike them down at a moment’s notice, and society also has a memory for certain acts it wants to remain constantly vigilant against, through laws and regulations.

So the lesson I hope biopharma takes from this extended analogy is the following: it’s too late to keep on doing the same things and hope no one notices. The system’s primed now and already starting to develop its long-term defenses. You’ll note I liken biopharma to commensal bacteria. I do think the industry serves a vital, important, useful role in helping society, just as our commensal microorganisms perform a host of beneficial functions for us. But as these kinds of rent-seeking behaviors proliferate—and not just on the side of generic drugs—the system as a whole will begin treating biopharma as more of a pathogen than a friend and the results could be very unpredictable. After all, the immune response doesn’t think about long-term consequences for the whole; it operates quickly and locally, often with unforeseen consequences. When excessively agitated by certain stimuli, the immune system can enter a chronic state of autoimmunity, creating long-lasting pain and fatigue.

The biopharma industry is blessed with the gifts of intelligence and foresight, and can take actions to head off a more stringent adaptive response from society by being more transparent and ramping down its most immunogenic actions—namely, pricing drugs to the limit of what the market will bear rather than based on a reasoned consideration of innovation and value. Because the alternative, as Robert Nelson of Arch Ventures laid out so starkly in Forbes, could be a crushing of all the good things biopharma can do in the rush to punish for the sins of a few.


1Maybe. The internet is notorious for its short attention span. Still, Bernie Sanders already had been calling out for reform on drug prices, so it may well be that this is a controversy that won’t get swept away by the next noisy thing. And, it must be said that at least one Republican candidate did respond to Turing’s price hike. Donald Trump, the candidate with the most experience as a provocateur, felt compelled to call Martin Shkreli a “spoiled brat.”

2I’m not an immunologist by training—I’m a geneticist. But I’ve been working with immunologists for the last seven years, which I think qualifies me to make broad, overly generalized and not entirely accurate analogies for the sake of a good story hook.

Baseball, regression to the mean, and avoiding potential clinical trial biases

This post originally appeared on The Timmerman Report. You should check out the TR.

It’s baseball season. Which means it’s fantasy baseball season. Which means I have to keep reminding myself that, even though it’s already been a month and a half, that’s still a pretty short time in the long rhythm of the season and every performance has to be viewed with skepticism. Ryan Zimmerman sporting a 0.293 On Base Percentage (OBP)? He’s not likely to end up there. On the other hand, Jake Odorizzi with an Earned Run Average (ERA) less than 2.10? He’s good, but not that good. I try to avoid making trades in the first few months (although with several players on my team on the Disabled List, I may have to break my own rule) because I know that in small samples, big fluctuations in statistical performance in the end  are not really telling us much about actual player talent.

One of the big lessons I’ve learned from following baseball and the revolution in sports analytics is that one of the most powerful forces in player performance is regression to the mean. This is the tendency for most outliers, over the course of repeated measurements, to move toward the mean of both individual and population-wide performance levels. There’s nothing magical, just simple statistical truth.

And as I lift my head up from ESPN sports and look around, I’ve started to wonder if regression to the mean might be affecting another interest of mine, and not for the better. I wonder if a lack of understanding of regression to the mean might be a problem in our search for ways to reach better health.
Continue reading

An Open Standard for APIs Could Lead us to Better Health

There’s a parable about the elephant and the rider that’s been used by Chip and Dan Heath, and that originated with Jonathan Haidt, to describe how humans make decisions. A person’s mind can be thought of as consisting of a rider, representing the rational part of human thinking, and the elephant she’s riding, representing emotion. Both of these play a role in how a person decides things, and many of us believe the rider–the rational part–is in charge. The rider taps the elephant with her guide stick, and the elephant obediently moves in that general direction or does a specific task, like carrying lumber from place to place.

Except that’s not how a lot of decisions actually get made. Instead, the elephant sees a bunch of bananas, or a herd of other elephants, or a nice cool river to bathe in, and goes that way instead. And the rider…well, the rider can’t do much about it except, after the fact, rationalize how she always wanted to go in that direction to begin with. Yeah, it was time for a bath, sure

This framing has stuck in my mind for years and it’s a really helpful way of looking at many of the odd things that people do or say, ranging from climate change denial, to believing genetically modified organisms are inherently evil, to smoking despite everything we know about the harms that result, to even saying that Paul Blart, Mall Cop II is really, you know, not that bad–really. And it also speaks to one of the more vexing problems we have in human health. Why do people keep doing things they really probably shouldn’t, and know they shouldn’t, if they want to stay healthy?

I’ve touched before on how the power of digital tools can help make it easier for us to make good decisions. OPower is doing this for power consumption and conservation, and with the advent of tools like Apple’s Healthkit and the proliferation of activity trackers, the time is right to do this for health. Continue reading