11 Comments

Do you think the NIH should use more innovative prizes or is that not an appropriate tool for the agency?

Expand full comment

There are many downsides of prizes. I think there are opportunities the NIH to use prizes, the trick is defining the "winning" outcome properly.

Expand full comment

Thanks for the interesting analysis, but I don't understand the logic here.

Claim: The NIH funds a lot of small, crappy trials

Data: The (Phase I) trial size is similar for NIH and for other funders

How does the data counter the claim? Couldn't one simply conclude that *every funder* funds too many small, crappy trials?

(I posted this comment on MR, as well.)

Expand full comment

As far as I can tell, OP is a strawman. Who is saying that the NIH is *worse* than others, such that the presented statistics refute that claim? What they actually seem to say is that *everyone* is funding too many small crappy trials:

> That quote is from a FDA higher up, but the story has been repeated by many others.

The first quote/citation talks about trials in general, specifically, the COVID vaccine trials, funded by every possible source, and doesn't mention NIH. The second one is about NIH specifically, suggesting how to reform NIH, but at no point I can see skimming it, does the author claim that the NIH is worse than anyone else or describe who gets it right (implying they think no one does, otherwise they would be invoking the ideal as an example for the reform). The third one doesn't mention NIH. And neither does the fourth one, which explicitly talks about it applying to all the 50,000+ (!) RCTs run annually.

So, none of them would be refuted by OP at all. If you sent them the graph and said, 'see, industry funds just as many small crappy trials as the NIH!', they would simply shrug and say, 'yes, that's what I said. We do too many small crappy trials.'

Expand full comment
author

The clear implication from Stuart's post (and from talking to him in person) is that the NIH in particular has a problem with trials that don't make any progress.

It was my honest sense that this was a common view, so comparing to industry is reasonable.

It's probably true that industry also funds too many small trials. I still find it interesting that the NIH is not particularly bad in this respect, also Simpson's paradox is always interesting.

Expand full comment

I don't think there's a 'clear implication' there, and he doesn't seem to think that either in his comments right here saying that "the problem is widespread".

If it is that common a view, that the NIH is uniquely bad in this respect, surely you can find one actual citation by someone of note making that specific criticism which would be refuted by anything you posted in OP, rather than citing 4 which don't and aren't. And why did you falsely cite so many instances instead of simply saying "I think this is a common view about the NIH and so won't bother with examples"?

Expand full comment

And yes, the problem is widespread as per Gwen's point.

FDA officials reviewed over 2,000 Covid trials from around the world, and only 5% met bare minimum standards for randomization and statistical power.

https://www.nature.com/articles/d41573-021-00037-3

Expand full comment

Just noticed that this Nature paper was discussed in the second link in the post above.

Expand full comment

An informative piece Maxwell. One of the issues that grapple with is the to properly align the spending of research funding with social imperatives.

The free market tends to underproduce R&D, filling the gap between what we should spend on research, and what we do, is the first challenge. The second, much more difficult challenge is using that funding effectively.

Expand full comment

Good to have this contemporary example of the Simpson paradox.

Expand full comment