Isn't it likely that the the bottom % of papers have negative impact? (e.g., Wakefield's MMR autism fraud paper). I think I agree with your overall point but steelmanning the argument for peer review would probably mean drawing an s-curve with the negative spiking left tail causing significant harm.
Yes you're right, some science probably does have negative impact. I agree with Mastroianni's argument that there's a much smaller range for bad effects than good ones. Great science lasts forever as a deep useful truth or as a stepping stone to new discoveries. Bad science is only bad until it's obsolesced. Spontaneous generation theory held back medical practice for a while but the bad from that theory is finite while the gains from germ theory is larger and still growing. So the positive gains science are uncapped but the negative potential is capped.
"Although the average input quality increases by the same amount as in the weak link model, the average final impact barely changes."
This isn't true - peer review which filters out the bottom half of research improves the average impact *more* in the increasing returns graph than in the linear graph. Try comparing the areas in the graphs.
Intuition: if every piece of research has nonnegative impact, then the best you can do by filtering out the bottom half of research is to double the average impact of research. That's what happens if the bottom half of research all has zero impact. And the closer the bottom half of research is to having zero impact, the closer you get to doubling. And increasing returns keeps the bottom half closer to zero, relative to the rest of the graph.
Some math which you can check by taking some integrals: If the lowest quality research has 0 value...
... and the graph is linear (y=kx), then removing the bottom half is a 1.5x multiplier on average value
... and the graph is quadratic (y=kx^2), then removing the bottom half is a 1.75x multiplier on the average value
... and the graphic is cubic (y=kx^3), then removing the bottom half is a 1.875 multiplier on the average value
Hmmm I see what you are saying. I created the graphs to visualize a relationship between the _range_ of two distributions. I wanted to get across the idea that if only differences on the top end of quality change final impact much, then filtering the bottom range of the quality distribution does not have much effect on the range of the impact distribution.
So say you're trying to predict the eventual impact of a scientific project. Knowing that this project got past peer review and is thus in the top half of the input quality distribution is very useful information in the weak-link model but not so much in the strong-link model.
When I wrote the quote "Although the average input quality increases by the same amount as in the weak link model, the average final impact barely changes." I saw that since the range of final impact changes less, the uniform average (.5(a+b)) also changes less.
Is it possible to have this non-linear relationship between the range of two distributions but have the distributions themselves both be uniform?
Isn't it likely that the the bottom % of papers have negative impact? (e.g., Wakefield's MMR autism fraud paper). I think I agree with your overall point but steelmanning the argument for peer review would probably mean drawing an s-curve with the negative spiking left tail causing significant harm.
Yes you're right, some science probably does have negative impact. I agree with Mastroianni's argument that there's a much smaller range for bad effects than good ones. Great science lasts forever as a deep useful truth or as a stepping stone to new discoveries. Bad science is only bad until it's obsolesced. Spontaneous generation theory held back medical practice for a while but the bad from that theory is finite while the gains from germ theory is larger and still growing. So the positive gains science are uncapped but the negative potential is capped.
"Although the average input quality increases by the same amount as in the weak link model, the average final impact barely changes."
This isn't true - peer review which filters out the bottom half of research improves the average impact *more* in the increasing returns graph than in the linear graph. Try comparing the areas in the graphs.
Intuition: if every piece of research has nonnegative impact, then the best you can do by filtering out the bottom half of research is to double the average impact of research. That's what happens if the bottom half of research all has zero impact. And the closer the bottom half of research is to having zero impact, the closer you get to doubling. And increasing returns keeps the bottom half closer to zero, relative to the rest of the graph.
Some math which you can check by taking some integrals: If the lowest quality research has 0 value...
... and the graph is linear (y=kx), then removing the bottom half is a 1.5x multiplier on average value
... and the graph is quadratic (y=kx^2), then removing the bottom half is a 1.75x multiplier on the average value
... and the graphic is cubic (y=kx^3), then removing the bottom half is a 1.875 multiplier on the average value
Hmmm I see what you are saying. I created the graphs to visualize a relationship between the _range_ of two distributions. I wanted to get across the idea that if only differences on the top end of quality change final impact much, then filtering the bottom range of the quality distribution does not have much effect on the range of the impact distribution.
So say you're trying to predict the eventual impact of a scientific project. Knowing that this project got past peer review and is thus in the top half of the input quality distribution is very useful information in the weak-link model but not so much in the strong-link model.
When I wrote the quote "Although the average input quality increases by the same amount as in the weak link model, the average final impact barely changes." I saw that since the range of final impact changes less, the uniform average (.5(a+b)) also changes less.
Is it possible to have this non-linear relationship between the range of two distributions but have the distributions themselves both be uniform?