Science should be more like this, in current times, yes.
But until much of academia is burned to the ground, or until science can be properly separated from modern academia, this will never be so. The current academic incentives are all wrong: low-quality research is rewarded and results in publications, whereas high-quality research (that takes time, and usually reveals that most exciting publications depend on p-hacking or other highly data-dependent analyses and selective presentations) is not published or actively blocked during peer review.
So instead you get BS arguments about how data can't be released for various privacy concerns (when in reality the vast majority of most datasets are trivial to scrub of identifying factors, and even in more complex datasets where you need to consider k-anonymity, it is still trivial to release data that allows replication of core analyses), and academic science is increasingly irrelevant unless it is tied to tech and industry, where producing junk actually has real negative economic and personal consequences.
I don't know what world this article / post lives in, but it isn't the messy world of actual reality.
Hear hear! There are so many obvious improvements to how almost everything is done. For instance, in medicine review articles as a class of articles largely represent a giant waste of time. RCTs flatten all their gathered data during publishing, summarizing complex trial data, which is gathered but never published, into a few numbers. Then review articles take a bunch of flattened data, discard the articles that don't fit the exact question they are reviewing, and then publish a doubly flattened conclusion. If any of the included articles turn out to have flaws, if treatments change in retrospect, if you are looking for the answer to a slightly different question or you are looking at a different subgroup, then the review is useless and has to be repeated.
All of these tens of thousands of man-hours could be replaced by a few GitHub repos, if only RCTs would just publish their damn data. Then you could just run and rerun the statistics on whatever subgroup you're looking for, instead of combing through decades of review articles answering slightly different questions, looking for the answer between the lines. With LLMs making mining of large scale datasets almost trivial (with the process most likely becoming trustworthy within a few years), the current status quo is looking more and more antiquated.
If you want to be even more radical, hospitals could just publish their data continuously. Of course, it is easy to point to the risks of doing so, but what's often ignored is the benefits. It is hard to overstate just how many medical mysteries a hospital encounters on a daily basis, how much unknown we are navigating in practice. The current norm is that 99.99% of these cases are never published, and are only ever thought about by a small group of people who happened to be at work. Particularly, when someone dies of something no one figured out, it is never published anywhere, because even if you tried it is not interesting reading material for a journal to publish. And no one ever tries because they're scared of being called out for a mistake. A hospital is essentially a continuously running and extremely interesting experiment, where 99.99999% of all results are thrown in the garbage, and the only published data is subject to extreme selection bias.
All of this could be different, and the risks involved are actually quite small in practice. It is easy to automatically anonymize data quite well, but extremely difficult to absolutely guarantee that it is anonymous. And since current ethical norms are extremely averse to any degree of risk, and usually entirely ignore potential benefits, we all suffer for it. It is not entirely unlikely that someone reading this post will one day die because of something that could have been prevented, had things been different.
Yup, strongly agree with all of this, especially the RCT stuff.
This has all been profoundly obvious for at least well over a decade or even two now. A consequence has been that too many serious people are driven away from academia and research, to the detriment of science generally.
I've no idea what to do about all this, because people have voiced obvious and easy solutions for decades, but they are all routinely ignored.
I think political lobbying for legal changes might be the only realistic pathway, in that current legislation (eg. GDPR in the EU) is extremely punitive even for minor violations.
I personally am trying to float using local LLMs to create anonymized case files and auto-suggest publishing cases in my hospital, which knowing how things work will probably never amount to anything.
Or if you want to float truly insane ideas I guess you can shop around with blackhat groups and see if anyone has stolen some juicy records/data during all the ransomware attacks and databreaches over the years, and do some rogue scientific publishing. Obviously that's crazy, but I have to admit that the notion of pirate scientists plundering and publishing data is hilarious to me.
D-Machine · · focus · HN ↗
But until much of academia is burned to the ground, or until science can be properly separated from modern academia, this will never be so. The current academic incentives are all wrong: low-quality research is rewarded and results in publications, whereas high-quality research (that takes time, and usually reveals that most exciting publications depend on p-hacking or other highly data-dependent analyses and selective presentations) is not published or actively blocked during peer review.
So instead you get BS arguments about how data can't be released for various privacy concerns (when in reality the vast majority of most datasets are trivial to scrub of identifying factors, and even in more complex datasets where you need to consider k-anonymity, it is still trivial to release data that allows replication of core analyses), and academic science is increasingly irrelevant unless it is tied to tech and industry, where producing junk actually has real negative economic and personal consequences.
I don't know what world this article / post lives in, but it isn't the messy world of actual reality.
stalfie · · focus · HN ↗
All of these tens of thousands of man-hours could be replaced by a few GitHub repos, if only RCTs would just publish their damn data. Then you could just run and rerun the statistics on whatever subgroup you're looking for, instead of combing through decades of review articles answering slightly different questions, looking for the answer between the lines. With LLMs making mining of large scale datasets almost trivial (with the process most likely becoming trustworthy within a few years), the current status quo is looking more and more antiquated.
If you want to be even more radical, hospitals could just publish their data continuously. Of course, it is easy to point to the risks of doing so, but what's often ignored is the benefits. It is hard to overstate just how many medical mysteries a hospital encounters on a daily basis, how much unknown we are navigating in practice. The current norm is that 99.99% of these cases are never published, and are only ever thought about by a small group of people who happened to be at work. Particularly, when someone dies of something no one figured out, it is never published anywhere, because even if you tried it is not interesting reading material for a journal to publish. And no one ever tries because they're scared of being called out for a mistake. A hospital is essentially a continuously running and extremely interesting experiment, where 99.99999% of all results are thrown in the garbage, and the only published data is subject to extreme selection bias.
All of this could be different, and the risks involved are actually quite small in practice. It is easy to automatically anonymize data quite well, but extremely difficult to absolutely guarantee that it is anonymous. And since current ethical norms are extremely averse to any degree of risk, and usually entirely ignore potential benefits, we all suffer for it. It is not entirely unlikely that someone reading this post will one day die because of something that could have been prevented, had things been different.
D-Machine · · focus · HN ↗
This has all been profoundly obvious for at least well over a decade or even two now. A consequence has been that too many serious people are driven away from academia and research, to the detriment of science generally.
I've no idea what to do about all this, because people have voiced obvious and easy solutions for decades, but they are all routinely ignored.
stalfie · · focus · HN ↗
I personally am trying to float using local LLMs to create anonymized case files and auto-suggest publishing cases in my hospital, which knowing how things work will probably never amount to anything.
Or if you want to float truly insane ideas I guess you can shop around with blackhat groups and see if anyone has stolen some juicy records/data during all the ransomware attacks and databreaches over the years, and do some rogue scientific publishing. Obviously that's crazy, but I have to admit that the notion of pirate scientists plundering and publishing data is hilarious to me.