IMHO (not a paper writer, but read a lot during my grad school years), the Genie is out of the bottle. The only way forward, as I see it, is using LLMs for reviews also. Basically, filter all submitted papers with an LLM and ask it to summarize it, find the biggest weaknesses and main strong points, etc. that a human can then use to review the paper. Basically, LLM-as-a-reviewer .
Personally, I would love to see a conference where people are explicitly encouraged to use LLMs for doing the work and writing the papers, and LLMs are used to review them too.
The entire point of an academic paper is to add to the sum of human knowledge. How can an LLM trained on a subset of human knowledge possibly even begin to accurate evaluate such a paper?
I trust an LLM to review that the language used in the paper is grammatically correct, but not to evaluate new information for accuracy.
1) Humans also are trained on a subset of human knowledge.
2)A lot of papers are just about experimenting something, and then applying simple stats. Eg empirical studies, around 1/3rd of published papers. Like, we tried this drug or did this experiment, from a sample size X here are the results. An expert is needed to maybe comment on the conclusion/hypothesis of the underlying suspected mechanism, but LLMs are still very useful on catching bad statistics or p hacking (so so common)
Schmidthuber has an answer: <a href="https://arxiv.org/abs/0812.4360" rel="nofollow">https://arxiv.org/abs/0812.4360
For a more practical approach you need to use proxies: <a href="https://zby.github.io/commonplace/articles/what-an-automated-reviewer-should-measure/" rel="nofollow">https://zby.github.io/commonplace/articles/what-an-automated...
mlmonkey · · focus · HN ↗
Personally, I would love to see a conference where people are explicitly encouraged to use LLMs for doing the work and writing the papers, and LLMs are used to review them too.
pwinnski · · focus · HN ↗
I trust an LLM to review that the language used in the paper is grammatically correct, but not to evaluate new information for accuracy.
MikhailTal · · focus · HN ↗
1) Humans also are trained on a subset of human knowledge. 2)A lot of papers are just about experimenting something, and then applying simple stats. Eg empirical studies, around 1/3rd of published papers. Like, we tried this drug or did this experiment, from a sample size X here are the results. An expert is needed to maybe comment on the conclusion/hypothesis of the underlying suspected mechanism, but LLMs are still very useful on catching bad statistics or p hacking (so so common)
zbyforgotp · · focus · HN ↗
For a more practical approach you need to use proxies: <a href="https://zby.github.io/commonplace/articles/what-an-automated-reviewer-should-measure/" rel="nofollow">https://zby.github.io/commonplace/articles/what-an-automated...