Publication in the Age of AI

Scientific publication is being overwhelmed by artificial intelligence. We have entered an age in which it is faster to write a paper than it is to properly review it. An artificial intelligence (AI) agent can search a public dataset for every pairwise relationship, keep the few that clear significance, and draft a manuscript almost before a referee has time to read the abstract. The cost of producing a plausible finding – backed by real data – is rapidly approaching zero while the cost of evaluation remains unchanged. This asymmetry threatens to flatten the signal-to-noise ratio in scientific literature.

The challenge here is not fraud; these (AI) authors are genuinely running reasonable analyses on real data. The issue is that these analyses can be run on a massive scale. Exploratory work like this can be meaningful, but only when replicated in new data, or when integrated with related findings.

More than the narrative

We believe that AI can be used as a mechanism to alleviate this AI-generated problem. However, it will require restructuring what it means to publish. We need to exploit AI for its strengths while ensuring it isn't used where it isn't trustworthy.

We propose placing data and its metadata at the center of publication. A complete research product should include the dataset (or the means to compute against it), the metadata needed to understand and reuse it, the analysis code with the details of the computing environment, and an interpretation or narrative that is understandable by people and ingestible by machines. We recognize that this is not a new idea – consider the research compendium and the FAIR (Findability, Accessibility, Interoperability, and Reusability) standard. However, AI makes this achievable in a way that is far less prescriptive (and restrictive) than ever before.

By taking advantage of the ability of AI to translate natural language plus metadata into analysis, we can construct a computing environment that facilitates the publication of robust research products while allowing data of essentially arbitrary structure to be both protected and broadly usable (Figure 1). This environment is:

• Secure – isolates the data in a secure environment (if so desired)
• Auditable – enables vetting by experts to guard against hallucination
• Flexible – can be used with essentially any data layout; allows arbitrary analyses
• Version controlled – technical bookkeeping such as versioning of data, computing environment, scripts, and chat logs is automatic
• Reviewer friendly – allows reviewers to automatically verify analyses through the direct submission of analysis scripts
• Transparent – the entire arc of an analysis can be tracked through examination of the agent chat logs

As these environments spread, it becomes possible to distribute analyses across many datasets. This allows early vetting of biological hypotheses based on what is already available, and focused experiments to fill in missing pieces. The idea of distributing analyses across many datasets is also not new. Consider the OMOP (Observational Medical Outcomes Partnership) data standard used by OHDSI (Observational Health Data Sciences and Informatics) to run the same analyses in a distributed way across many databases. The flexibility of AI allows federated management of data and compute environments as well as distributed analysis across much more disparate datasets.

Data marketplaces

Incentives, not technology, will decide whether data gets shared and how research publication changes. The institutions that hold the most valuable data – health systems, pharmaceutical companies, biobanks – have no reason to give it away. Asking them to do so is futile and devalues their contributions.

The publication model we propose lets the data stay where it is. The holder (i) controls access, (ii) publishes metadata rich enough for an outside analyst, or an AI, to write an analysis, and (iii) runs that analysis inside its own enclave and returns the result. Compute travels to the data so that the data is never out in the open. When governed appropriately to avoid re-identification, this converts data you cannot share into both a financial asset for the owner and a research asset for the community.

This approach ensures that data sharing does not become a tax on the large players; it is instead an instrument that incentivizes broader collaboration and data access. Building a clean, well-documented dataset becomes financially viable. Analyses spanning many modalities – claims, genomics, imaging, clinical notes – become possible, and each holder contributes without losing control.

Peer review

Vetting research products looks different from reviewing articles. Reproducing figures and tables is mechanical and automatic; this allows reviewers to spend their effort where judgment is required.

An AI "reviewer" can:

• Search for relevant public datasets the authors missed and run confirmatory analyses
• Directly test whether results are robust to reasonable analytic variation (see multiverse or specification curve analysis)
• Determine whether the metadata suffices to link the data to other published datasets

Even though much of the validation can be done by the AI, there are questions that aren't answered by reproduction. These are the more complex, subjective questions for which human reviewers are better suited.

• Do the statistics support the written claims?
• Does the arc of exploration (available in the chat logs with the AI) warrant a strong or a weak conclusion?
• Is the narrative accurate, and does it matter to patients, medicine, and/or society?

Conclusion

When a complete research product is published, our entire scientific literature changes shape. Null results do not require a special venue; they are simply available. Findings are no longer single events; they instead transform into a living record represented by evidence that updates as data arrives. Meta-analyses are available on demand and with statistical backing, facilitating rapid understanding of the cutting edge of research in any field.

Most importantly, much of the rote part of the work associated with peer review can be automated. Widely shared data, carrying the metadata needed to use it, opens approaches to discovery that we have not yet imagined.