Lessons from an AI-assisted review

When we reviewed the literature on anthropogenic heat in urban climate systems, AI supported the whole of a rigorous, multi-author review, from the first screening of records through information extraction to the final synthesis, rather than just helping one researcher with code or exploratory analysis. As far as we know, few reviews have been assembled this way. Here we set out three lessons we learned, and the questions that remain open, so that other teams can evaluate, adapt and critically assess the workflow.

Lesson oneVersion control is the essential guardrail

Bringing AI into a scientific workflow raises a basic problem: AI systems are fast and scale easily, but their mistakes can be hard to spot, and by the time one is found, later analyses may already rest on it. Human errors cause the same problem, though they may spread at a different speed and scale. Version control guards against both. It does not prevent mistakes; it limits their damage by keeping a traceable history of every change, as it has long done in software development, where Git is now standard practice.

For example, an early version of the extraction prompt was too generous in classifying papers as containing evidence of anthropogenic heat feedback, and a manual check of a small subset of papers exposed the problem. We fixed it with a new prompt version rather than by editing the old one: the affected papers were re-extracted under the revised prompt, each result was linked to the prompt version that produced it, and the superseded run was kept beside the new one. The re-analysis also turned up three papers that needed reclassifying. The original error, its correction and the evidence for the correction all stayed in the record.

Tellingly, the most consequential silent error in the project came from a human process, not from the AI. A summary statistic derived from the corpus was miscalculated and went unnoticed for months. Because every intermediate state had been kept, the changes that introduced the error could be traced and put right. We then gathered all summary statistics into a single authoritative file checked automatically, so that any future inconsistency surfaces as part of the recorded workflow.

These cases show that version control is most valuable when it is continuous, capturing not only final outputs but the intermediate states that lead to them. Scientific papers usually preserve the analyses that worked and leave out the discarded approaches, revised decisions and failed attempts. Part of the reason is the effort a complete record takes, so important context disappears simply because nobody wrote it down. Working with AI lowers that burden: an effective AI-assisted workflow needs explicit prompts, schemas and model settings, and these are kept alongside each run as a matter of course.

Version control is also most valuable when every contributor can use it. Git is a powerful way to track changes, but its steep learning curve makes it an impractical interface for many researchers, especially those who have never used it. In a collaborative review whose contributors come from different technical backgrounds, requiring everyone to work through Git would shut some people out. Letting them work outside the shared record instead would bring back the very problem version control exists to prevent: silent, unrecoverable changes made outside the workflow. So the task is not only to keep a version-controlled record but to make it usable by everyone. We built a companion web interface onto the same record, where reviewers can read a paper beside its extraction, submit decisions, and have those decisions written into the shared history automatically.

In short, the project was built around version control from the start. One GitHub repository holds everything: the corpus, search queries, prompts, schema, every AI run and every review decision. Each piece of evidence in the review can be traced to the paper it came from, the run that produced it and the reviewer who validated it. The same record keeps human and machine contributions in step: the manuscript, schema and prompts evolve together, so every component works from the same shared state.

Lesson twoDelegate what we can judge well

Record-keeping is an important but mostly infrastructural use of AI. The bigger step is letting AI take part in the research itself: reading papers, extracting information and weighing evidence. That delegation is appropriate only when two conditions hold: (1) the AI can do the task, and (2) the researchers understand the task well enough to judge both the output and how it was produced. The second condition does the real work. Effective oversight means more than inspecting outputs: a researcher has to know in advance what a scientifically valid and useful result looks like, and what a sound route to it involves.

Screening, a core task in this project, met both conditions. Each paper was judged on its title and abstract alone, well within the AI's capability. Applying one inclusion criterion consistently across thousands of records is exactly where human judgement struggles to stay consistent, while an AI system applies the same rule the same way at scale. A reviewer could check any AI decision, because the inclusion criteria were clear and accessible. Checking every decision by hand would have cancelled most of the efficiency gained, so we audited a targeted sample instead. Of 90 sampled high-confidence rejections, none met the inclusion criteria, which puts the residual miss rate below 4% (see Text S1.5 of the supporting information). Delegation worked here because we understood the task well enough to evaluate what the AI produced.

Identifying and evaluating evidence of anthropogenic heat feedback, by contrast, initially failed the second condition. At the start of extraction we asked the AI to find papers that studied anthropogenic heat feedback and to describe the feedback mechanisms they reported. It found many papers discussing feedback-like effects, but we had no clear criteria for deciding whether they fell within the review's scope. Once we had developed the forcing–response–feedback framework, we narrowed the focus to papers that reported a feedback parameter directly or gave enough information to derive one, which gave us a clear criterion to judge AI outputs against. The lesson is that a task should not be handed to AI until people have a clear basis for evaluating the result. A researcher who cannot assess code quality should not rely blindly on AI-generated code, and in the same way scientific tasks should be delegated only when researchers know what counts as a valid and useful result within the scope of their work.

Lesson threeHuman in the loop should improve the loop

Once a task is delegated, people should stay in the loop, not only to oversee it and assess the outputs but, more importantly, to use what they see to improve the workflow. If a person is there only to inspect each AI output, the cost grows in step with the size of the corpus and the workflow learns little. If what they observe changes how the AI works in the next iteration, every review decision becomes a chance to improve the workflow.

Extracting sensitivity values was one example. Early on, a reviewer auditing the sensitivity results found papers that had been given sensitivity values on ambiguous evidence. Some described a temperature response to urbanisation in general, or to a combination of urban parameters, without isolating the contribution of anthropogenic heat flux. That finding led to criteria for when a reported sensitivity can be attributed to anthropogenic heat flux: requirements on the reported temperature response, on attribution to anthropogenic heat flux rather than wider urban effects, and on consistency between the spatial and temporal scales of the temperature response and of the heat flux perturbation. The criteria went straight into the extraction prompt, so later runs applied them systematically. The result was not just a correction to individual records but a better instrument.

The same mechanism worked elsewhere in the project. Human review showed, for instance, that the original schema did not record whether a paper estimated anthropogenic heat flux itself or took it from cited work, so this became a required field for every extraction. The cumulative effect was substantial: 627 of the 959 reviewer sign-off decisions (65%) changed the extraction. Being able to carry such improvements across the whole corpus is a major advantage of AI-assisted workflows. Once a revised criterion is in a prompt or schema, it is applied the same way across thousands of papers, which reduces inconsistent interpretation as the rules evolve. Systematic AI errors are informative too: a repeated error pattern points to a weakness in the instrument rather than a one-off mistake in a record, and shows where to improve the workflow.

ClosingWhat carries forward

This human-in-the-loop workflow has limits. AI systems remain partly opaque, runs may not be exactly reproducible, and human oversight is still essential. How well the approach transfers is also uncertain, since this experience comes from one review team and one research topic. Still, the principle looks broadly applicable: route papers by confidence, bring in reviewers where scientific judgement is needed, record the reasoning behind decisions, and let those decisions improve the instrument as well as the outputs. In this model AI does not replace expert judgement; it extends its reach, applying continuously refined criteria across large bodies of literature.

We are cautiously optimistic about working this way, not only for what it made possible but because it points to an emerging way of doing research built on increasingly capable collaboration between people and AI. It let a small team work at a scale that would once have been hard to reach. The main investment is not the tooling, which can often be adopted in days, but learning how to work well with AI as its capabilities keep changing, and that knowledge and experience carry forward.