A systematic review starts with thousands of records and two humans who must read every one. PRISMA-Local fine-tunes a small open model to put the eligible ones near the top — on hardware you control, because the records often cannot leave the building.
The problem
A review's database search returns several thousand records. Two reviewers read every title and abstract against the protocol, independently, and reconcile. It takes weeks.
What makes the task unusual is the asymmetry. One missed eligible study can invalidate the review; a hundred extra full-text checks costs an afternoon. So the useful output is not a yes/no classifier. It is a ranking with a stopping rule — put the likely includes at the top, and tell the reviewer when they can stop reading.
The measure that follows from this is Work Saved over Sampling: the fraction of the corpus a reviewer never has to open, once a chosen recall target is met.
Unpublished search exports, embargoed records and institutional agreements frequently prohibit sending bibliographic data to a third-party API. A screening assistant that requires one is a screening assistant that half of the teams who need it cannot use.
The largest model here trains on a single 16 GB consumer GPU and serves from a free Colab T4. The demo below goes further and runs in the browser tab.
The method
Each record is rendered into a prompt holding the review protocol and the candidate's title and abstract, ending in a single instruction: reply with exactly one word, INCLUDE or EXCLUDE. The model generates one token. The score is not the word it picked but the raw margin between the two logits:
score = logit(INCLUDE) − logit(EXCLUDE)That single change is what turns a classifier into a ranker. A hard label gives you two buckets; the margin gives you an ordering over all 13,665 records, which is what a stopping rule needs. It also has to be computed carefully — the first attempt collapsed nearly every record onto the same value.
4-bit NF4 quantisation of Qwen2.5-7B-Instruct with rank-16 adapters. The only way a 7B fine-tunes inside 16 GB of VRAM.
Qwen2.5-3B-Instruct in bf16, trained twice on disjoint data to give two independent screeners — the machine analogue of dual human screening.
Batch scoring with both adapters mounted on one copy of the base weights, plus prefix caching over the shared protocol header.
73 SYNERGY reviews: 67 train, 6 held out completely. The model never sees a single record from a held-out topic, so what is measured is transfer to a brand-new review — the situation a team is actually in on day one, before anyone has labelled anything. Training draws 8,000 records with negatives downsampled to nine per positive; the test set keeps each review's natural prevalence, which ranges from 0.53% to 23.8%.
| Held-out review | Records | Includes | Prevalence |
|---|
Results
Mean over the six held-out reviews. Ties are broken pessimistically — within a group of equal scores, positives are ranked last — so a model gets no credit for includes a reviewer could not have reached.
| System | AUC | Recall@10% | Screened@95% | WSS@95% | NNS@95% | Distinct scores |
|---|
Read this table with the last column. Only LoRA 3B (A) was rescored in fp32. The other three LLM rows are the original bf16 runs, where 13,665 records collapse onto a few dozen values — which costs them recall@10% specifically. The table compares one corrected system against three uncorrected ones, and it flatters row A. Rescoring the rest is unfinished work, not a footnote.
Fraction of eligible studies found after reading the top x% of the ranking. The diagonal is reading in random order.
The average hides a lot. Every system does badly on Adamo_2021, and the LLMs swing widely across topics.
Two different GPUs, so this is not a controlled comparison — the vLLM figures come from a free Colab T4 and the HuggingFace ones from an RTX 4080 Super. At 5.88 records per second an 8,000-record review takes 23 minutes on the T4.
Demo
Load a held-out review, or drop in your own RIS or CSV export. This is the TF-IDF cross-topic row of the table above, not an imitation of it — the same model, fitted by src/export_browser_model.py on the same 67 training reviews and shipped here as weights.
Pruning it to 40,000 features so it fits in a web page costs 0.003 AUC (0.697 → 0.693). The JavaScript reproduces scikit-learn's analyzer to 2×10⁻⁶ across 300 checked records. Everything runs locally in your browser; no record is uploaded anywhere, which is the same property the project argues for on the server.
Your browser calls this address directly, so vLLM has to allow this origin: --allowed-origins '["https://asifuddin01.github.io"]'. Nothing is proxied through this site.
Findings
A respectable AUC of 0.708 sat next to a recall@10% of 0.229. The instinct is to blame the model. The cause was the float path: taking the decision through a bf16 softmax collapsed 13,665 records onto 45 distinct scores. Recomputing the same margin in fp32 and storing it raw gave 13,619.
Same adapter, same records, same prompts. Only the arithmetic differs:
| Run | Distinct | AUC | Recall@10% | WSS@95% |
|---|---|---|---|---|
| A — bf16 softmax | 45 | 0.708 | 0.229 | 0.312 |
| A — vLLM fp16 margin | 5,836 | 0.710 | 0.290 | 0.334 |
| A — HF fp32 margin | 13,619 | 0.710 | 0.289 | 0.331 |
| B — bf16 softmax | 36 | 0.657 | 0.109 | 0.246 |
| B — vLLM fp16 margin | 3,967 | 0.659 | 0.167 | 0.286 |
AUC moves by 0.002. Recall@10% moves by 26% for screener A and 53% for screener B. AUC is a rank statistic that averages tied ranks, so a mountain of ties barely dents it. Top-k selection has to break those ties, and a reviewer reading down the list gets whatever order the tie group happens to sit in. The metric that survived the bug is the one most screening papers lead with.
Every record in a review shares a prefix: the system message and the protocol header, everything before that record's title. Cache it once and the rest of the review skips re-computing it.
Measured on 800 records: 191.5 s without the cache, 137.2 s with it — a 1.40× speedup, 28.4% less wall-clock time. The shared prefix is 98 tokens of a 428-token mean prompt, 23%. The saving slightly exceeds the prefix's share of the tokens, which is the right direction: only one token is generated, so the run is essentially all prefill, and attention over the cached prefix is quadratic in position — the tokens skipped are cheaper to skip than their count suggests.
The record body still dominates. The header is not where the win is, and a longer protocol would move this number a lot.
Correction to an earlier draft. These were previously written as a 300–400 token header inside a ~900-token prompt, capping the saving near 35–40%. Measured against the tokenizer, no prompt in the test set reaches 900 tokens (mean 428, max 690) and the 1024-token limit never binds. The figures above are the measured ones.
What this is not
Published WSS@95% figures for screening tools mostly come from within-review active learning: the tool gets labelled seeds from the very review being screened and retrains as the human works. That is a substantially easier problem than the one measured here, and quoting the two side by side without saying so would be misleading.
So both are computed. Active learning reaches WSS@95% of 0.659. The best fine-tuned model, given no labels from the review at all, reaches 0.331 — and a plain TF-IDF logistic regression trained on the same 67 reviews reaches 0.294.
The gap is the finding. A local fine-tuned 3B decisively beats zero-shot prompting (0.331 vs 0.129), which is worth knowing. It does not yet beat a linear model on the same data, and it is half as good as the technique practitioners already use. Reporting that plainly is the point of computing both.
The criteria field is empty for all 73 reviews in this build of the corpus, and review_title holds a citation key. The protocol block the model actually saw was:
### REVIEW PROTOCOL
Topic: Butink 2023
Eligibility criteria:
See review topic.So the model could learn what an includable record looks like in general, not whether one matches this protocol. That is a fair description of the ceiling these numbers sit under, and supplying real criteria is the first experiment to run next.
One epoch, rank-16 adapters, no hyperparameter search. Title and abstract only. Throughput spans two GPUs. Six held-out reviews is a small sample to average over and the spread is wide — LoRA 3B (A) ranges from AUC 0.39 on Adamo_2021 to 0.89 on Butink_2023, so the headline mean hides a system that fails outright on some topics.
Run it
The adapter is 119 MB and mounts on stock Qwen2.5-3B-Instruct. One GPU, no fine-tuning needed.
git clone https://github.com/asifuddin01/prisma-local
cd prisma-local && git lfs pull
vllm serve Qwen/Qwen2.5-3B-Instruct \
--enable-lora \
--lora-modules screener=./adapters/lora-3b-a \
--max-lora-rank 16 \
--enable-prefix-caching
pip install -r requirements.txt gradio
python app/app.pyThen either use the Gradio app, or point the demo above at http://localhost:8000 and keep this interface.
Every metric in this page recomputes from the per-record scores in results/scores/. No GPU required.
pip install -r requirements.txt
python src/export_site_data.pyThe three stages are notebooks: training and HF scoring on a 16 GB Windows GPU, vLLM on a free Colab T4, analysis on a laptop CPU. Only the adapters travelled between them — which is the practical argument for LoRA as a deployment format, and what made the split viable when WSL was not available.
The SYNERGY records are not redistributed here beyond the two held-out reviews the demo ships. results/split.json pins the exact 67/6 review split.