An AI agent writes better-rated health notes by learning from past corrections

An AI agent called EvoNote produced health-related Community Notes that scored better than existing human-written notes under a GPT-4.1-based utility judge. This is a benchmark result about candidate notes, not proof that an AI fact-checker is more accurate or more effective at correcting beliefs in the real world.

A new paper from the National University of Singapore introduces EvoNote, an LLM agent that generates health Community Notes by accumulating lessons from its own past correction episodes. The system doesn't just retrieve similar cases — it distills what went wrong and right in each episode into reusable, phase-tagged memory items, then draws on those items in future corrections. The result is a system that improves continuously as it processes more cases.

The anchoring number: only 10.1% of health-related Community Notes in the wild are ultimately rated Helpful by the crowd — out of 48,295 notes analyzed.

Many Human Notes Had Not Reached a Crowd Verdict

The crowd-sourced model has two compounding failure modes. First, it's slow: the median time from post creation to a first note is 6.1–7.1 hours, and then another 6.4–7.3 hours pass before the crowd reaches a helpfulness verdict. These delays can limit timely correction, but the benchmark does not measure how much harm occurs before a note appears.

Second, most notes in the collected dataset had not reached a verdict at the time of observation. Of 48,295 health-related notes collected through October 2025, only 13.12% received any crowd-derived helpfulness label. The remaining 86.88% were marked "Needs More Ratings" — an unresolved status, not a judgment that the note is unhelpful.

Even the notes that do get rated aren't reliably good. When evaluated under a hierarchical judge that checks evidence relevance, evidence correctness, and communication quality separately, only 38.33% of human-written notes pass. Crowd-rated helpfulness, it turns out, doesn't catch systematic evidence failures — notes can earn community approval while citing weak or tangentially relevant sources.

How EvoNote Accumulates and Reuses Correction Experience

EvoNote runs four coordinated agents in a closed loop: a Claim Analyzer, a Note Writer, a Social Utility Judge, and a Memory Evolver. After each correction episode, the Memory Evolver distills what happened into structured memory items — each carrying a Trigger (what situation prompted the action), an ActionStrategy (what the agent did), and a Phase label (which step in the pipeline the lesson belongs to). On the next similar case, the system retrieves only the items relevant to the current phase rather than replaying full past trajectories.

This design matters because health misinformation has recurring patterns. A post making an unsupported claim about a medical mechanism will appear dozens of times in different forms. If an agent successfully corrects one instance, the lessons from that correction — which sources were authoritative, how to frame the rebuttal clearly — should transfer to the next. Without cross-episode memory, each correction starts from scratch.

The paper illustrates the cost of this gap directly. A timeline analysis shows that an earlier helpful note for a medical mechanism overclaim contained reusable lessons that later notes for structurally similar posts never accessed. Those later notes were rated Not Helpful. The lesson existed in the system's history; it just wasn't being retrieved.

Across 1,200 episodes in the MM-HealthCN benchmark, EvoNote accumulated 4,277 memory items. Ablations confirm that ongoing accumulation matters: a frozen memory bank — loaded once and never updated — achieves a 84.75% win rate, while the continuously evolving version reaches 89.59%. The gap isn't enormous, but it's consistent, and it grows as the system processes more cases.

Performance Against Human Notes and Competing Agents

EvoNote scored 89.6% on the pairwise comparison with human-written notes; wins count as one and ties as half. The GPT-4.1-based judge agreed with human majority votes on 98 of 100 sampled pairs. This score measures judged note utility, not factual accuracy or correction success after publication. That preference holds even when the human note is already good: in cases where human notes pass the hierarchical quality judge, EvoNote still wins the pairwise comparison 74.1% of the time. When human notes fail the judge, EvoNote produces a helpful alternative in 77.0% of those cases.

On a separate set of 300 recent posts that never received a crowd verdict — the Needs More Ratings cases that represent the bulk of the backlog — EvoNote produces helpful notes 82.0% of the time.

The speed advantage is substantial. EvoNote takes a mean of 112.6 seconds per case on a single H200 GPU. DeepResearch, a strong agentic baseline, takes 153.0 seconds and achieves a 78.09% win rate. ReMem, which asks the agent to infer reusable lessons on the fly rather than storing them explicitly, takes 118.3 seconds and reaches 74.50%. EvoNote is faster than both and more accurate than either — the structured memory retrieval is doing real work, not just adding overhead.

On evidence quality specifically, EvoNote leads across all five measured dimensions: source quality (72.93), domain diversity (93.32), semantic diversity (22.56), average URLs cited (1.97), and multi-source rate (68.92%). The notes aren't just preferred stylistically — they're grounded in broader, higher-quality evidence.

Multimodal Handling and Where the System Breaks

Health misinformation on X isn't just text. EvoNote handles image and video posts through a caption-based preprocessing step — Nanonets Docstrange for images, Gemini-2.5-Flash for video — rather than feeding visual content directly to the language model. Caption quality validation on 200 sampled instances showed majority-vote accuracy of 0.95 for images and 0.96 for video, suggesting the preprocessing is reliable enough to support downstream reasoning.

The alternative approach — direct visual language model inference using MedGemma-27B — failed badly. Of 400 image cases tested, 63 runs remained invalid even after repair heuristics, producing ungrounded answers, schema mismatches, or timeouts. Even the valid outputs underperformed the caption-based pipeline and fell below the no-memory LLM baseline. The lesson is that multimodal grounding requires explicit pipeline design; a capable vision model alone isn't sufficient.

The deeper risk is memory propagation. EvoNote learns from its own trajectories, which means a flawed correction — one that cites a poor source or misrepresents a mechanism — can generate a memory item that influences future corrections. The authors flag this directly: memory auditing, decay, and deletion are unresolved requirements for any real deployment. A system that improves through experience can also entrench its mistakes through experience. There's also an automation bias concern specific to health contexts: notes that look authoritative and well-sourced may be trusted even when the underlying evidence is incomplete or selectively cited.

What This Means for Health Correction at Scale

The practical implication isn't that EvoNote should replace human annotators — the paper doesn't claim that, and the caveats around memory propagation and automation bias are real. The unresolved-note backlog is a reason to investigate faster candidate drafting. This study did not test whether recruiting more volunteers would solve that backlog, and its benchmark results do not establish that automated notes should bypass human oversight.

For teams working on health misinformation infrastructure, the more actionable finding is the memory architecture itself. Phase-specific, action-level memory items that accumulate across episodes outperform both full-trajectory retrieval and on-the-fly lesson inference — and they do it faster. If you're building correction or fact-checking pipelines, the design question isn't whether to use retrieval-augmented generation, but what unit of experience to retrieve. EvoNote's answer — structured lessons tied to specific workflow phases, not raw past outputs — is worth testing in other domains.

The system currently handles English health claims only, and the authors note that political, financial, or sociocultural misinformation may require different utility criteria entirely. But the core mechanism — an agent that gets better at a task by distilling its own correction history — is domain-agnostic. The health domain just happens to be where the need is most acute and the human pipeline most visibly broken.