DragonSlayer

We built a tool that generates radiology reports ~5x faster than PowerScribe. We also built RadReportBench, a benchmark that measures an AI’s ability to generate signable radiology reports from real dictations in a single shot.

Introduction

Radiology is a profession near and dear to my heart. Both my parents are radiologists. My grandmother was a radiologist, and I spent most of my childhood in the radiology center my grandfather owned.

Over the decades, I’ve seen the radiology industry fall apart for patients and for doctors.

My dad used to serve his patients. He put on his scrubs, showed up to his practice, read cases and walked patients through their findings with love and care. Now, radiologists are shoved into small dark rooms, get lousy tech, and for process efficiency’s sake, they crunch reports while others share the news with patients.

Our deep dive into the field made me realize radiology looks a lot like coding did ~12 months ago, when putting AI in the loop made us 10x more productive and prolific.

The same way good engineers now spend far more time talking to customers and working on only the hardest problems, radiologists should be able to spend far more time with patients and still read an order of magnitude more cases. Hopefully, that leads to a world where people have access to cheaper and more frequent imaging.

We were surprised to find that a large part of the bottleneck preventing this future is purely technical.

Three big buckets of innovations are required. First, AI needs to be able to read medical imaging with more nines of reliability (from 9x.xx% to 99.999%). Second, we need delightful and ubiquitous imaging hardware (rooting for you, David Holz¹!). Lastly, we need to automate image reporting. Incredible teams are working on the first two.

DragonSlayer is focused on the third bucket. A few other good teams are pushing on this lever too, like Rad AI, RADPAIR, and DeepHealth. Microsoft owns PowerScribe and is starting to support AI-generated impressions.

Unfortunately, if you’re deep in AI, you will be saddened to learn what the current state of the art looks like in radiology.

The current problem

Radiologists spend ~60 minutes a day dictating and signing reports.

Radiologists used to have transcriptionists. A radiologist would read an ultrasound upper abdomen and simply say, “Abdomen pain. CBD four. Gallbladder absent. Rest negative”. Doctors would dictate for ~5 seconds and their transcriptionist would return a signable report.

This changed ~30 years ago, when the first speech recognition software (Dragon!)² hit the industry. Private practice gave way to consolidated groups that cut costs (like transcriptionists) with new technology. The tech was not ready, because real transcriptionists were never speech-to-text machines, but wonderful speech-to-intent professionals.

Today, about 80% of radiologists use PowerScribe^3,4 (which still runs Dragon speech recognition).

With PowerScribe, that 5-second ultrasound example now takes ~25 seconds, but the problem gets worse on harder modalities.

Here is how radiologists use PowerScribe to generate a realistic PET/CT report (Fig. 1).

Yes, you are seeing that right. Doctors have to manually click into each section of a template, hit record, perfectly transcribe each word (they even have to say things like “period” between sentences), and repeat 5-10+ times a report.

This case takes about 10 minutes to read, but generating this report takes ~5 minutes in PowerScribe. That means radiologists can spend up to one third of their reading time on filling out reports. That same case should have only taken 30 seconds to dictate to a transcriptionist.

Obviously, a doctor should just speak like they were talking to a good transcriptionist and the final report should appear for approval.

That’s exactly how DragonSlayer works.

Fixing the problem with DragonSlayer

DragonSlayer is a speech-to-report generation tool that can save radiologists 30-50 minutes a day.

It takes in audio dictation, searches among your templates and macros, makes precise edits to generate the final report for approval, and lets you train it with your styles and preferences.

DragonSlayer turns the reporting time for that single PET/CT case from 5 minutes to ~50 seconds (6x improvement).

DragonSlayer gives radiologists the transcriptionist support they were robbed of decades ago.

Here is a demo of me using DragonSlayer for both cases (Fig. 2).

Just like human transcriptionists of old, it understood the intent behind what I said and was far more accurate and faster as a result (Table 1).

The technical challenge

Yes, it is realistic to speak that fast to a good transcriptionist. Listen to this real 8-second dictation snippet from a radiologist (Audio 1) to get a feel for how hard this can be.

Being allowed to dictate at this speed and at this level of abstraction can make radiologists 10x faster, but it is a very difficult technical problem to solve.

Doctors mumble through their own vocabulary at extreme speeds and correct themselves. Each dictation must find its matching template and macros (among hundreds). Each report then requires multiple precise edits and uncaught errors carry catastrophic consequences. Speed matters, so we don’t have the luxury of verification loops.

Peter and I tackled these problems the same way all hard AI problems are solved, with a purpose-built benchmark that fully encapsulates the problem.

Building RadReportBench to improve DragonSlayer

At Off Policy, we believe building exceptional agent products requires hill-climbing purpose-built and expert-level evals.

Evals for radiology dictation to report generation didn’t exist, so we built one. We made sure that it measured the full job of the doctor, and every problem radiologists ran into on DragonSlayer or PowerScribe: accuracy, speed, cost, and style.

This way, improving our score on RadReportBench directly improves DragonSlayer for our doctors.

We collaborated closely with professional radiologists to build RadReportBench.

It has 55 audio dictations and matching final reports signed off by radiologists (38 ultrasounds, 7 MRIs, 4 X-rays, 4 CTs, and 2 PET/CTs). The cases are synthetic, but directly inspired by real cases.

Because a doctor might sign off on variations of the same report, a verifier agent grades each final report against the signed-off golden report.

On DragonSlayer, frontier models are close to human level transcriptionists, and orders of magnitude cheaper. Opus 5.5 is the state of the art on the Pareto frontier, reaching 81% in 17 seconds (Fig. 3a). 71% of its reports were signable (GPT-6.1 Sol: also 71%, 3.5x cheaper), and about a quarter were perfect.

A 10 is (nearly) identical to the golden report. An 8-9 has only small differences, but is still signable. Below 8 can’t be signed. In the demo, Opus 5.5 low scored a 10 and a 7 (a 9 after one voice edit) in ~10 and ~50 seconds.

Costs were the original reason the industry moved away from transcriptionists. A transcriptionist costs about 10% of what the doctor makes per case: $1-2 for ultrasounds, $3-5 for MRIs and CTs.

GPT-6 Luna high reaches 76% at less than 1/5th of a penny ($0.002), and Opus 5.5 medium 81% at about $0.06 a case (Fig. 3c). It turns out speed is a bigger constraint than cost.

How we improved accuracy and what we learned

We landed on the following schematic after weeks of hill-climbing the benchmark and iterating against radiologist user testing.

Improvement 1/3 – Bitter lesson x 2

Early on, one of the biggest gains was moving from dedicated speech-to-text models like Parakeet to Gemini 3.7 Flash. If you were to look at STT benchmarks⁵, you’d think that ElevenLabs’ Scribe, MAI Transcribe, or Gemini 3.5 Transcribe would perform best. After all, they are specifically trained to transcribe speech. However, none of them could piece together extremely rapid and highly technical speech, the way that radiologists would dictate to a competent transcriptionist. In contrast, Gemini 3.7 Flash is trained as an LLM, and when it was provided with a single line context (“You are a radiologist transcriptionist”), it did far better than all of these models.

As a consequence, Gemini 3.7 Flash scored 7% higher and stopped sending us to the wrong template. This felt like our first bitter lesson⁶. You don’t want good small models generating exact text; a smarter multimodal model that understands context and intent generalized and scored better.

Then Opus 5.5 class models arrived and showed us a new bitter lesson. Models are now so good that they can do the full job of reasoning over and building from confusing literal transcriptions from smaller local models. The extra step was superfluous. Scores stayed at 80% either way, and moving back to a local model cut transcription time by over 90% (0.2 s vs ~3 s).

Improvement 2/3 – Edit format

The bar for accuracy in radiology looks more like self-driving than coding.

DragonSlayer must maximize accuracy under a speed constraint that limits the use of verification loops.

We tested three edit approaches: full rewrites, coding style patches (as in Codex⁷), and exact edits (as in Claude Code⁸).

Switching to exact edits cut failed edits from 30% of reports to under 1%, raised the average score on a control case 28% (7.5 to 9.6 out of 10), and made voice edits 40% faster.

{ "old_text": "measuring _____ mm", "new_text": "measuring 4 mm" }

@@ FINDINGS:
- CBD: The common bile duct is normal in size, measuring _____ mm.
+ CBD: The common bile duct is normal in size, measuring 4 mm.

FINDINGS:
LIVER: Normal in size and echotexture.
CBD: The common bile duct is normal in size, measuring _____ mm.
PANCREAS: Visualized portions are unremarkable.

FINDINGS:
LIVER: Normal in size and echotexture.
CBD: The common bile duct is normal in size, measuring 4 mm.
PANCREAS: Visualized portions are unremarkable.

Improvement 3/3 – Splitting search from work

Our first version did everything in one call: the transcript, a doctor’s 235 templates and 87 macros, about 93k tokens. We split it in two: a quick search call that picks the right template, then a writing call that sees only those templates, every macro, and the doctor’s rules. The writing prompt shrank 4.3x and scores rose about 3%, for roughly one extra second.

DragonSlayer performance improvements after hill-climbing RadReportBench

Each of these improvements significantly improved our doctors’ user tests. When we started, the median case required at least one edit before radiologist approval. We also had a ~30% breaking error rate.

By the end, over 80% of DragonSlayer generated reports were accepted without a single edit (in line with the 89% our radiologists would sign, Table A1), and 95%+ after a single edit.

Many of the hardest cases fail because of hard to decipher audio (Appendix 2).

Next steps

DragonSlayer has reached a performance point where first-time user testers are pretty consistently shocked and request access. But medicine is a heavily regulated and slower moving industry, where the buyers of technology don’t always have the same incentives as the users (the same reason radiologists have been treated more and more like factory workers).

We believe we can hill-climb this frontier model class towards ~90% accuracy while still improving speed. Despite being multiples faster than PowerScribe, the dream speed from transcription completion to report is closer to < 3 seconds. Accuracy and speed are bottlenecks we love to solve, but we’d like to work with partners on PACS / RIS integration and aligning buyers with radiologists.

In an ideal world, everyone would get a full imaging workup as often as twice a year, at little to no cost, and catch more scares before they became terminal.

We are a long way from that future, but advancements in AI medical reading, new hardware, and report automation will make it possible. DragonSlayer is our first step, giving radiologists back 30-50 minutes a day.

Hopefully, that means doctors and patients can spend a bit more time together again, the way my dad and his patients used to.

We’re looking for a small number of partners who believe in the same wonderful future for radiology, from medical groups to research labs.

Reach out to [email protected] if you are interested in collaborating on research, licensing our technology, or deploying DragonSlayer for your practice.