Benchmark ● seed 20260731
Measured, not asserted.
Three aggregators, four simulated indexers, 30 repetitions per query. The harness, the corpus and the method are in the repository, so you can run it yourself and disagree with a number instead of with us.
make -C bench matrix
99.5%
Recall
NZBHydra2 95.8% on the same corpus
21.1%
Duplicate rate
50.6% fewer repeated rows than NZBHydra2's 42.6%
3.3 MB
Idle memory
209× less than NZBHydra2's 689 MB
7 ms
Cold start
206× faster to first correct answer
Where it is a draw: at realistic indexer latency our p95 is 232 ms against their 236 ms. That gap is smaller than the drift between two runs of the same build, so it is a tie, and the table below marks it as one.
Coverage and deduplication
The two numbers that hold each other honest.
Either alone is trivially gamed: return the catalogue untouched and recall is perfect, merge everything into one row and duplication is. Every chart here starts at zero and keeps its reference bars on the same axis, because a cropped axis is how a three-point gap gets drawn as a rout.
Recall — distinct uploads returned
Of every unique upload that exists in the corpus, the share the aggregator gave back. Higher is better.
- prowlarr syncs indexers instead of aggregating them; it does not deduplicate by design
All three are high and the gap is small. It holds on all four corpora, but read the stability charts before treating it as a headline.
Duplicate rate — repeated rows in the answer
Of every row returned, the share repeating an upload already in the answer. Lower is better.
- prowlarr syncs indexers instead of aggregating them; it does not deduplicate by design
- no merging the corpus served untouched — the score any deduplicator has to beat
A tenth of the corpus is generated past the merge tolerances both products use, so nothing can reach zero. The dashed line is that floor, and our distance to it is a gap left to close, not a verdict.
Recall with one indexer down
The same run with one of the four simulated indexers failing. 5.4% of uploads live only on it and are unreachable by anyone.
- prowlarr syncs indexers instead of aggregating them; it does not deduplicate by design
Not falling over when one indexer does is where the prior research put the real value, so it gets measured instead of asserted.
Latency
A tie at the tail, and one scenario we lose.
A fan-out cannot finish before its slowest indexer, and the mocks answer in 80–230 ms. Both aggregators sit on that floor, so the realistic scenario describes the harness more than the products. It is published with that warning rather than used as an argument.
p95, indexers at realistic latency
What a client waits for on a slow search. Lower is better.
- prowlarr 5244 ms, of which ~150 ms is the indexer. The rest is internal to Prowlarr's manual-search endpoint, which Sonarr and Radarr do not use in normal operation
The lead between the two aggregators is smaller than the drift between two runs of the same build. That is a tie, and it is marked as one.
p50 with the indexers answering instantly
The wait removed, leaving the aggregator's own work: parse, bucket, dedup, serialise. Lower is better.
NZBHydra2 is ahead here, outside the tie band. It is on the page because a benchmark that shows only the rows its author wins is an advert.
CPU per request
Read from the container cgroup. Time spent waiting on an indexer cannot hide inside this one. Lower is better.
The honest efficiency metric: work actually done, not time spent waiting for somebody else to do theirs.
And the speed is conditional. Fetching a window's pages at once costs about 85% more API calls, so Nexarr only does it against indexers with no quota cap. If your indexers are on free plans with a daily call limit you get the sequential walk and none of that speed — deliberately. Halving your searches per day to save 200 ms is a bad trade, and we would rather protect the budget than the benchmark.
Footprint
What it costs to leave running.
An aggregator idles most of the day waiting to be asked something. NZBHydra2 has not run on a JVM since its v5 in 2023 — it compiles to a native binary with GraalVM — so this is one compiled binary against another, not the "heavy Java" argument that stopped being true three years ago.
Idle memory (RSS)
Median of seven samples per subject, taken with every container idle.
Container image size
Cold start to first correct answer
Median of three restarts. One restart alone once reported 265 ms for Nexarr against a real 7 ms, and that figure reached a published table — a single observation is not a measurement.
Results
| Metric | nexarr | nzbhydra2 | prowlarr 5 reps |
|---|---|---|---|
| Recall (unique uploads) | 0.995 | 0.958 | 0.992 |
| Duplicate rate | 0.211 | 0.426 | 0.589 |
| Recall, one indexer failing | 0.850 | 0.797 | — |
| Latency p50 | 198 ms tie | 199 ms tie | 5206 ms |
| Latency p95 | 232 ms tie | 236 ms tie | 5244 ms |
| p50, indexers instant isolates the aggregator's own work | 4.0 ms | 3.7 ms | — |
| CPU per request | 11 ms | 24 ms | — |
| Image size | 26 MB | 285 MB | 221 MB |
| Idle memory | 3 MB | 689 MB | 123 MB |
| Cold start | 7.0 ms | 1444 ms | 1739 ms |
A highlighted cell is the best value for that row. Where the lead is inside the drift we measure between runs of the same build, the row is marked a tie instead — a 3 ms lead on a 230 ms measurement is not a finding. Prowlarr does not deduplicate by design and is not an aggregator; its row values describe a different job.
How to read this
Recall and duplicate rate, together or not at all.
Recall is the share of distinct uploads returned. Duplicate rate is the share of returned rows that repeat one already in the answer. A tool that merges too eagerly looks like it returns fewer results and is in fact worse, which is why neither number appears on this page without the other.
Both Nexarr and NZBHydra2 gate a merge on the same tolerances — 1% of size, two hours of age — and a tenth of the corpus is generated beyond them deliberately, so no parser can reach zero. On this corpus the floor is 9.5%, merging nothing at all scores about 60.5%, and our 21.1% sits between the two.
Does it hold on another corpus?
Four seeds, and the two metrics do not hold up equally.
One corpus gives one number, and one number cannot tell a real difference from a lucky draw. The measurement was repeated against 4 independently generated corpora. Each band below spans a subject's worst and best seed; each dot is one seed.
Recall across 4 corpora
share of distinct uploads returned — higher is better
Duplicate rate across 4 corpora
share of returned rows repeating one already in the answer — lower is better
Seeds: 20260731, 20260801, 20260802, 20260803. Each is an independently generated corpus, not a repeat of the same one.
Deduplication holds. The gap between us is around 20 points on every seed, roughly three times either product's own spread across corpora, and the two bands come nowhere near overlapping. That is a property of the implementations, not of one lucky draw.
Recall is thinner than the headline suggests. We come out ahead on all four seeds, but by 0.9 to 2.4 points — comparable to how much NZBHydra2 alone moves between corpora. Our worst seed and its best seed very nearly touch, which the chart shows and an average would have hidden. Consistent in direction, and we would not claim more than that from four corpora.
Method
How these numbers were made.
Every step below is in the repository — generator, mocks, compose files, driver and scripts. Nothing here contacts a real indexer, which is both a hard rule of the project and the reason the run is reproducible at all: a measurement that spends somebody's API quota can only be taken once.
- 01
The corpus is constructed, never labelled
500 canonical uploads are generated from seed 20260731, then each is projected onto one or more of four simulated indexers by applying naming transformations taken from public conventions: separators, tag order, group abbreviations, language suffixes, PROPER/REPACK, extension noise. Two entries are the same upload because they were generated from the same upload — nobody judged them alike, so nobody's judgement is in the ground truth.
Our own dedup tuning corpus (testdata/dedup) is deliberately not used here. Scoring a parser against the corpus it was tuned on measures memory, not deduplication.
- 02
A tenth of the corpus is unmergeable on purpose
Copies of one upload diverge by ±0.3% in size and ±15 minutes in age — the divergence a real indexer shows, since the date comes from the same NNTP headers and the size differs only by segment and par2 accounting. 10% are generated past the merge tolerances both products use (1% size, 2 hours age) as hard cases.
That sets a floor: on this corpus no implementation can score below 9.5% duplicates. Merging nothing at all scores about 60.5%. Both bounds are on the chart, so the numbers between them can be read.
- 03
The indexers are simulated, and they misbehave on purpose
Four mock Newznab servers, each holding a controlled subset of the corpus, with configurable latency and jitter, real pagination, and injected faults: HTTP 500, timeouts, HTML where XML belongs, inflated result totals, exhausted quota, a short page mid-catalogue. Every subject meets the identical failure script.
Whether a request fails is a pure function of that request — indexer, type, query, category, offset, limit — not of how many arrived before it. It used to be arrival order, which meant a build that paginated differently drew different failures and moved the degraded score by ~0.003 for reasons that had nothing to do with resilience.
- 04
Four scenarios, because one number answers the wrong question
Realistic latency (indexers answer in 80–230 ms) measures what a client waits for. Zero latency removes the wait and leaves only the aggregator's own work — parse, bucket, dedup, serialise. CPU per request is read from the container cgroup, which nothing can mask by waiting on somebody else. Degraded runs the whole set with one indexer down.
In the realistic scenario a fan-out cannot finish before its slowest indexer, so every subject sits on a ~150–230 ms floor built by our mocks. That scenario's wall-clock cannot support a comparative claim in either direction, and it is published saying so.
- 05
Repetition, warm-up, and a host that has to be quiet
30 repetitions of each of the 111 queries per subject, the first 3 discarded as warm-up, p50 and p95 reported — never a bare mean. Subject order is randomised so thermal drift is shared. The host is recorded in the result file, load average included: it was linux/amd64, 16 CPUs.
The harness refuses to start above a quarter of the cores in load. A full matrix once ran beside an unrelated LLM server holding 783% CPU: every timing came out roughly four times worse for both subjects, while coverage was bit-identical — so the metrics one sanity-checks first looked perfect. Those figures came within an hour of being published.
- 06
Every correction to the harness is logged, including the flattering ones
The method document was written before anything was run, and each change since is recorded with its date, its cause, and who it favours. The mock used to reject requests with no API key, which made NZBHydra2 disable all four indexers and return nothing — a spectacular result, entirely our bug. The driver used to request one page, which penalised the subject that honours its own advertised page ceiling: fixing that improved our numbers.
Both directions are documented at the same length, on purpose. A correction log that only contains fixes helping the author is not a log, it is a filter.
Run it yourself
Four commands, one Docker host, no account required.
Needs Docker with Compose and a Go toolchain. Expect about two hours for the full matrix — Prowlarr answers in ~5 s and dominates the wall-clock — or a few minutes with the quick pass.
git clone https://github.com/nexarr-app/nexarr
cd nexarr/bench
make up # 4 mock indexers + all three aggregators, in Docker
make configure # point NZBHydra2 and Prowlarr at the mocks
make bench # the measured run -> out/results.json
make report # render the markdown report
make down # tear it all down, volumes includedOr unattended, exactly as the published run was made:
./scripts/run-matrix.sh # everything (~2h)
QUERIES=30 ./scripts/run-matrix.sh # quick pass
./scripts/multi-seed.sh # repeat on independent corpora
The knobs are SEED, UPLOADS,
INDEXERS, REPS and QUERIES. Change
the seed and you get a different corpus with the same construction — the
cheapest way to check whether a result of ours was a lucky draw.
The matrix script stops if the machine is busy. That is the guard, not an
obstacle: set ALLOW_BUSY_HOST=1 only for a coverage pass,
where load does not change the answer, and never for timings.
What this cannot tell you
- Synthetic corpus: ground truth is exact, the distribution is not Usenet's.
- Prowlarr syncs indexers rather than aggregating them; it does not deduplicate by design, and it does torrents, which Nexarr does not.
- No real indexer is contacted, which is what makes the run reproducible.
- Speculative pagination only speeds up indexers with no quota cap. On a capped account the walk is sequential and the latency is unchanged, by design.
- The benchmark's author wrote one of the three subjects; every correction to the harness is logged in the method.