What oracle-anchored means

Fidelity is a comparison, so a fidelity score is only ever a statement about two things: the output being measured, and the thing it was measured against. Change the second one and the number changes without a single line of the engine changing. That is why a score handed to you without its anchor is not a weak claim — it is not yet a claim at all.

This page is the reader’s guide. It explains the three questions worth asking of any published rendering score, including ours, and it publishes no number itself. The method behind our own figures is the fidelity methodology; this is the shorter, prior question of how to read one.

Question one: what was it measured against?

Three anchors are in common use, and they answer three different questions.

Self-anchored. The output is compared to the same engine’s earlier output. This is a genuinely useful regression signal — it catches the day a refactor moves a table — but it measures stability, not correctness. An engine can be perfectly self-consistent and consistently wrong, and a self-anchored score will never say so.

Anchored to another renderer. The output is compared to what a second implementation produces for the same file — commonly a free office suite such as headless LibreOffice, because it is scriptable and free to run at corpus scale. This is a real and honest measurement, and it answers a real question: how close is this output to that renderer’s output? What it cannot answer is how close either one is to Office, because the reference renderer has its own layout engine and its own divergences. Measuring against it inherits them. A high score against a renderer that itself differs from Excel is not evidence about Excel — it is evidence about agreement, and the two are not the same thing.

Anchored to the oracle. The output is compared to what the source application itself prints for that workbook. For Rendlio Sheets that is Microsoft Excel’s own print output, and the reason it is the right reference is that it makes the score falsifiable: when the engine and Excel disagree, the engine is wrong by definition and the difference is our defect, not a judgement call. Nothing about a self-anchored or renderer-anchored score can be wrong in that way.

The point that matters when reading someone’s published figure: the anchor is a property of the test harness, not of the engine. The same engine, the same output, re-anchored, produces a different number. So the anchor has to be published alongside the score, or the score cannot be interpreted — and if it is not stated, the honest reading is that you do not know what was measured.

Question two: who produced it, and can you re-run it?

Two different artifacts get called “fidelity evidence”, and they carry very different weight.

A corpus score is a claim about a fixture set the publisher chose. Publishing one is good practice and better than publishing nothing — most converters, including commercial ones costing real money, publish no measurement at all. But it is self-generated by construction, and its value to you depends entirely on what travels with it:

the corpus itself, and its content hash, so you can prove you hold the same inputs;
the anchor, named;
the rubric — the tolerances a page had to meet to count as matching;
enough of the harness to re-run the whole thing yourself.

With those four, a corpus score is checkable by a stranger. Without them it is a number on a page, and a reader is being asked to extend trust rather than to verify.

A per-conversion report is a different kind of artifact entirely, and the difference is that it is not about a fixture set — it is about your file. Rendlio Sheets emits one with every conversion: each feature it met, its status, a warning code where something deviated, the locations, and an exit code your pipeline can gate on. It is produced by the same run that produced your PDF, so there is no selection step in which an inconvenient case could be left out. A corpus score is a claim about our fixtures; the report is a claim about your document, and it is the one we are willing to be held to per file.

That is also the order the assurance stack should be read in. The per-conversion report comes first, because it is the artifact that speaks about your file rather than about our fixtures. Then determinism — identical input and options producing identical bytes, which is also the property a downstream signing step depends on, since it lets that step re-derive what it signed and compare rather than take it on trust. Then the hardening and security posture around a file you did not write. Archival output such as PDF/A-2b matters and is included, but it is a capability, not the argument: several free tools ship it, and a conformant PDF that lays the page out differently from Excel is still the wrong document.

Question three: the average, or the worst case?

An average pass rate answers “how did this fixture set do overall”. It is the right summary for tracking progress and the wrong one for deciding whether to trust a document, because the documents that matter are not average.

Audit-grade work fails on its worst page. One invoice with a dropped column, one statement whose multi-page table collapses onto a single sheet, one archived record that no longer matches the workbook it came from — any of those costs more than a hundred correct conversions saved. A score that reports only the central tendency is silent about exactly the case that decides the purchase.

Two structural consequences follow, and they govern what this site is allowed to publish:

No aggregate figure, ever. There is no single headline fidelity number for Rendlio Sheets and there will not be one. Which number you would get depends on how you weight features, and that weighting is precisely the judgement a reader is trying not to delegate. Scores are per rendering area, and the failures are published with the passes.
The distribution is part of the claim. An area publishes its pass rate together with what the misses were, in the same four-status vocabulary your own compatibility reports use. “Where does it break” is the question a buyer is actually asking.

Why the shape of the exam matters

There is a related failure mode that no single number exposes, and it is worth naming because it is easy to fall into honestly, without anyone intending to mislead.

An engine tuned against a fixture set it can see will get very good at that fixture set. If layout decisions are calibrated to make known cases come out right — a width table nudged here, a spacing adjustment there — the score on those cases rises without the underlying model getting any more correct. The result is a high score on the fixtures and a much lower one on documents drawn from somewhere else. The gap between those two is the useful measurement, and a publisher who only ever scores the visible set cannot see it either.

So the set a score came from matters as much as the anchor it was measured against, and three kinds are worth telling apart — a question to put to any publisher, ourselves included:

A public corpus — the practice exam. It ships with the method, so anyone can re-run it. Anyone can also tune against it, the publisher very much included, which is exactly why its score alone is not the claim.
A set held back — the exam nobody tunes against, because nobody sees it before the run. Its whole function is to catch the difference between an engine that models layout and one that has memorised a fixture set.
Your own workbook — the final. It is the only one that decides anything for you, and for Rendlio Sheets it is the one the playground and the per-conversion report exist to answer.

An engine that is genuinely modelling the format scores similarly on all three. One that has been fitted to the visible cases does not, and the drop is the finding. What our own corpus contains is the fidelity methodology’s to state, not this page’s; this page only names the question worth asking of it.

What this means for a number on this site

The rules are the same ones the fidelity methodology sets out, restated as what a reader can hold us to:

Rule	What it rules out
Every figure names its anchor, its corpus and that corpus’s hash	A score you cannot locate the inputs for
Every figure is generated from a committed harness result file, never typed	A number that drifted from the run behind it, or never had one
No figure without a completed run	Projections, extrapolation from a neighbouring area, rounding up to the next milestone
No aggregate percentage, in any form	One headline number standing in for the whole format
Per-area scores publish the misses with the passes	An average that hides its own worst case

Today that means there is no fidelity number on this site at all, and the absence is the correct state rather than an omission. The public scoreboard ships at launch, and every figure on it names its anchor, its corpus hash and the method that produced it — nothing goes on that board measured a different way from anything else on it, ours included, because a scoreboard that scores its own author by a second rubric is not a scoreboard.

Two things about it follow from the method rather than from a date. An engine that is not ours joins the board only once measuring it the same way has cleared the review that comparative publication waits on — a comparative figure is a claim about someone else, and it does not ship on our schedule alone. And re-running the board is the step after that: the corpus publishes at GA as an Apache-2.0 repository, and the hash each figure has carried since launch is what pins that corpus to the inputs the board was scored on. From GA, anyone holding an Office licence can stop reading our figures and check them.

The five questions, as a checklist

Worth asking of any published rendering score, on any site:

What was it measured against — the same engine’s earlier output, another renderer, or the source application itself?
Is the corpus published, and hashed, so you can hold the same inputs?
Is the rubric published — what tolerance counted as a match?
Is it the average or the worst case, and are the failures published with the passes?
Could a stranger re-run it and get the same answer?

A score that answers all five is evidence. A score that answers none is an adjective with a decimal point in it. Most published figures sit somewhere between, and knowing where is the whole skill.

Next steps

Fidelity methodology — the oracle, the rubric, the corpus and the provenance rules behind our own figures
Fidelity evidence — the per-area board those rules govern
Compatibility report — the per-conversion artifact, its statuses and CI gating recipes
Playground — run the standard against your own workbook, which is the only score that decides anything

View as Markdown