Is your ONNX model the same model you trained?
Leafparity compares an XGBoost, LightGBM or scikit-learn tree model with its ONNX conversion over every possible input. It either proves the two always agree, or shows you exactly which inputs get a different answer.
- original
- x ≤ -17.066402760988257 double
- converted
- x ≤ -17.066402435302734 float32
- missing
- original treats 0 as missing (
zero_as_missing), converted evaluates it as an ordinary number
From a real report: LightGBM with zero_as_missing=True, converted with onnxmltools. Both predictions were produced by the real runtimes.
One input, two models, two different answers.
Pick an input and follow it through the original tree and the converted one. Most inputs take the same path. A few don't, and those are the ones Leafparity finds.
A simplified one-tree illustration of the case in the report above. Leafparity doesn't try inputs one by one like this: it analyses every split, so it finds all such inputs, including ones nobody would think to test.
Test sets rarely hit the difference. Production traffic does.
When a conversion goes wrong, it goes wrong in slivers of the input space one float step wide: around a threshold, at exactly zero, or only for missing values. Comparing predictions on held-out data almost never lands there.
Rounded thresholds
A double threshold stored as float32 moves slightly. Inputs in the gap take a different branch.
Zero treated as missing
LightGBM's
missing_type=Zerois ignored by the converter, so an exact 0.0 goes the other way.NaN routed differently
scikit-learn decides per node where NaN goes. The converted model sends it the other way at some of those nodes.
A proof, or the exact inputs that break it.
Leafparity doesn't sample. It reproduces each runtime's arithmetic, including every cast, preprocessing step, comparison and missing-value rule, and walks both models together through every tree.
Proof that the models agree on every input
For every possible input, both models reach corresponding leaves. Outputs can differ only by the stated floating-point rounding allowance.
Every disagreement, with a witness
Each place in each tree where the models differ, the exact set of affected inputs, and a concrete input you can run yourself. You also get the largest difference found and a proven upper bound.
If a model uses something Leafparity can't analyse exactly, it says so and stops (exit code 2). It never guesses. The exit codes let you use it as a CI step.
Found in real conversions, including the official tutorial.
This is the sklearn-onnx documentation's own "Issues when switching to float" example (StandardScaler and a decision tree). Its test set suggests the error tops out around 156. The real maximum, proven by walking every split, is 556, more than three and a half times higher. Leafparity also certifies the tutorial's CastTransformer fix as equivalent for float32 inputs without NaN, and shows the fix does not hold for float64 inputs. See the complete report below.
| Setup | What Leafparity reports |
|---|---|
LightGBM with zero_as_missing=True, converted with onnxmltools | An input of exactly 0.0 takes a different path in every tree. Raw scores differ by up to 295. |
| scikit-learn trees (1.3 and later) that receive NaN, converted with skl2onnx | scikit-learn decides per node where NaN goes; the converted model sends it the other way at some of those nodes. The predicted class flips for some inputs, verified on both runtimes. |
| LightGBM with double thresholds, served through float32 ONNX | Narrow bands of float64 inputs next to thresholds are routed differently at almost every node. With --input-dtype float32 it tells you whether your float32 data can reach them at all. |
Each case is reproduced in the examples folder of the repository.
View the complete, unedited report for this case
Exactly what examples/01_sklearn_onnx_float_tutorial.py prints when you run it yourself. Nothing shortened or reworded. The bar chart above and the hero example both come from a report like this one.
Test-set comparison (what the tutorial does): max |difference| = 155.52
==============================================================================
leafparity 0.1.0 - equivalence report
==============================================================================
Original : scikit-learn StandardScaler -> DecisionTreeRegressor, 1 trees
Converted: ONNX TreeEnsembleRegressor (1 trees, post_transform=NONE, input float32)
Domain : every float32 input vector (10 features), no missing values
VERDICT: NOT EQUIVALENT
Not equivalent: 10 distinct problem(s) at 362 place(s) in the trees. For some
inputs the raw outputs differ by 555.835 (proven to be the largest
possible difference).
Self-check (exact model vs. real runtimes; inputs constructed to reach every split node, on and beside its boundary):
original : 5620 tree decisions, 0 mismatches
converted: 5620 tree decisions, 0 mismatches
Examined : 10754 joint regions; 615 of 905 matched split nodes have different exact rules
Cross-check: 4000 boundary inputs run through both real runtimes, max |raw diff| 236.29, all within proven bounds
PROBLEMS (largest effect first; 10 problem(s), 362 place(s) in the trees)
------------------------------------------------------------------------------
#1 a narrow band of values next to a threshold is routed differently
(threshold or input rounded to lower precision) [feature 'f3']
occurs at 72 split node(s) in 1 tree(s); largest effect of a single tree: 555.835
example - tree 0:
original node 1442: float32(x) <= -1.0622187554836273 (double) -> left, missing -> right
converted node 1442: x <= -1.0622187852859497 (float32) -> true branch, missing -> false branch
inputs routed differently here: [-8.5, -8.5] (1 representable value)
witness: {f3=-8.5, f0=1.0, f1=5.0, f2=-2.0, f4=4.0, f5=-26.0, f6=135.0, f7=-114.0, ... (2 more)}
original predicts 413.213; converted predicts -142.622 (verified by running both real runtimes)
#2 a narrow band of values next to a threshold is routed differently
(threshold or input rounded to lower precision) [feature 'f8']
occurs at 96 split node(s) in 1 tree(s); largest effect of a single tree: 493.681
example - tree 0:
original node 538: float32(x) <= -1.0806240439414978 (double) -> left, missing -> right
converted node 538: x <= -1.0806241035461426 (float32) -> true branch, missing -> false branch
inputs routed differently here: [-278.0, -278.0] (1 representable value)
witness: {f8=-278.0, f0=2.0, f1=-2.0, f2=-3.0, f3=-15.0, f4=-24.0, f5=47.0, f6=204.0, ... (2 more)}
original predicts -383.305; converted predicts 110.376 (verified by running both real runtimes)
WORST CASE over the whole domain (raw output = prediction)
------------------------------------------------------------------------------
output 0, original - converted: attained 555.835, guaranteed <= 555.835 (proven maximum)
output 0, converted - original: attained 493.681, guaranteed <= 493.681 (proven maximum)
input attaining the largest difference:
witness: {f0=0.0, f1=3.509999990463257, f2=0.0, f3=-8.5, f4=0.0, f5=-20.5, f6=135.0, f7=1.5, ... (2 more)}
original predicts 413.213; converted predicts -142.622 (verified by running both real runtimes)
NOTES
- preprocessing modelled exactly: StandardScaler
Completed in 0.8s.
Tutorial's fix, float32 inputs, missing values excluded: EQUIVALENT - Equivalent on the whole analysed input domain: the two models take corresponding paths for every possible input, and their raw outputs differ by at most 0.000111 (floating-point rounding of leaf values and sums).
Tutorial's fix, float64 inputs, missing values excluded: NOT EQUIVALENT - Not equivalent: 10 distinct problem(s) at 905 place(s) in the trees. For some inputs the raw outputs differ by 565.348 (proven to be the largest possible difference).
Tutorial's fix, float32 inputs, missing values allowed: NOT EQUIVALENT - Not equivalent: 10 distinct problem(s) at 225 place(s) in the trees. For some inputs the raw outputs differ by 720.826 (proven to be the largest possible difference).
Every report checks its own work.
Three checks run on every analysis, in this order. The report shows the result of each one.
- 1
Self-check before any analysis
Inputs are built to reach every split node and sit exactly on its boundary and one float step either side, including zeros, tiny values and NaN. They run through the real libraries and through Leafparity's model. If a single decision differs, for example because a library update changed its arithmetic, Leafparity refuses to certify.
- 2
Witness verification
Every reported problem, and the worst case, comes with an input that has been run through both real runtimes. The report states whether the observed difference matches the prediction.
- 3
Independent cross-check
Thousands of boundary inputs run through both runtimes without using the analysis engine at all. Every observed difference must fall within the proven bound.
What it checks today.
Original model
- XGBoost
gbtreebooster, numerical splits. Regression, binary, multiclass.- LightGBM
- gbdt and rf, numerical splits, all missing-value modes. Regression, binary, multiclass.
- scikit-learn
- Decision trees, random forests, extra trees, gradient boosting. Optionally after a scaler or
CastTransformerin a Pipeline.
Converted model
- ONNX
ai.onnx.mlTreeEnsembleRegressor and TreeEnsembleClassifier (opset 3 and earlier), optionally after Cast, Scaler or arithmetic with constants. Float or double input, as produced by skl2onnx and onnxmltools.
Refused rather than guessed: categorical splits, XGBoost dart, LightGBM linear trees, ONNX opset-5 TreeEnsemble, PMML, compiled code (m2cgen, treelite) and SQL. Need one of these? Tell us.
Run it on your own machine.
Leafparity is open source (Apache 2.0). Your models never have to leave your computer or your company network.
# 1. install pip install leafparity[all] # 2. compare your original model with its ONNX conversion leafparity check model.joblib model.onnx
Exit code 0 means proven equivalent. Exit code 1 means not equivalent, with the exact inputs that differ. Exit code 2 means the model uses something Leafparity cannot analyse exactly yet, so it says so instead of guessing. A typical small model takes a few seconds. The README lists every option.
Want a certificate you can file, or a second pair of eyes?
- STEP 1
Send both models under an NDA
The original model file and the
.onnxfile, plus whether production feeds float32 or float64 values. We sign an NDA first if you ask for one. - STEP 2
We run the check
Within two business days you get the full report: the verdict, every problem found, a witness input for each, the proven worst case, and a plain language explanation of each cause.
- STEP 3
Fix or certify
If the models differ, the report shows where to fix the conversion, and once you have fixed it we run the check again at no charge. If they match, you have a record you can file with your model documentation.
About your model files
By default your files stay with you: Leafparity runs locally, so we never see your model unless you choose to send it. For a certificate we need the original model and its ONNX conversion, nothing else. Never your training data, never your customers' data. Files are deleted once the report is sent; ask and we'll confirm it in writing. Prefer not to send anything? Run the check yourself and send us only the report if you want a second opinion. The report lists split thresholds and example inputs, so read it before you share it.
How you pay
Payment is upfront, by card or bank transfer via a PayPal invoice sent to you after we confirm the request. If leafparity can't produce a verdict for your model because it uses a construct we don't yet support (exit code 2), you get a full refund, no questions asked.
Pay per model, or check every release.
Founding-customer rates. Early customers keep their rate when prices go up.
Per-model certificate
One model, one conversion, one answer.
You can run the same check yourself for free. The certificate is us running it, explaining it and re-checking it for you. It is not a signature from an accredited body, and it does not say the model itself is accurate.
- Full EQUIVALENT or NOT EQUIVALENT proof, with every problem and a verified witness
- Each finding explained in plain language: the cause, whether it comes from the converter, and the version or setting that fixes it
- One free re-check after you fix the conversion
- A report you can file with your model documentation, with a scope note on what was and was not checked
CI plan
Coming soonWaiting on the public package release.
- GitHub Action as a drop-in CI step
- Fail builds only above a proven difference you set
- Unlimited checks on your models
Enterprise
Volume pricing for several teams and models.
- 10 certificates per year across your organization, billed yearly, more at an agreed rate
- Priority turnaround and support
- Direct line for new model formats, and first access once the CI plan ships
Find out before production does.
Run the check yourself, or send us the model and its ONNX conversion under an NDA. You get a proof that they agree, or the exact inputs where they don't.
- Original model (XGBoost, LightGBM or scikit-learn)
- The converted
.onnxfile - Optional: a small sample of real inputs, for realistic witnesses