SplatRs: We Gave the Model 4× More Pixels and It Got Worse
Last week ended with a fast trainer and a winning recipe, and one obvious next move: train on full-resolution photos instead of half-size ones. The model literally cannot learn detail it has never seen, and the new tile rasterizer had turned a 3-day full-res run into an overnight one.
What followed was five runs, three bugs, a mechanism hunt — and a conclusion we didn’t expect. Every number below traces to a committed run log.
Lesson 1 — Look at your training data. Actually look at it.
While assembling a report we noticed magenta smudges in the sky of a saved training target. The photos were clean — three other decoders proved it. The culprit was the JPEG backend our Rust image library is version-locked to: patchy chroma-plane corruption on ~0.2% of pixels, bit-identical on every load, in every training and eval image, for weeks. One loader swap fixed it.
The epilogue is the interesting part. We re-ran the champion configuration on clean data expecting a shift — and got 16.88 dB vs 16.91, a statistical tie (structural similarity actually improved). Because training and grading saw the same corrupted bits, the damage nearly cancelled. Every A/B verdict survived. The lesson isn’t “corruption is harmless” — it’s that paired controls are what let you keep three weeks of conclusions after finding a data bug.
Lesson 2 — Your GPU will lie by omission
The first full-res attempt died at iteration 5,501. Not slowly — the render watchdog caught it, saved the model, and aborted loudly. The cause: our GPU device was created with the library’s default limits, capping any buffer binding at 128 MiB while the hardware supports 4 GiB. At 400k gaussians and full resolution, the tile rasterizer’s pair list needed 256 MiB; every command after it silently failed validation.
Raising the limit exposed a second bug hiding behind the first: above ~16.7M tile/gaussian pairs, the sort needed 65,536 workgroups. The API’s hard cap is 65,535. One over. Both fixes landed with pre-flight guards and bit-exactness gates, and the pipeline then ran five consecutive 30k-iteration full-res runs without a single fault.
The infrastructure lesson: the failure you can see costs a morning; the one returned as Ok costs a training run.
Lesson 3 — More supervision made the model worse, and we can prove it
Same scene, same gaussian budget, same 30k iterations, same 980×545 scoring. The only change: training on full-resolution photos instead of half-size. Every full-res arm plateaued around 13 dB while the half-res model scores 14.66 on the identical check.
We excluded the obvious outs first: it isn’t the metric (scoring the same model at both resolutions differs by 0.06 dB), and it isn’t the loss (L2 and perceptual arms both hit the ceiling). Then the mechanism hunt, one flag per arm:
| arm (all 30k iters, 400k cap) | final dB | what it established |
|---|---|---|
| DSSIM recipe | 11.91 | periodic opacity resets cut the high-opacity DSSIM population 17× — at full-res rates it never recovers (fog) |
| + gentler reset floor | 11.42 | population health fixed, quality worse — mechanism confirmed, root cause refuted |
| L2 loss | 12.86 | stable flat plateau — the ceiling is loss-independent |
| + halved densify threshold | 13.19 | densification really was starved at full-res (first event 8× bigger) — real, but only +0.33 |
| + resolution-scaled anti-aliasing | 12.85 | the last measured suspect, cleanly refuted |
My favorite row is the second one: we fixed the thing we thought was the root cause, and the model got worse. That’s what a refutation looks like when your harness is honest. It exhausted the suspects we could measure, so full-res is shelved — with five checkpoint grids, per-500-iteration renders, and a written list of unmeasured suspects for whenever it reopens. A negative result with preserved evidence is a result.
Lesson 4 — The discipline is what made failing cheap
Five overnight runs to a clean shelving decision, and none of them was wasted motion:
- Every arm changed exactly one flag against a same-binary control.
- Monitors reported the decisive intermediate number — often killing suspense at iteration 6,000 instead of 30,000.
- Seeded runs made trajectories reproducible enough that two runs matching to the fourth decimal at iteration 101 was routine.
- Every verdict — especially the refutations — went into the plan document so nothing gets re-run by accident.
The full-res chapter “failed,” but it fixed three bugs that affected everything, hardened the GPU pipeline, and bought a map of the territory for next time.
Where that leaves us
Back on the main quest with a clean baseline: 16.88 dB against splatfacto’s 22.21 on the identical held-out protocol — an honest 5.33 dB to close, and we know its rough decomposition (≲2 dB capacity, the rest recipe).
Two single-flag arms are in flight as this is written: the gaussian budget raised to 400k at half-res (the old cap was a throughput limit from the pre-tile-rasterizer era, not a choice), and splatfacto’s actual shipping densification signal (absgrad at threshold 0.0008 — our earlier refutation tested 0.0004, the wrong point of their convention).