Turning a PDF into clean Markdown sounds like an OCR problem until you try it on real documents. Two-column papers, tables that run across pages, a formula sitting in the middle of a sentence. Our team has been training a small model for exactly this for a while, and we just released the code and weights, so here is what we learned along the way.
The architecture split mattered more than we expected. Layout analysis is its own model, and you can swap it for a different one. Recognition and reading order are predicted together in one pass of the main model, which removes a whole sequential step. That split, together with how we post-trained it, is where most of the accuracy and speed came from.
For post-training we used a loop we call RSI. The model predicts on a large batch of documents, and each sample is sorted as correct, wrong or uncertain (uncertain means repeated predictions disagree). A VLM writes down the reasons behind the wrong and uncertain ones, the reasons get grouped into clusters, and each cluster becomes the plan for the next round of targeted training data. We tracked progress by the repair rate of each cluster in the following round, which told us much more than watching one benchmark number.
Where it landed: in our comparison against published scores it came out first on olmOCR-Bench, OmniDocBench and DocParsingBench, and it was the fastest of the 21 models we timed on the same 100 pages. Multi-column pages are still a weaker spot (4th in that olmOCR-Bench category), and running it locally takes an NVIDIA GPU with 24 GB of VRAM.
The model is OXR, from SoMarkAI. Repo, weights and the full benchmark report are in the first comment. Happy to go into detail on the training loop, it was the most interesting part to build.
