What is Doclang?
Your Documents Were Never Built for AI, and That's the Real Bottleneck
Every team building with LLMs eventually hits the same wall, and it's rarely the model. It's the documents.
You'd think by 2026 this would be a solved problem. It isn't. If you've ever watched a PDF parser turn a clean financial table into a wall of flattened text, or seen a figure just quietly vanish somewhere in the pipeline, you already know what I'm talking about. This is not an edge case. It's the default experience of feeding real-world documents into an AI system.
The Problem Nobody Wants to Admit
Here's the thing that took me a while to actually internalize: PDF was never built for machines to understand. It was built to tell a printer where to put ink. DOCX was built for humans clicking around in Word. Markdown was built so people could write readable plain text. HTML was built for browsers to render a page.
Not one of these formats was designed with the question "how would a language model read this" in mind. That question didn't exist when these formats were created.
So what happens in practice? Your pipeline pulls in a contract, an invoice, a regulatory filing, a research paper, and a parser has to guess. It guesses at reading order. It guesses at whether that block of numbers is a table or just text that happens to have some whitespace in it. It strips out metadata because there's nowhere sensible to put it. Figures get dropped entirely because there was no clean way to represent "there is an image here and here is what it's connected to."
And then everyone acts surprised when the model hallucinates a fact that was sitting right there in the document, just mangled beyond recognition by the time it reached the model. The uncomfortable truth is that a lot of what gets blamed on "model quality" is actually a document quality problem wearing a different hat. Teams spend more engineering hours wrangling preprocessing pipelines than they do actually building the product they set out to build.
Why Duct-Taping Existing Formats Doesn't Fix It
The obvious first move is to just get better at parsing. Better OCR, better table extraction, a smarter chunking strategy. And that helps, some. But it doesn't fix the underlying issue, because the problem isn't that parsers are bad at their job. The problem is that they're translating from a format that was never meant to carry semantic meaning in the first place.
Think about what a table actually is inside a PDF. It's not a table. It's a collection of positioned text fragments that happen to visually line up into rows and columns when rendered. The PDF format has no concept of "this is a table with three columns and a merged header cell." That structure is an illusion created for your eyes. A parser has to reverse-engineer that illusion back into structured data, and reverse-engineering a guess is never as reliable as just having the structure natively encoded in the first place.
Every additional translation layer between the source document and the model is another place for information to get lost, another place for two different tools to disagree about what the document actually said, and another place where your pipeline behaves differently depending on which parser you happened to plug in that quarter.
Here's roughly what that pipeline looks like today in most teams, and where it actually breaks:
Every red step in that chain is a place where the model never had a fair shot, because the information it needed was already gone by the time the chunking step even started.
Who's Actually Behind This, and Why It's Not Just Another Vendor Format
The document tooling space is already crowded with parsers, so it's fair to ask why this one is worth paying attention to. The short answer is who's in the room and how the thing is governed.
DocLang isn't a single company's product with a roadmap decided behind closed doors. It's developed as a Joint Development Foundation Project under the Linux Foundation, specifically an LF AI & Data project, which means the specification itself is public and no single vendor gets to unilaterally steer where it goes. The working group founding it includes IBM, NVIDIA, Red Hat, ABBYY, and HumanSignal, a mix that spans document intelligence, hardware and AI infrastructure, enterprise open source, and data labeling. That's a meaningfully different set of incentives than one startup trying to lock you into its own parsing API.
And it's not purely theoretical either. Docling and ABBYY FineReader Engine, both real, widely used document processing tools, already produce DocLang output natively. That's the detail that actually matters more than the spec being well written: it tells you the standard is being built with real production tools in mind from day one, not designed in isolation and hoping adoption follows later.
What a Document Format Built for AI Actually Looks Like
This is the gap DocLang is built to close, and it's worth being precise about what kind of thing it actually is, because it's easy to mistake it for just another parser in a crowded field of parsers.
It isn't. DocLang is a standard, not a tool. The distinction matters more than it might sound like at first. A parser converts a document into whatever proprietary format that specific tool decided to output, and you're stuck with that tool's opinions baked into your pipeline forever. A standard is something any parser, any converter, any downstream system can implement, the same way JSON became the shared language for structured data regardless of which programming language or database you're using, or the way HTML became the shared language for the web regardless of which browser is rendering it.
DocLang is trying to be that shared language, specifically for the layer between "raw document" and "what an LLM actually needs to reason over it correctly."
Concretely, every element in a DocLang document carries three things natively: a semantic tag describing what it is, a bounding box describing where it sits on the page, and its position in the reading order. A heading isn't just a heading, it knows its level and where it sits. A table isn't flattened text, it retains its full grid structure, including merged cells, using a compact token scheme called OTSL that represents a table's structure in a handful of tokens instead of the dozens HTML would need for the same thing. Metadata, things like author, PII flags, and permissioning rules for retrieval, lives inside the document's own header rather than in some separate sidecar file that inevitably gets out of sync with the content it's describing.
And it doesn't stop at text documents. The same underlying primitives extend to audio transcripts, images, and video, so a speaker turn in a transcript or a scene boundary in a video gets represented with the same structural approach as a heading or a table in a written document. One consistent mental model across every content type your pipeline touches, rather than a different bespoke format for each modality.
A Concrete Example: The Same Table, Two Ways
Abstract descriptions of "structure preservation" only go so far. Here's an actual quarterly financial table and what happens to it in each world.
What a typical parser hands your model today, after a PDF table gets flattened:
Q3 2024Financial Re
port Net Revenue42
M51M39M Figure3.2
omitted author:null
That's not a mangled example for effect. This is genuinely close to what comes out the other end of a lot of production pipelines once a multi-column table with a merged header meets a parser that's just trying its best. The revenue figures are in there somewhere, but the model now has to guess which number belongs to which quarter, and the referenced figure is just gone.
What the same table looks like as a DocLang document:
<doclang>
<heading level="1">
<location value="48"/><location value="40"/>
<location value="420"/><location value="72"/>
Q3 Financial Summary
</heading>
<table>
<location value="48"/><location value="88"/>
<location value="420"/><location value="168"/>
<ched/>Quarter<ched/>Revenue<ched/>YoY<nl/>
<fcel/>Q3 2024<fcel/>$42M<fcel/>+18%<nl/>
</table>
</doclang>The heading knows its own level and exact page position. The table isn't a guess, it's an actual grid, with header cells (ched) and data cells (fcel) explicitly marked out, row by row, using the OTSL scheme mentioned earlier. Instead of the roughly 28 structural tokens HTML would need to describe the same table, OTSL does it in about 5. That's not just a neatness win, it's fewer tokens burned per document at ingestion time, which adds up fast across a real document corpus.
Nothing here is inferred. The model isn't pattern-matching its way back to "oh, I bet 42M is revenue for Q3." It's just reading what's there.
Why This Actually Matters for Production AI
It's easy to read "yet another document format" and shrug. But the practical implications are bigger than that framing suggests.
If your model doesn't have to guess at structure, it stops hallucinating structure. A table that's actually encoded as a table, with its real grid intact, produces far more reliable extraction than a table that got smashed into a string of numbers the model has to pattern-match its way through. Reading order that's explicit rather than inferred means a two-column layout doesn't get silently scrambled into nonsense the way plenty of parsers still manage to do today.
There's also a governance angle that matters more the bigger your organization gets. Compliance and legal teams want rules defined once and enforced everywhere, not scattered across a dozen sidecar config files that drift out of sync with the actual documents. Baking permissioning and PII flags directly into the document's own metadata means every downstream system that reads the document also reads the rules that apply to it, automatically, without anyone having to remember to check a separate system first.
And because it's an open standard rather than a proprietary tool, you're not locking your pipeline to one vendor's roadmap. Swap out your parsing tool, your retrieval system, your downstream agent framework, and as long as everything speaks the same standard, your documents stay portable. That's a genuinely different guarantee than what you get from any single parsing product on the market today, however good that product is at what it does.
Worth noting: this isn't a fringe side project. It's governed as a Joint Development Foundation Project under the Linux Foundation, an LF AI & Data project, with a working group that includes IBM, NVIDIA, Red Hat, ABBYY, and HumanSignal, among others. Existing tools like Docling and ABBYY FineReader Engine already support producing DocLang output natively, which is the kind of early real-world adoption that actually matters more than a nicely designed spec sitting alone on GitHub.
What This Changes for RAG Specifically
RAG is where this problem quietly does the most damage, because a retrieval pipeline is only as good as the chunks it has to work with, and chunking is exactly where structure loss compounds.
Picture a standard enterprise RAG pipeline ingesting that same financial report. If the table came in as flattened text, your chunker has no idea where one logical unit ends and another begins. It might split the table mid-row. It might chunk the heading away from the numbers it introduces. The embedding for that chunk now represents a garbled fragment instead of a coherent unit of meaning, and no amount of clever reranking downstream recovers information that was already lost at the chunking step.
With structure preserved natively, a chunker can respect the actual boundaries of a table, a section, or a figure caption instead of guessing where to cut based on character count. Retrieval quality goes up not because the embedding model got smarter, but because the thing being embedded actually makes sense as a unit. And because permission metadata and PII flags travel inside the document itself rather than in a separate access-control layer, a RAG system can enforce who's allowed to see a retrieved chunk without needing a second system to cross-reference every single time. That's a real, direct fix for one of the hardest problems in enterprise RAG: permission enforcement that was bolted on after the fact instead of designed in from the start.
Citations get more trustworthy too. When a chunk retains its exact bounding box and page location, a RAG answer can point back to precisely where on the page a claim came from, not just "somewhere in this 40-page PDF."
The Honest Take
None of this means document parsing suddenly becomes a solved, boring problem overnight. Standards take time to get adopted broadly, and the value of a shared format only really shows up once enough of the ecosystem, parsers, retrieval systems, agent frameworks, actually speaks it. A spec with a handful of early adopters is a promising start, not a finished migration.
But the underlying diagnosis is hard to argue with. If your AI pipeline's accuracy ceiling is being set by how badly your documents get mangled before they ever reach the model, no amount of better prompting or bigger context windows fixes that. You're optimizing the wrong layer. The fix has to happen upstream, at the format the document is represented in, not downstream in cleverer workarounds for a format that was never built to carry meaning in the first place.
If you're building anything that touches contracts, invoices, research papers, or regulatory filings at scale, this is worth understanding now rather than after you've spent another year of engineering time patching around a format problem that a shared standard could have solved from the start.
Learn more or read the full specification at doclang.ai.