Datalab has released OmniExtractBench, an open benchmark designed to fix bias and opacity in structured document extraction. The new benchmark pools 620 documents from four existing suites, testing how accurately systems fill JSON schemas from PDFs using a single deterministic scorer that explains every decision.
The release comes as various extraction vendors publish their own leaderboards, which Datalab argues are difficult to compare or audit fairly. OmniExtractBench provides a shared yardstick with a scorer that installs from PyPI under an Apache 2.0 license, while its data and code are hosted on Hugging Face and GitHub. The benchmark addresses common industry flaws such as vendor bias, opaque evaluation harnesses, unclear scoring logic, and narrow document variety.
Datalab evaluated ten system configurations on the full corpus, revealing distinct performance patterns among models and vendors. Datalab's accurate mode led with 93.85 accuracy, while models like GPT 5.6-sol leaned toward missing values and LlamaExtract leaned toward invented values. This tool matters because it establishes a transparent, standardized evaluation method for structured document extraction across the industry.