Exploring DOGE: A Leap in Visual Document Understanding

You know how sometimes we struggle to understand the tiny details in documents? Think of a chart with lots of numbers or a PDF with complex texts. That's where multimodal large language models (MLLMs) come in. They're supposed to help us make sense of these things, but they've been falling short, especially in the world of visual documents. Why? Well, there just aren't enough detailed datasets and comprehensive benchmarks to help these models improve. Enter the DOcument Grounding and Eferring data engine, or DOGE-Engine. This awesome tool produces two kinds of high-quality data: one that helps with basic text localization and recognition, and another that fine-tunes MLLMs to smoothly handle grounding and referring tasks during conversations and reasoning.

With the DOGE-Engine, we built DOGE-Bench. This super useful tool offers seven different grounding and referring tasks across three types of documents—charts, posters, and PDFs. It's a fantastic way to thoroughly evaluate these models. Using the data created by our engine, we developed a top-notch baseline model called DOGE. This model is a game-changer—it can accurately refer to and recognize texts at multiple levels within document images. Plus, we're making the code, data, and model available to everyone. Isn't that cool?

Actions