Vision Models

The VlmPipeline in Docling allows you to convert documents end-to-end using a vision-language model.

Docling supports vision-language models which output:

• DocTags (e.g. SmolDocling), the preferred choice
• Markdown
• HTML

Quick Start

For running Docling using local models with the VlmPipeline:

docling --pipeline vlm FILE

from docling.datamodel.base_models import InputFormat
from docling.document_converter import DocumentConverter, PdfFormatOption
from docling.pipeline.vlm_pipeline import VlmPipeline

converter = DocumentConverter(
    format_options={
        InputFormat.PDF: PdfFormatOption(
            pipeline_cls=VlmPipeline,
        ),
    }
)

doc = converter.convert(source="FILE").document

To extract chart data after VLM conversion, enable chart enrichment. Docling classifies pictures before chart extraction and skips fields already present in the VLM output.

docling --from image --to dclx IMAGE.png --pipeline vlm \
    --vlm-model mineru2_pro --enrich-chart-extraction

The Python SDK uses VlmPipelineOptions(do_chart_extraction=True) for the same behavior. Picture classification is enabled automatically when chart extraction is requested. Picture description and classification can also be enabled with --enrich-picture-description and --enrich-picture-classes.

The default chart model is Granite Vision 4.1 4B. On Apple Silicon, it uses MLX when mlx-vlm>=0.7.0 and MPS are available; otherwise it uses Transformers. To use the older CSV-only ibm-granite/granite-vision-3.3-2b-chart2csv-preview model, add --chart-extraction-preset granite_vision_v3_3 to the command. In Python, set chart_extraction_options=ChartExtractionVlmEngineOptions.from_preset("granite_vision_v3_3") on VlmPipelineOptions or PdfPipelineOptions.

To require MLX explicitly, select --chart-extraction-preset granite_vision_v4_mlx. This preset loads the same official ibm-granite/granite-vision-4.1-4b repository. In Python, use ChartExtractionVlmEngineOptions.from_preset("granite_vision_v4_mlx").

Available local models

By default, the vision-language models are running locally. Docling allows to choose between the Hugging Face Transformers framework and the MLX (for Apple devices with MPS acceleration) one.

The following table reports the models currently available out-of-the-box.

Inference time is computed on a Macbook M3 Max using the example page tests/data/pdf/2305.03393v1-pg9.pdf. The comparison is done with the example compare_vlm_models.py.

For choosing the model, the code snippet above can be extended as follow

from docling.datamodel.base_models import InputFormat
from docling.document_converter import DocumentConverter, PdfFormatOption
from docling.pipeline.vlm_pipeline import VlmPipeline
from docling.datamodel.pipeline_options import (
    VlmPipelineOptions,
)
from docling.datamodel import vlm_model_specs

pipeline_options = VlmPipelineOptions(
    vlm_options=vlm_model_specs.SMOLDOCLING_MLX,  # <-- change the model here
)

converter = DocumentConverter(
    format_options={
        InputFormat.PDF: PdfFormatOption(
            pipeline_cls=VlmPipeline,
            pipeline_options=pipeline_options,
        ),
    }
)

doc = converter.convert(source="FILE").document

Other models

Other models can be configured by directly providing the Hugging Face repo_id, the prompt and a few more options.

For example:

from docling.datamodel.accelerator_options import AcceleratorDevice
from docling.datamodel.pipeline_options import VlmPipelineOptions
from docling.datamodel.pipeline_options_vlm_model import InlineVlmOptions, InferenceFramework, ResponseFormat, TransformersModelType

pipeline_options = VlmPipelineOptions(
    vlm_options=InlineVlmOptions(
        repo_id="ibm-granite/granite-vision-3.2-2b",
        prompt="Convert this page to markdown. Do not miss any text and only output the bare markdown!",
        response_format=ResponseFormat.MARKDOWN,
        inference_framework=InferenceFramework.TRANSFORMERS,
        transformers_model_type=TransformersModelType.AUTOMODEL_IMAGETEXTTOTEXT,
        supported_devices=[
            AcceleratorDevice.CPU,
            AcceleratorDevice.CUDA,
            AcceleratorDevice.MPS,
            AcceleratorDevice.XPU,
        ],
        scale=2.0,
        temperature=0.0,
    )
)

Remote models

Additionally to local models, the VlmPipeline allows to offload the inference to a remote service hosting the models. Many remote inference services are provided, the key requirement is to offer an OpenAI-compatible API. This includes vLLM, Ollama, etc.

More examples on how to connect with the remote inference services can be found in the following examples:

• vlm_pipeline_api_model.py

Native dots inline formatting

Native dots.ocr and dots.mocr JSON text can contain Markdown bold and italic markers. Docling maps this emphasis and inline HTML to styled text spans. Mixed text uses inline groups. Titles, heading levels, list items, and text block labels are preserved. Formula blocks, inline math, and code keep their literal content.

Markdown emphasis uses the existing marko dependency. For slim installations, install it with pip install 'docling-slim[format-markdown]'.

Native MinerU and dots captions

MinerU and dots caption blocks are linked when exactly one immediately adjacent native block maps to a compatible table or picture. MinerU's table_caption, image_caption, and code_caption labels constrain the owner type; generic captions from either model consider all three types. Both preceding and following blocks are considered, without geometry or text-based guesses. Skipped blocks remain barriers. Captions with no compatible neighbor or two compatible neighbors remain standalone text in DocLang. Caption text and its own provenance are preserved.

Code captions and captions containing formatted child runs remain standalone: the minimum supported Docling Core version cannot faithfully serialize those associations in DocLang. A container's existing caption takes precedence; additional captions remain standalone because its DocLang head holds one caption.

Chandra HTML output

Use ResponseFormat.CHANDRA_HTML with CHANDRA_OCR_LAYOUT_PROMPT for Chandra's HTML layout blocks. Docling scales each block's data-bbox coordinates from 0–1000 to the source page size. At most one document item representing an annotated block receives its box. Derived children remain unlocated unless their source HTML element declares its own data-bbox; the model does not provide separate coordinates for each word or table cell.

The converter preserves paragraphs, heading levels, nested lists, table spans, inline formatting, links, math, and code whitespace. Tables are recognized by their markup even inside blocks labeled Text, Form, or Figure. Cells with structured content use RichTableCell references. Form regions retain checkbox and radio states and fillable text values without inferring key–value links from visual proximity.

Picture descriptions from img alt are stored in PictureItem.meta.description. Chart tables, diagram code, and other picture content remain children of the picture. Explicit <chem> content is stored as SMILES molecule metadata. Captions and footnotes are linked when nested under their picture or table; separate layout blocks remain unlinked. Separate table fragments remain separate tables. CSS layout and styling without corresponding document primitives are not reconstructed.

To retain picture pixels, enable generate_picture_images or generate_page_images on VlmPipelineOptions. With page images retained, picture.get_image(document) can crop the picture using its provenance. When exporting Markdown, use traverse_pictures=True to include picture children. Use JSON to inspect all metadata and rich document structure; individual export formats may omit some of these details.

Recognizable HTML without layout blocks is recovered with a warning and without invented coordinates. Invalid bounding boxes also produce a warning while their content is retained. Nonempty prose or JSON responses without HTML transcription produce a page-specific inference error and PARTIAL_SUCCESS, rather than an unreported empty result. An explicitly labeled Blank-Page may be empty.