# Validate llama.cpp language-model and mmproj projector pairs

A llama.cpp language GGUF and mmproj must form a supported multimodal pair. A projector from a similar model is not interchangeable merely because its file size or family name resembles the intended artifact.

Author: Retrivium team
Updated: 2026-11-01
Canonical URL: https://retrivium.ai/journal/llama-cpp-mmproj-pair-validation/
Language: en

A llama.cpp language GGUF and mmproj must form a supported multimodal pair. A projector from a similar model is not interchangeable merely because its file size or family name resembles the intended artifact. Preserve both sources and test a known image with a specific question. Account for encoder offload separately: projector placement and language-layer placement are different controls affecting memory and input-processing time.

## Identify the projector's role

The [multimodal guide](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/docs/multimodal.md) describes model-plus-`mmproj` loading and offload controls. The projector connects image representation with the language component, so compatibility concerns architecture and data transformation, not just free VRAM. Preserve model-card information, revision, and the documented pairing. Random replacement after an error is not a compatibility test. Absence of an immediate crash does not establish that visual content was interpreted correctly or that the language model received the expected representation. Functional validation must cross that boundary.

## Establish the multimodal path

The [libmtmd README](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/mtmd/README.md) explains tool evolution and consolidated multimodal support. Older model-specific examples may target another executable or development stage. Match commands and component names to the selected build. State which endpoint and input types the tested path supports. A successful text-only language-model request does not validate image processing because it can bypass the projector completely. Keep text liveness and multimodal acceptance as different fixtures rather than treating the former as evidence for the whole pipeline.

## Use a known image fixture

Create a simple image containing two clearly distinguishable shapes and ask about their positions. Preserve the file, question, and acceptable structural result instead of relying on a changing remote image. This is a diagnostic fixture, not a vision-quality benchmark. Submit it with the documented pair and verify that media processing actually occurs. If the answer merely reflects prompt text, inspect media loading and the encoder stage before changing sampling or declaring the projector incompatible without further evidence.

## Validate independent offload

Record language-model and projector placement separately. Where supported, compare projector offload disabled and enabled while holding other settings fixed. Changed GPU memory use does not necessarily mean that language-layer placement changed. Reuse the same image and preprocessing configuration. Moving an encoder to CPU does not always free enough memory or improve speed; it changes execution location and transfer costs, which depend on model, input, and hardware. Measure those effects rather than inferring them from the switch name.

## Preserve the accepted pair

Keep language GGUF, projector, revisions, build, placement map, and media fixture. State tested input sizes and runtime limits. Repeat visual acceptance after either artifact changes, not merely a text health request. If official pairing evidence is absent, label the combination unverified rather than universally supported. The record should expose encoder and language boundaries and reproduce the complete media-to-answer path. It cannot establish every VLM's quality from one image or equate successful tensor loading with correct visual interpretation across arbitrary projectors and backend versions.

## Related reading

[GPU hosting for multimodal models: count more than text](/journal/multimodal-gpu-hosting/); [Qwen3-VL VRAM: Test Images of Different Complexity](/journal/qwen3-vl-vram-requirements/).