Multi-Modal and Frontier Models

The transition from text-only to multi-modal systems is not an incremental improvement — it is a different category of engineering problem. You are no longer aligning a single latent space to human preferences; you are learning to bridge modality gaps between fundamentally different signal distributions while preserving semantic coherence across them. This file covers what is actually deployed in production, what is genuinely useful today, and where the research demos have not yet caught up to real-world reliability.

---

1. Why Multi-Modal Matters (and Why It’s Harder Than It Looks)

A purely text-based model cannot see a diagram, analyze a chart, transcribe speech, or understand a video. The real world is not text-only. The engineering challenge is cross-modal alignment: meaningful regions of the visual latent space must map to corresponding semantic regions in the language latent space so that “a cat sitting on a red couch” in text and an image of exactly that refer to the same conceptual point in some shared representational geometry.

The practical consequence: Multi-modal models are generally larger, more expensive to serve, and more complex to evaluate. The failure modes (hallucinated image content, OCR errors, cross-modal confusion) are harder to catch than text hallucinations because humans often can’t verify the model’s visual processing.

---

2. CLIP: The Foundational Architecture

CLIP (Contrastive Language-Image Pre-Training, arXiv 2103.00020, Radford et al., OpenAI 2021) established the template that most multi-modal systems still use.

Architecture:

Image → Image Encoder (ViT) → Image Embedding [512d]
                                      ↓
                           Contrastive Loss: maximize
Text  → Text Encoder (Transformer) → Text Embedding [512d]   cosine similarity
                                      ↑                        of matching pairs
                                      minimizes distance
                                      of non-matching pairs

Training: 400M image-text pairs from the internet. No human labels. The contrastive objective forces the model to learn which text describes which image by pulling matching pairs together and pushing non-matching pairs apart in embedding space.

Why it matters: CLIP embeddings enable zero-shot image classification (compare image embedding to text description embeddings, take argmax) and are the visual backbone of nearly every subsequent multi-modal model.

Limitations: CLIP struggles with counting, spatial reasoning (“object X is to the left of object Y”), and understanding fine-grained attributes. It sees relationships statistically, not geometrically.

---

3. LLaVA: Open-Source Vision-Language Models

LLaVA (Large Language and Vision Assistant, arXiv 2304.08485, Liu et al., 2023) demonstrated that you don’t need to train a vision-language model from scratch. You can connect an existing vision encoder to an existing LLM with a small projection layer, trained on visual instruction data.

Architecture:

Image → CLIP ViT-L/14 → Visual Features [256 tokens × 1024d]
                              ↓
                    Projection MLP (2-layer, trained)
                              ↓
                    Visual Token Sequence [256 × 4096d]
                              ↓
          [System Prompt] + [Visual Tokens] + [Text Input] → LLaMA/Mistral

Training pipeline:

1. Stage 1 (Feature alignment, ~1 day on 8xA100): Freeze LLM + CLIP, train only the projection MLP on 595K image-text pairs. Teaches the MLP to map visual features into the LLM’s token space.

2. Stage 2 (Instruction tuning, ~15 hours on 8xA100): Unfreeze LLM, train on 150K visual instruction pairs. Teaches the model to follow visual instructions.

LLaVA-1.5 (arXiv 2310.03744): Swapped MLP projection for a simple linear layer + used CLIP ViT-L/14@336px. Achieved 85.9% on VQAv2. LLaVA-NeXT (2024): Higher resolution (up to 1344×1344 via dynamic tiling), improved OCR performance.

Practical note: LLaVA-1.6-Mistral-7B runs on 16GB VRAM. Useful for local deployment where you need basic vision capabilities without GPT-4V pricing.

---

4. GPT-4V and Gemini: Frontier Closed Models

GPT-4V (OpenAI, 2023) and GPT-4o (2024):

• Architecture: Not published. Believed to be early-fusion transformer with visual tokens interleaved with text tokens.

• GPT-4o adds native audio input/output (not just speech-to-text + text-to-speech).

• Vision capabilities: OCR, document understanding, chart analysis, spatial reasoning, code from screenshots.

• Pricing (as of 2025): $2.50/$10.00 per 1M input/output tokens (text); images ~$0.00213 per 512×512 tile.

• Latency: ~2-4s for vision queries.

Gemini 1.5 Pro (Google, 2024):

• 1M (and later 2M) context window — enabling full-length video and entire codebases as context.

• Native multi-modal: text, images, video, audio processed natively (not converted to text first).

• Video understanding: Can analyze 1-hour videos (natively, not via sampled frames).

• Flash variant: significantly cheaper, somewhat lower quality.

• Key advantage over GPT-4V: video and audio native support.

Claude 3.5 Sonnet (Anthropic, 2024-2025):

• Strong vision benchmarks, particularly for document analysis and diagram understanding.

• Competitive with GPT-4V on most practical vision tasks.

---

5. Vision Benchmark Reality Check

Benchmark	What It Tests	GPT-4V	Gemini 1.5 Pro	LLaVA-1.6-34B
VQAv2	Visual QA	77.2%	73.2%	79.3%
MMBench	Multi-modal reasoning	75.8%	73.9%	68.9%
OCRBench	Text in images	645/1000	680/1000	568/1000
MathVista	Math + visuals	49.9%	52.1%	46.2%
MMMU	College-level multi-modal	56.8%	58.5%	44.7%

What these numbers obscure: All frontier models perform significantly worse on domain-specific images (medical, microscopy, specialized diagrams) that were underrepresented in training data. Your production use case almost certainly differs from these benchmarks. Evaluate on your actual data.

---

6. Audio-Language Models

Whisper (OpenAI, arXiv 2212.04356): Transformer encoder-decoder for speech-to-text. Trained on 680K hours of weakly labeled speech. Available in 5 sizes (39M → 1.5B parameters). Excellent for most languages; struggles with heavy accents, domain-specific terminology, and overlapping speech.

import whisper

model = whisper.load_model("large-v3")  # 1.5B params, best accuracy
result = model.transcribe("audio.mp3", language="en")
print(result["text"])
# Runtime: ~20-30 seconds for a 5-minute clip on A100
# VRAM: ~10GB for large-v3

Whisper + LLM pipeline (most common production pattern):

Audio → Whisper → Transcript → LLM → Response
                  (+ timestamps for grounding)

Limitation: This pipeline cannot understand tone, emotion, or non-verbal audio cues. For those, you need models with native audio understanding (Gemini, GPT-4o audio mode).

---

7. Video Understanding

Video understanding is the frontier where production deployments are sparse and research demos are abundant. Mapping to current deployment reality:

Capability	Production-Ready	Research Demo	Notes
Video transcription	✅	—	Whisper + speaker diarization
Short clip QA (<2 min)	✅	—	Gemini Flash, GPT-4V
Long video summarization	⚠️	—	Gemini 1.5M context, but expensive
Temporal grounding	❌	✅	“Find the moment when X happens”
Video generation	❌ Production	✅ Research	Sora, Runway, but unreliable
Action recognition	✅	—	Specialized models (VideoMAE)

The honest status of video generation (2025): Sora and competitors produce impressive demos but are not reliable for commercial content production without significant human curation. Temporal consistency degrades beyond 10-15 seconds. Physics simulation is still broken for complex interactions.

---

8. Multi-Modal RAG

Standard RAG works on text chunks. Multi-modal RAG extends this to images, tables, and charts — which is where most enterprise document understanding actually lives.

ColPali (arXiv 2407.01449): Rather than extracting text from PDF pages (losing layout/visual information), ColPali embeds the page as an image directly using a vision model. Achieves significantly better retrieval on document-heavy benchmarks than text-extraction pipelines.

# Conceptual multi-modal RAG pipeline
from byaldi import RAGMultiModalModel  # ColPali wrapper

# Index documents as images (preserves layout, tables, charts)
RAG = RAGMultiModalModel.from_pretrained("vidore/colpali")
RAG.index(
    input_path="./documents/",
    index_name="my_docs",
    store_collection_with_index=True
)

# Retrieve by visual similarity
results = RAG.search("quarterly revenue breakdown", k=3)

# Pass retrieved page images to vision LLM
for result in results:
    # result.base64 is the page image
    # Pass to GPT-4V or LLaVA for answer extraction
    pass

---

9. What’s Actually Deployed vs. Research Demos

Honestly deployed in production (2025):

• Document OCR + understanding: GPT-4V, Claude 3.5 Sonnet — reliable

• Chart/figure analysis: GPT-4V — reliable for standard charts; fails on custom formats

• Speech transcription: Whisper — production-grade

• Image classification with natural language: CLIP-based pipelines — production-grade

• Alt-text generation for accessibility: GPT-4V, LLaVA — production-grade

In production but with significant caveats:

• Visual QA on medical/scientific images: Models hallucinate with confidence. Require expert validation loops.

• Video understanding for long content: Gemini 1.5 Pro works but at $3.50/1M tokens with 1M context queries, cost becomes significant fast.

• Multi-modal agents (e.g., “look at this UI screenshot and click the button”): Works in controlled environments; brittle in the real world.

Research demos not yet production-reliable:

• Video generation for commercial use

• Real-time audio conversation with full emotion understanding

• 3D understanding and spatial reasoning

• Multi-modal reasoning that matches text-only reasoning quality

---

10. What Most People Get Wrong

“Multi-modal capability benchmarks measure research performance, not production reliability.”

The gap between MMMU scores and real-world deployment performance is consistently 15-25% on domain-specific data. The reasons:

1. Your domain’s images are different from internet images

2. OCR quality degrades sharply on low-resolution, handwritten, or domain-specific notation

3. Multi-modal models tend to confabulate image details confidently — harder to catch than text hallucination

4. Evaluation is expensive (no automatic metrics as reliable as BLEU/ROUGE for text)

The second mistake: Treating vision as a preprocessing step (“first convert image to text, then run NLP”). For documents with rich layout, tables, or diagrams, this discards structural information that was load-bearing. ColPali-style end-to-end visual embedding frequently outperforms OCR pipelines on real enterprise documents.

---

Papers

Paper	arXiv ID	What It Contributes
CLIP	2103.00020	Contrastive visual-language pretraining
LLaVA	2304.08485	Connecting vision encoder to LLM via projection
LLaVA-1.5	2310.03744	MLP connector + improved data pipeline
Flamingo	2204.14198	Interleaved image-text pretraining
InstructBLIP	2305.06500	Instruction-tuned visual QA
Whisper	2212.04356	Speech recognition at scale
ColPali	2407.01449	Document retrieval via visual embeddings
VideoMAE	2203.12602	Efficient video understanding

---

Return to README.md · Previous: 04_llm_agents.md · Next: 06_phase_projects.md