Multimodal & OCR

Japanese OCR, vision models, speech-to-text on RTX 3060 12GB

Japanese OCR: Decision Tree

PaddleOCR PP-OCRv5

github.com/PaddlePaddle/PaddleOCR

• Japanese explicitly supported (unified Chinese/English/Japanese framework)
• PP-OCRv5_server_rec is the high-accuracy server variant (~2M params)
• 13% accuracy boost over v4, 109 languages
• Nearly instant for individual images
• Production-ready, battle-tested

PaddleOCR-VL

• 0.9B parameter vision-language model for document parsing
• 94.5% accuracy on OmniDocBench
• 111 languages, seal recognition, text spotting
• Robust to skew, warping, scanning artifacts, illumination
• Significantly more capable than traditional PaddleOCR for complex layouts

Qwen 2.5 VL for OCR

Available in 3B, 7B, 32B, 72B. The 3B model is the sweet spot for 12GB VRAM:

Fully supported in llama.cpp with mmproj. At ~6-8GB in BF16, fits your GPU.

# Qwen 2.5 VL 3B with vision
llama-server -hf ggml-org/Qwen2.5-VL-3B-Instruct-GGUF \
  -ngl 99 -fa -c 4096

Qwen3.5-4B with Vision

The all-in-one option: OCR + translate + converse in 201 languages.

Gemma 4 26B-A4B Multimodal

unsloth GGUFs | Model card

The "multi lingual/audio/image/video" claim is partially wrong for 26B-A4B. Audio requires the smaller E2B/E4B models. "E" stands for "effective parameters" (not e2b.dev sandbox).

140+ languages, 35+ with out-of-the-box quality. Japanese included. 256K context.

Quantizations That Fit

Vision in llama.cpp (mmproj)

What are mmproj files?

The multimodal projector (mmproj) is a separate GGUF file that bridges visual input to the language model. It projects image features (from SigLIP or ViT) into the LLM's token embedding space.

Key Flags

Models with Working Vision in llama.cpp

Qwen3.5 is NOT yet listed. Audio models: Ultravox 0.5 (1B, 8B), Voxtral Mini 3B, Qwen2.5 Omni (3B, 7B).

Speech-to-Text

Parakeet TDT 0.6B v2 (English)

HuggingFace — NVIDIA's FastConformer-TDT architecture

• 16kHz mono audio, up to 24 minutes per segment
• Automatic punctuation, capitalization, word-level timestamps
• English only — no Japanese
• v3 adds 25 European languages (still no Japanese)

pip install nemo_toolkit[asr]

import nemo.collections.asr as nemo_asr
model = nemo_asr.models.ASRModel.from_pretrained("nvidia/parakeet-tdt-0.6b-v2")
output = model.transcribe(['audio.wav'], timestamps=True)
print(output[0].text)

LFM2.5-VL-450M (Liquid AI)

Blog | 450M param vision-language model

• 8 languages including Japanese
• OCRBench: 684 (mediocre vs Qwen's 850+)
• Best for: object detection, bounding boxes (RefCOCO-M: 81.28)
• Speed: 242ms on Jetson Orin for 512x512 images
• Q4 would be ~0.25GB — trivial VRAM
• Gated model — apply for access on HuggingFace

Not recommended for rich OCR or description — a 3-4B VLM produces much better output. Useful only for lightweight structured vision tasks.

Tesseract vs VLM-based OCR

From r/LocalLLaMA discussion:

For Japanese manga/comics: VLM-based OCR is strongly preferred. Text appears in vertical columns, speech bubbles, and artistic fonts that Tesseract handles poorly.

Recommended Setup

Daily OCR Pipeline (Gallery/Manga app)

1. PaddleOCR PP-OCRv5 for fast pre-screening (instant, CPU)
2. Qwen 2.5 VL 3B for complex images needing context (llama.cpp, ~6GB VRAM)
3. Both can run simultaneously — PaddleOCR on CPU, Qwen on GPU

Voice Pipeline

1. Parakeet TDT 0.6B v2 for English STT (2GB VRAM)
2. Whisper large-v3 for Japanese STT (3GB VRAM)
3. Feed transcription to Qwen3.5-4B for processing (2.2GB VRAM)
4. Total: ~7GB VRAM — fits comfortably with room for context