microsoft/Mage-VL · Hugging Face

Log In<br>Sign Up

","pad_token":"","unk_token":null},"chat_template_jinja":"{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}system\nYou are a helpful assistant.\n{% endif %}{{ message['role'] }}\n{% if message['content'] is string %}{{ message['content'] }}\n{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}\n{% endif %}{% endfor %}{% if add_generation_prompt %}assistant\n{% endif %}"},"createdAt":"2026-07-25T08:29:09.000Z","discussionsDisabled":false,"discussionsSorting":"recently-created","downloads":0,"downloadsAllTime":0,"id":"microsoft/Mage-VL","isLikedByUser":false,"availableInferenceProviders":[],"showHuggingChatEntry":false,"inference":"","lastModified":"2026-07-26T11:43:51.000Z","likes":11,"pipeline_tag":"image-text-to-text","library_name":"transformers","librariesOther":[],"trackDownloads":true,"model-index":null,"private":false,"repoType":"model","gated":false,"tags":["transformers","safetensors","mage_vl","image-text-to-text","multimodal","vision-language-model","mage-vl","video-understanding","streaming","conversational","custom_code","license:apache-2.0","region:us"],"tag_objs":[{"id":"image-text-to-text","label":"Image-Text-to-Text","type":"pipeline_tag","subType":"multimodal"},{"id":"transformers","label":"Transformers","type":"library"},{"id":"safetensors","label":"Safetensors","type":"library"},{"id":"mage_vl","label":"mage_vl","type":"other","clickable":true},{"id":"multimodal","label":"multimodal","type":"other","clickable":true},{"id":"vision-language-model","label":"vision-language-model","type":"other","clickable":true},{"id":"mage-vl","label":"mage-vl","type":"other","clickable":true},{"id":"video-understanding","label":"video-understanding","type":"other","clickable":true},{"id":"streaming","label":"streaming","type":"other","clickable":true},{"id":"conversational","label":"conversational","type":"other","clickable":true},{"id":"custom_code","label":"custom_code","type":"other","clickable":true},{"id":"license:apache-2.0","label":"apache-2.0","type":"license"},{"type":"region","label":"🇺🇸 Region: US","id":"region:us"}],"transformersInfo":{"auto_model":"AutoModelForImageTextToText","custom_class":"modeling_mage_vl.MageVLForConditionalGeneration","pipeline_tag":"image-text-to-text"},"widgetData":[{"text":"Hi, what can you help me with?"},{"text":"What is 84 * 3 / 2?"},{"text":"Tell me an interesting fact about the universe!"},{"text":"Explain quantum computing in simple terms."}],"safetensors":{"parameters":{"BF16":4741793792},"total":4741793792,"sharded":true,"totalFileSize":9483741656},"hasBlockedOids":false,"region":"us","isQuantized":false},"discussionsStats":{"closed":0,"open":1,"total":1},"query":{},"inferenceContextData":{"billableEntities":[],"entityName2Providers":{}},"hasQuantizations":false,"copyToBucketNamespaces":[]}">

Mage-VL<br>An Efficient Codec-Native Streaming Multimodal Foundation Model

Mage-VL is a codec-native, proactive-streaming multimodal foundation model for image and video understanding, whose visual encoder is trained entirely from scratch at a compact 4B scale. It targets a modern Moravec's paradox of VLMs — strong at complex offline reasoning, yet slow and compute-heavy on simple real-time streaming perception. Instead of decoding video into uniformly-sampled frames and pushing a dense grid of patch tokens through a frozen web-pretrained ViT, Mage-VL follows the structure of modern video codecs: it separates a stream into anchor (I) frames and predicted (P) frames , keeps every anchor patch, and retains only the predicted-frame patches where the codec spends bits — the regions carrying real motion and new detail. This codec-aligned sparsity cuts visual tokens by over 75% while preserving spatio-temporal context, yielding up to 3.5× wall-clock inference speedup over uniform frame sampling.

The system pairs two components :

Mage-ViT — a from-scratch Codec-ViT visual encoder that allocates tokens by codec-derived spatio-temporal importance, on a shared 16×16 patch grid with 3D rotary position encoding. It is codec-agnostic : the same interface accepts a traditional codec (H.264/AVC, HEVC/H.265) via motion vectors + residual energy, or a neural codec (DCVC-RT) via its learned rate map — no architecture or retraining change.

Qwen3-4B causal decoder — a Qwen3-4B-Instruct-2507 language backbone (the only pretrained component) that consumes Mage-ViT's variable-length token...