EximiusLabs/fusion-embedding-1-2b-preview · Hugging Face

Log In<br>Sign Up

One model. One vector space. Text, image, video, audio — and PDF.

An open-weight multimodal embedding model that extends a state-of-the-art<br>vision-language embedding base with audio — without modifying a single base weight.

Fusion Embedding 1 extends Qwen3-VL-Embedding-2B<br>with an audio modality. A trained connector (~16M parameters) maps frozen<br>Qwen2.5-Omni audio-tower features into the<br>base model's embedding space; the base model itself is unmodified. The result is a single<br>embedding space covering text, images, video, and audio , with retrieval supported in<br>any direction between modalities.

Highlights

Leads every unified embedding model we measured on audio↔text. On a single<br>cross-modal protocol, this model exceeds ImageBind, LanguageBind, and Gemini<br>Embedding 2 on audio↔text in both directions, and both language-bound baselines on<br>emergent audio↔image (full tables below).

Unmodified base. Only the connector is trained; the base model's parameters are<br>byte-identical to the original release, so its text/image/video retrieval performance<br>(MMEB-V2) carries over unchanged.

Emergent cross-modal alignment. The connector is trained exclusively on audio–text<br>pairs. Audio→image retrieval nonetheless reaches R@10 0.407 over 696 VGGSound candidates<br>(chance: 0.014) with no audio-visual pairs in training — alignment to text places audio<br>in the space the base already shares across modalities.

Matryoshka representation. Embeddings truncate to {2048, 1536, 1024, 512, 256, 128,<br>64} dimensions with renormalization.

Compact distribution. This repository ships the connector and normalization<br>statistics (~60 MB); the frozen towers are downloaded from their original repositories.<br>The parameter count shown for this repository (16.4M) is the trained connector —<br>model.safetensors and the .pt checkpoint contain the same weights; inference.py<br>loads the .pt.

This is a research preview , currently at v0.3 : the v0.2 contrastive stage (484K<br>pairs) followed by a connector-only in-domain fine-tune on the AudioCaps train split.<br>Earlier versions remain downloadable via the v0.1-preview and v0.2-preview tags;<br>v0.3-preview pins the current version. All are compared below; pin a tag if you build<br>on this model.

Architecture

A perceiver-resampler (width 384, 64 latent queries) translates frozen audio-tower frames<br>into the base model's input embedding space; its outputs occupy placeholder positions in<br>the input stream, mirroring the base model's image-token mechanism. Training is<br>contrastive (InfoNCE over the Matryoshka ladder, symmetric, with a full-corpus<br>frozen-text negative bank — 484K captions at v0.2) against the base model's text<br>embeddings in its native input format. v0.3 adds a second, connector-only fine-tuning<br>stage on the AudioCaps train split (400 steps at a reduced learning rate), warm-started<br>from the v0.2 checkpoint.

Input formatting. All inputs use the base model's chat-template format (instruction in<br>the system turn, content in the user turn, last-token pooling). Embedding quality is<br>sensitive to this formatting; use the templates in inference.py. For cross-modal<br>ranking, per-modality mean-centering of the gallery is recommended (FusionEmbedder.center).

Evaluation

Cross-modal retrieval — versus unified embedding models

VGGSound-AV, 696 audio/video-frame pairs (chance R@10 = 0.014). R@10 shown as<br>audio-side → other / other → audio-side:

Model<br>audio↔image<br>audio↔text<br>text↔image

ImageBind-Huge<br>0.718 / 0.720<br>0.404 / 0.348<br>0.243 / 0.282

LanguageBind<br>0.365 / 0.415<br>0.547 / 0.331<br>0.221 / 0.283

Gemini Embedding 2 (API, 2026-07-09)<br>0.312 / 0.316<br>0.379 / 0.374<br>0.273 / 0.366

fusion-embedding-1-2b-preview v0.1<br>0.368 / 0.388<br>0.555 / 0.592<br>0.331 / 0.319

fusion-embedding-1-2b-preview v0.2<br>0.418 / 0.440<br>0.588 / 0.631<br>0.331 / 0.319

fusion-embedding-1-2b-preview v0.3<br>0.407 / 0.428<br>0.625 / 0.645<br>0.331 / 0.319

ImageBind trains directly on audio–image pairs, so that pair is its supervised direction;<br>its audio–text alignment is emergent. LanguageBind trains audio against language (its<br>audio↔text is supervised; the value shown is its best readout, using the audio branch's<br>own text tower); its audio↔image is emergent. This model trains on audio–text only; its<br>audio–image alignment is emergent. All models evaluated with identical clips, frames, and<br>scoring, using the released imagebind_huge checkpoint and revision-pinned LanguageBind<br>checkpoints (LanguageBind_Audio_FT + LanguageBind_Image). Note on LanguageBind: its<br>branches fine-tune separate copies of the text tower, which diverge (mean caption cosine<br>0.55 between the audio and image branches' text embeddings) — the cross-branch binding<br>weakens, which is consistent with its emergent audio↔image score. This model's shared<br>space cannot drift by construction (the base is frozen; every training run asserts<br>parameter-level identity). Gemini Embedding 2 is Google's natively multimodal embedding<br>API...