All You Need to Know about JEPA | Overshoot

PlatformDocumentationBlogJoin us

← Back to all postsBeyond Labels and Pixels

Traditionally, supervised vision systems learn from labeled examples. For instance, an image is paired with a labeled category such as "dog," "car," or "airplane." In training, the model then learns to adjust its parameters until it can predict those classes reliably.

While labeled learning has been crucial to modern AI, it also provides only a narrow description of the visual world. A given photograph can contain information about objects, materials, spatial relationships, motion, lighting, and physical structure. These elements are far more than a single category can convey.

In contrast, self-supervised learning offers a different approach. Instead of merely relying on human-written labels, the model constructs a learning task from the raw data itself. It might hide part of an image, for example, and learn to predict what is missing. Many self-supervised methods either reconstruct the missing input or train different views of the same image to produce similar representations. A Joint-Embedding Predictive Architecture, or JEPA, explores a different goal:

Instead of predicting the missing pixels, can a model predict what the missing region represents?

Learning Without Labels

One useful way to understand visual self-supervised learning is through three broad architectural approaches: generative architectures, invariance-based joint-embedding architectures, and joint-embedding predictive architectures.

Generative architectures

Reconstruction-based generative methods learn by reconstructing missing or corrupted parts of an input. For example, in a task like masked image modeling, a model receives the visible image patches and tries to recover the missing pixel values.

A Masked Autoencoder, or MAE, follows this approach. While this provides a clear training objective, it can require the model to reproduce details that are difficult to infer and may not be important for understanding the scene. If part of a dog is hidden, the model may need to predict exact colors, textures, shadows, and strands of fur. For many semantic tasks, the more useful information may be that the missing region belongs to the dog.

div]:my-0">

Masked Autoencoder reconstructing missing image patches from visible context.

Invariance-based joint-embedding architectures

Invariance-based methods train related views of the same image to produce similar representations. Two versions of the same image might have different colors, scale, or framing, but the model learns to recognize what remains consistent between them.

It is important to point out that these methods do not reconstruct the original pixels. Rather, they are actually learning representations that remain stable across selected transformations.

Joint-embedding predictive architectures

A JEPA also operates in representation space. Yet its objective is predictive and not purely invariant. At a high level, it uses the representation of one part of an input to predict the representation of another.

In place of reconstructing exact pixels, JEPA learns what information about a missing region can be inferred from the context that surrounds it.

If the Pixels Are Uncertain, What Should the Model Predict?

For an illustrative example, let's suppose part of an image is hidden.

A pixel-reconstruction system tries to recover the missing RGB values. A JEPA tries to predict the vector that an encoder would produce for that missing region.

A raw image patch contains hundreds of pixel values that describe its precise color and texture. Once these patches go through a vision encoder, each one is represented by a learned vector. Because a Vision Transformer allows patches to interact through attention, the resulting vector can reflect how the patch relates to the rest of the image.

Given this, consider an image of a dog standing on grass. If the dog's torso is hidden, the exact pattern of its fur may be impossible to determine from the surrounding patches. There can be many different pixel arrangements that form a plausible torso.

Thus, the predictable information is more general. This missing region probably contains part of the dog, which connects its head to its legs, and occupies a particular position in the scene.

JEPA is designed to capture this kind of predictable structure, but in a way where the model does not have to resolve every uncertain pixel-level detail. The learned representation space is not given in advance. Rather, this space starts unstructured and develops through training as the encoders and predictor repeatedly solve the prediction task.

How I-JEPA Learns From an Image

How can a model create a prediction target without being shown the missing content?

I-JEPA applies the JEPA idea to static images. The underlying architecture has three main components: a context encoder, a target encoder, and a predictor, all of which use...