To enable communication between a model that understands images and one trained on language, the most common recipe requires massive collections of image-text pairs. The same holds true for other domains: molecules and descriptions, audio and transcripts, documents and structured data. But building these pairings is costly, requires specialized expertise, and often runs up against availability or licensing limitations. A paper published on arXiv by Julien Kordes and Reuben Dorent tackles precisely the less convenient, yet practically very frequent scenario: reliable pairs are scarce—on the order of 100 to 500—while unpaired data is abundant.
The preprint, deposited on October 8, 2026, describes a system for aligning representations produced by two independently trained unimodal encoders. The goal is not to train a large multimodal model from scratch, but to build a bridge between numerical spaces that already exist, making elements from different modalities comparable. If the bridge works, a text description can be matched with an image, or a text with a molecule, even when starting from separate encoders.
The proposed method is called Distilled Optimal Transport and combines a common ground obtained from the few known pairs, global matching between datasets, and a final stage that makes the result applicable to new examples. The authors report results superior to competing methods with the same number of anchor examples on the Flickr30k, MS-COCO, and ChEBI-20 benchmarks. An essential caveat remains, however: this is a preprint, meaning the results have not yet undergone peer review.
The hidden problem behind two separate encoders
An encoder transforms an input into a numerical representation, often a vector that preserves features considered relevant by the model. Two encoders can each be highly effective in their own domain while remaining completely incompatible with one another. A vector associated with a photograph and one associated with its caption are not automatically comparable simply because they share a similar mathematical form.
To bridge this gap, conventional multimodal training uses a large number of paired examples: a photo of an object and the sentence describing it, for instance. The model gradually learns to bring correct pairs closer together and push incorrect ones apart. When annotated associations number only a few hundred, this approach becomes fragile: they are not enough to cover linguistic variety, visual categories, or edge cases, while a simple nearest-neighbor comparison can produce unbalanced matches.
Kordes and Dorent's work stems from a practical observation. Even if complete pairs are lacking, many images may be available on one side and plenty of texts on the other. These collections do not need to originate from the same elements: it is enough that they represent the two distributions to be connected. The only set for which observing both modalities remains necessary is that of the initial anchors.
Anchors become a shared coordinate system
The first step involves using the available pairs as reference points. Instead of attempting to directly align the original spaces of the two encoders, each element is described through its similarity to the anchors. An image is thus interpreted relative to the anchor images; a text, relative to the corresponding anchor texts.
Because the anchors are paired across the two modalities, these similarity lists provide a sort of shared coordinates. The mechanism is important because it shifts the comparison from the absolute meaning of the vectors—which can vary radically from one encoder to another—to internal relationships relative to a small, known core. In other words, the method attempts to determine where to place an unobserved element by looking at how it aligns with the same reference points.
This choice does not eliminate the informational limit imposed by the small number of pairs. If the anchors are poorly representative of the domain, the system has fewer signals to distinguish relevant concepts, classes, or structures. The contribution of this work lies in attempting to make better use of the few existing associations and bringing the vast volumes of unpaired data into the alignment as well.
Why optimal transport can avoid distorted matches
After constructing the shared coordinates, the method connects the two point clouds using a single entropic optimal transport problem computed over the entire available dataset. Optimal transport is a family of mathematical techniques used to find a way to map or distribute mass between two sets while minimizing a problem-defined transport cost.
Here, each match is not determined by looking at the nearest candidate in isolation. Instead, the transport plan considers the overall distribution of points across both sides. Its marginal constraints impose limits on the amount of assigned mass, preventing a small number of elements from becoming the recurring target of countless associations. This is the phenomenon known as hubness: in high-dimensional spaces, certain points can spuriously appear close to an excessive number of other points, degrading match retrieval.
The authors link this aspect to single-step remedies studied in bilingual lexicon induction, the research area that aims to extract correspondences between words in different languages from distinct representation spaces. The parallel is not merely terminological: both problems suffer from the tendency of certain candidates to act as universal attractors. The use of global constraints seeks to correct that flaw directly during assignment.
From the transport plan to a usable system
A matching computed on a closed collection has an obvious limitation: it does not automatically indicate how to handle the next incoming document, image, or molecule. For this reason, the procedure does not stop at the transport plan. The result is distilled into two lightweight encoders, one for each side, with the goal of transforming the obtained alignment into an inductive capability.
This is a critical step for real-world applications. A transductive system can only perform well on elements already included in the computation; an inductive system, by contrast, produces a mapping applicable to new inputs. The preprint therefore proposes a three-stage pipeline: anchor-based representation, global matching of unpaired data, and distillation into leaner models. However, the available abstract does not provide sufficient detail to evaluate computational costs, parameter sensitivity, or behavior on significantly larger collections.
The reported evaluations cover image-text alignment on Flickr30k and MS-COCO, two well-established benchmarks for the relationship between images and descriptions, along with molecule-text alignment on ChEBI-20. The inclusion of the molecular domain is significant because it suggests the approach does not rely solely on the regularities of visual language. At the same time, the three benchmarks alone are not sufficient to prove that the method transfers with equal effectiveness to every combination of encoder, language, industry sector, or data distribution.
What could change and what remains to be proven
If confirmed by further testing, results of this kind could make integrating specialized models more accessible. Laboratories, companies, and organizations that possess extensive yet sparsely annotated archives might not need to create millions of new pairings to connect pre-trained systems. The potential advantage primarily concerns contexts where paired data requires human review, scientific expertise, or complex agreements to be shared.
However, a key issue remains: the alignment inherits the quality of the baseline encoders and the selected anchors. Imbalanced data, biased representations, or ambiguous initial pairs can propagate errors throughout the entire procedure. Independent comparisons, rare-class analyses, and evaluations beyond the cited benchmarks will therefore be crucial, along with the availability of reproducible implementations and protocols.
For now, the contribution should be seen as a research proposal with a clear message: in multimodal alignment, a scarcity of pairs does not necessarily mean giving up on unlabeled data. The challenge shifts to constructing a global constraint robust enough to leverage those collections without turning superficial similarities into incorrect associations.
