[Paper Review] Nomic Embed: Training a Reproducible Long Context Text Embedder

Zach Nussbaum, John X. Morris+2 more|arXiv (Cornell University)|Feb 2, 2024

Translation Studies and Practices23 citations

TL;DR

This technical report presents nomic-embed-text-v1, an open-source, 8192-token long-context text embedding model with 137M parameters that surpasses OpenAI ada-002 and text-embedding-3-small on short and long-context benchmarks, along with fully auditable training data, code, and weights.

ABSTRACT

This technical report describes the training of nomic-embed-text-v1, the first fully reproducible, open-source, open-weights, open-data, 8192 context length English text embedding model that outperforms both OpenAI Ada-002 and OpenAI text-embedding-3-small on the short-context MTEB benchmark and the long context LoCo benchmark. We release the training code and model weights under an Apache 2.0 license. In contrast with other open-source models, we release the full curated training data and code that allows for full replication of nomic-embed-text-v1. You can find code and data to replicate the model at https://github.com/nomic-ai/contrastors.

Motivation & Objective

• Demonstrate a fully reproducible long-context text embedding model with auditable data, weights, and code.
• Show that nomic-embed-text-v1 outperforms OpenAI ada-002 and text-embedding-3-small on short and long-context benchmarks.
• Provide a transparent training pipeline and dataset to enable end-to-end replication and auditability.

Proposed method

• Adapt BERT-base architecture with 8192 token support via architectural changes (rotary positional embeddings, SwiGLU, Flash Attention).
• Train nomic-bert-2048 with masked language modeling on BooksCorpus and 2023 Wikipedia dump.
• Perform unsupervised contrastive pretraining on 235M curated text pairs with InfoNCE loss and in-batch negatives.
• Fine-tune with supervised contrastive learning on BEIR-compatible datasets using hard negatives for retrieval tasks.
• Conduct final supervised contrastive fine-tuning with task prefixes and a limited number of negatives.

Experimental results

Research questions

• RQ1 Can a fully auditable, open-data long-context embedding model achieve competitive or superior performance to closed-source baselines on both short and long-context tasks?
• RQ2 What training regimen and architectural choices enable effective long-context (8192-token) embeddings while maintaining trainability and reproducibility?
• RQ3 How does adding long-context data and multi-stage contrastive learning impact MTEB, LoCo, and Jina long-context benchmarks?
• RQ4 Does releasing data loaders and curations enable end-to-end replication and auditability of the embedding model?

Key findings

• Nomic-embed-text-v1 (137M parameters) with 8192 context length surpasses ada-002 and text-embedding-3-small on short and long-context benchmarks (MTEB, LoCo, Jina LC).
• The ablated variant nomic-embed-text-v1-ablated shows performance drop when FEVER/HotpotQA/MEDI data are removed, indicating the impact of these datasets.
• Long-context benchmarks (Jina LC and LoCo) show nomic-embed-text-v1 beating jina-embeddings-v2-base-en across sequence lengths and outperforming ada-002 on several datasets.
• Nomic-bert-2048 achieves competitive GLUE results, comparable to MosaicBERT, demonstrating strong base pretraining with long-context adaptations.
• The full training stack (weights, code, data) is released under Apache 2.0, enabling end-to-end replication and auditability.