Stop Waiting on PyTorch: How to Generate Fast Local Vector Embeddings with FastEmbed in 5 Lines of Python

Last updated: November 7, 2026 · 4-minute read

If you are building a Retrieval-Augmented Generation (RAG) pipeline, local semantic search, or document deduplication tool in Python, you have almost certainly reached for sentence-transformers.

And you have almost certainly paid the heavy tax that comes with it.

Installing sentence-transformers drags in torch, torchvision, and hundreds of megabytes of CUDA and C++ shared libraries. Your clean, lightweight Docker container balloons from 120MB to nearly 2.5GB. Cold-start times in serverless functions (like AWS Lambda or Google Cloud Run) jump to fifteen seconds just to load the PyTorch runtime into memory.

Worst of all, if you are running on standard developer laptops or cloud CPU instances without an expensive dedicated GPU, PyTorch's execution is notoriously heavy and unoptimized.

There is a vastly faster, featherweight alternative: FastEmbed.

Maintained by the team at Qdrant, FastEmbed strips out PyTorch entirely and runs embedding models on ONNX Runtime. It installs in seconds, boots instantly, uses a fraction of the memory, and is built for fast multi-core CPU inference — Qdrant positions it as the lighter, faster route for exactly this job.

Here is how to set it up in five lines of Python and ditch PyTorch bloat forever.

Why PyTorch Is Overkill for Embeddings

When you train a deep learning model, you need PyTorch’s dynamic computation graphs, autograd engine, and backpropagation machinery.

However, generating embeddings is pure inference. You are passing static text through a frozen, pre-trained transformer model (like BGE, MiniLM, or Nomic) and reading the output vectors.

Running full PyTorch just for embedding inference is like chartering a cargo freight train to deliver an envelope. ONNX Runtime, by contrast, compiles the model weights into an optimized compute graph specifically tuned for your CPU architecture (AVX-512, NEON, or Apple Metal).

Treat the table as orders of magnitude, not lab benchmarks. Wheel sizes depend on platform, and timings move with hardware — but the direction of every row is structural: a training framework on one side, an inference runtime on the other. Figures reflect typical installs as of November 2026.

Step 1: Install FastEmbed in Seconds

Because it doesn't depend on PyTorch, FastEmbed installs almost instantly:

# Or standard pip pip install fastembed

Step 2: Generate Dense Embeddings in 5 Lines

Using FastEmbed is ridiculously straightforward. By default, it uses the battle-tested BAAI/bge-small-en-v1.5 model (a compact 384-dimensional model that punches far above its weight):

# 1. Initialize the embedding model (downloads quantized ONNX weights automatically on first run) model = TextEmbedding(modelname="BAAI/bge-small-en-v1.5")

# 2. Your documents documents = [ "Model Context Protocol (MCP) connects AI assistants to local tools.", "FastEmbed generates text embeddings using ONNX Runtime without PyTorch.", "DuckDB is a columnar SQL analytics database for local files.", ]

# 3. Generate embeddings as a memory-efficient generator embeddings = list(model.embed(documents))

print(f"Generated {len(embeddings)} vectors.") print(f"Vector dimension: {len(embeddings[0])}") print(f"First 5 dimensions: {embeddings[0][:5]}")

Terminal output: text Generated 3 vectors. Vector dimension: 384 First 5 dimensions: [-0.0341, 0.0812, -0.0195, 0.0456, -0.0711]

Step 3: Fast Streaming for Massive Datasets

One of FastEmbed’s cleanest design choices is that .embed() returns a Python generator rather than buffering millions of floating-point numbers into RAM.

If you are embedding a 500,000-row CSV file, you can stream batches directly into SQLite, DuckDB, or a vector store without triggering an out-of-memory crash:

# Streams batches of 256 items across all CPU cores for vector in model.embed(streamlargedataset(), batchsize=256): savetodatabase(vector)

FastEmbed automatically detects your available CPU cores and parallelizes tokenization and inference across worker threads.

Beyond Dense Vectors

Modern search systems frequently combine dense embeddings (for semantic concepts) with sparse embeddings (like BM25 or SPLADE for exact keyword matching).

FastEmbed supports dense, sparse, and late-interaction models out of the box with the same zero-PyTorch architecture:

# Generate BM25 / SPLADE sparse vectors for hybrid search sparsemodel = SparseTextEmbedding(modelname="Qdrant/bm25")

sparseembeddings = list(sparsemodel.embed(["fast local vector search"])) print(sparseembeddings[0]) # Output: SparseEmbedding(indices=[1402, 2891, ...], values=[0.84, 1.12, ...])

Pro Tips for Production Deployments