Qdrant builds dataset to benchmark vector retrieval at scale

Open-source value

Qdrant's origins are in open-source technology.

Founded in 2021 and based in Berlin and New York City, the vendor began as an open-source project published on GitHub before launching a proprietary version of its vector database in 2023. Meanwhile, with both versions of its vector database built with the open-source Rust programming language, Qdrant remains closely tied to the open-source community.

This dataset enables organizations to accurately measure recall … on realistic, non-synthetic web data under true production scale.

Developing a public dataset for benchmarking vector retrieval capabilities reflects the vendor's recognition that public benchmarks that measure the rising demands being placed on vector retrieval systems are needed, according to Nathan LeRoy, a research engineer at Qdrant Labs.

"Vector search workloads have already hit billion vector scale. Enterprises need benchmarks to reflect this so that they can make informed decisions at this scale," he told TechTarget. "Ideally, such benchmarks and their datasets are public and accessible, so it's easy to make comparisons and draw conclusions across search systems."

While Qdrant recognizes the value of the open-source tools, McKnight noted that vendors sometimes release open-source capabilities even if they aren't closely aligned with the open-source community.

Google, for example, developed the Agent2Agent protocol in May 2025 to provide an open standard for connecting and orchestrating agents and donated it to the Linux Foundation.

However, the reasons vendors develop open-source capabilities are not always magnanimous, McKnight continued. While it shows that the vendor is committed to its market and is aligned with cutting-edge thought, it also can lead to greater visibility and attract new potential customers.

"It isn't uncommon for vendors to release open-source benchmarking datasets and tools, especially ones they are likely good at competitively," he said.

Michael Ni, an analyst at Constellation Research, similarly noted that Qdrant's launch of an open-source dataset for benchmarking vector retrieval is beneficial to enterprises as they build AI tools, as well as the vendor itself.

"A public dataset like Fineweb-10B is an ecosystem and category-building move," he told TechTarget. "Qdrant is effectively subsidizing the cost of evaluation … and position[ing] Qdrant as central to that evaluation. Qdrant gains mindshare, creates de facto standards and evaluation criteria, and increases the odds [potential customers] ask the kinds of questions Qdrant is already optimized to answer."

Up to the test

Qdrant Fineweb-10B is aimed at enabling organizations to benchmark vector retrieval at enterprise scale.

Toward that end, Qdrant built the dataset using vector embeddings of real documents rather than synthetic data, and a corpus -- a collection of data -- of 10 billion documents so it is large enough to ensure that size-dependent behavior shows up during testing. In addition, the dataset includes a grounded truth query set, which is a list of search prompts paired with their ideal responses against which an enterprise can measure the accuracy of its benchmark tests.

Ni noted that Fineweb-10B is not the first public dataset for benchmarking vector retrieval. However, the size of the new dataset differentiates it from others.

"This is an important extension of existing benchmark work, not a brand-new category of benchmarking," he said. "Qdrant has contributed a much larger, more application-like retrieval workload with known correct answers that allow engineering teams' tests to go beyond throughput to how accurately the system behaves as corpus size, filtering and retrieval complexity increase."

Meanwhile, the dataset is appropriately constructed to aid organizations seeking to better understand the effectiveness of their vector retrieval systems, Ni continued, while adding that it is only one part of what should be a broader evaluation process.

"Qdrant-Fineweb-10B certainly solves the problem Qdrant says it is solving, measuring whether vector retrieval remains accurate and performant at very large scale," Ni said. "At the same time, companies still need to test against their own data, business filters, re-ranking and downstream agent outcomes as part of a total enterprise retrieval approach versus just testing a retrieval engine."

McKnight similarly noted that Qdrant-Fineweb-10B is well constructed, though limited in scope.

"The dataset seems well-designed for pairing realistic web-text distributions with exact ground truth across dense, sparse and filtered queries," he said. "However, fully leveraging it seems to require substantial compute and storage infrastructure to ingest and index 10 billion records, and it does not test real-time streaming updates."

Looking ahead

As Qdrant plans product development, speed, accuracy and openness are focal points, according to LeRoy. In April, the vendor's Qdrant Cloud update included features aimed at improving performance.

"As 2026 draws to a close, we’ll continue to push the limits of vector search for speed, accuracy, and being deployable anywhere," LeRoy said.

Focusing on speed and accuracy is wise, according to Ni.

He noted that vector database specialists such as Qdrant and Pinecone now face competition from broad-based data management providers, including hyperscale cloud vendors AWS and Oracle, that provide vector search. As a result, specialists need to distinguish themselves.

"Vector search is becoming table stakes," Ni said. "Most enterprises can get it from databases they already own, so specialists like Qdrant have to drive what makes them distinct and valuable to the market by delivering better retrieval quality, latency and cost at scale."

McKnight, meanwhile, suggested that better integrating feedback loops and adding to its metadata management capabilities would help Qdrant better serve existing customers and perhaps draw in new ones.

"By natively integrating generative feedback loops, [Qdrant would remove] the need for teams to build these pipelines via external APIs," he said. "Additionally, broadening its rich scalar metadata capabilities to include multidimensional matrices and tensors would enhance support for complex data structures."

Eric Avidon is a senior news writer for Informa TechTarget and a journalist with more than three decades of experience. He covers analytics and data management.