VectorDBBench

VectorDBBench(VDBBench): A Benchmark Tool for VectorDB

What is VDBBench

VDBBench is not just an offering of benchmark results for mainstream vector databases and cloud services, it's your go-to tool for the ultimate performance and cost-effectiveness comparison. Designed with ease-of-use in mind, VDBBench is devised to help users, even non-professionals, reproduce results or test new systems, making the hunt for the optimal choice amongst a plethora of cloud services and open-source vector databases a breeze.

Understanding the importance of user experience, we provide an intuitive visual interface. This not only empowers users to initiate benchmarks at ease, but also to view comparative result reports, thereby reproducing benchmark results effortlessly. To add more relevance and practicality, we provide cost-effectiveness reports particularly for cloud services. This allows for a more realistic and applicable benchmarking process.

VectorDBBench provides repeatable workloads for insertion, vector search, filtered search, and full-text search. Its registered vector datasets range from classic SIFT and GIST corpora to Cohere embeddings, OpenAI embeddings generated from the public C4 dataset, and the VIBE and VDBBench multimodal datasets hosted on Hugging Face. Together, these datasets cover different corpus sizes, dimensions, modalities, distance metrics, and query distributions.

Prepare to delve into the world of VDBBench, and let it guide you in uncovering your perfect vector database match.

VDBBench is sponsored by Zilliz，the leading opensource vectorDB company behind Milvus. Choose smarter with VDBBench - start your free test on zilliz cloud today!

Leaderboard: https://zilliz.com/benchmark

🎈 Announcement 🎈

June 2026 update: Full Text Search has landed in VectorDBBench. We now benchmark BM25-style retrieval across supported backends, starting with MS MARCO and HotpotQA datasets, payload profiles, recall, QPS, and load metrics ready to compare. See the VectorDBBench Full Text Search Release Note for the full rollout details and caveats.

September 2026 update: VectorDBBench can now run registered datasets hosted on Hugging Face through the existing Performance case. The initial catalog includes all 24 VIBE HDF5 datasets and VDBBench multimodal Parquet datasets at 1M, 10M, and 100M corpus vectors.

vectordbbench milvushnsw \
  --case-type Performance \
  --dataset-name glove-200-cosine \
  --k 100

See the Hugging Face datasets release note for the complete catalog, data flow, provenance, constraints, and configuration details.

Quick Start

Prerequirement

python >= 3.11

Install

Install vectordb-bench with only PyMilvus

pip install vectordb-bench

Install the specific database client

pip install 'vectordb-bench[pinecone]'

All the database client supported

Optional database client	install command
pymilvus, zilliz_cloud (default)	pip install vectordb-bench
qdrant	pip install vectordb-bench[qdrant]
pinecone	pip install vectordb-bench[pinecone]
weaviate	pip install vectordb-bench[weaviate]
elastic, aliyun_elasticsearch	pip install vectordb-bench[elastic]
pgvector, pgvectorscale, pgdiskann, alloydb, vectorchord, lakebase_vector	pip install vectordb-bench[pgvector]
pgvecto.rs	pip install vectordb-bench[pgvecto_rs]
redis	pip install vectordb-bench[redis]
valkey	pip install vectordb-bench[valkey]
memorydb	pip install vectordb-bench[memorydb]
kividb	pip install vectordb-bench[kividb]
chromadb	pip install vectordb-bench[chromadb]
cockroachdb	pip install vectordb-bench[cockroachdb]
awsopensearch	pip install vectordb-bench[opensearch]
aliyun_opensearch	pip install vectordb-bench[aliyun_opensearch]
mongodb	pip install vectordb-bench[mongodb]
tidb	pip install vectordb-bench[tidb]
vespa	pip install vectordb-bench[vespa]
oceanbase	pip install vectordb-bench[oceanbase]
hologres	pip install vectordb-bench[hologres]
tencent_es	pip install vectordb-bench[tencent_es]
alisql	pip install vectordb-bench[alisql]
polardb	pip install vectordb-bench[polardb]
polardb_pg	pip install vectordb-bench[pgvector]
doris	pip install vectordb-bench[doris]
zvec	pip install vectordb-bench[zvec]
endee	pip install vectordb-bench[endee]
lindorm	pip install vectordb-bench[lindorm]
volc_mysql	pip install vectordb-bench[volc_mysql]
adbpg	pip install vectordb-bench[adbpg]

Run

init_bench

OR:

Run from the command line.

vectordbbench [OPTIONS] COMMAND [ARGS]...

To list the clients that are runnable via the commandline option, execute: vectordbbench --help

$ vectordbbench --help
Usage: vectordbbench [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  polardbpghnsw
  pgvectorhnsw
  pgvectorivfflat
  vectorchordrq
  volcmysqlhnsw
  test
  weaviate

To list the options for each command, execute vectordbbench [command] --help

milvusautoindex and zillizautoindex accept --level from 1 to 10 (YAML: level) on servers that support AutoIndex search levels. If omitted, Milvus sends no level, preserving the server's default behavior; Zilliz Cloud keeps its existing default of 1 and always sends it. Results record the value in task_config.db_case_config.level.

vectordbbench milvusautoindex \
  --uri http://localhost:19530 \
  --level 2 \
  <other options>

milvushnsw accepts --vector-data-type (YAML: vector_data_type) to store vectors as FLOAT32 (default) or BFLOAT16. The index type stays HNSW. With --skip-drop-old --skip-load, the run fails if the vector field of the collection has a different data type or dimension. Results record the value in task_config.db_case_config.vector_data_type.

vectordbbench milvushnsw \
  --uri http://localhost:19530 \
  --vector-data-type BFLOAT16 \
  <other options>

Use --nq (default: 1) to set the number of query vectors per concurrent search request. Milvus and Zilliz Cloud support batching for single-tenant vector performance cases, including filtered searches. This example reuses a collection containing the selected dataset and a matching HNSW index:

vectordbbench milvushnsw \
  --uri http://localhost:19530 --collection-name VDBBench \
  --case-type Performance768D1M --m 16 --ef-construction 128 --ef-search 128 \
  --k 10 --nq 2 \
  --skip-drop-old --skip-load

YAML uses nq: 2. The value is saved in task_config.case_config.nq in result JSON. Result views distinguish non-default NQ values in case names so different batch sizes stay separate. Omitting --num-concurrency uses the configured concurrency list. Each request contains exactly nq vectors, wrapping through the query dataset as needed.

• qps and conc_qps_list count successful query vectors per second. Successful request throughput is QPS divided by nq. Failed batches or responses missing a query's result list contribute zero.
• conc_latency_* measures the whole successful batch request.
• serial_latency_* and recall use NQ=1 and concurrency=1.

Other backends and streaming, full-text, cold-latency and multi-tenant cases reject nq > 1 before data loading.

Use --note or --note-file to preserve deployment, resource, client, network, and constraint context in each result JSON under task_config.db_config.note. The options are mutually exclusive. Prefer --note-file for structured or multiline context, and never include credentials, tokens, or sensitive connection details.

vectordbbench zillizautoindex \
  --note-file ./run-context.json \
  <other options>

$ vectordbbench pgvectorhnsw --help
Usage: vectordbbench pgvectorhnsw [OPTIONS]

Options:
  --config-file PATH              Read configuration from yaml file
  --drop-old / --skip-drop-old    Drop old or skip  [default: drop-old]
  --load / --skip-load            Load or skip  [default: load]
  --search-serial / --skip-search-serial
                                  Search serial or skip  [default: search-
                                  serial]
  --search-concurrent / --skip-search-concurrent
                                  Search concurrent or skip  [default: search-
                                  concurrent]
  --case-type [CapacityDim128|CapacityDim960|Performance768D100M|Performance768D10M|Performance768D1M|Performance768D10M1P|Performance768D1M1P|Performance768D10M99P|Performance768D1M99P|Performance1536D500K|Performance1536D5M|Performance1536D500K1P|Performance1536D5M1P|Performance1536D500K99P|Performance1536D5M99P|Performance1536D50K]
                                  Case type
  --db-label TEXT                 Db label, default: date in ISO format
                                  [default: 2024-05-20T20:26:31.113290]
  --note TEXT                     Run context stored with each result
                                  [default: ""]
  --note-file FILE                Read run context from a UTF-8 text file
  --dry-run                       Print just the configuration and exit
                                  without running the tasks
  --k INTEGER                     K value for number of nearest neighbors to
                                  search  [default: 100]
  --concurrency-duration INTEGER  Adjusts the duration in seconds of each
                                  concurrency search  [default: 30]
  --num-concurrency TEXT          Comma-separated list of concurrency values
                                  to test during concurrent search  [default:
                                  1,10,20]
  --concurrency-timeout INTEGER   Timeout (in seconds) to wait for a
                                  concurrency slot before failing. Set to a
                                  negative value to wait indefinitely.
                                  [default: 3600]
  --user-name TEXT                Db username  [required]
  --password TEXT                 Db password  [required]
  --host TEXT                     Db host  [required]
  --db-name TEXT                  Db name  [required]
  --maintenance-work-mem TEXT     Sets the maximum memory to be used for
                                  maintenance operations (index creation). Can
                                  be entered as string with unit like '64GB'
                                  or as an integer number of KB.This will set
                                  the parameters:
                                  max_parallel_maintenance_workers,
                                  max_parallel_workers &
                                  table(parallel_workers)
  --max-parallel-workers INTEGER  Sets the maximum number of parallel
                                  processes per maintenance operation (index
                                  creation)
  --m INTEGER                     hnsw m
  --ef-construction INTEGER       hnsw ef-construction
  --ef-search INTEGER             hnsw ef-search
  --quantization-type [none|bit|halfvec]
                                  quantization type for vectors (in index)
  --table-quantization-type [none|bit|halfvec]
                                  quantization type for vectors (in table). If
                                  equal to bit, the parameter
                                  quantization_type will be set to bit too.
  --reranking / --skip-reranking  Enable reranking for HNSW search for binary
                                  quantization
  --reranking-metric [L2|COSINE|IP|DP]
                                  Distance metric for reranking  [default:
                                  COSINE]
  --quantized-fetch-limit INTEGER
                                  Limit of fetching quantized vector ranked by
                                  distance for reranking                 --
                                  bound by ef_search
  --custom-case-name TEXT         Custom case name i.e. PerformanceCase1536D50K
  --custom-case-description TEXT  Custom name description
  --custom-case-load-timeout INTEGER
                                  Custom case load timeout [default: 36000]
  --custom-case-optimize-timeout INTEGER
                                  Custom case optimize timeout [default: 36000]
  --custom-dataset-name TEXT
                                  Dataset name i.e OpenAI
  --custom-dataset-dir TEXT       Dataset directory i.e. openai_medium_500k
  --custom-dataset-size INTEGER   Dataset size i.e. 500000
  --custom-dataset-dim INTEGER    Dataset dimension
  --custom-dataset-metric-type TEXT
                                  Dataset distance metric [default: COSINE]
  --custom-dataset-file-count INTEGER
                                  Dataset file count
  --custom-dataset-use-shuffled / --skip-custom-dataset-use-shuffled
                                  Use shuffled custom dataset or skip  [default: custom-dataset-
                                  use-shuffled]
  --custom-dataset-with-gt / --skip-custom-dataset-with-gt
                                  Custom dataset with ground truth or skip  [default: custom-dataset-
                                  with-gt]
  --help                          Show this message and exit.

Run polardb_pg from command line

polardb_pg supports HNSW with PQ, SQ4, SQ8, RaBitQ, and Graph Cache. This PostgreSQL client is separate from the existing MySQL-compatible PolarDB commands and reuses the pgvector dependency extra:

pip install 'vectordb-bench[pgvector]'

PolarDB-specific internal quantization and Graph Cache options require PolarDB vector extension 0.8.3.1 or later. Use --skip-graph-cache when Graph Cache is not configured on the server.

Example: Run hnsw index test

vectordbbench polardbpghnsw \
  --case-type Performance1024D1M \
  --db-label polardb-pg-rabitq8 \
  --user-name postgres --password '<password>' \
  --host localhost --port 5432 --db-name vectordb \
  --m 16 --ef-construction 256 --ef-search 200 \
  --quantization rabitq --quantization-nbits 8 \
  --iterative-scan off --graph-cache

To list the options for polardb_pg, execute vectordbbench polardbpghnsw --help. The following are some PolarDB-specific command-line options.

  --quantization [pq|sq4|sq8|rabitq]
  --pq-m INTEGER
  --train-samples INTEGER
  --quantization-nbits [1|4|8]
  --graph-cache / --skip-graph-cache
  --graph-cache-timeout INTEGER
  --iterative-scan [off|strict_order|relaxed_order]

Run VectorChord (vchordrq) from command line

VectorChord is a PostgreSQL extension for scalable vector similarity search using IVF + RaBitQ indexing. It is fully compatible with pgvector data types and provides faster queries and index builds.

vectordbbench vectorchordrq \
  --user-name postgres --password '<password>' \
  --host localhost --port 5432 --db-name vectordb \
  --case-type Performance1536D50K \
  --lists 1000 --probes 10 --epsilon 1.9 \
  --spherical-centroids --build-threads 8 \
  --max-parallel-workers 15

Key VectorChord-specific options:

Option	Description
--lists	Number of IVF lists for vchordrq index
--probes	Number of probes during search (default: 10)
--epsilon	Reranking precision factor, 0.0-4.0 (default: 1.9)
--residual-quantization	Enable residual quantization
--spherical-centroids	L2-normalize centroids (recommended for cosine/IP)
--build-threads	Number of threads for index building (1-255)
--degree-of-parallelism	Degree of parallelism for index build (1-256)
--max-parallel-workers	Sets max_parallel_workers & max_parallel_maintenance_workers
--max-scan-tuples	Max tuples to scan before stopping (-1 for unlimited)

Run awsopensearch from command line

vectordbbench awsopensearch --db-label awsopensearch \
--m 16 --ef-construction 256 \
--host search-vector-db-prod-h4f6m4of6x7yp2rz7gdmots7w4.us-west-2.es.amazonaws.com --port 443 \
--user vector --password '<password>' \
--case-type Performance1536D5M --number-of-indexing-clients 10  \
--skip-load --num-concurrency 75

To list the options for awsopensearch, execute vectordbbench awsopensearch --help

$ vectordbbench awsopensearch --help
Usage: vectordbbench awsopensearch [OPTIONS]

Options:
  # Sharding and Replication
  --number-of-shards INTEGER      Number of primary shards for the index
  --number-of-replicas INTEGER    Number of replica copies for each primary
                                  shard
  # Indexing Performance
  --index-thread-qty INTEGER      Thread count for native engine indexing
  --index-thread-qty-during-force-merge INTEGER
                                  Thread count during force merge operations
  --number-of-indexing-clients INTEGER
                                  Number of concurrent indexing clients
  # Index Management
  --number-of-segments INTEGER    Target number of segments after merging
  --refresh-interval TEXT         How often to make new data available for
                                  search
  --force-merge-enabled BOOLEAN   Whether to perform force merge operation
  --flush-threshold-size TEXT     Size threshold for flushing the transaction
                                  log
  --engine TEXT                   type of engine to use valid values [faiss, lucene, s3vector]
  # Memory Management
  --cb-threshold TEXT             k-NN Memory circuit breaker threshold

  --ondisk                        Ondisk mode with binary quantization(32x compression)
  --oversample-factor             Controls the degree of oversampling applied to minority classes in imbalanced datasets to improve model performance by balancing class distributions.(default 1.0)

  # Quantization Type
  --quantization-type TEXT        which type of quantization to use valid values [fp32, fp16, bq]
  --help                          Show this message and exit.

Run awsopensearch serverless from command line

OpenSearch Serverless (AOSS) is a serverless deployment option for Amazon OpenSearch Service. VDBBench supports AOSS with the --serverless flag, which uses AWS SigV4 authentication and automatically skips unsupported operations (cluster settings, force merge, manual refresh, warmup API).

Prerequisites:

• AWS credentials configured (via ~/.aws/credentials, environment variables, or IAM role)
• Serverless dependencies installed (included in the opensearch extra): pip install 'vectordb-bench[opensearch]'. This installs opensearch-py, boto3, and requests-aws4auth.
• IAM identity policy allowing aoss:APIAccessAll on the collection
• AOSS Data Access Policy granting index/collection permissions to the IAM principal

Example: Run performance test on OpenSearch Serverless

vectordbbench awsopensearch --db-label aoss --insert-batch-size 100 \
  --serverless --aws-region us-east-1 \
  --host <collection-id>.aoss.us-east-1.on.aws --port 443 \
  --case-type Performance768D1M \
  --m 16 --ef-construction 200 --ef-search 40 \
  --number-of-shards 8 --number-of-replicas 0 \
  --engine faiss --metric-type cosine \
  --num-concurrency 80,100,120

OpenSearch Serverless-specific options:

Option	Description
--serverless	Enable OpenSearch Serverless mode (uses AWS SigV4 auth)
--aws-region	AWS region for the AOSS collection (default: us-east-1)
--insert-batch-size	Number of vectors per Serverless bulk request (default: 100)

Notes:

• --user and --password are not needed for Serverless mode
• --engine is accepted but ignored internally (AOSS manages the engine)
• --force-merge-enabled, --refresh-interval, --flush-threshold-size, and --cb-threshold are ignored for Serverless
• Keep --insert-batch-size small enough for the Serverless bulk API request limits

Run Elastic Cloud from command line

Elastic Cloud supports multiple index types: HNSW, HNSW_INT8, HNSW_INT4, and HNSW_BBQ.

Example: Run HNSW index test

vectordbbench elasticcloudhnsw --db-label elastic-cloud-test \
--cloud-id <your-cloud-id> --password '<your-password>' \
--m 16 --ef-construction 100 --num-candidates 100 \
--case-type Performance768D1M --number-of-shards 1 \
--number-of-replicas 0 --refresh-interval 30s

Example: Run HNSW_INT8 index test

vectordbbench elasticcloudhnswint8 --db-label elastic-cloud-int8 \
--cloud-id <your-cloud-id> --password '<your-password>' \
--m 16 --ef-construction 200 --num-candidates 200 \
--case-type Performance1536D50K --element-type float

Example: Run HNSW_INT4 index test

vectordbbench elasticcloudhnswint4 --db-label elastic-cloud-int4 \
--cloud-id <your-cloud-id> --password '<your-password>' \
--m 16 --ef-construction 200 --num-candidates 200 \
--case-type Performance768D10M --use-rescore --oversample-ratio 2.0

Example: Run HNSW_BBQ index test

vectordbbench elasticcloudhnswbbq --db-label elastic-cloud-bbq \
--cloud-id <your-cloud-id> --password '<your-password>' \
--m 16 --ef-construction 200 --num-candidates 200 \
--case-type Performance1536D5M --use-routing --use-force-merge

Example: Run Label Filter Performance test

vectordbbench elasticcloudhnsw --db-label elastic-cloud-label-filter \
--cloud-id <your-cloud-id> --password '<your-password>' \
--case-type LabelFilterPerformanceCase \
--dataset-with-size-type "Medium OpenAI (1536dim, 500K)" \
--label-percentage 0.001 \
--m 16 --ef-construction 128 --num-candidates 100 \
--num-concurrency 1,5 --number-of-shards 1

To list all options for Elastic Cloud, execute vectordbbench elasticcloudhnsw --help. The following are Elastic Cloud-specific command-line options:

$ vectordbbench elasticcloudhnsw --help
Usage: vectordbbench elasticcloudhnsw [OPTIONS]

Options:
  # Connection
  --cloud-id TEXT                 Elastic Cloud ID  [required]
  --password TEXT                 Elastic Cloud password  [required]

  # HNSW Index Parameters
  --m INTEGER                     HNSW M parameter  [default: 16]
  --ef-construction INTEGER       HNSW efConstruction parameter  [default: 100]
  --num-candidates INTEGER        Number of candidates for search  [default: 100]
  --element-type [float|byte]     Element type for vectors (float: 4 bytes, byte: 1 byte)  [default: float]

  # Index Configuration
  --number-of-shards INTEGER      Number of shards  [default: 1]
  --number-of-replicas INTEGER    Number of replicas  [default: 0]
  --refresh-interval TEXT         Index refresh interval  [default: 30s]
  --merge-max-thread-count INTEGER
                                  Maximum thread count for merge  [default: 8]
  --use-force-merge BOOLEAN       Whether to use force merge  [default: True]
  --use-routing BOOLEAN           Whether to use routing  [default: False]
  --use-rescore BOOLEAN           Whether to use rescore  [default: False]
  --oversample-ratio FLOAT        Oversample ratio for rescore  [default: 2.0]

  # Common Options
  --case-type [CapacityDim128|CapacityDim960|Performance768D100M|...]
                                  Case type
  --db-label TEXT                 Db label, default: date in ISO format
  --k INTEGER                     K value for number of nearest neighbors to search  [default: 100]
  --num-concurrency TEXT          Comma-separated list of concurrency values  [default: 1,5,10,20,30,40,60,80]
  --help                          Show this message and exit.

Run OceanBase from command line

Execute tests for the index types: HNSW, HNSW_SQ, or HNSW_BQ.

vectordbbench oceanbasehnsw --host xxx --port xxx --user root@mysql_tenant --database test \
--m 16 --ef-construction 200 --case-type Performance1536D50K \
--index-type HNSW --ef-search 100

To list the options for oceanbase, execute vectordbbench oceanbasehnsw --help, The following are some OceanBase-specific command-line options.

$ vectordbbench oceanbasehnsw --help
Usage: vectordbbench oceanbasehnsw [OPTIONS]

Options:
  [...]
  --host TEXT                     OceanBase host
  --user TEXT                     OceanBase username  [required]
  --password TEXT                 OceanBase database password
  --database TEXT                 DataBase name  [required]
  --port INTEGER                  OceanBase port  [required]
  --m INTEGER                     hnsw m  [required]
  --ef-construction INTEGER       hnsw ef-construction  [required]
  --ef-search INTEGER             hnsw ef-search  [required]
  --index-type [HNSW|HNSW_SQ|HNSW_BQ]
                                  Type of index to use. Supported values:
                                  HNSW, HNSW_SQ, HNSW_BQ  [required]
  --help                          Show this message and exit.

Execute tests for the index types: IVF_FLAT, IVF_SQ8, or IVF_PQ.

vectordbbench oceanbaseivf --host xxx --port xxx --user root@mysql_tenant --database test \
--nlist 1000 --sample_per_nlist 256 --case-type Performance768D1M \
--index-type IVF_FLAT --ivf_nprobes 100

To list the options for oceanbase, execute vectordbbench oceanbaseivf --help, The following are some OceanBase-specific command-line options.

$ vectordbbench oceanbaseivf --help
Usage: vectordbbench oceanbaseivf [OPTIONS]

Options:
  [...]
  --host TEXT                     OceanBase host
  --user TEXT                     OceanBase username  [required]
  --password TEXT                 OceanBase database password
  --database TEXT                 DataBase name  [required]
  --port INTEGER                  OceanBase port  [required]
  --index-type [IVF_FLAT|IVF_SQ8|IVF_PQ]
                                  Type of index to use. Supported values:
                                  IVF_FLAT, IVF_SQ8, IVF_PQ  [required]
  --nlist INTEGER                 Number of cluster centers  [required]
  --sample_per_nlist INTEGER      The cluster centers are calculated by total
                                  sampling sample_per_nlist * nlist vectors
                                  [required]
  --ivf_nprobes TEXT              How many clustering centers to search during
                                  the query  [required]
  --m INTEGER                     The number of sub-vectors that each data
                                  vector is divided into during IVF-PQ
  --help                          Show this message and exit.                       Show this message and exit.

Run Hologres from command line

It is recommended to use the following code for installation.

pip install 'vectordb-bench[hologres]' 'psycopg[binary]' pgvector

Execute tests for the index types: HGraph.

vectordbbench hologreshgraph --host Hologres_Endpoint --port 80 --insert-batch-size 10000 \
--user ACCESS_ID --password ACCESS_KEY --database DATABASE_NAME \
--m 64 --ef-construction 400 --case-type Performance768D10M \
--index-type HGraph --ef-search 400 --k 10 --num-concurrency 1,60,70,75,80,90,95,100,105,110,115,120,125,130 \
--serial-cooldown 3

To list the options for Hologres, execute vectordbbench hologreshgraph --help, The following are some Hologres-specific command-line options.

$ vectordbbench hologreshgraph --help
Usage: vectordbbench hologreshgraph [OPTIONS]

Options:
  [...]
  --host TEXT                     Hologres host
  --user TEXT                     Hologres username  [required]
  --password TEXT                 Hologres database password
  --database TEXT                 Hologres database name  [required]
  --port INTEGER                  Hologres port  [required]
  --m INTEGER                     hnsw m  [required]
  --ef-construction INTEGER       hnsw ef-construction  [required]
  --ef-search INTEGER             hnsw ef-search  [required]
  --index-type [HGraph]           Type of index to use. Supported values:
                                  HGraph [required]
  --help                          Show this message and exit.

Run Zvec from command line

vectordbbench zvec --path Performance768D10M --db-label 16c64g-v0.1 \
    --case-type Performance768D10M --num-concurrency 12,14,16,18,20 \
    --quantize-type int8 --ef-search 118 --is-using-refiner

To list the options for zvec, execute vectordbbench zvec --help

  --path TEXT                     collection path  [required]
  --m INTEGER                     HNSW index parameter m.
  --ef-construction INTEGER       HNSW index parameter ef_construction
  --ef-search INTEGER             HNSW index parameter ef for search
  --quantize-type TEXT            HNSW index quantize type, fp16/int8
                                  supported
  --is-using-refiner              is using refiner, suitable for quantized
                                  index, recall `ef-search` results then
                                  refine with unquantized vector to `topk`
                                  results

Run Doris from command line

Doris supports ann index with type hnsw from version 4.0.x

vectordbbench doris --http-port=8030 --port=9030 --db-name=vector_test --case-type=Performance768D1M --insert-batch-size=1000000 --stream-load-rows-per-batch=500000

Using flag --session-var, if you want to test doris with some customized session variables. For example:

vectordbbench doris --http-port=8030 --port=9030 --db-name=vector_test --case-type=Performance768D1M --insert-batch-size=1000000 --stream-load-rows-per-batch=500000 --session-var enable_profile=True

Mote options:

--m INTEGER                     hnsw m
--ef-construction INTEGER       hnsw ef-construction
--username TEXT                 Username  [default: root; required]
--password TEXT                 Password  [default: ""]
--host TEXT                     Db host  [default: 127.0.0.1; required]
--port INTEGER                  Query Port  [default: 9030; required]
--http-port INTEGER             Http Port  [default: 8030; required]
--db-name TEXT                  Db name  [default: test; required]
--ssl / --no-ssl                Enable or disable SSL, for Doris Serverless
                                SSL must be enabled  [default: no-ssl]
--index-prop TEXT               Extra index PROPERTY as key=value
                                (repeatable)
--session-var TEXT              Session variable key=value applied to each
                                SQL session (repeatable)
--stream-load-rows-per-batch INTEGER
                                Rows per Doris stream-load request; when
                                omitted, the Doris client default is used
--no-index                      Create table without ANN index

Run Lindorm from command line

Lindorm supports index types: hnsw, ivfpq, or ivfbq.

Example: Run hnsw index test

vectordbbench lindormhnsw --case-type Performance768D10M --index-name <index_name> --k 10 \
--host <lindorm_host> --port <lindorm_port> --user <username> --password <password> --m 32 \
--ef-construction 400 --ef-search 150 --reorder-factor 2

Example: Run ivfpq index test

vectordbbench lindormivfpq --case-type Performance768D10M \
--index-name <index_name> --k 10 --host <lindorm_host> --port <lindorm_port> \
--user <username> --password <password> --lists <nlist> --probes <nprobe> \
--m 32 --ef-construction 500 --ef-search 200 --reorder-factor 2

Example: Run ivfbq index test

vectordbbench lindormivfbq --case-type Performance768D10M --index-name <index_name> \
--k 10 --host <index_name> --port <lindorm_port> \
--user <username> --password <password> --lists <nlist> --probes <nprobe> \
--exbits 2 --m 32 --ef-construction 500 --ef-search 200 --reorder-factor 2

To list the options for Lindorm, execute vectordbbench lindormhnsw --help, The following are some Lindorm-specific command-line options.

  --host TEXT                     host connection string  [required]
  --port INTEGER                  Db Port  [required]
  --user TEXT                     Db username  [required]
  --password TEXT                 Db password  [required]
  --index-name TEXT               Db index name  [required]
  --filter-type TEXT              post_filter|pre_filter|efficient_filter
  --number-of-regions INTEGER     Vector number of regions
  --m INTEGER                     hnsw m  [required]
  --ef-construction INTEGER       hnsw ef-construction  [required]
  --ef-search INTEGER             hnsw ef-search  [required]
  --reorder-factor INTEGER        reorder factor

Run ADBPG (Aliyun AnalyticDB for PostgreSQL) from command line

ADBPG Nova uses the fastann/Nova vector index engine with USING ann syntax.

Example: Run novamr index benchmark (BioASQ 1M, 1024-dim)

vectordbbench adbpgnova --case-type Performance1024D1M --k 10 \
--host <adbpg_host> --port 5432 --db-name postgres \
--user-name <username> --password <password> \
--algorithm novamr --hnsw-m 48 --ef-construction 600 \
--ef-search 130 --max-scan-points 5000 --quantize-rescore-amp 2.0

Example: Run from config file

vectordbbench adbpgnova --config-file adbpg_bioasq1m_novamr.yml

To list the options for ADBPG, execute vectordbbench adbpgnova --help. The following are some ADBPG-specific command-line options.

  --user-name TEXT                Db username  [required]
  --password TEXT                 Postgres database password  [$POSTGRES_PASSWORD]
  --host TEXT                     Db host  [required]
  --port INTEGER                  Postgres database port  [default: 5432]
  --db-name TEXT                  Db name  [required]
  --algorithm TEXT                algorithm  [default: novamr]
  --hnsw-m INTEGER                hnsw_m  [default: 16]
  --ef-construction INTEGER       ef_construction  [default: 200]
  --ef-search INTEGER             ef_search  [default: 100]
  --max-scan-points INTEGER       max scan points  [default: 2000]
  --quantize-rescore-amp FLOAT    fastann.quantize_rescore_amp  [default: 1.0]
  --nova-adaptive-gamma FLOAT     fastann.nova_adaptive_gamma  [default: 0.0]
  --auto-reduction/--no-auto-reduction  Index WITH auto_reduction=on  [default: False]

Run PolarDB from command line

PolarDB supports index types: faiss_hnsw_flat, faiss_hnsw_pq, and faiss_hnsw_sq.

Example: Run faiss_hnsw_flat benchmark

vectordbbench polardbhnswflat \
  --case-type Performance768D1M \
  --username <db_user> \
  --password '<db_password>' \
  --host <db_host> \
  --port 3306 \
  --m 16 \
  --ef-construction 256 \
  --ef-search 256 \
  --insert-workers 64 \
  --num-concurrency '10,20,40,60,80' \
  --concurrency-duration 60 \
  --task-label <task_label> \
  --db-label <db_label> \
  --skip-search-serial \
  --post-load-index

To list the options for PolarDB, execute vectordbbench polardbhnswflat --help. The following are some PolarDB-specific command-line options.

  --username TEXT                  Username  [required]
  --password TEXT                  Password
  --host TEXT                      Db host  [default: 127.0.0.1]
  --port INTEGER                   Db Port  [default: 3306]
  --database TEXT                  Database name  [default: vectordbbench]
  --m INTEGER                      M parameter (max_degree) in HNSW
  --ef-construction INTEGER        ef_construction parameter in HNSW
  --ef-search INTEGER              polar_vector_index_hnsw_ef_search session variable
  --insert-workers INTEGER         Number of concurrent threads for data insertion
  --post-load-index / --inline-index
                                   Create index after load or inline at table creation

Run VolcMySQL from command line

VolcMySQL is a MySQL-compatible service with a native VECTOR type and an HNSW vector index (created via SECONDARY_ENGINE_ATTRIBUTE). Optional quantization is configurable through --quant-algorithm (NONE, SQ, PQ) and --quant-type (16_bit, 8_bit, 4_bit, binary).

vectordbbench volcmysqlhnsw \
  --case-type Performance1536D50K \
  --username <db_user> \
  --password '<db_password>' \
  --host <db_host> \
  --port 3306 \
  --m 16 \
  --ef-construction 128 \
  --ef-search 100 \
  --quant-algorithm SQ \
  --quant-type 16_bit \
  --num-concurrency '10,20,40,60,80' \
  --concurrency-duration 30 \
  --task-label <task_label> \
  --db-label <db_label>

To list the options for VolcMySQL, execute vectordbbench volcmysqlhnsw --help. The following are some VolcMySQL-specific command-line options.

  --username TEXT                  Username  [required]
  --password TEXT                  Password  [required]
  --host TEXT                      Db host  [default: 127.0.0.1]
  --port INTEGER                   DB Port  [default: 3306]
  --m INTEGER                      M parameter in HNSW vector indexing
  --ef-search INTEGER              Session variable loose_hnsw_ef_search
  --ef-construction INTEGER        HNSW ef_construction
  --quant-algorithm [NONE|SQ|PQ]   Quantization algorithm
  --quant-type [16_bit|8_bit|4_bit|binary]
                                   Quantization type

Note: vectors are loaded and queried over the raw-binary VECTOR path by default; the client auto-probes server support and falls back to to_vector() text when unavailable. Set VDB_BINARY_VEC=0 to force the text path or 1 to force binary.

Run HouyiDB from command line

HouyiDB is a memory-first vector database with an HNSW index, a PostgreSQL-compatible pgwire endpoint and an HTTP benchmark API. Install the optional client dependencies with pip install vectordb-bench[houyidb]. The client loads and searches over pgwire by default and switches to the HTTP API for filtered cases or when --transport http is selected.

HouyiDB builds a cosine index, so the client rejects datasets whose distance metric is not COSINE. Because the server has no remote drop for the main index, a drop_old=True case is rejected when the index is not empty: start HouyiDB from an empty data directory (and empty WAL) before a fresh load. Before measuring, the client reads the effective hnsw_m, hnsw_ef_construction and distance_metric from /v1/index/status and aborts the run if they do not match the case configuration. With --transport pgwire, start the server with matching build parameters, e.g. --pgwire-hnsw-m 24 --pgwire-hnsw-efc 400; --no-verify-build-params is available only for older servers that do not report these fields.

vectordbbench houyidb \
  --case-type Performance1536D50K \
  --host <db_host> \
  --port 15432 \
  --http-port 17000 \
  --transport pgwire \
  --m 24 \
  --ef-construction 400 \
  --ef-search 256 \
  --num-concurrency '10,20,40,60,80' \
  --concurrency-duration 30 \
  --task-label <task_label> \
  --db-label <db_label>

To list the options for HouyiDB, execute vectordbbench houyidb --help. The following are some HouyiDB-specific command-line options.

  --host TEXT                      HouyiDB host  [required]
  --port INTEGER                   HouyiDB pgwire port  [default: 15432]
  --http-port INTEGER              Benchmark HTTP port  [default: 17000]
  --transport [pgwire|http]        Data plane  [default: pgwire]
  --use-tls / --no-use-tls         Use HTTPS for HTTP requests  [default: no-use-tls]
  --api-key TEXT                   Optional token: HTTP Bearer or pgwire password
  --timeout-sec INTEGER            Per-request timeout in seconds, including pgwire statement execution  [default: 600]
  --verify-build-params / --no-verify-build-params
                                   Verify the server's effective HNSW build parameters before measuring  [default: verify]
  --metric-type [COSINE]           Distance metric (HouyiDB builds a cosine index)  [default: COSINE]
  --m INTEGER                      HNSW m  [default: 24]
  --ef-construction INTEGER        HNSW ef_construction  [default: 400]
  --ef-search INTEGER              HNSW ef_search  [default: 256]

Filtered cases use the HTTP data plane because that is where HouyiDB accepts scalar labels and id filters.

Using a configuration file.

The vectordbbench command can optionally read some or all the options from a yaml formatted configuration file.

By default, configuration files are expected to be in vectordb_bench/config-files/, this can be overridden by setting the environment variable CONFIG_LOCAL_DIR or by passing the full path to the file.

The required format is:

commandname:
   parameter_name: parameter_value
   parameter_name: parameter_value

Example:

pgvectorhnsw:
  db_label: pgConfigTest
  user_name: vectordbbench
  password: vectordbbench
  db_name:  vectordbbench
  host: localhost
  m: 16
  ef_construction: 128
  ef_search: 128
milvushnsw:
  skip_search_serial: True
  case_type: Performance1536D50K
  uri: http://localhost:19530
  m: 16
  ef_construction: 128
  ef_search: 128
  drop_old: False
  load: False
elasticcloudhnsw:
  db_label: elastic-cloud-hnsw
  cloud_id: <your-cloud-id>
  password: <your-password>
  case_type: Performance768D1M
  m: 16
  ef_construction: 100
  num_candidates: 100
  number_of_shards: 1
  number_of_replicas: 0
  refresh_interval: 30s
  element_type: float

Notes:

• Options passed on the command line will override the configuration file*
• Parameter names use an _ not -
• For LabelFilterPerformanceCase and NewIntFilterPerformanceCase, you must specify dataset_with_size_type in addition to case_type

Using a batch configuration file.

The vectordbbench command can read a batch configuration file to run all the test cases in the yaml formatted configuration file.

By default, configuration files are expected to be in vectordb_bench/config-files/, this can be overridden by setting the environment variable CONFIG_LOCAL_DIR or by passing the full path to the file.

The required format is:

commandname:
  - parameter_name: parameter_value
    another_parameter_name: parameter_value

Example:

pgvectorhnsw:
  - db_label: pgConfigTest
    user_name: vectordbbench
    password: vectordbbench
    db_name:  vectordbbench
    host: localhost
    m: 16
    ef_construction: 128
    ef_search: 128
milvushnsw:
  - skip_search_serial: True
    case_type: Performance1536D50K
    uri: http://localhost:19530
    m: 16
    ef_construction: 128
    ef_search: 128
    drop_old: False
    load: False
elasticcloudhnsw:
  - db_label: elastic-cloud-hnsw-test-1
    cloud_id: <your-cloud-id>
    password: <your-password>
    case_type: Performance768D1M
    m: 16
    ef_construction: 100
    num_candidates: 100
  - db_label: elastic-cloud-label-filter-0.1
    cloud_id: <your-cloud-id>
    password: <your-password>
    case_type: LabelFilterPerformanceCase
    dataset_with_size_type: "Medium OpenAI (1536dim, 500K)"
    label_percentage: 0.001
    m: 16
    ef_construction: 128
    num_candidates: 100
    num_concurrency: "1,5"

Notes:

• Options can only be passed through configuration files
• Parameter names use an _ not -
• For LabelFilterPerformanceCase and NewIntFilterPerformanceCase, you must specify dataset_with_size_type in addition to case_type

How to use?

vectordbbench batchcli --batch-config-file <your-yaml-configuration-file>

Leaderboard

Introduction

To facilitate the presentation of test results and provide a comprehensive performance analysis report, we offer a leaderboard page. It allows us to choose from QPS, QP$, and latency metrics, and provides a comprehensive assessment of a system's performance based on the test results of various cases and a set of scoring mechanisms (to be introduced later). On this leaderboard, we can select the systems and models to be compared, and filter out cases we do not want to consider. Comprehensive scores are always ranked from best to worst, and the specific test results of each query will be presented in the list below.

Cloud Leaderboard

VectorDBBench now includes Cloud Leaderboard cases for production-oriented cloud vector database evaluation. These cases complement the original raw-performance leaderboard by measuring behaviors that matter for managed services:

• CloudInsertCase: insert throughput plus searchable and indexed readiness delays.
• CloudPayloadSearchCase: search performance when responses return IDs only, scalar metadata, or vectors.
• CloudMultiTenantSearchCase: tenant-routed search for SaaS-shaped workloads.
• CloudColdLatencyCase: cold and warm serial latency for first-query and cache-sensitive serving paths.

The May 2026 release note explains why the Cloud Leaderboard was added, what changed, which systems were tested this round, and how to run each new case: docs/release/2026-05-cloud-leaderboard.md.

Scoring Rules

1. For each case, select a base value and score each system based on relative values.

   • For QPS and QP$, we use the highest value as the reference, denoted as base_QPS or base_QP$, and the score of each system is (QPS/base_QPS) * 100 or (QP$/base_QP$) * 100.
   • For Latency, we use the lowest value as the reference, that is, base_Latency, and the score of each system is (base_Latency + 10ms)/(Latency + 10ms) * 100.

   We want to give equal weight to different cases, and not let a case with high absolute result values become the sole reason for the overall scoring. Therefore, when scoring different systems in each case, we need to use relative values.

   Also, for Latency, we add 10ms to the numerator and denominator to ensure that if every system performs particularly well in a case, its advantage will not be infinitely magnified when latency tends to 0.

2. For systems that fail or timeout in a particular case, we will give them a score based on a value worse than the worst result by a factor of two. For example, in QPS or QP$, it would be half the lowest value. For Latency, it would be twice the maximum value.

3. For each system, we will take the geometric mean of its scores in all cases as its comprehensive score for a particular metric.

Build on your own

Install requirements

pip install -e '.[test]'

pip install -e '.[pinecone]'

Run test server

python -m vectordb_bench

OR:

init_bench

OR:

If you are using dev container, create the following dataset directory first:

# Mount local ~/vectordb_bench/dataset to contain's /tmp/vectordb_bench/dataset.
# If you are not comfortable with the path name, feel free to change it in devcontainer.json
mkdir -p ~/vectordb_bench/dataset

After reopen the repository in container, run python -m vectordb_bench in the container's bash.

Check coding styles

make lint

To fix the coding styles automatically

make format

How does it work?

Result Page

This is the main page of VDBBench, which displays the standard benchmark results we provide. Additionally, results of all tests performed by users themselves will also be shown here. We also offer the ability to select and compare results from multiple tests simultaneously.

The standard benchmark results displayed here include all 15 cases that we currently support for 6 of our clients (Milvus, Zilliz Cloud, Elastic Search, Qdrant Cloud, Weaviate Cloud and PgVector). However, as some systems may not be able to complete all the tests successfully due to issues like Out of Memory (OOM) or timeouts, not all clients are included in every case.

All standard benchmark results are generated by a client running on an 8 core, 32 GB host, which is located in the same region as the server being tested. The client host is equipped with an Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz processor. Also all the servers for the open-source systems tested in our benchmarks run on hosts with the same type of processor.

Run Test Page

1. Initially, you select the systems to be tested - multiple selections are allowed. Once selected, corresponding forms will pop up to gather necessary information for using the chosen databases. The db_label is used to differentiate different instances of the same system. We recommend filling in the host size or instance type here (as we do in our standard results).
2. The next step is to select the test cases you want to perform. You can select multiple cases at once, and a form to collect corresponding parameters will appear.
3. Finally, you'll need to provide a task label to distinguish different test results. Using the same label for different tests will result in the previous results being overwritten. Now we can only run one task at the same time.

Module

Code Structure

Client

Our client module is designed with flexibility and extensibility in mind, aiming to integrate APIs from different systems seamlessly. As of now, it supports Milvus, Zilliz Cloud, Elastic Search, Pinecone, Qdrant Cloud, Weaviate Cloud, PgVector, VectorChord, Redis, Chroma, CockroachDB, VolcMySQL, etc. Stay tuned for more options, as we are consistently working on extending our reach to other systems.

Benchmark Cases

We've developed lots of comprehensive benchmark cases to test vector databases' various capabilities, each designed to give you a different piece of the puzzle. These cases are categorized into several main types:

Capacity Case

• Large Dim: Tests the database's loading capacity by inserting large-dimension vectors (GIST 100K vectors, 960 dimensions) until fully loaded. The final number of inserted vectors is reported.
• Small Dim: Similar to the Large Dim case but uses small-dimension vectors (SIFT 500K vectors, 128 dimensions).

Search Performance Case

• XLarge Dataset: Measures search performance with a massive dataset (LAION 100M vectors, 768 dimensions) at varying parallel levels. The results include index building time, recall, latency, and maximum QPS.
• Large Dataset: Similar to the XLarge Dataset case, but uses a slightly smaller dataset (10M-1024dim, 10M-768dim, 5M-1536dim).
• Medium Dataset: A case using a medium dataset (1M-1024dim, 1M-768dim, 500K-1536dim).
• Small Dataset: For development (100K-768dim, 50K-1536dim).

LAION-100M Large-TopK

Performance768D100M selects its query and ground-truth files from --k:

Requested K	Query file	Ground-truth file	Queries	GT width
1..1,000	test.parquet	neighbors.parquet	1,000	1,000
1,001..100,000	test_nq200.parquet	neighbors_top100k_nq200.parquet	200	100,000
100,001..1,000,000	test_nq200.parquet	neighbors_top1m_nq200.parquet	200	1,000,000

K must be positive, and LAION-100M rejects values above 1,000,000. Integer-filter runs use the smallest published GT width that covers K:

Integer filter rate	Published GT widths	Maximum K
50%, 60%, 70%, 80%, 90%, 95%, 98%, 99%	100K, 1M	1M
99.5%	100K, 500K	500K
99.8%	100K, 200K	200K
99.9%	100K	100K

K up to 1,000 keeps the original 1,000-query filtered artifacts; larger K uses test_nq200.parquet. Other integer filter rates and label-filter runs above K=1,000 are rejected before database initialization. VDBBench validates query IDs, row counts, and GT width before issuing a search.

vectordbbench milvusautoindex --uri http://localhost:19530 --case-type NewIntFilterPerformanceCase --dataset-with-size-type "Large LAION (768dim, 100M)" --filter-rate 0.99 --k 1000000

Wide GT remains in Parquet/Arrow form and is opened inside the serial-search subprocess one query row at a time. Results include primary recall@K, recall_at for the available cutoffs among 100, 1K, 10K, 100K, and 1M, plus serial and concurrent p50/p95/p99 latency. Concurrent throughput continues to use the configured fixed-duration phase.

For Milvus and Zilliz Cloud performance runs with K above 16,384, VDBBench automatically creates new collections with query_mode=large_topk before creating the vector index. Reused collections are validated and rejected when that property is missing or incompatible. The target database, selected mode, and requested K are written to the run log. Self-hosted Milvus must be 2.6.14 or later, the release that introduced the query_mode=large_topk collection property; earlier servers accept the property without honoring it. Other backends must already permit the requested K; VDBBench forwards K unchanged and does not alter their collection properties.

Performance Response Payloads

Every vector search performance case can be configured for either an IDs-only response or a response that also includes each result vector. IDs only remains the default. Run the scenarios separately from the CLI:

vectordbbench milvusautoindex --uri http://localhost:19530 --case-type Performance768D100M --k 1000000 --payload-profile ids_only
vectordbbench milvusautoindex --uri http://localhost:19530 --case-type Performance768D100M --k 1000000 --payload-profile vector

The frontend can select one or both scenarios for Milvus and Zilliz Cloud. Each scenario produces independent P99 latency, QPS, and recall metrics. qps remains the highest observed QPS among the configured concurrency levels; VDBBench does not discover a backend concurrency limit.

Filtering Search Performance Case

• Int-Filter Cases: Evaluates search performance with int-based filter expression (e.g. "id >= 2,000").
• Label-Filter Cases: Evaluates search performance with label-based filter expressions (e.g., "color == 'red'"). The test includes randomly generated labels to simulate real-world filtering scenarios.

Full Text Search Performance Case

• FullTextSearchPerformance: Measures BM25-style text retrieval over raw text documents. The case inserts documents, runs the backend optimization or index-readiness step, then measures recall, latency, and QPS for text queries.
• Datasets: The initial FTS benchmark uses MS MARCO and HotpotQA in small, medium, and large corpus sizes.
• Ground truth: Recall, MRR, and NDCG are computed against positive semantic relevance labels from ir_datasets.
• Payload profiles: FTS supports IDs-only responses and text payload responses so users can compare pure retrieval throughput against response-size overhead.

Streaming Cases

• Insertion-Under-Load Case: Evaluates search performance while maintaining a constant insertion workload. VDBBench applies a steady stream of insert requests at a fixed rate to simulate real-world scenarios where search operations must perform reliably under continuous data ingestion.

Each case provides an in-depth examination of a vector database's abilities, providing you a comprehensive view of the database's performance.

Dataset selection for search performance

Registered datasets

Use the Performance case with --dataset-name to select a registered dataset. The catalog includes the existing VectorDBBench datasets, all 24 VIBE HDF5 datasets, and the VDBBench multimodal Parquet datasets at 1M, 10M, and 100M corpus vectors. Source and storage-format handling are internal to the registration. See the Hugging Face datasets release note for the complete new catalog, constraints, and data-loading behavior.

Custom local datasets

Through the /custom page, users can customize their own performance case using local datasets. After saving, the corresponding case can be selected from the /run_test page to perform the test.

We have strict requirements for the data set format, please follow them.

• Folder Path - The path to the folder containing all the files. Please ensure that all files in the folder are in the Parquet format.

  • Vectors data files: The file must be named train.parquet and should have two columns: id as an incrementing int and emb as an array of float32.
  • Query test vectors: The file must be named test.parquet and should have two columns: id as an incrementing int and emb as an array of float32.
    • We recommend limiting the number of test query vectors, like 1,000. When conducting concurrent query tests, Vdbbench creates a large number of processes. To minimize additional communication overhead during testing, we prepare a complete set of test queries for each process, allowing them to run independently. However, this means that as the number of concurrent processes increases, the number of copied query vectors also increases significantly, which can place substantial pressure on memory resources.
  • Ground truth file: The file must be named neighbors.parquet and should have two columns: id corresponding to query vectors and neighbors_id as an array of int.

• Train File Count - If the vector file is too large, you can consider splitting it into multiple files. The naming format for the split files should be train-[index]-of-[file_count].parquet. For example, train-01-of-10.parquet represents the second file (0-indexed) among 10 split files.

• Use Shuffled Data - If you check this option, the vector data files need to be modified. VDBBench will load the data labeled with shuffle. For example, use shuffle_train.parquet instead of train.parquet and shuffle_train-04-of-10.parquet instead of train-04-of-10.parquet. The id column in the shuffled data can be in any order.

Goals

Our goals of this benchmark are:

Reproducibility & Usability

One of the primary goals of VDBBench is to enable users to reproduce benchmark results swiftly and easily, or to test their customized scenarios. We believe that lowering the barriers to entry for conducting these tests will enhance the community's understanding and improvement of vector databases. We aim to creat