I'm working with a few hundred million embeddings with dimensions 512 and 768, and I need a vector database that can perform similarity searches quickly and accurately. I'm looking to avoid using a GPU server, preferring a setup with just CPU and SSD/NVMe storage. I've found that pg_vector struggles to handle the load—I tried HNSW, but it just freezes up. My current setup has about 150GB of RAM, and while I could scale it a bit, I'd prefer not to expand to terabytes. My ideal solution would utilize NVMe storage efficiently and have intelligent indexing options. I've tested Qdrant and Milvus, but both have failed during data upload. It seems like most databases are built for smaller payloads that fit in RAM, and while FAISS is an option, it's primarily GPU-focused and requires managing persistence myself. I'm currently using ps_vector with IVFFlat, but the search quality is quite poor. Is there a better solution out there?
2 Answers
Have you checked out AWS OpenSearch? They have a plugin for embeddings and a KNN algorithm that might fit your needs. It could handle a large data set pretty well if you configure it right!
This seems like more of an architecture issue. Are all your vectors stored in a single table column? If that's the case, you might want to segment your dataset. Pre-filtering the embeddings could speed up the similarity search significantly.
My entire database is already a specific data segment. It’s not feasible to split it further because it's focused on a particular feature that necessitates the similarity search.