Exploring the Best Vector Database
George Chalkiadakis
25 Jul 2024
Exploring the Best Vector Database for My-Buddy.AI: A Deep Dive into Our Vectordb Research
Hello, my name is George Chalkiadakis and In this article, I’d like to share insights from our recent research and experimentation with various vector databases (vectordb) over the past few months. Our goal was to identify the best vectordb solution to support our needs, focusing on performance, scalability, and client satisfaction.
The Search for the Ideal Vectordb
Our journey began with an extensive evaluation of several leading vector databases, including Qdrant, PGVECTOR, Milvus, Chroma, Relyt, Weaviate, and Pinecone. Given the complexity of our application and the specific requirements of our customers, finding the right fit was crucial.
Embedding Model and Vector Characteristics
We utilize the OpenAI text-embedding model large 3, which has the following characteristics:
- Embedding Vector Size: Text-embedding-3-large
- Embedding Vector Length: 3072 dimensions
- Size per Embedding Vector: Each dimension is represented by a 4-byte (32-bit) floating-point number.
The size of one vector embedding can be calculated as:
Size per Embedding = Vector Length × Size per Dimension
Size per Embedding = 3072 × 4 bytes = 12288 bytes = 12 KB
Evaluating Client Satisfaction
Our customers frequently use complex queries, such as filtering by cost (cheapest, most expensive, etc.). Through our evaluations, we identified that only Weaviate and Pinecone met the highest levels of client satisfaction. This led us to focus our efforts on these two options, aiming to establish a strong long-term partnership with the best fit.
Deep Dive into Weaviate
Initial Impressions and Challenges
We began our in-depth evaluation with Weaviate. While their cloud solution offered a basic interface, it lacked transparency regarding scaling and operational insights. To better understand its capabilities, we set up a local cluster on Azure.
Documentation and Configuration Hurdles
We encountered several challenges due to incomplete documentation, particularly in configuring critical environment values and setting up effective metrics and monitoring. Here are some of the specific issues we faced:
- Configuration of Critical ENV Values: We needed guidance on properly configuring crucial environment values such as ASYNC_INDEXING, GOGC, QUERY_MAXIMUM_RESULTS, TRACK_VECTOR_DIMENSIONS, and REINDEX_VECTOR_DIMENSIONS_AT_STARTUP.
- Metrics and Monitoring: Implementing effective metrics and monitoring required understanding which metrics to track and how to manage them using tools like Grafana.
- Tuning Guidance: We sought detailed explanations on tuning Weaviate’s configuration options, akin to managing a Postgres database.
- Advanced Configuration and Scaling: We explored options for vertical and horizontal scaling and fine-tuning search result quality to balance speed, quality, and consistency.
- System-Level Tuning: We considered any operating system-level configurations that could enhance performance.
Despite these challenges, Weaviate demonstrated great potential, and we continue to discuss a potential partnership with them.
Exploring Pinecone
Quick Integration and User Experience
Our evaluation of Pinecone revealed a different experience. The integration process was smooth, and its multi-tenant support was implemented quickly. Pinecone’s user interface is intuitive and provides a serverless architecture, which means we only pay for what we use. However, we did notice some performance issues that Weaviate did not exhibit, which raised some concerns.
Improving Similarity Search with Dotproduct Metric
We used the dotproduct metric for vector similarity search, which significantly improved our search results. This enhancement in search capability made Pinecone a strong contender in our evaluation process.
Conclusion and Next Steps
Both Weaviate and Pinecone have proven to be strong candidates for our needs. We are in the final stages of evaluation and negotiations to determine the best fit for My-Buddy.AI.
Each has its strengths and areas for improvement, and we are carefully considering these factors in our decision-making process.
Stay tuned for our final decision, where I will share which vector database we choose and the reasons behind our choice.
Thank you for reading, and I look forward to sharing more updates soon.