feat: qa vector db

[nsantacruz]

This pull request introduces vector search and indexing capabilities to the project, supporting both pgvector (PostgreSQL extension) and Qdrant (a vector database). It adds new API endpoints for vector search and indexing, provides helper functions for interacting with both backends, and includes infrastructure changes to support Qdrant in both Docker Compose and Helm deployments. Additionally, scripts for initializing and benchmarking the vector databases are included.

API and Backend Functionality:

• Added new API endpoints api/v3/vector-search/<backend> and api/v3/vector-index/<backend> in sefaria/urls_shared.py and implemented corresponding VectorSearch and VectorIndex view classes to handle search and indexing requests for both pgvector and Qdrant backends. ([[1]](https://github.com/Sefaria/Sefaria-Project/pull/3393/files#diff-8f355aa80bd3ead68ed63deaf8856b8ae9f3d71a6234d67dfe6cbcdb984e19e0R79-R80), [[2]](https://github.com/Sefaria/Sefaria-Project/pull/3393/files#diff-7fb412808754eb12ee1b172c34c44eb9ce58b3bba43d62ffcb986ad7917588a8R71-R105))
• Implemented helper functions in sefaria/helper/vector_search.py for searching and indexing embeddings in both pgvector and Qdrant, including connection management and payload formatting. ([sefaria/helper/vector_search.pyR1-R63](https://github.com/Sefaria/Sefaria-Project/pull/3393/files#diff-b9c1eba71d65731c93bd67ebbd7ade5fbc3bf3308798892dc229c37e2b9e2494R1-R63))

Infrastructure and Configuration:

• Updated Docker Compose to use the pgvector/pgvector:pg17 image for PostgreSQL and added a Qdrant service with persistent storage. ([[1]](https://github.com/Sefaria/Sefaria-Project/pull/3393/files#diff-e45e45baeda1c1e73482975a664062aa56f20c03dd9d64a827aba57775bed0d3L37-R37), [[2]](https://github.com/Sefaria/Sefaria-Project/pull/3393/files#diff-e45e45baeda1c1e73482975a664062aa56f20c03dd9d64a827aba57775bed0d3R47-R58))
• Added a Helm chart template (helm-chart/sefaria/templates/qdrant.yaml) and related configuration (helm-chart/sefaria/values.yaml) to support deploying Qdrant in Kubernetes environments. ([[1]](https://github.com/Sefaria/Sefaria-Project/pull/3393/files#diff-90f91964db768066b22da07d30ec6e487cca2d09b558055050749d34df67b7c1R1-R78), [[2]](https://github.com/Sefaria/Sefaria-Project/pull/3393/files#diff-f456df0f77726a2691872d2aafe48fcc6416f34f51ba3652644a9e56ed096125R93-R104))
• Introduced new settings in sefaria/settings.py for Qdrant host, port, collection name, and vector dimensions. ([sefaria/settings.pyR313-R317](https://github.com/Sefaria/Sefaria-Project/pull/3393/files#diff-527bb6e93208e6b52293bab4c87f3ef11938f72d3b22b04a36df5c077c1016e5R313-R317))

Utility Scripts:

• Added scripts/init_vector_dbs.py to initialize the pgvector table and Qdrant collection for storing embeddings. ([scripts/init_vector_dbs.pyR1-R56](https://github.com/Sefaria/Sefaria-Project/pull/3393/files#diff-59d33840dfa99e488ff4febdd47d7585024df3a2e4f9536943b06118d34bf35dR1-R56))
• Added scripts/benchmark_vector_dbs.py to benchmark latency between pgvector and Qdrant using pre-embedded query vectors via the new API endpoints. ([scripts/benchmark_vector_dbs.pyR1-R82](https://github.com/Sefaria/Sefaria-Project/pull/3393/files#diff-6cf414a305699141dbdce8532c122f844ae6a90bd86684969c297b8fc027153eR1-R82))