[Bug] Potential BM25 Implementation Discrepancy: Missing (k1 + 1) Multiplier

[ep9io]

Is this a new bug?

• I believe this is a new bug
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I noticed that the _encode_single_document method may be missing the standard (k1 + 1) multiplier in the numerator of the BM25 term frequency calculation. I wanted to raise this for clarification, as it differs from the standard BM25 formula (https://en.wikipedia.org/wiki/Okapi_BM25).

Pinecone Text Source: https://github.com/pinecone-io/pinecone-text/blob/main/pinecone_text/sparse/bm25_encoder.py#L120

Current Implementation

    def _encode_single_document(self, text: str) -> SparseVector:
        indices, doc_tf = self._tf(text)
        tf = np.array(doc_tf)
        tf_sum = sum(tf)

        tf_normed = tf / (
            self.k1 * (1.0 - self.b + self.b * (tf_sum / self.avgdl)) + tf
        )
        return {
            "indices": indices,
            "values": tf_normed.tolist(),
        }

Expected Implementation

    def _encode_single_document(self, text: str) -> SparseVector:
        indices, doc_tf = self._tf(text)
        tf = np.array(doc_tf)
        tf_sum = sum(tf)

        tf_normed = (tf * (self.k1 + 1.0)) / (
            self.k1 * (1.0 - self.b + self.b * (tf_sum / self.avgdl)) + tf
        )
        return {
            "indices": indices,
            "values": tf_normed.tolist(),
        }

Expected Behavior

Unless k1=0, the expected behaviour is to factor in k1+1 in the nominator (tf_normed).

Steps To Reproduce

This is a formula question so no steps to reproduce.

Relevant log output

Environment

- **OS**:
- **Language version**:
- **Pinecone client version**:

Additional Context

No response

[akelch11]

Hi @ep9io ,

Pinecone-text's Bm25 implementation outputs a normalized score in between 0 - 1, and does this by omitting the (k1 + 1) factor in the scoring formula, such that the frequency term $\frac{f(q_i,D)}{f(q_i, D)+k_1(1-b+b \frac{|D|}{avgdl}}$ is bounded in the limit by 1. This forum post has a blurb explaining more: https://community.pinecone.io/t/normailized-score-when-using-hybrid-search-which-is-only-supported-now-with-dot-product-instead-of-cosine-similarity/3051/2

More broadly, pinecone-text is no longer being maintained. Since its release we've released greater support for sparse vectors and hybrid search / cascading retrieval, including our own pinecone-sparse-english-v0 embedding model. https://www.pinecone.io/blog/cascading-retrieval/

[ep9io]

Thanks for your response and for sharing that.

There really is no standard way of normalising BM25 since each method has its own trade offs. I have not seen the removal of k1 plus 1 as an approach before. I suppose it could reduce computation, but I would be concerned about losing the saturation effect. I have not tested it myself, so I am not sure if or how much the quality drops. I have used post hoc normalisation like Min Max scaling before, and from the link you shared, it looks like that is the approach being used by dividing by the maximum possible score.

Thanks also for letting me know it is not being maintained. I did not know that, so it is good to be aware. I read the link you shared on sparse english v0 and it sounds interesting. I have only used BGE M3 before. I will give v0 a try one of these days. 😀