Reindexing Produces Different Result On The Same Query Vector

Hello All,

I am using Opendistro ES for knn-based vector search. I insert vectors as per below settings and mappings:

settings = {
    "index": {
	    "knn": True,
	    "knn.algo_param.ef_search": 1024,
	    "knn.algo_param.ef_construction": 1024,
	    "knn.algo_param.m": 32,
	    "max_inner_result_window": 6,
    }
}

mappings = {
    "properties": {
	    "doc_id": {"type": "keyword"},
	    "doc_text": {"type": "text"},
	    "feature_vector": {"type": "knn_vector", "dimension": 512},
    }
}

For searching my documents, I am using approximate KNN. What I have observed that, everytime I reindex my data (upsert data into the index), I get different score for some documents in the result list.
My search query:

request = {
    "query": {
	    "bool": {
		    "must": [
			    {
				    "knn": {
					    "feature_vector": {
						    "vector": query_vector.tolist(),
						    "k": n_results,
					    }
				    }
			    }
		    ],
		    "should": [
			    {
				    "match": {
					    "doc_text": {
						    "query": "entity1",
						    "operator": "and",
					    }
				    }
			    },
			    {
				    "match": {
					    "doc_text": {
						    "query": "entity2",
						    "operator": "and",
					    }
				    }
			    }
		    ],
	    }
    },
}

For example,
Scenario 1:

  • I indexed data into a new Index.
  • Used KNN Vector search.
  • In the result list, say doc1 gets the score 0.59

Scenario 2:

  • I reindex the data.
  • Used KNN Vector search.
  • In the result list, doc1 gets a different score (0.58).

In between these two scenarios, only reindexing step differs, rest all are same. The consequence is that I get different precision/recall score.

Can anyone explain the change in the observation?

I have tried using Brute-force KNN search as well. In this case, getting the non-deterministic result is less frequent.

docker version used: amazon/opendistro-for-elasticsearch:1.13.1
language: Python
library used: elasticsearch = “^7.8.1”

If you need any details, please let me know.

Thanks in advance,
Abhash Sinha

Hi @abhash

I have not seen this before yet. I will investigate and get back to you here.

Jack

Hi @abhash

Could you run the query without the additional should match sections? So just:

request = {
    "query": {
	    "knn": {
			"feature_vector": {
				"vector": query_vector.tolist(),
			        "k": n_results,
		    }
	    }
    },
}

I want to check if the basic knn score is changing after reindex or if it could be related to some of the other query parameters.

Hi @jmazane,

Thanks for your reply. I tried your suggestion and here is one comparison in the scores of a document from result list before and after reindexing:
score before reindexing: 0.46198926
score after reindexing: 0.4640275516326531

Thanks @abhash

That is pretty strange. One more thing, could you provide the sample query vector and the index vector (corresponding to the score) before and after reindex? I want to check if the data has been changed at all during reindex.

Also, could you run the script score as well before and after reindex and provide the scores returned? I want to check if the scores are the same before and after reindex for ANN and score script.

Hi @jmazane,

I am using vector of size 512, I am not sure if that is a good idea to post the whole vector here. But I can assure you that I have compared the vectors before and after reindexing and they have not changed.

The score of the same document before and after reindexing while using script KNN:
before: 0.4600089755102041
after: 0.4600089755102041

In this case, we see the deviation in result set less frequently as mentioned earlier.

Hi @abhash

So far, I am unable to reproduce with random data.

Could you provide the query vector and the indexed document vector for the scores mentioned above? You can put them in a gist.

Jack

Hi @jmazane
Here is a link to Gist.
It contains document vector before and after reindexing in different files. There is one file for query vector and a file for my observations.