I am working on a website, where I need to search through an index with millions of documents, and be able to display them as pages of 500 documents. With pagination we need to support the jump to a random page.
I looked at Elastic Search documentation and it says we should use search_after for deep pagination. Documentation till 7.2 also says
search_after is not a solution to jump freely to a random page but rather to scroll many queries in parallel. It is very similar to the scroll API but unlike it, the search_after parameter is stateless, it is always resolved against the latest version of the searcher. For this reason the sort order may change during a walk depending on the updates and deletes of your index.
Deep, random pagination is a pretty classic problem. scroll is compounded by having to keep the state around and to manage that. And search_after isn’t a silver bullet, I don’t think that after 7.2 search_after changed that much, what did change is that 7.10 added Point In Time (PIT) in X-Pack (not Open Distro!) that made search_after more sane (search_after suffers badly in data sets with moving data).
What is your pagination sort context? Can you integrate that into your query? Have you considered aggregations to paginate?