Let's suppose we have a fairly large database of products for e.g. 50 K Mobile Phone. We store these data in Elastic Search. Now I have a Product Listing page for mobile phone, there I list all mobile 10 at a time using pagination with their basic details. That page also have a filter section like Brand, Price Range, RAM, Avg. Rating, Release date and lot more specifications.
Now When I fetch for mobile of Samsung company and 6 GB ram, I fire a elastic query and got results and their total count. So here bring the count query become complex, Total count depends on filter and this type of query increases load on system.
I want a system which will compute count for filters once and save it somewhere so I don't need to calculate count for same filter again, thus reducing complexity overhead for same filters again and again. Let me know with your knowledge How I can solve this problem or how should I maintain my system?
Any reference or article would also be appreciated.
Elastic search provides its own caching techniques and you can you use those setting for caching a specific query
GET /my_index/_search?request_cache=true
{
"size": 0,
"aggs": {
"popular_colors": {
"terms": {
"field": "colors"
}
}
}
}
Here is a link for more details. Mind it that you have this much hardware configuration for caching of elasticsearch. https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-request-cache.html#_enabling_and_disabling_caching_per_request.
Also if you have multiple data nodes, and there are few indexes on whom queries and aggregation are performed frequently, and you have different hardware configs of different nodes, then you should look into elasticsearch hot and cold nodes concept, and put you imp indexes in hot nodes, instead of caching everything
https://www.elastic.co/blog/hot-warm-architecture-in-elasticsearch-5-x
could we add a cache layer and cache the most used filter count. and increment and decrement the count whenever the inventory is updated. this way we can avoid computation. which wont be possible in case of elasticsearch as it will invalidate the cache when the inventory is updated. this way we can avoid too many hits on elasticsearch.
Could you please update above post with the sample of the complex count query you are using? It would be helpful to identify issue you are facing. Thanks!