I know there are a couple of posts asking the same questions. I have tried them all but after a couple of sleepless nights, I had to ask the question and I will be very appreciative if someone can help me with the problem.
So I am trying to get the reviews from a yelp restaurants page using Python. The reviews are generated dynamically using javascript. I tried every possible way with scrapy, requests-html, scrapy_splash, beautifulSoup. Nothing worked. I realized that even when using Splash by itself through the browser using localhost:8050, I still get the same problem. I tried to disable the private mode and increase the wait time but that did not fix the problem. I also tried it on both firefox and chromium, still the same.
Here is the page I am testing https://www.yelp.com/biz/brendas-french-soul-food-san-francisco-5
You will notice that the divs containing the reviews are missing.
Any help will be much appreciated.
The reviews you are searching for are loaded by js requesting this endpoint
https://www.yelp.com/biz/lJAGnYzku5zSaLnQ_T6_GQ/review_feed?rl=en&q=&sort_by=relevance_desc&start={offset value}
Which shows 10 comments and whole load of other data.