I need to scrape name, latitude & longitude of list of restaurants from https://food.grab.com/ph/en/restaurants and save them to a csv. This is 'load more' type website: 8 more restaurants will be added in the search results for every lick on 'load more' button.
When you inspect element, you have a body tag which contains a div followed by a list of script tags. The below code starts from second script tag.
<script async="" id="__NEXT_PAGE__/restaurants" src="/_next/static/~k5VPZk5KBtLKJOP7fLbR/pages/restaurants.js"></script>
<script async="" id="__NEXT_PAGE__/_app" src="/_next/static/~k5VPZk5KBtLKJOP7fLbR/pages/_app.js"></script>
<script async="" src="/_next/static/chunks/146.5cc25d7743407960833f.js"></script>
<script src="/_next/static/runtime/webpack-562f11f96da3918c3d37.js" async=""></script>
<script src="/_next/static/chunks/styles.4fc2149595e9293ccd52.js" async=""></script>
For restaurants loaded after clicking 'load more', the latitude/longitude information seems to be stored in external javascript file stored in script tag with id="NEXT_PAGE/restaurants" (or the other external js files right below).
I am new to Javascript. I went through the first js file but unable to figure out how to get the required info. I couldn't find any variables with restaurant names or locations mentioned explicitly. Also skimmed through other js files but can't find anything. Not sure if I should look for script tags in the header instead. Can someone answer this?
It has nothing to do with JavaScript files.
JavaScript receives relevant information by sending http requests to the server
These requests are visible
You must repeat the same requests with Python
in devtools go to network tab, check all request. when you click on load more btn a request sent to https://portal.grab.com/foodweb/v2/search
see response in response section is:
{"searchResult":{"searchID":"910af096fe304265a0f3d4a3e8b5e2d8","totalCount":515,"searchMerchants":[{"id":"2-C2MFREUZN6WKLN","address":{"name":"Wai Ying Fastfood - Binondo [Available for LONG-DISTANCE DELIVERY]"},"latlng":{"latitude":14.602068853150598,"longitude":120.97598854230432},"estimatedDeliveryTime":30,"merchantBrief":{"description":"Prices are all VAT inclusive. Prices may also vary or be subject to change by the merchant.","cuisine":["Asian","Noodles","Quick Bites"],"photoHref":"https://d1sag4ddilekf6.azureedge.net/compressed/merchants/2-C2MFREUZN6WKLN/hero/12a517f18aed4fc7ac9d10ef754193fa_1617944281957721088.jpeg","isIntegrated":true,"openHours":{"open":true,"displayedHours":"08:00-20:00","sun":"08:00-20:00","mon":"08:00-20:00","tue":"08:00-20:00","wed":"08:00-20:00","thu":"08:00-20:00","fri":"08:00-20:00","sat":"08:00-....
get restaurant_id from this response, each restaurant_id stored with id key
then when you click on one restaurants a request sent to https://portal.grab.com/foodweb/v2/merchants/2-CYKCVZNZJTDFLE?latlng=14.5995,120.9842
where 2-CYKCVZNZJTDFLE is restaurant_id
and response is :
{
"merchant": {
"ID": "2-CYKCVZNZJTDFLE",
"latlng": {
"latitude": 14.6094796,
"longitude": 120.992802
},
"name": "Yellow Cab Pizza - Espana [Available for LONG-DISTANCE DELIVERY]",
....
"businessType": "FOOD",
},
"promotions": null
}
you can get latitude and longitude from here
make the https://portal.grab.com/foodweb/v2/search request and https://portal.grab.com/foodweb/v2/merchants/{id} with python, but insure all of the http headers must be same with http headers that in the chrome dev tools