I'm using scrape-it in my node.js scraping tool (for identifying proper keyword usage) but being identified as a bot by some websites and not getting any content. Is there a way to configure a known user agent header for the GET request to bypass the block?
You can set the headers, including User-agent, by passing an options object to scrape-it:
scrapeIt({
url: "http://example.com"
, headers: { "User-agent": "known-user-agent-of-choice" }
},
{
// some scrapeHTML options ...
})
.then(
// some code ...
);