This seems to be possible in Scrapy Scrapy and respect of robots.txt but is there a simple way to do this in Puppeteer?
I haven't found an easy way to build in "respect the Robots" into Puppeteer commands.
I don't believe puppeteer has anything built in but you can use puppeteer to access robots.txt and then use any of a number of npm modules to parse robots.txt to see if you're allowed to get any particular URL. For example, here's how you might use robots-txt-parser:
const robotsParser = require('robots-txt-parser')
const robots = robotsParser()
// Now inside an async function
// (or not if using a version of Node.js that supports top-level await)
await robots.useRobotsFor('https://example.com/')
if (await robots.canCrawl(urlToVisit)) {
// Do stuff with puppeteer here to visit the URL
} else {
// Inform the user that sadly crawling that URL is forbidden
}