I have a Python web scraper using the HTMLParser module. The website it scraps is http://consulta.siiau.udg.mx/wco/sspseca.consulta_oferta?ciclop=202120&cup=D&mostrarp=100000&ordenp=2
Now I need to do the same but web browser based using javascript, so I tried fetching the raw HTML using axios but I keep getting 'Access to XMLHttpRequest has been blocked by CORS policy'.
What I have tried is
axios.post('http://consulta.siiau.udg.mx/wco/sspseca.consulta_oferta', {
ciclop: '202120',
cup: 'D',
mostrarp: 10000,
ordennp: 2,
})
.then((response) => {
console.log(response)
})
And
axios.get('http://consulta.siiau.udg.mx/wco/sspseca.consulta_oferta?ciclop=202120&cup=D&mostrarp=100000&ordenp=2',
{ crossdomain: true }
)
.then((response) => {
console.log(response)
})
I am aware that in javascript they normally use a headless browser like the one inside Puppeteer, but since this project is a website I can't use Node.js modules.
Right now the solution I implemented is to have a server running a Flask API that handles the html fetching and then sends it back to the client for processing, but it would be a relieve for my server performance if the client could do this on his side.
In short, you can't use the fetch API or XMLHTTPRequest to access resources that aren't allowed by the browser's cross-origin policy.
For security reasons, browsers restrict HTTP requests initiated from scripts. A web application can only request resources from the same origin the application was loaded from unless the response from other origins includes the right CORS headers.
You can use a headless browser (puppeteer) to access a web page, for example using:
await page.goto('https://example.com');
This is equivalent to putting http://example.com in the browser bar and pressing ENTER. Any scripts on that page are subject to the same cross-origin restrictions as before. However, telling the browser to visit http://example.com isn't a cross-origin request in itself.