I need to download some 300 files from a direct download link. When the link is opened directly in the browser, an automatic pdf download gets triggered. The file gets downloaded and the browser doesn't go anywhere. The links are as follows:
www.link.com/store/item/123
In the link, the 123 part would be changed on every loop.
I was thinking of using puppeteer (with goto), but I guess since visiting the link automatically triggers the download of the pdf and doesnt actually go to the page, it fails.
This is what I tried, but its not working at all:
const links = ['123', '456'];
(async () => {
const browser = await puppeteer.launch({
headless: false //preferably would run with true
});
links.forEach( async link => {
const page = await browser.newPage();
await page.goto(
linkBeginning + link
);
await browser.close();
})
})();
I searched around, but I could not really find this specific case, all other cases are more focused on the user side or have the target file in the actual link (like xx/store/doc.pdf). Not quite sure if that makes a difference though. I would just need a script that will get me the pdf files for a one time run.
If anyone has a solution in php/python that would work as well, as this is just a one off thing.
edit: ended up doing it in html
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Document</title>
<script src="sku.js"></script>
<script>
const linkStart = 'https://www.sols-europe.com/gb/pdfpublication/pdf/product/sku/';
sku.forEach(element => {
document.write('<a target = "_blank" class="click" href="' + linkStart + element.id +'">'+ element.id +'</a></br>')
});
</script>
</head>
<body></body>
</html>
<script>
const clickInterval = setInterval(function () {
const el = document.querySelector('.click:not(.clicked)');
if(el){
el.classList.add('clicked');
el.click()
} else {
clearInterval(clickInterval);
}
}, 2000);
</script>
Your placement for browser.close() inside loop isn't a good thing.
So i moved it outside forEach and change it to page.close() instead.
const links = ['123', '456']
const linkBeginning = 'https://www.link.com/store/item/'
;(async () => {
const browser = await puppeteer.launch({
headless: false //preferably would run with true
})
links.forEach( async link => {
const page = await browser.newPage()
const session = await page.target().createCDPSession()
await session.send('Page.setDownloadBehavior', {
behavior: 'allow',
downloadPath: './pdf/'
})
await page.goto(linkBeginning + link)
await page.close() // Don't use browser.close() inside loop
})
await browser.close() // Use here instead
})()
You don't need puppeteer to do this, and you can achieve it fairly easily in NodeJS:
import http from "https";
import fs from "fs";
(async () => {
const skus = ["00548", "03575"];
const filesPromiseArray = skus.map(
(sku) =>
new Promise((resolve, reject) => {
const file = fs.createWriteStream(`${sku}.pdf`);
const request = http.get(`https://www.sols-europe.com/gb/pdfpublication/pdf/product/sku/${sku}`, (response) =>
response.pipe(file)
);
file.on("finish", resolve);
file.on("error", reject);
})
);
try {
await Promise.all(filesPromiseArray);
} catch {
console.log("There was an error downloading one of the files");
}
})();
skus, we are using .map() to transform them into an array of requests..map() we're creating a Promise which will be successful (resolve) when the file finishes downloading, or unsuccessful (reject) if the download errors.await all of the requests that we have just created. If one of them fails it will log.If you are using CommonJS ("type":"commonjs", in your package.json), replace the two imports with:
const http = require('https');
const fs = require('fs');