I am currently writing a simple program that grabs the name of a song from my discord bot, finds the video and passes it to a function to convert to mp3. My problem is that puppeteer dosen't click on the video and instead just returns the search page link.
Here is my code to grab the link and pass it through download:
async function findSongName(stringWithName){
let stringName = stringWithName.replace(commands.play, '')
const link = 'https://www.youtube.com';
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto(link)
await page.type('ytd-searchbox#search.style-scope.ytd-masthead', stringName);
page.keyboard.press('Enter');
await page.click('yt-interaction#extended');
console.log(page.url())
await browser.close()
}
It sounds like you want to get the title and URL of the top result for a YT search. For starters, you don't need to start at the YT homepage. Just navigate to https://www.youtube.com/results?search_query=${yourQuery} to speed things up and reduce complexity.
Next, if you view the page source of /results, there's a large (~1 MB) global data structure called ytInitialData that has all of the relevant results in it (along with a lot of other irrelevant stuff, admittedly). Theoretically, you could grab the page with Axios, parse out ytInitialData with Cheerio, grab your data using plain array/object JS and skip Puppeteer entirely.
Of course, using the YT search API is the most reliable and proper way.
Since you're using Puppeteer, though, the data can be pulled out of the "#items a#video-title" elements as follows:
const puppeteer = require("puppeteer");
const searchYT = async (page, searchQuery) => {
const encodedQuery = encodeURIComponent(searchQuery);
const url = `https://www.youtube.com/results?search_query=${encodedQuery}`;
await page.goto(url);
const sel = "a#video-title";
await page.waitForSelector(sel);
return page.$$eval(sel, els =>
els.map(e => ({
title: e.textContent.trim(),
href: e.href,
}))
);
};
let browser;
(async () => {
browser = await puppeteer.launch({headless: true});
const [page] = await browser.pages();
await page.setRequestInterception(true);
page.on("request", req => {
req.resourceType() === "image" ? req.abort() : req.continue();
});
const results = await searchYT(page, "stack overflow");
console.log(results);
})()
.catch(err => console.error(err))
.finally(() => browser?.close())
;
Output (for the search term "stack overflow"):
[
{
title: 'Stack Overflow is full of idiots.',
href: 'https://www.youtube.com/watch?v=I_ZK0t9-llo'
},
{
title: "How To Use Stack Overflow (no, ForrestKnight, it's not full of idiots)",
href: 'https://www.youtube.com/watch?v=sMIslcynm0Q'
},
{
title: 'How to use Stack Overflow as a Beginner ?',
href: 'https://www.youtube.com/watch?v=Vt-Wf7d0CFo'
},
{
title: 'How Microsoft Uses Stack Overflow for Teams',
href: 'https://www.youtube.com/watch?v=mhh0aK6yJgA'
},
// ...
]
Since you only want the first result, then it's here, but if you want more than the initial batch, either work through ytInitialData as described above or scroll the page down with Puppeteer.
Now that you have a video URL that you want to make into an mp3, I'd recommend youtube-dl. There are Node wrappers you can install to access its API easily, such as node-youtube-dl which was the first result when I searched and I've never used before.