I am new with scraping so I watched many videos and tried to understand some of it. So I am scraping answers from Stckoverflow by browsing every page that have the same keywords. It worked with the screenshots but what I really need is data in an array or json file. this is my function:
const getAnswerFromQuestion = async (website, query, page) => {
console.log("Website", website);
await page.goto(website, ["load", "domcontentloaded", "networkidle0"]);
const popUp = (await page.$x("//button[@title='Dismiss']"))[0];
if (popUp) await popUp.click();
const generateId = shortid.generate();
const screenshotPath = `${generateId}.png`;
const acceptedAnswer = await page.$(".accepted-answer");
const { data } = await axios.get(website);
const $ = cheerio.load(data);
const acceptedAnswer2 = $('.accepted-answer');
const answers = [];
acceptedAnswer2.find('.answercell.post-layout--right').each((i, element) => {
const $element = $(element);
const answer = {};
answer.answerer = $element.find('.user-details > a').text();
answer.content = $element.find('.s-prose.js-post-body').text();
answers.push(answer);
console.log(answers);
});
if (!acceptedAnswer) return;
const pathToSaveScreenshot = `/answers/${query}/`;
await createDir(pathToSaveScreenshot);
await acceptedAnswer.screenshot({
path: `.${pathToSaveScreenshot}${screenshotPath}`,
});
};
I am trying to get the username of the answer and the content of the answer.
It works but it adds the last answer data from the last browse of the site in the answers array.