Empresas
Empregos
  • Sobre nós
  • Soluções
    • Publicação de vagas
      Publique sua vaga e receba candidatos qualificados em 48h.
    • Avaliações de candidatos
      Mais de 500 testes técnicos e psicológicos, mais anti-fraude.
    • Headhunting
      Busca executiva personalizada do início ao fim.
    • Folha de Pagamento + EOR
      Dispersão de folha e EOR em mais de 15 países da LATAM.
  • Preços
  • Empregos

0

289
Visualizações
Axios & Cheerio - Visit a list of URLS contained in JSON file, and add extracted data into that object?

I'm attempting to build a scraper with Axios & Cheerio.

I'm needing to do this in two parts. The first part I've already completed, which visits a site and writes me a JSON file that contains a bunch of objects like so:

[
    {
        "count": 0,
        "title": "Result One",
        "url": "http://www.resultone.com"
    },
    {
        "count": 1,
        "title": "Result Two",
        "url": "http://www.resulttwo.com"
    },
    {
        "count": 2,
        "title": "Result Three",
        "url": "http://www.resultone.com"
    },
]

Now for the second part, I need to read this JSON file, visit each URL listed, extract some data from the page and add it to the current JSON object in the original file.

Once the JSON file is created, I can run the following:

let json_url_list = require('./' + outputFile);

// Loop over the URLS
for(i=0; i<json_url_list.length; ++i) {
    let url = json_url_list[i].url;        
    
    // Run a function here to visit the URL and extract data
    getNewData(url)
}

Along with a function like so:

// Create new function to visit each of the URLs captured.
const getNewData = async(url) => {
    try {
        const response = await axios.get(url)
        const $ = cheerio.load(response.data);
        
        // Get the data here (using page title for example)
        const title = $('title').text();
        
        // TODO: Add the new data above to the original JSON object in the file we're reading from


        return false;

    } catch (error) {
        console.error(error)
    }
}

But, this is where I run out of ideas on how to make this work... Could anyone point me in the right direction?

Thank you!

about 4 years ago · Juan Pablo Isaza
1 Respostas
Responde à pergunta

0

Try this.

https://stackblitz.com/edit/js-tmmnqj?file=index.js

// Create new function to visit each of the URLs captured.
const getNewData = async (url) => {
  try {
    const response = await axios.get(url);
    const $ = cheerio.load(response.data);

    // Get the data here (using page title for example)
    const title = $('title').text();
    console.log(title);

    // TODO: Add the new data above to the original JSON object in the file we're reading from

    return false;
  } catch (error) {
    console.error(error);
  }
};



(async () => {
  // Loop over the URLS
  for (const jsonData of json_url_list) {
    let url = jsonData.url;

    // Run a function here to visit the URL and extract data
    const data = await getNewData(url);
  }
})();

about 4 years ago · Juan Pablo Isaza Relatório
Responde à pergunta
Encontrar trabalhos remotos

Descubra a nova forma de encontrar um emprego!

melhores empregos
Principais categorias de trabalho
Empresas
Postar vaga Preços Comercial
Jurídico
Termos e Condições Política de privacidade
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Recomende algumas ofertas para mim
Preciso de ajuda