Empresas
Empleos
  • Sobre nosotros
  • Soluciones
    • Publicación de vacantes
      Publica tu vacante y recibe candidatos calificados en 48h.
    • Evaluación de candidatos
      500+ pruebas técnicas y psicológicas, más anti-fraude.
    • Headhunting
      Búsqueda ejecutiva a la medida de principio a fin.
    • Nómina + EOR
      Dispersión de nómina y EOR en más de 15 países de LATAM.
  • Precios
  • Empleos

0

280
Vistas
Axios & Cheerio - Visit a list of URLS contained in JSON file, and add extracted data into that object?

I'm attempting to build a scraper with Axios & Cheerio.

I'm needing to do this in two parts. The first part I've already completed, which visits a site and writes me a JSON file that contains a bunch of objects like so:

[
    {
        "count": 0,
        "title": "Result One",
        "url": "http://www.resultone.com"
    },
    {
        "count": 1,
        "title": "Result Two",
        "url": "http://www.resulttwo.com"
    },
    {
        "count": 2,
        "title": "Result Three",
        "url": "http://www.resultone.com"
    },
]

Now for the second part, I need to read this JSON file, visit each URL listed, extract some data from the page and add it to the current JSON object in the original file.

Once the JSON file is created, I can run the following:

let json_url_list = require('./' + outputFile);

// Loop over the URLS
for(i=0; i<json_url_list.length; ++i) {
    let url = json_url_list[i].url;        
    
    // Run a function here to visit the URL and extract data
    getNewData(url)
}

Along with a function like so:

// Create new function to visit each of the URLs captured.
const getNewData = async(url) => {
    try {
        const response = await axios.get(url)
        const $ = cheerio.load(response.data);
        
        // Get the data here (using page title for example)
        const title = $('title').text();
        
        // TODO: Add the new data above to the original JSON object in the file we're reading from


        return false;

    } catch (error) {
        console.error(error)
    }
}

But, this is where I run out of ideas on how to make this work... Could anyone point me in the right direction?

Thank you!

about 4 years ago · Juan Pablo Isaza
1 Respuestas
Responde la pregunta

0

Try this.

https://stackblitz.com/edit/js-tmmnqj?file=index.js

// Create new function to visit each of the URLs captured.
const getNewData = async (url) => {
  try {
    const response = await axios.get(url);
    const $ = cheerio.load(response.data);

    // Get the data here (using page title for example)
    const title = $('title').text();
    console.log(title);

    // TODO: Add the new data above to the original JSON object in the file we're reading from

    return false;
  } catch (error) {
    console.error(error);
  }
};



(async () => {
  // Loop over the URLS
  for (const jsonData of json_url_list) {
    let url = jsonData.url;

    // Run a function here to visit the URL and extract data
    const data = await getNewData(url);
  }
})();

about 4 years ago · Juan Pablo Isaza Denunciar
Responde la pregunta
Encuentra empleos remotos

¡Descubre la nueva forma de encontrar empleo!

Top de empleos
Top categorías de empleo
Empresas
Publicar vacante Precios Comercial
Legal
Términos y condiciones Política de privacidad
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Recomiéndame algunas ofertas
Necesito ayuda