Question

0

316

Vistas

Nodejs web scraping- How to get the contents of all args

I am trying to create a html web scraper, and email the contents of what was found.

I am working with a music site, and am passing the names of artists through the command line, which should then access the website, searching for the html tags of song entries on the charts. The email sent should be the list of songs by that particular artist or any song they're featured on , out of the first 25 songs on the site.

My code however only emails and outputs when one argument is passed, but when more than one are entered, the contents aren't displayed in the email:

Heres a snippet of my code/jquery loop:

var request = require('request');
var cheerio = require('cheerio');
var nodemailer = require('nodemailer');
var process = require('process');
var artists = process.argv.slice(2).toString();
var creds= require('./credentials.json');


var transporter = nodemailer.createTransport(creds);

request('http://www.popvortex.com/music/charts/top-rap-songs.php', function (error, response, html) {
    if (!error && response.statusCode == 200) {
        var $ = cheerio.load(html);
        
        var data="";
        //var artist= $(this).children('em.artist').text();
        //var song = $(this).children('cite.title').text();
            
        $('p.title-artist').each(function(i, element) {
            var artist= $(this).children('em.artist').text();
            var song = $(this).children('cite.title').text();
            
            if(i<25 && (artist.includes(artists) || song.includes(artists))) {
                var artist= $(this).children('em.artist').text();
                var song = $(this).children('cite.title').text();
                data+=artist + ' : ' + song;
                console.log(artist + ' : '+ song);
            }
            
        })
                var mailOptions = {
                    from: creds.user,
                    to : 'jguffer4234@gmail.com',
                    subject: 'Your artists are: '  + artists.toString(),
                    text: data,
                        
                }

command line input/output

Email still sends, when more than one artist is entered but the songs they are included in are not.

Latto email

Latto Jack email, lacking body with all of their songs

Can anyone help? And help with my formatting of the email(line by line) would also be helpful.

about 4 years ago · Juan Pablo Isaza

2 Respuestas

Responde la pregunta

0

Responde la pregunta

Encuentra empleos remotos

Answer 1 · 2022-06-30T12:29:00.320Z

I believe that the issue is in this line:

if(i<25 && (artist.includes(artists) || song.includes(artists))) {

I think the use of the method string's includes is incorrect.

Instead try with:

if(i<25 && (artists.includes(artist) || artists.includes(song))) {

Based on the documentation, I created an example to show how it works:

let list = ["a", "b", "c"];
console.log("Case 1: ", "a".includes(list)); // false
console.log("Case 2: ", list.includes("a")); // true

While I'm not sure if this will solve the whole issue you are having, I'm hoping it will help you out at least on that.

Answer 2 · 2022-06-30T12:29:00.329Z

Ok. As promised, here is an alternative solution.

Since the request package has being deprecated, I'm using axios instead.

Please notices, that I broke the problem into small pieces.

This allowed me to provide names that describe what is being done, it make it easier to read, plus allows me to test each part as I code.

In the package.json, I did:

{
  "name": "experiment1",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "start": "node index.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "axios": "^0.27.2",
    "cheerio": "^1.0.0-rc.10"
  }
}

The command line I'm using for example is:

~/Experiment1$ npm start "Jack Harlow" "Big Energy"

The results are:

> experiment1@1.0.0 start
> node index.js "Jack Harlow" "Big Energy"

data: {
  'Jack Harlow': 'First Class',
  'Latto & Mariah Carey': 'Big Energy (Remix) [feat. DJ Khaled]',
  Latto: 'Big Energy',
  'Lil Nas X & Jack Harlow': 'INDUSTRY BABY'
}

Note: Please notices that duplicates aren't included.

Here is the code:

const axios = require('axios').default;
const url = 'http://www.popvortex.com/music/charts/top-rap-songs.php';
axios.get(url)
  .then((response) => {
    let data = findArtistAndSong(response.data);    
    console.log("data:", data);
  })
  .catch((error) => {
    console.error(error);
  });

const cheerio = require('cheerio');
const LIST_LIMIT = 25;
function findArtistAndSong(html){
  const $ = cheerio.load(html);
  const data = {};
  $('p.title-artist').each(function(i, element) {
    let artist= $(this).children('em.artist').text();
    let song = $(this).children('cite.title').text();            
    if (i < LIST_LIMIT && doesArtistOrSongMatch(artist, song)) {        
      data[artist] = song;
    }      
  });
  return data;
}

function doesArtistOrSongMatch(artist, song){
  return isMatch(artist) || isMatch(song);
}

const arguments = process.argv.slice(2);
const isMatch = (search) => {
  return arguments.some(item => search.includes(item));
};

Please let me know if you have any questions.

0

316

Nodejs web scraping- How to get the contents of all args

2 Respuestas

0

0

Encuentra empleos remotos

Andres GPT