0
506
ViewsWhy does phantomJS 2.1.1 render this page as background color with no content, but foxshot 1.2.0 renders it correctly?
I like to keep track of Massachusetts lottery results (make sure the results are really random). I can bring up the page of results in any browser and copy-paste the lines of text, and run the text through a filter written in php or gawk to extract the numbers for each date. But I thought it would be nicer to automate the process using phantomJS to get the dates and numbers directly from the DOM.
I haven't been able to make this work because the elements containing the results do not appear to be in the DOM accessible through phantomJS. I don't know if I'm doing something wrong (I probably am) or if it's a problem with phantomJS (which I know is no longer maintained). But I also tried using the example of rendering a URL to a .png image directly from the Page Loading example in the quickstart, and the image came out as just background color with no text, so I'm thinking it's a problem with phantomJS. Or does that example need some extra code to wait for all the included scripts to complete?
Here is the example javascript with the URL in question and image filename hardwired:
var page = require('webpage').create();
page.open('https://www.masslottery.com/tools/past-results/mega-millions?start_date=2021-09-03&end_date=2021-09-28', function(status) {
console.log("Status: " + status);
if(status === "success") {
page.render('example.png');
}
phantom.exit();
});
Running phantomjs on the above javascript produces this file named example.png as follows:
But running foxshot I see the following:
$ foxshot 'https://www.masslottery.com/tools/past-results/mega-millions?start_date=2021-09-03&end_date=2021-09-28'
no valid dimensions provided, defaulting to 1024x768
loading site...
1024x768
And it produces a file named screenshot_1024x768.png as follows:
Of course foxshot doesn't provide a way to access the DOM or otherwise access the text displayed in the image. Any suggestions?
1 answers
Answer question0
It goes like this with puppeteer
//npm i puppeteer
const puppeteer = require('puppeteer')
!(async () => {
//open browser
const browser = await puppeteer.launch()
//open browser page
const page = await browser.newPage()
//open url
await page.goto('https://www.masslottery.com/tools/past-results/mega-millions?start_date=2021-09-03&end_date=2021-09-28')
//wait for table to appear
await page.waitForSelector('.multi-col-stacking-table')
//iterate over table rows on page
const data = await page.$$eval('.multi-col-stacking-table tbody tr', d =>
d.map(d => ({
//format date as YYYY-MM-DD
date: new Date(d.querySelector('.past-results-row-draw-date').innerText).toJSON().slice(0, 10),
//collect winning numbers
winningNumber: Array.from(d.querySelectorAll('.winning-number-ball-circle, .winning-number-ball-circle-no-border')).map(d => +d.innerText),
//get multiplier as number
multiplier: +d.querySelector('.winning-number-ball-multiplier').innerText.slice(0, -1),
//parse jackpot into number - remove all non-digits and multiply by 1M
jackpot: +d.querySelector('.jackpot').innerText.replace(/[^\d]/g, '') * 1e6
}))
)
console.log(data)
//optional screenshot
await page.screenshot({path: 'screenshot.png'})
await browser.close()
})()
//output
/*
[
{
date: '2021-09-27',
winningNumber: [ 18, 30, 43, 68, 69, 22 ],
multiplier: 4,
jackpot: 22000000
},
{
date: '2021-09-23',
winningNumber: [ 17, 21, 27, 43, 56, 15 ],
multiplier: 3,
jackpot: 20000000
},
...
]
*/