0
700
ViewsAccessing javascript variable in Puppeteer
I'm trying to render a page using puppeteer, extract the name of a dynamic javascript variable on the page, and return the variable as an object. I'm using puppeteer and node.js to do this.
I'm able to get the name of the javascript variable that is being generated on page but anytime I try to print it out to the console I am getting a "undefined" error. I've spent the better part of the morning trying to figure this out but I'm stumped. Here's the code below.
const puppeteer= require('puppeteer');
var url = 'https://www.website.com';
async function run () {
try {
const browser = await puppeteer.launch({
headless: true,
args: ['--no-sandbox']
});
const page = await browser.newPage();
await page.goto(url);
const flashVariableName = await page.evaluate( () => {
var start = document.documentElement.innerHTML.indexOf('var flashvariable_') + 'var '.length;
var tempDoc = document.documentElement.innerHTML.substr(start);
var end = tempDoc.indexOf(' =');
return tempDoc.substr(0, end);
});
console.log(await page.evaluate( (flashVariableName) => flashVariableName ));
await browser.close();
} catch(e){
console.log("Error Occurred", e)
}
}
run();
flashVariableName is returning the right name for the javascript variable on the page but how can I actually get access to the varialble?
3 answers
Answer question0
You have two issues.
This is the first one:
console.log(await page.evaluate( (flashVariableName) => flashVariableName ));
// ^^^^^^^^^^^^^^^^^
// |
// +--> this is undefined
If you look at the documentation for Page#evaluate
What you're missing is the ...args bit which supplies parameters to the pageFunction. It should have been this instead:
console.log(await page.evaluate( (flashVariableName) => flashVariableName, flashVariableName ));
// ^^^^^^^^^^^^^^^^^
But that is not enough because flashVariableName is just a string at this point. It is the name of a variable but not the variable itself. You still need to "evaluate" it:
console.log(await page.evaluate( (varName) => window[varName], flashVariableName ));
However that only works if the variable you are trying to read is available in the global scope (i.e. window in your case) AND it can be serialised (e.g. if it refers to a function or a DOM element that won't be possible)
0
The return value from page.evaluate() method must be serializable. Other ways it will return undefined. Right now you are just returning a String. You can see the definition here https://pptr.dev/#?product=Puppeteer&version=v5.5.0&show=api-pageevaluatepagefunction-args
let theVariableName = tempDoc.substr(0,end);
return {theVariable: theVariableName}
After this you can access your variable in your flashVariableName variable.
0
I have found the best easy way. See the bellow example
const flatFormData = await page.evaluate("__platform_data__");
console.log(flatFormData);