Trying to execute some JavaScript via a scraping API, with the goal of scrolling through dynamically loading pages, then parsing the full response. I tested the script in the console at an Auction site (with single page ticked) and it worked as expected.

//Keep scrolling until the previous doc height is equal to the new height
 const infinite_scroll = async () => {
    let previous = document.body.scrollHeight;
    while (true){
        window.scrollTo(0, document.body.scrollHeight);
        // Fully Wait Until Page has loaded, then proceed
        await new Promise(r => setTimeout(r, 2000)); 
        let new_height = document.body.scrollHeight;
        console.log('%s Previous Height: ',previous)
        console.log(' ')
        console.log('%s New Height: ',new_height)
        if (new_height == previous){
            break
        }
        else{
            previous = new_height
        }
    }
};
infinite_scroll();

In the network tab it shows 11 successful XHR calls are made from '372610?pn=6&ipp=10' to '372610?pn=16&ipp=10' ( 'pn', representing the equivalent to a page number and 'ipp', items per page)

However when I try to pass this script to Scrapfly, the API I'm using, It makes only 3 XHR calls and the last one times out, So instead of getting the entire page I get an extra 20 items.

Using the API Demo (would need to sign up for a free account!) It can be reproduced by adding the script, setting the Rendering time to 10000 ms, and adding the following headers;

emailcta : pagehits%3D1%26userdismissed%3Dfalse

cookie: UseInfiniteScroll=true

Looking at the details for the last XHR call, it times out and has a null response. The request headers for the call is identical to the two previous successful ones so I'm not exactly sure what the issue is.

JS_Rendering time doesn't seem to affect this or the wait time within the infinite scroll function. All the docs say is:

"We provide a way to inject your javascript to be executed on the web page.

You must base64 your script.Your Javascript will be executed after the rendering delay and before the awaited selector (if defined)." They encode the script in base64 then execute it but the result is drastically different from the one in console