So...I need to scrape some public data for an academic MBA dissertation from FB public pages (correlating readability with engagement etc etc)
The problem is that FB is soooo hard to scrape!
I have tried a lot!
So I am sitting and trying to extract some juice with good old chrome console! I don't really care about the end product cause I will clean it up with regex in Gsheets.
But the problem I am facing is that FB gives me only the 10 most close posts to where I am in the page. If I am in the beginning it will give me up to 5 posts. if I am in post 50 it will show me 45-55 approx. You get the picture! Any ideas?
Here is my code:
This is for getting all the posts and store in the allelements array
let current = document.querySelector('#mount_0_0_pK > div > div:nth-child(1) > div > div.rq0escxv.l9j0dhe7.du4w35lb > div > div > div.j83agx80.cbu4d94t.d6urw2fd.dp1hu0rb.l9j0dhe7.du4w35lb > div.l9j0dhe7.dp1hu0rb.cbu4d94t.j83agx80 > div.bp9cbjyn.j83agx80.cbu4d94t.d2edcug0 > div.rq0escxv.d2edcug0.ecyo15nh.k387qaup.r24q5c3a.hv4rvrfc.dati1w0a.cxgpxx05 > div > div.rq0escxv.l9j0dhe7.du4w35lb.hpfvmrgz.g5gj957u.aov4n071.oi9244e8.bi6gxh9e.h676nmdw.aghb5jc5.gile2uim.pwa15fzy.fhuww2h9 > div > div > div > div:nth-child(1)');
let nextSibling = current.nextElementSibling;
let allelements = []
allelements.push(current)
while(nextSibling) {
console.log(nextSibling);
nextSibling = nextSibling.nextElementSibling;
allelements.push(nextSibling)
}
And this is simply for getting the posts' juice
allelements.forEach((element, index)=>{console.log(index, element.innerText)})
The end result is that the allelements array, stores all the posts but it only gives data for the very proximate ones of the position I am in. I tried to make it work with window.scroll() but this is an async function and it cannot work. When I tried to work it synchronously, I got a stack size error...
any ideas??? Maybe there is a way to override this function of loading the content of certain posts?
Thank you very much!
To directly address your problem of trying to scrape FB posts in the developer console I wrote a functioning synchronous script that saves all posts to an array and their innerText to another array as it scrolls down through someone's timeline. It will also log a running tally of how many posts have been scraped and then output the array of all collected text upon completion.
NOTES:
console.clear();
const allText = [];
const allPosts = [];
(function scrollAndScrape() {
console.log('Total posts scraped:', allPosts.length)
if ($x('//span[contains(text(), "Born on")]').length > 0) {
console.log('Webscrape complete', '\n', allText)
} else {
let visiblePosts = $x('//div[@data-pagelet="ProfileTimeline"]/div')
for (let i = 0; i < visiblePosts.length; i++) {
if (!allPosts.includes(visiblePosts[i])) {
allPosts.push(visiblePosts[i])
allText.push(visiblePosts[i].innerText)
}
window.scrollByLines(100)
}
setTimeout(scrollAndScrape, 2000);
}
})()
The output looks something like this:
...
Total posts scraped: 176
Total posts scraped: 180
Total posts scraped: 187
Total posts scraped: 194
Total posts scraped: 200
Webscrape complete
> Array(200) [TEXT BLOB, TEXT BLOB...]
The text blobs will include lots of special string characters such as new line characters (\n) but once you run your second code snippet (see code block below) the text will print nicely in the console. Additionally, you can always format the postText as you wish using regex.
allText.forEach((postText, index)=>{console.log(index, postText)})
To address the more general issue of Facebook being difficult to scrape I would recommend using Selenium. Its compatible with Java, JavaScript, Python, and Ruby. Selenium is an automated browser tool that supports multiple drivers (firefox, chrome, etc.) and can run with or without a GUI. Selenium is well-documented with a large following so it is easy to find guides, tutorials, etc. Everything my JS snippet above does can be replicated with Selenium. You can follow the same logic:
sendKeys method to trigger the Page Down key event)