Empresas
Empleos
  • Sobre nosotros
  • Soluciones
    • Publicación de vacantes
      Publica tu vacante y recibe candidatos calificados en 48h.
    • Evaluación de candidatos
      500+ pruebas técnicas y psicológicas, más anti-fraude.
    • Headhunting
      Búsqueda ejecutiva a la medida de principio a fin.
    • Nómina + EOR
      Dispersión de nómina y EOR en más de 15 países de LATAM.
  • Precios
  • Empleos

0

114
Vistas
FB loads only certain posts' data

So...I need to scrape some public data for an academic MBA dissertation from FB public pages (correlating readability with engagement etc etc)

The problem is that FB is soooo hard to scrape!

I have tried a lot!

  1. Octoparse does not scrape all posts because of post being hidden on scroll
  2. Web scrapper (the cute chrome plugin) has the same problem even though it gets much more and much better than the Octoparse overkill!
  3. I don't wanna build puppeteer app cause I am almost sure it will need the double verification so it wont work either.

So I am sitting and trying to extract some juice with good old chrome console! I don't really care about the end product cause I will clean it up with regex in Gsheets.

But the problem I am facing is that FB gives me only the 10 most close posts to where I am in the page. If I am in the beginning it will give me up to 5 posts. if I am in post 50 it will show me 45-55 approx. You get the picture! Any ideas?

Here is my code:

This is for getting all the posts and store in the allelements array

let current = document.querySelector('#mount_0_0_pK > div > div:nth-child(1) > div > div.rq0escxv.l9j0dhe7.du4w35lb > div > div > div.j83agx80.cbu4d94t.d6urw2fd.dp1hu0rb.l9j0dhe7.du4w35lb > div.l9j0dhe7.dp1hu0rb.cbu4d94t.j83agx80 > div.bp9cbjyn.j83agx80.cbu4d94t.d2edcug0 > div.rq0escxv.d2edcug0.ecyo15nh.k387qaup.r24q5c3a.hv4rvrfc.dati1w0a.cxgpxx05 > div > div.rq0escxv.l9j0dhe7.du4w35lb.hpfvmrgz.g5gj957u.aov4n071.oi9244e8.bi6gxh9e.h676nmdw.aghb5jc5.gile2uim.pwa15fzy.fhuww2h9 > div > div > div > div:nth-child(1)');
let nextSibling = current.nextElementSibling;
let allelements = []
allelements.push(current)
while(nextSibling) {
    console.log(nextSibling);
    nextSibling = nextSibling.nextElementSibling;
    allelements.push(nextSibling)
}

And this is simply for getting the posts' juice

allelements.forEach((element, index)=>{console.log(index, element.innerText)})

The end result is that the allelements array, stores all the posts but it only gives data for the very proximate ones of the position I am in. I tried to make it work with window.scroll() but this is an async function and it cannot work. When I tried to work it synchronously, I got a stack size error...

any ideas??? Maybe there is a way to override this function of loading the content of certain posts?

Thank you very much!

about 4 years ago · Juan Pablo Isaza
1 Respuestas
Responde la pregunta

0

To directly address your problem of trying to scrape FB posts in the developer console I wrote a functioning synchronous script that saves all posts to an array and their innerText to another array as it scrolls down through someone's timeline. It will also log a running tally of how many posts have been scraped and then output the array of all collected text upon completion.

NOTES:

  • You can choose how long you want the script to pause before scrolling down again. I chose two seconds because some posts were taking a while to load
  • My script stops collecting posts once it detects the Facebook generated event "Born on..." since that is the first item on the timeline. You can choose any other criteria you see fit to detect the end of the page. I just found this one to be easiest
  • It's more robust to not use CSS selectors since the class names and ids change every time a new instance of Facebook is loaded. A concise alternative is to use xpath instead, which you'll see below.
  • You can have multiple browser windows open, but this script will only work if its the active tab of its respective browser window.
console.clear();
const allText = [];
const allPosts = [];
(function scrollAndScrape() {
    console.log('Total posts scraped:', allPosts.length)
    if ($x('//span[contains(text(), "Born on")]').length > 0) {
        console.log('Webscrape complete', '\n', allText)
    } else {
        let visiblePosts = $x('//div[@data-pagelet="ProfileTimeline"]/div')
        for (let i = 0; i < visiblePosts.length; i++) {
            if (!allPosts.includes(visiblePosts[i])) {
                allPosts.push(visiblePosts[i])
                allText.push(visiblePosts[i].innerText)
            }
            window.scrollByLines(100)
        }
        setTimeout(scrollAndScrape, 2000);
    }
})()

The output looks something like this:

...
Total posts scraped: 176
Total posts scraped: 180
Total posts scraped: 187
Total posts scraped: 194
Total posts scraped: 200
Webscrape complete
> Array(200) [TEXT BLOB, TEXT BLOB...]

The text blobs will include lots of special string characters such as new line characters (\n) but once you run your second code snippet (see code block below) the text will print nicely in the console. Additionally, you can always format the postText as you wish using regex.

allText.forEach((postText, index)=>{console.log(index, postText)})

To address the more general issue of Facebook being difficult to scrape I would recommend using Selenium. Its compatible with Java, JavaScript, Python, and Ruby. Selenium is an automated browser tool that supports multiple drivers (firefox, chrome, etc.) and can run with or without a GUI. Selenium is well-documented with a large following so it is easy to find guides, tutorials, etc. Everything my JS snippet above does can be replicated with Selenium. You can follow the same logic:

  1. Collect all visible posts in an array
  2. Scroll down (an easy way to do this is by using the sendKeys method to trigger the Page Down key event)
  3. Collect any new visible posts and save them to the array
about 4 years ago · Juan Pablo Isaza Denunciar
Responde la pregunta
Encuentra empleos remotos

¡Descubre la nueva forma de encontrar empleo!

Top de empleos
Top categorías de empleo
Empresas
Publicar vacante Precios Comercial
Legal
Términos y condiciones Política de privacidad
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Recomiéndame algunas ofertas
Necesito ayuda