I am building a web scraper using cypress to scrape all apps from Atlassian marketplace
https://marketplace.atlassian.com/search
The problem is the page has load more button which need to be clicked to load more apps. When I did this in cpypress it take for ever to scroll and load all apps.
My code:
import { times } from 'lodash'
describe('Scraping Marketplace', () => {
it('Visits The Marketlace and scrape all data to csv', () => {
cy.visit('https://marketplace.atlassian.com/addons/top-rated');
cy.get('.css-yki7vm-AkButtonWrapper > button').find('span').should('exist').then(() => {
times(10000, () => {
if (cy.get('.css-yki7vm-AkButtonWrapper > button').find('span').should('exist')) {
cy.get('.css-yki7vm-AkButtonWrapper > button').find('span').click({force: true});
} else {
return false;
}
});
})
})
})