I am trying to scrape all translated text from multiple webpages. The page has to translate fully before I start the scraping and in order to do that I have to scroll through the page. I am using selenium with Edge and it translate feature. The problem here is that it only translates text that is visible and there are some text that dont show until some buttons are pressed.
This is my code:
options = EdgeOptions()
options.add_experimental_option('excludeSwitches', ['enable-logging'])
options.use_chromium = True
options.add_extension('adblock.crx')
capabilities = DesiredCapabilities.EDGE
prefs = {
"translate_whitelists": {input_lang:output_lang},
"translate":{"enabled":"true"}}
options.add_argument('--lang=en')
options.add_experimental_option("prefs", prefs)
driver = Edge(executable_path=PATH, options=options, desired_capabilities=capabilities)
driver.maximize_window()
driver.get(url)
# This is to scroll through the page
if input_lang != output_lang:
total_height = int(driver.execute_script("return document.body.scrollHeight"))
divisions = total_height // 20
for i in range(1, total_height, divisions):
driver.execute_script("window.scrollTo(0, {});".format(i))
time.sleep(3)
driver.execute_script(f"window.scrollTo(0, {total_height});")
time.sleep(4)
Is there a way I can find all hidden element with text and loop click through them so they become visible? Any help will be appreciated
Use element.getAttribute("textContent") instead of getText()
getText() uses innerText and considers the element display property, if its not displayed , then no content will be retrieved
While textContent doesn't care whether the element is displayed , it retries the content as it is