I am trying to web scrape using Java and Selenium but for one part of what I need to web scrape, it requires me to access a shadow DOM element. I am using Javascript executor to access it and I am entering the querySelector snippet needed to retrieve what I am looking for but I am getting a "Cannot read properties of null (reading 'ShadowRoot')" error. Is anyone familiar with this and how to solve it?
Here is my code snippet:
String pagePdfUrl = (String) js.executeScript("document.querySelector('pdf-viewer').shadowRoot.getElementById('content').querySelector('embed').getAttribute('original-url')");
js is the JavascriptExectuor variable.
Thank you!
While using querySelector() to read a ShadowRoot this error message...
Cannot read properties of null
...implies either of the following circumstances:
In case of the first and second scenario, you won't be able to access the ShadowRoot. Where as in the third case you need to wait for sometime e.g. time.sleep(3) before you attempt to access the ShadowRoot and it's elements.
In Selenium 4.0, for Chromium versions 96+ you can use the getShadowRoot() method and avoid the JavaScript entirely.
A working example:
driver.get("http://watir.com/examples/shadow_dom.html");
WebElement shadowHost = driver.findElement(By.cssSelector("#shadow_host"));
SearchContext shadowRoot = shadowHost.getShadowRoot();
WebElement shadowContent = shadowRoot.findElement(By.cssSelector("#shadow_content"));
Source: https://titusfortner.com/2021/11/22/shadow-dom-selenium.html