We try to scrape a website using Selenium, JAVA, eg. try to change product color at https://www.ergodyne.com/shax-6054-pop-up-tent-sidewall-kit-includes-10ftx10ft
How to find out/check when the in-site JS has changed the HTML upon the arrival of the AJAX response ?
How big should be delay after AJAX/XHR response and JS replaced HTML ?
The AJAZ queries are quite complicated to try to get data from them...
Link for JSON response: https://jsonformatter.org/298d31
Using org.openqa.selenium.support.ui.ExpectedConditions and org.openqa.selenium.support.ui.WebDriverWait. See https://www.guru99.com/implicit-explicit-waits-selenium.html
Code:
package tests;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
public class IgorSavinking {
public static String userDir = System.getProperty("user.dir");
public static String chromedriverPath = userDir + "\\resources\\chromedriver.exe";
public static void main(String[] args) {
System.setProperty("webdriver.chrome.driver", chromedriverPath);
ChromeOptions options = new ChromeOptions();
options.addArguments("--ignore-certificate-errors");
options.addArguments("--start-maximized");
options.addArguments("--disable-notifications");
WebDriver driver = new ChromeDriver(options);
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
driver.get("https://www.ergodyne.com/shax-6054-pop-up-tent-sidewall-kit-includes-10ftx10ft");
driver.findElement(By.id("cookie-policy-accept")).click();
String colorBefore = driver.findElement(By.tagName("legend")).getText().substring(5);
System.out.println("Current color is: " + colorBefore);
driver.findElement(By.xpath("//label[contains(@for,'edit-purchased-entity-0-attributes-attribute-color-17')]")).click();
new WebDriverWait(driver, 30).until(ExpectedConditions.numberOfElementsToBe(By.xpath("div[@class='message']"), 0));
String colorAfter = driver.findElement(By.tagName("legend")).getText().substring(5);
System.out.println("Current color is: " + colorAfter);
System.out.println("Color has been changed: " + !colorBefore.equals(colorAfter));
driver.quit();
}
}
Output:
Starting ChromeDriver 97.0.4692.71 (adefa7837d02a07a604c1e6eff0b3a09422ab88d-refs/branch-heads/4692@{#1247}) on port 38237
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
Úno 02, 2022 8:02:15 DOP. org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Current color is: Lime
Current color is: Blue
Color has been changed: true