Before I spend too much time researching this I though I would ask the question here to make sure that it is even possible what I am trying to do.
Context: I am trying to help a friend who is a truck driver. He uses a dispatch website to select his freights. The website lists everything that is available as a list but it does not let him filter by zip code. Basically he has to manually scroll this big list looking for deliveries. If he finds something on his zip code or around it he selects it.
Question: Is it possible to auto-scroll the website, filter by zip code and select an item, if it matches?
There are no available APIs that would allow me to do this programmatically. So I though maybe there is a way to scroll the website using JavaScript and select base on a particular field (zip code).
Is this even possible? Is there a better way to go about this?
Answering the question: Selenium Webdriver can be used for this.
var driver = new ChromeDriver();
driver.Navigate().GoToUrl("https://en.wikipedia.org/wiki/List_of_most_visited_websites");
IJavaScriptExecutor js = driver as IJavaScriptExecutor;
// scroll by element - find element by link text and store in variable "Element"
WebElement Element = (WebElement)driver.FindElement(By.LinkText("United States"));
js.ExecuteScript("arguments[0].scrollIntoView();", Element);
System.Threading.Thread.Sleep(5000);
Console.Read();
Selenium webdriver would allow a programmer to access the web page and scrape desired content based on a field. However, some sites prohibit scraping data and it depends on the specific site/fields.
You can scroll to each element using execute_script something like this:
driver.execute_script("arguments[0].scrollIntoView(true);", ele)
where ele is a web element.
you could just put the above line inside a for loop or infinite while loop and scroll until a condition is met.
Sample code:
j = 0
while True:
ele = driver.find_elements(By.XPATH, "xpath of all row")
driver.execute_script("arguments[0].scrollIntoView(true);", ele[j])
#ele[j].text
#ele[j] will represent an web element, so you can retrieve text form that like above.
j = j +1
if ele[j].text == "zip code":
break
else:
continue