I am currently trying to create a python script that can go into Menards website and grab all their products descriptions, SKU, and price and then imports them into an excel file. So I can then create pivot tables or do other stuff with their products. Currently, I was able to make it work by using XPath and etc. But this method is not very robust since it relies on the specific path for each different webpages. What I want to do is make something more robust that can be used on the website I'm looking at. These are the websites I am referring to.
https://www.menards.com/main/electrical/conduit-conduit-fittings-raceways/conduit/c-6423.htm?queryType=allItems&rid=ipKkIrbnch&shippingOptions_facet=Pickup+at+Store+Eligible'
https://www.menards.com/main/electrical/conduit-conduit-fittings-raceways/conduit-fittings-supports/c-9538.htm queryType=allItems&shippingOptions_facet=Pickup+at+Store+Eligible&sortby=priceAsc'
https://www.menards.com/main/electrical/light-switches-dimmers-outlets/light-switches/c-6324.htm?queryType=allItems&Spec_Color%2FFinishFamily_facet=Ivory&Spec_Color%2FFinishFamily_facet=Light+Almond&Spec_Color%2FFinishFamily_facet=White&shippingOptions_facet=Pickup+at+Store+Eligible&sortby=priceAsc/'
This is the code that I have currently have in trying to figure out a more robust design.
driver.get(website.get('IRW'))
driver.minimize_window()
description = driver.find_elements(By.CSS_SELECTOR, ".search-item")
print(len(description))
for ll in range(0, 1):
print(description[ll].text)
Currently, this code just locates all of the elements on the webpage by its class name "search-items". Which locates most of the elements and such but however I get a lot of garbage with the results I want. Here is an example of what I am output of the code:
> 11 Click here to go to detail page Bestseller 12 Variations Available
> 12 Gauge NM-B Cable with Ground Wire Click to add item "12 Gauge NM-B
> Cable with Ground Wire" to the compare list Compare Click to add item
> "12 Gauge NM-B Cable with Ground Wire" to the compare list Add To List
> Click to add item 12 Gauge NM-B Cable with Ground Wire to your list
> Sku # 3691660 $25.79 You Save $3.19 with Mail-In Rebate More
> Information Shipping CHOOSE VARIATION None
I want to avoid having to replace the current string and do a lot of loops in order to get the values since all of this garbage is not consistent on every webpage. I am hoping that is a better way to locate the elements. Any ideas thanks!
You can use the css child > and descendant combinators to specify direct children or descendants (respectively) of parents with class search-item, in a sequence that isolates just the hrefs of interest
links = [i.text for i in
driver.find_elements_by_css_selector("#search-items > .search-item .details > a")]
Thank you QHarr I was able to figure out how to access other classes with this specific class. The line of code that allows me to do this is as follows:
#for description
description = [i.text for i in driver.find_elements(By.CSS_SELECTOR, "#search-items > .search-item .details > a > .row.pt-5.pb-sm-5 > .multilines-3.text-truncate-multilines.xs-single-col-8.col-12 > .font-weight-bold.text-dark")]
print(description)