I have a scrape on a JS driven HTML page that updates found products on the same page. I want to grab the products and parse them out. I want to then move onto the next page, which I have already managed using Watir. The problem is that Watir cannot see the first page's items. Each item is formatted in the HTML below. Here is my code after, then the result:
## HTML
<div data-jplist-item="" class="itemSpace">
<div class="itemContainer">
<a href="/en/Products/All/Tipless Chine" target="_blank" title="Click here to learn more about this product">
<div class="name itemName">(1711) Red Chine, Tips Removed<span style="display:none;">boat part, DELTAUPDATE</span></div>
<div class="saleDiscount" style="display:none;">25.575</div>
<div class="itemNo" style="display:none;">1711</div>
<div class="saleNote">SALE</div></a>
</div>
</div>
## code
##### a is the browser
item = a.div(class: "itemSpace")
ap item.inspect
puts item.class
puts
item2 = a.div(class: "itemContainer")
ap item2.inspect
puts item2.class
## result
"#<Watir::Div: located: false; {:class=>\"itemSpace\", :tag_name=>\"div\"}>"
Watir::Div
"#<Watir::Div: located: false; {:class=>\"itemContainer\", :tag_name=>\"div\"}>"
Watir::Div
As you can see, the div is not located. I'm under the impression Watir manages JS returns, so i don't know why it isn't visible.
Anybody know what I'm doing wrong? Different gem (Ferrum) needs to be used?
The located: false does not mean that Watir cannot see the element or that the element does not exist. It just means that the Watir has not tried to find the element yet. Once you perform an action on the element, then it will be located: true.
For example, if you inspected the class attribute (rather than the object's class), the :located will change to true.
item = a.div(class: "itemSpace")
p item.inspect
#=> "#<Watir::Div: located: false; {:class=>\"itemSpace\", :tag_name=>\"div\"}>"
p item.class_name # note that this is #class_name not #class
#=> "itemSpace"
p item.inspect
#=> "#<Watir::Div: located: true; {:class=>\"itemSpace\", :tag_name=>\"div\"}>"
As to why Watir no longer sees the first page of results, that would depend on what happens to them when loading the second page of results. If the first page of results is removed from the page (ie user can no longer see them), then Watir would not be able to see it either. You'd want to capture the first page of results before going to the next page.