I need to fill the three fields of the form in this page in order to scrape the records of the resulting search, however, I haven´t figured out how to change the values to apartamento, compra usado and Medellín (Antioquia), respectively. Note that the first field is like a dropdown list with checkbox options and the third one is a text input, but you have to select one of the options displayed in order to get the correct results in the search. I'm using the Rselenium package in R but I think it might be easier using native javascript with RSelenium. I'm also open to suggestions in python or any other language.
script <- "
document.querySelector('[class = \"m2-select__single-value css-1uccc91-singleValue\"]').textContent='Compra usado';
document.querySelector('[name = \"businessType\"]').setAttribute('value', 'venta/usado');
document.getElementsByClassName('form-control')[0].setAttribute('value', 'Medellín (Antioquia)');
"
remDr$executeScript(script)
inmueble <- remDr$findElements(using = 'css', '#propertyTypes .css-1hwfws3')
inmueble$sendKeysToElement(list("Casas"))
ciudad <- remDr$findElement(using = 'css', '.form-control')
ciudad$sendKeysToElement(list("Medellín (Antioquia)"))
buscar_boton <- remDr$findElement(using = 'css', '#btnSearch')
buscar_boton$sendKeysToElement(list(key = 'enter'))
EDIT: Added the code I have so far, it changes the values in the html but when I click enter, it does the search with the default values.
You could use httr and make the same search request as the webpage does. You need to pick up an API key en route from one of the JavaScript files:
library(httr)
library(stringr)
library(magrittr)
params <- list(
"realEstateBusinessList" = "venta",
"realEstateStatusList" = "usado",
"locationsList" = "Medellín (Antioquia)",
"realEstateTypeList" = "apartamento",
"from" = "0",
"size" = "50"
)
key <- httr::GET("https://www.metrocuadrado.com/results/_next/static/chunks/commons.3c50d9b2b91c3e061be4.js") %>%
content(as = "text") %>%
stringr::str_match(., '"X-Api-Key":"(.*?)"') %>%
.[, 2]
headers <- c(
"user-agent" = "Mozilla/5.0",
"accept" = "application/json, text/plain, */*",
"x-api-key" = key
)
r <- httr::GET(url = "https://www.metrocuadrado.com/rest-search/search", httr::add_headers(.headers = headers), query = params) %>%
content()
r$results