2 times drop-down menu selection in a row with RSelenium

Question

I am trying to scrape this website. I got a problem with text fields because text fields and drop-down menus combined. The site does not allow to enter the names of cities directly. For example, I only want to take the port pointers (ex: Port of Shanghai, China), firstly, I must enter "Shanghai, China", and need to click the first option, then need to click "Port of Shanghai, China". (actually, ports are the second option). With the following codes, I can select the country name but not the port.

> remote_driver$findElement(using = 'id', value ='from')$sendKeysToElement(list("Shanghai, China"))$clickElement()
> remote_driver$findElement(using = "css", value = "#from-autocomplete > div > div > div > div:nth-child(1)")$clickElement()
> remote_driver$findElement(using = "css", value = "#from-autocomplete > div > div > div > div.fvlk5rSgKI--Xo4zMn3an > div:nth-child(2)")$clickElement()

But in the second selection for port names (3rd line), the port name has been erased from the text field.

Thanks in advance!

Answer 1

Fist of all, this "drop-downs menu" are more appropriate to be called "suggestion results". So here is my answer and hope it works because I don't have RSelenium to test it. I added intentionally some sleep just to give enough time for the components to load (delete the sleeps if it works without them after you test it). I also modified the css selectors to be less breakable:

> remote_driver$findElement(using = 'id', value ='from')$sendKeysToElement(list("Shanghai, China"))$clickElement()
> Sys.sleep(2)  
> remote_driver$findElement(using = "css", value = "#from-autocomplete [tabindex="0"]")$clickElement()
> Sys.sleep(2)  
> remote_driver$findElement(using = "css", value = "#from-autocomplete [tabindex="1"]")$clickElement()

Answer 2

It seems that the website's JavaScript is interfering with the second click event, which is erasing the previously entered value. To overcome this issue, you can try the following approach:

1. Use the sendKeysToElement method to enter the city name and wait for a while using the Sys.sleep function or remote_driver$client$setTimeout method to give the website's JavaScript enough time to process the input and populate the dropdown menu.

2. Use the findElements method to get a list of all the options available in the dropdown menu.

3. Use a loop to iterate through the list of options and check for the desired option, for example the port name.

4. When the desired option is found, use the clickElement method to select it.

Here is an example code:

remote_driver$findElement(using = 'id', value ='from')$sendKeysToElement(list("Shanghai, China"))
Sys.sleep(5)
options <- remote_driver$findElements(using = "css", value = "#from-autocomplete > div > div > div > div")
for(option in options) {
  if(grepl("Port of Shanghai, China", option$getElementText())) {
    option$clickElement()
    break
  }
}

The Sys.sleep function is just used to wait for the dropdown menu to appear. The actual wait time might be different based on your system and internet speed, so you may need to adjust it accordingly.