1

I have been doing Selenium automation around 2013 - 2014. By that time there were only 2 bindings available (or maybe I knew only about 2) - Java and C#. I remember I would have to write so much code before even the browser would launch.

I haven't been in the browser/ test automation field up until now when my manager asked me to quickly write a program that would run through one of our website and collect some information.

I know during all this time there were so many independent selenium libraries developed, that one can easily lose count on them - puppeteer (JavaScript), selenium with Python (https://pypi.org/project/selenium/), different npm packages, commercial solutions and so on.

In you most honest opinion, what library you would pick up today (doesn't matter the language) for most easiness in setting it up, usability and maintenance?

5
  • 3
    @Tired_Of_Testing, Are you sure you need a Selenium for your tasks? Can web scraping tools (like Python beautiful soup) do the job? Another thing I would like to point out that Puppeteer isn't Selenium and doesn't use Selenium. Mar 31, 2020 at 3:37
  • 1
    Protractor is easy to set up and use
    – PDHide
    Mar 31, 2020 at 6:01
  • @GKalnytskyi but I will need to login, click on some buttons before actually pulling data from the site. Isn't selenium for? Mar 31, 2020 at 18:55
  • @Tired_Of_Testing, If you need to interact with the page before scraping the use of browser interaction tools is valid. Sorry for doubting your choice. Mar 31, 2020 at 23:28
  • A little outdated but you can get the overall picture from Design Decisions for Perfect JavaScript Testing Framework by Michael Bodnarchuk
    – Rsf
    Apr 1, 2020 at 6:17

1 Answer 1

2

As @Gkalnytskyi has already pointed out, web scraping is the area you should approach for your objective. Python and BeautifulSoup together with the headless Chrome driver are a combination that you could go with.

BeautifulSoup is a Python library for pulling data out of HTML and XML files. Chrome Headless is a socalled headless web browser that can be used for headless website testing and page automation. Hence, when accessing webpages, you won't see any visible interactions with the browser as it is usually the case.

Here's an code example that shows how you could (that you would have to adjust):

import os  
from selenium import webdriver  
from selenium.webdriver.common.keys import Keys  
from selenium.webdriver.chrome.options import Options  
from bs4 import BeautifulSoup 


chrome_options = Options()  
chrome_options.add_argument("--headless")  

# Initialize the chrome driver with the above specified options
driver = webdriver.Chrome(executable_path=os.path.abspath(“chromedriver"),   
chrome_options=chrome_options)  

# This is the url that you want to scrape data from
url = 'http://www.yourpagewithsomecooldata.com/'
driver.get(url)

soup = BeautifulSoup(driver.page_source, 'lxml')

# Now you can access a web element from the scraped page source like this
a_list_of_matching_spans = soup.find_all('span', class_ = 'avalidclassname')

# Now do whatever you like with the data of the span elements

Selenium needs to be locally installed/available as well.

I hope this helps.

4
  • isn't PhantomJS deprecated project? Mar 31, 2020 at 18:54
  • I haven't used for a quite long time to be honest. I've just researched a bit and it seems like you're right. I will adjust my answer with another headless browser (headless Chrome) that should be used instead.
    – Mh_tm92
    Mar 31, 2020 at 19:00
  • 2
    @TiredOfTesting, You shouldn't confuse PhantomJS and Chrome headless feature. They both serve the same purpose, but headless Chrome is native feature of base Chrome now. And yes the author of PhantomJS stopped project development after Google introduced/released headless mode. (FYI, Firefox can also run headless). Mar 31, 2020 at 23:34
  • @GKalnytskyi thank you for the clarification! :)
    – Mh_tm92
    Apr 1, 2020 at 8:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.