How to go through all pages at a certain domain with Selenium WebDriver?

Question

I need to verify a certain thing on multiple pages on a certain site. The thing is it is quite tedious to manually add links to each page for testing to a script. Is there a way the Selenium WebDriver can sweep all the links to all the pages on a certain site? Or any other tool for that matter. I am using Java language.

Answer 1

You want your test script written in Java with Selenium Webdriver to automatically go through all the webpages in a website and verify a certain text/element.

To do this you will first need to get all links from a webpage. You can use this piece of code for doing this,

System.setProperty("webdriver.chrome.driver","./src/resources/chromedriver");
WebDriver driver = new ChromeDriver();
driver.manage().window().maximize();
driver.get("https://example.com/");
 
//Storing the links in a list and traversing through the links
List<WebElement> links = driver.findElements(By.tagName("a"));
 
// This line will print the number of links and the count of links.
System.out.println("No of links are "+ links.size());  
      
//checking the links fetched.
for(int i=0;i<links.size();i++)
{
    WebElement E1= links.get(i);
    String url= E1.getAttribute("href");
}

You can add all the links found in an Array. Then you can iterate through the array to open each link and verify the text/element you need to verify.

You can also fetch all links in each page and verify with your array to check if there are any move pages which were not in the first page and perform nested checks to ensure entire website coverage.

Edit: You might want to avoid the social media links or any other third party links. So, when adding the links in the array, first check whether the first part of the URL (The domain) matched that of the website you intend to test. Otherwise the script will start to crawl the third party webpages.