0

Script for testing broken links using Selenium C#.

  • Please edit your answer to explain the context in which you are checking for broken links. This answer is likely to be closed because you have not provided enough information for a good answer. – Kate Paulk Jul 5 at 12:29
2

I personally wouldn't use Selenium to do this unless I had no other choice. It would be much more efficient to use a dedicated link crawler.

That said, if I had to use Selenium, I'd do something on these lines (no guarantees on code correctness - I'm doing this from memory early in the morning):

// First get all web elements with an href attribute
List<IWebElement> links = driver.FindElements(By.Xpath(//*[@href]));
// Loop through them all and click each one
for(int i = 0; i < links.Count, i++)
{
    IWebElement link = links[i];
    link.click();
    // check that the result of clicking the link is not a 404 or other error page
    // The exact check will depend on the site you're using
    // Go to the previous page
    driver.Navigate().Back();
    // all the links will be stale so refresh the list
    driver.Navigate().Refresh();
    links.Clear();
    links = driver.FindElements(By.Xpath(//*[@href]));
}

The reason for this logic goes this way:

Navigating to each link then going back will disconnect the elements in the list from the currently in-memory DOM, so it's not possible to use the simpler foreach looping. Since it's still possible to iterate through the list using the index, and the current index stays in memory, the for loop works best.

Because the current value of the iterator i isn't changed when the list is refreshed, the next iteration will move to the next link in the list.

I'd recommend a separate method to check whether a page is valid, because some sites use custom error pages, where others let the browser defaults handle things. What you use for your site will be up to you.

It's important to note that this method will not work in all circumstances. If you are using it on a page that required a form submit, you will get browser notifications prompting you to resend the original form. This may or may not cause other failures, depending on your site. If there is a tight no-caching policy for the site, it may not be possible to go back because the page will have expired. This is particularly common with financial sites. Finally, if the site relies heavily on scripting, it's possible that the page itself is a scripted artifact which has to be generated through another page. In that case, the browser Back() function will not return you to the original page.

1

You could use selenium to crawl the site for all link elements:

    private static List<string> GetLinksWithinElement(IWebElement element, IWebDriver driver)
    {
        HashSet<string> links = new HashSet<string>();
        IList<IWebElement> listOlinks = element.FindElements(By.TagName("a"));

        foreach (var i in listOlinks)
        {
            links.Add(i.GetAttribute("href"));
        }
        return links.ToList();
    }

Split list of links into multiple threads to reduce runtime:

    private static void GetUrlListResponses(IWebDriver driver, List<string> links, ExtentTest test)
    {
        List<string> thrd1 = new List<string>();
        List<string> thrd2 = new List<string>();
        List<string> thrd3 = new List<string>();
        List<string> thrd4 = new List<string>();

        for (int i = 0; i < links.Count; i++)
        {
            if( i < (links.Count / 4))
            {
                thrd1.Add(links[i]);
            }
            else if (i < (links.Count / 4)*2)
            {
                thrd2.Add(links[i]);
            }
            else if (i < (links.Count / 4) * 3)
            {
                thrd3.Add(links[i]);
            }
            else
            {
                thrd4.Add(links[i]);
            }
        }

        if (links.Count > 2)
        {
            Thread thread1 = new Thread(() =>
            {
                LinkValidation(driver, test, thrd1, "");
            });
            Thread thread2 = new Thread(() =>
            {
                LinkValidation(driver, test, thrd2, "");

            });
            Thread thread3 = new Thread(() =>
            {
                LinkValidation(driver, test, thrd3, "");

            });
            Thread thread4 = new Thread(() =>
            {
                LinkValidation(driver, test, thrd4, "");

            });
            thread1.Start();
            thread2.Start();
            thread3.Start();
            thread4.Start();

            thread1.Join();
            thread2.Join();
            thread3.Join();
            thread4.Join();

        }
        else
        {
            LinkValidation(driver, test, links, "blah");
        }
    }

iterate through each sublist (thread). This will use the next method to output any links that return bad response codes:

private static void LinkValidation(IWebDriver driver, List<string> links, string name)
    {
        foreach (var link in links)
        {
            try
            {
                string response = GetUrlResponse(link);
                if (response.Contains("False") || response.Contains("ERROR"))
                {
                    Console.WriteLine(link);
                }
            }
            catch
            {
                Console.WriteLine(link)
            }
        }
    }

Method to check response code:

private static string GetUrlResponse(string url)
{
    if (url == null)
        throw new ArgumentNullException("url");

    if (url.IndexOf(':') < 0)
        url = "http://" + url.TrimStart('/');

    if (!Uri.IsWellFormedUriString(url, UriKind.Absolute))
        return "False";

    var request = (HttpWebRequest)WebRequest.Create(url);
    request.Method = "HEAD";

    try
    {
        request.Proxy = null;
        using (var response = request.GetResponse() as HttpWebResponse)
        {

            if (response.ResponseUri != request.RequestUri)
            {

                if (response.ResponseUri.ToString().Contains("error"))
                {
                    return "ERROR (redirect)";

                }
                return "(Redirect)";

            }
            if (response != null && response.StatusCode == HttpStatusCode.OK)
                return "True";

            return "False";
        }
    }
    catch (WebException)
    {
        return "False";
    }
}

If you want to run everything:

List<string> links = GetLinksWithinElement(driver.FindElement(By.ClassName("main-wrapper")), driver);
GetUrlListResponses(driver, links, test);
0

This article contains code to extract broken links from all website pages but that is using by Java Selenium not by C#. If you want you just replace java code with C#. It contains all links with their current status. http://mundrisoft.com/tech-bytes/code-to-identify-all-web-broken-links-of-a-website/

-2
       String loginpage = "https://www.lipsum.com/";
        HttpWebRequest hwr = null;
        String url = null;
        IList<IWebElement> tags = driver.FindElements(By.TagName("a"));
        Console.WriteLine(tags.Count());
        for (int i = 0; i < tags.Count(); i++)
        {
            url = tags.ElementAt(i).GetAttribute("href");
            if (url == null)
            {
                Console.WriteLine(url + "    url is not configured");
                continue;
            }
            if (!url.StartsWith(loginpage))
            {
                Console.WriteLine(url + "       url is in another domain");
                continue;
            }
            hwr = (HttpWebRequest)WebRequest.Create(url);
            try
            {
                var response = (HttpWebResponse)hwr.GetResponse();

                int responsecode = (int)response.StatusCode;

                if (responsecode != 200)
                {
                    Console.WriteLine(url + "  Its a broken link  ");
                }
                else
                {
                    Console.WriteLine(url + "  Its a safe link");
                }
            }
            catch (WebException e)
            {
                e.StackTrace.ToString();
            }
        }
  • 2
    This script is very bad example because: - servers can send response 200 even if link is wrong - server can send other response tha 200 but link will be correct - nowadays link can be not only "a" tags – Michal Dobrzycki Jul 4 at 15:42
  • 1
    @MichalDobrzycki - agreed. Some websites have a pattern where the response code is a 302 and redirect. Some links use a javascript redirect and have the pattern <a href='#' onclick='dosomething()'>. Any html tag that can legally have an href attribute can be a link. – Kate Paulk Jul 5 at 11:56

Not the answer you're looking for? Browse other questions tagged or ask your own question.