URL Verification software

Question

I need to find a program, that can take a list of URLs and actually run them to see where they go.

For example:

Take - www.example.com and see if it goes to www.example.com (no error/redirects) or if it goes to a redirect (i.e. sub.example.com) or if it throws an error such as 404 or 500.

The list is about 900 domains long and we used to copy and paste each one into a browser and see what the end result was, then document what it returned. This has now become a pain since the client base is increasing extremely fast. A desktop app would be ideal, we would have to consider web apps.

Answer 1

Xenu Link Sleuth (http://home.snafu.de/tilman/xenulink.html) can do that and more.

It's easy. It's fast. Best of all, it's free! (Free as in $0, but not Open Source)

Answer 2

If you are comfortable writing up a script, it probably will take less than 50 lines to do what you want.

If you want to use a programming language like python or java or something, you can use one of the URL libraries and make a request to the url. If the response code is 200, that URL passes your test. If not, you can mark it.

You can also write up a quick bash script using curl

Answer 3

Although I like Joe's answer, I've seen this fail a few times. I've seen it happen a few times where for a 'quick fix' a developer has put in javascript redirects on page load when they don't have access to the server itself. In these kind of circumstances, I've found even using something like Selenium useful.

It could be as simple as the following;

[Test]
public void FindSites()
{
  var listOfSites[] = new list[]{'site1', 'site2', 'site3'}
  var driver = new FireFoxDriver();
  foreach (var site in listOfSites)
    {
    driver.Navigate().GoToUrl(site);
    if(site != driver.Url)
      Console.WriteLine("Attempted to navigate to {0} 
        but was redirected to {1}", site, driver.Url);
    }
}

It may not be horribly efficient, but, it works.