I need to find a program, that can take a list of URLs and actually run them to see where they go.

For example:

Take - www.example.com and see if it goes to www.example.com (no error/redirects) or if it goes to a redirect (i.e. sub.example.com) or if it throws an error such as 404 or 500.

The list is about 900 domains long and we used to copy and paste each one into a browser and see what the end result was, then document what it returned. This has now become a pain since the client base is increasing extremely fast. A desktop app would be ideal, we would have to consider web apps.

  • Is there some programming language/environment that you'd prefer the solution to run under? For example C#/Windows or JAVA/Linux or ???
    – dcaswell
    Sep 17, 2013 at 23:34

3 Answers 3


Xenu Link Sleuth (http://home.snafu.de/tilman/xenulink.html) can do that and more.

It's easy. It's fast. Best of all, it's free! (Free as in $0, but not Open Source)

  • This looks perfect for what we need, but the company doesn't like using free open source programs. Ive sent it up the ladder anyways just in case. Thanks for the comment. Sep 16, 2013 at 18:22
  • @Duke03 - Perfect, because Xenu isn't open source! See: home.snafu.de/tilman/xenulink.html#FAQ Sep 16, 2013 at 18:24
  • Right I realized that, but still the company recognizes anything to be free as a potential danger to our system. Sep 16, 2013 at 18:26
  • 1
    @Duke03 - You are talking about tools to assist testing, right? My company also dislikes open source when used in Production systems. But they let us choose whatever we wish for test tools. Sep 16, 2013 at 18:28
  • I don't get the Free is a danger issue, but I do use Xenu to go through our site to help me find dead links on occasion so I can have them resolved through Content Editing or redirects. It's very nice, but spend time with the reporting because the default HTML one is not Business User friendly. I do my own summary of links using PowerShell to link the reports together for easy readability
    – MichaelF
    Sep 16, 2013 at 20:24

If you are comfortable writing up a script, it probably will take less than 50 lines to do what you want.

If you want to use a programming language like python or java or something, you can use one of the URL libraries and make a request to the url. If the response code is 200, that URL passes your test. If not, you can mark it.

You can also write up a quick bash script using curl

  • I could write a script, but do you know of any examples anywhere that could get me started? Also what other languages could i write this in? Not great with those two but i could give it a shot. Thanks for the comment, Sep 16, 2013 at 18:21
  • @Duke03, For Java see Apache HTTP Client with example here.
    – dzieciou
    Sep 16, 2013 at 18:57
  • @Suchit, great answer. One thing, the OP wanted also to recognize URL redirections. As there are many different approaches for redirection on client- and server-side: meta refresh, java script redirection, etc., his script would need to parse it somehow from HTTP response. Any library that can simplify that?
    – dzieciou
    Sep 16, 2013 at 19:03
  • @Duke03 - Take a look at this answer on SE too (php) - stackoverflow.com/a/3520085/781810. The response also has a link to a pretty good article. May not be the exact response you are looking for, but some good information. Sep 17, 2013 at 16:36

Although I like Joe's answer, I've seen this fail a few times. I've seen it happen a few times where for a 'quick fix' a developer has put in javascript redirects on page load when they don't have access to the server itself. In these kind of circumstances, I've found even using something like Selenium useful.

It could be as simple as the following;

public void FindSites()
  var listOfSites[] = new list[]{'site1', 'site2', 'site3'}
  var driver = new FireFoxDriver();
  foreach (var site in listOfSites)
    if(site != driver.Url)
      Console.WriteLine("Attempted to navigate to {0} 
        but was redirected to {1}", site, driver.Url);

It may not be horribly efficient, but, it works.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.