Are there any tools to test whether the response code is not equal to 404 or etc by reading the URLs from a CSV file

Question

I want to check the status code of a URL by reading the URL from a CSV file and "hitting it" with a browser.

How can I achieve this functionality? Are there any tools for that?

Answer 1

There are several solutions.

The simplest would probably be to list your links in HTML format, and then run that page through a link checker tool. Here's an open source example.

Another solution is to set up a testing project in Visual Studio (or a similar environment). You can provide source data in CSV format, see this topic. It's fairly easy to have Selenium (or Coded UI) open the webpage and check the contents for a 404 message.

Answer 2

A nice automation question.

This question has already been asked on Stack Overflow, in Check the server response code, then export to csv. I don't know how good your python coding skills are, but most of the code is given on that page. It seems to aim to achieve exactly what you want to do. It probably will need tidying up and debugging first though :-( However, as rSF suggests, it should only take an hour or two.

import csv

with open("urls.csv", 'r') as csvfile:
    urls = [row[0] for row in csv.reader(csvfile)]

import urllib2
for url in urls:
    try:
        connection = urllib2.urlopen(url)
        print connection.getcode()
        connection.close()
    except urllib2.HTTPError, e:
        print e.getcode()

As an aside, there is an online tool of the same functionality, HTTP Status Code Checker, although coding that up using that programatically is probably more trouble than it is worth.

Answer 3

Or just code it in your favorite language- Python, Perl, JAVA etc.

I estimate it in less than an hour of coding, verifying it and having a cup of coffee

Answer 4

I think that you should be able to use Apache JMeter for it. JMeter acts on protocol level so in any case it will be faster than using a browser. Besides JMeter is designed for load testing so you will be able to check all the URLs in matter of seconds.

Components you'll need are:

• HTTP Request
• CSV Data Set Config

Answer 5

I'm pretty new to coding, but I wrote some code that I believe does what you want. The output is a .csv file with two fields.

import csv
import urllib2

file_to_open = raw_input('Please enter the name of the csv file containing the URLs: ')
file_to_write = file_to_open[0:-4] + '-review.csv'

with open(file_to_open, 'r') as old_csvfile:
    urls = [row[0] for row in csv.reader(old_csvfile)]

with open(file_to_write, 'wb') as new_csvfile:
    new_csv_writer = csv.writer(new_csvfile, delimiter=',')
    new_csv_writer.writerow(['URL'] + ['Code'])
    for url in urls:
        try:
            connection = urllib2.urlopen(url)
            new_csv_writer.writerow([url] + [connection.getcode()])
            connection.close()
        except urllib2.HTTPError, e:
            new_csv_writer.writerow([url] + [e.getcode()])

Answer 6

I came across similar requirement so I used core Java's Connection URL API, URLConnection. It was much faster:

String  myString = null;
URLConnection urlConn = null;
try{
    URL url = new URL("enter url here");
    urlConn = url.openConnection();
    urlConn.setUseCaches(false);
    HttpURLConnection httpConn = (HttpURLConnection)urlConn;
    InputStream is=null;
    if (httpConn.getResponseCode() >= 400) {
        if(httpConn.getErrorStream()!=null){
            is = httpConn.getErrorStream();
        }else{System.out.println("no error");}
    } else {
        is = httpConn.getInputStream();
    }
    myString = IOUtils.toString(is, "UTF-8");
    System.out.println(myString);
}catch(Exception e){
    throw new RuntimeException("Exception while calling url:",e);
}