How to test a huge list of domains redirect to HTTPS?

Question

I have got a huge list of domains/URLs (about a 1000). The task is to test if each one of them, if accessed with http://domain.com then it redirects to https://domain.com.

All the domains that do not redirect are to be reported to devs.

One way of doing this would be,

1. To open each URL with Selenium WebDriver's get(url); method.
2. To get the current URL from the browser using getCurrentUrl();.
3. Compare the to to ensure the latter begins with HTTPS.

But, I guess there should be a faster way to do this without needing to use Selenium.

Any ideas?

Answer 1

You can use CURL for it. For instance, if you run

curl -Ls -o /dev/null -w %{url_effective} http://sqa.stackexchange.com/

it will show https://sqa.stackexchange.com/, which is the redirected URL.

The flags mean the following:

-L         Follow redirects
-s         Silent mode. Don't output anything
-o FILE    Write output to FILE instead of stdout
-w FORMAT  What to output after completion (url_effective is the final URL)

With some simple scripting, you can check if the returned strings form CURL start with https

Answer 2

Searched for a bit on the internet and from few websites gather and compiled this bit of code which reads domains from csv, checks redirection and then writes result in a csv,

public static void main(String[] args) {
    try {
        FileReader filereader = new FileReader(System.getProperty("user.dir") + "//data/file.csv");
        CSVParser parser = new CSVParserBuilder().withSeparator(';').build();
        CSVReader csvReader = new CSVReaderBuilder(filereader).withCSVParser(parser).withSkipLines(1).build();
        List<String[]> allData = csvReader.readAll();
        List<String[]> data = new ArrayList<String[]>();
        URL url;
        int count = 0;
        for (String[] row : allData) {
            for (String cell : row) {
                int responseCode = 0;
                String Location = null;
                url = new URL("http://" + cell);
                try {
                    HttpURLConnection.setFollowRedirects(false);
                    HttpURLConnection huc = (HttpURLConnection) url.openConnection();
                    responseCode = huc.getResponseCode();
                    if (responseCode == HttpURLConnection.HTTP_SEE_OTHER || responseCode == HttpURLConnection.HTTP_MOVED_PERM || responseCode == HttpURLConnection.HTTP_MOVED_TEMP) {
                        Location = huc.getHeaderField("Location");
                    }
                    if (Location != null && Location.startsWith("/")) {
                        Location = url.getProtocol() + "://" + url.getHost() + Location;
                    }
                } catch (Exception e) {
                    e.printStackTrace();
                }
                count++;
                if (Location != null && Location.startsWith("https")) {
                    data.add(new String[] { Integer.toString(count), String.valueOf(url), Integer.toString(responseCode), Location, "Pass" });
                } else {
                    data.add(new String[] { Integer.toString(count), String.valueOf(url), Integer.toString(responseCode), Location, "Fail" });
                }
            }
        }
        DateFormat df = new SimpleDateFormat("yyyy-MM-dd-HH-mm-ss");
        String currentDate = df.format(new Date());
        File file = new File(System.getProperty("user.dir") + "//data/report-" + currentDate + ".csv");
        FileWriter outputFile = new FileWriter(file);
        CSVWriter writer = new CSVWriter(outputFile, ',', CSVWriter.NO_QUOTE_CHARACTER, CSVWriter.DEFAULT_ESCAPE_CHARACTER, CSVWriter.DEFAULT_LINE_END);
        String[] header = { "#", "URL", "Response Code", "Redirected To", "Status" };
        writer.writeNext(header);
        writer.writeAll(data);
        writer.close();
    } catch (
    Exception e) {
        e.printStackTrace();
    }
}