I am QA automation engineer. We have so many clients for whom we are doing web scraping. We automate their daily task to save their time and resource.

Every client demands that they should able to run script anytime they want So currently what we are doing :

  1. Install Eclipse in client computer
  2. Install Java and configure Selenium and other required JARs in client computer
  3. Transfer script to their computer
  4. Setup Jenkins there
  5. Setup TestNG there

We want to get rid from above process. Is there any other way by which client can access script and run by URL or something?

Above process is time consuming also.

Please suggest best possible ways.

  • I don't understand what you mean by "script". Do you clients demand your test code, or do they demand your web-scraping code? Please edit your question to make that point clear.
    – user246
    Apr 12, 2016 at 13:55
  • @user246 - Yes I am talking about web scraping code. Apr 12, 2016 at 13:58

2 Answers 2


You do not need Eclipse, Jenkins, or TestNG to scrape a website.

Ask a software developer to help you package Selenium, your Java code, and any other necessary resources into an executable Jar file. If you can't find any developers, try Googling for "executable Jar file".

Your executable Jar file will require a main class. If you are accustomed to running your web-scraping code under TestNG, your main is currently the TestNG test runner. You will need to restructure that code to be invoked by your own main class instead.

I assume you can figure out how to distribute your executable jars to your demanding clients.

  • Thanks for detail answer. If I deliver JAR file to client then how can I set schedule to run that JAR file. Schedule like run once in a day or weekly etc. Or client will have to run it by double click on it? Apr 12, 2016 at 14:16
  • That depends on the operating system.
    – user246
    Apr 12, 2016 at 14:22

With JUnit I have set up a web-based runner/control unit that allows the end user to access and run the tests.

It is extremely primitive and it would run on the server instead of on their machine but here is the code that I used in order to achieve this:

class TestsService {
    UriInfo ui;

    public Result runSpec(@QueryParam("spec") String spec) {
        JUnitCore jUnitCore = new JUnitCore();
        try {
            Class test = Class.forName(spec)

            return jUnitCore.run(test);

        } catch (Exception ex) {

    public Result runAll() {
        JUnitCore jUnitCore = new JUnitCore();
        try {
            Reflections reflections = new Reflections("Tests.*");

            Class[] allClasses =

            return jUnitCore.run(allClasses);

        } catch (Exception ex) {

This created a rest layer using jersey, personally using a built-in jetty server. From there I built out an AngularJS page that displayed all of the historically ran Tests from the system.

An example of the jetty server:

public class App {
    public static void main(String[] args) throws Exception {

        ServletContextHandler context = new ServletContextHandler(ServletContextHandler.SESSIONS);
        FilterHolder holder = new FilterHolder(new CrossOriginFilter());
        holder.setInitParameter("allowedMethods", "GET,POST,HEAD,OPTIONS");
        context.addFilter(holder, "/*", EnumSet.of(DispatcherType.REQUEST));

        Server jettyServer = new Server(9898);

        ServletHolder jerseyServlet = context.addServlet(ServletContainer.class, "/*");
        // Tells the Jersey Servlet which REST service/class to load.

        try {
        } finally {
  • Thanks for answer. Can you please tell me if I use your given code then how client can access it? I mean what I need to deliver to client to run program? Apr 12, 2016 at 14:00
  • After rereading your question, User246's answer is likely better as you do not require the actual tests to be ran but rather just need to collect the data. Just compiling into a JAR and sending them the jar would be suffice.
    – Paul Muir
    Apr 12, 2016 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.