How to find out if a website is performing A/B Testing

Question

A have given website say - https://sqa.stackexchange.com/ I need to find out if there is A/B testing being run on the websites. I have come across some tools like, - Ghostery - Conversion.com's Optimizely Chrome

Ghostery - Just scans the webpage for different things and also reports back if there any A/B testing that is running on the site. Optimizely is method which Ghostery tells us that the page is running Optimizely.

Conversion.com's Optimizely is a Chrome plugin, which checks if Optimizely is running on the webpage.

Please let me know if there are any efficient ways to find out if a webpage is conducting A/B testing and if so what elements are under the A/B testing.

Answer 1

There is no certain way from a black box perspective, especially if the tests are not just 50/50 binary switches. For example if a test is being run at 5% / 95% (as we do in my company) that it will only show up, on average, once every 20 runs.

Given the above, there are a few things I can think of that you can do:

1. If the ab test is performed via URL parameters that you can see, e.g. ?&AbTest=show_details:true

2. Visit the page / workflow multiple times and observe any differences. This may require a tool, one that does screenshot comparisons such as GhostInspector

3. Create a simple test in seleniumIDE and run it repeatedly to observe changes

4. Write selenium based tests in a programming languages and run multiple times to see if different results are ever located.

5. Thinking outside of the box (whether black or white) - ask the company if they are doing this!

6. Look in network tools for page source, network tab and cookies and look for anything containing the name ab, abtest ab_test, ab_, etc.

Answer 2

I agree with some of the earlier comments the best way to know is to either ask or observe (which is much harder). If you know what A/B testing tool the site is using it's possible to do some research within documentation to help understand how things are possible stored. (Optimizely for example.)

From a browser standpoint you can look for a few things (this will depend on how the A/B tests are implemented):

1. If the tests are stored in localStorage then they could possibly appear under DevTools > Application > localStorage for that particular page. Or for that matter Session Storage.
2. Look for a possible URL parameter (Optimizely uses a URL parameter for opting out like http://www.example.com/?optimizely_opt_out=true)
3. Use a visual comparison tool like Applitools to take a screen shot of the page and then run several tests to determine if anything on the page changes. Changes are usually recorded as failed tests.