Testing using real data of the customer

Question

Is it common to use real data of the customer to perform testing? What policies companies apply regarding using real data of the customer for testing purposes? Is there any legislations regarding such issues?

Answer 1

Depends on your definition of testing, anonymized data is widely used by Microsoft and others for monitoring and testing in production, it's the basis for A/B testing or monitoring for example.

In Europe the GDPR does not allow usage of private data, but the GDPR does not apply to anonymised information and anonymised data can be used without consent. Anonymised data is defined as “data rendered anonymous in such a way that the data subject is not or no longer identifiable.”

Be careful though, you need to be careful on how data is anonymised and make sure it is really irreversible.

Answer 2

I wouldn't say it is common to use real data in testing, although the customer might provide a subset of "real" data in order to facilitate the process.

Apart from the privacy and business issues, there are also the legal ones, e.g. General Data Protection Regulation (GDPR) has been enforced since 25 May 2018 in Europe (but I think every company dealing with another company from the EU should take notice).

GDPR takes a very strict approach (fines) when dealing with personal data (and personal data is practically any data concerning a person - so this is a very broad definition) so it is better to just use test data, at least in this context.

Answer 3

It depends

In some industries it's not feasible to test without customer data. Sometimes it's not possible to properly anonymize said data - I test software that uses the US social security number for a large number of lookups. That means that any method of anonymization must ensure that the social security number of a given person must produce the same result across multiple tables while retaining the standard social security number format and not violating US regulations on what constitutes a valid social security number. That's just one example of a case where not using real data is somewhere between difficult and infeasible.

Another example - from the same software (which is also legacy software dating from 2002) - is a situation where data syncs between multiple systems. If your application is not the system of record (that is, the master system), you may not be able to anonymize data without losing the ability to sync to the master system. The software I test is a web application that sends updated data to a mainframe and receives updated information from the mainframe. If the data used by the web application differs too much from the data on the mainframe, the mainframe can't work with the data sent by the web application, and the entire sync breaks down.

Another situation where customer data may be necessary is where the customer's configuration is sufficiently unusual/unique enough that it's not feasible to mimic that configuration in order to reproduce a reported bug. In a previous job I had this happen, where there were four separate bugs that produced the same outcome as the customer's actual bug, and I found all of them before I was able to use the customer's data to reproduce the actual bug.

Scalability issues can be challenging without a database the size of a customer one - if the test database is of a modest size while the customer's data set is over a terabyte, it may be impossible to reproduce that customer's problems with the test database. I've seen this happen, and had to sign the non-disclosure agreements that went with the customer mailing us a copy of their database to work with. In those cases, we agreed to user their data only for as long as we needed it to reproduce and fix the problem they were having. That was acceptable with the PCI compliance rules at the time - I don't know if it would be acceptable now since I'm not currently working with software that requires that standard.

That said, email addresses are always changed, usually to one of our internal email addresses so we can test the emails that should be sent to customers, and purge the database of real users whenever we take a copy of the production database to use for testing.

Answer 4

Principles related to the processing of personal data

Personal data shall be obtained only for one or more specified and lawful purposes, and shall not be further processed in any manner incompatible with that purpose or those purposes.

So as per data protection law applied, production data can’t be processed for ulterior purposes from when it was originally obtained, without explicit permission from the data subject, an unrealistic scenario.

Answer 5

I would not recommend using real customer data for testing. On argument was already mentioned: unlawful use of customer data.

Another problem which can arise: leaking confidential business information. Imagine this scenario: due to a bug or misconfiguration, your test system suddenly sends out real email. Real persons now get a newsletter for a new product your company is developing with critical details.

Better be safe, use dedicated test data with adresses, emails and mobile phone numbers reserved for your testing only.

Answer 6

There are situations that are almost impossible to fully test without live data of some kind. However, we may not need to use customer data. I have tested with my live credit and debit cards when I needed to do an "End to End" test of a system. There are bugs that show up only in such a test. A small amount used in a test can be written off based on how much the customer is paying for quality software (but talk to accounting first).

Answer 7

Others have mentioned the GDPR, but there are also industry specific rules and constraints.

If you’re dealing with payment data, such as credit cards and cardholder data, the PCI DSS has an explicit prohibition on using actual customer data for testing.

Also, the US government has developed draft guidelines on this exact topic: DPIAC Privacy Recommendations on the Use of Live Data 4 in Research, Testing, or Training (pdf file). They are not official yet, but you can consider them a good starting point.

Answer 8

As others have highlighted its definitely a very grey area!

Most companies I've worked for do use real customer data (with or without the customers knowledge!). Mainly because sometimes with complex systems, certain issues are unable to be reproduced with test data. On numerous occasions i have found issues we are unable to reproduce without using real data.

We should always try to at least obfuscate/anonymise the data as best as we can

Answer 9

Is it common to use real data of the customer to perform testing?

Yes, in smaller engineering teams this is very common. They just blindly copy production data into a testing environment. In larger corporates hopefully they have pragmatic internal policies to guide this. Certainly if the data is of sensitive personal nature.

What policies companies apply regarding using real data of the customer for testing purposes?

• A thorough ISO 27001/2 implementation would probably cover how data is available to testers. This article shows depth on the subject, resulting in two conclusions:
  1. Development, testing, and change management require clear written information security policies.
  2. The organization must enforce the policies in all projects and have evidence.

Is there any legislations regarding such issues?

• General Data Protection Regulation: https://gdpr-info.eu/
• The medical and financial industries also have a lot of acts and regulations that might influence this. I think it is much to sum up as each industry has its own standards.

Data could also be owned by someone else than the owner of the software. You might need formal sign off to use real data in a test environment from a client for example.

I would suggest you get legal counsel from someone who understand the domain you are building and testing software in. Together research risks, contracts and the law.