Any one have any good recommendations/guideline policies about using production data within testing environments (sensitive vs non-sensitive)?
The amount of data cleaning you need to do will depend on your application and regulatory environment. As a general guideline, any financial data must be scrubbed even if encrypted (if financial data isn't encrypted on your production system that's a whole other issue). Any personally identifiable data should be scrubbed even if encrypted. Personally identifiable data for children may be subject to stricter rules than personally identifiable data for adults.
Things to watch for:
- Data scrubbing isn't just deleting anything sensitive. You'll need to replace it with something that meets validation and other data integrity rules.
- The algorithm used to transform data should be one-way only.
- Once transformed, your data should still be subject to the same business rules and still meet business rule standards. This gets particularly tricky when there is cross-validation being performed (e.g. does the state match the postal code).
- Transformed data used to identify records needs to stay internally consistent. For instance, if a common lookup in your application is by social security number, then User X's social security number must be transformed to the same value in every part of the database it's found. (Yes, I know it's not good practice to use something like SSN as a key, but that doesn't mean you won't find tons of places that do this, my workplace among them).