Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I'm a Java developer by trade. I was 'brought-up' in what you could call best practices. Then I took my current job. I had a choice between the Java/SOA team, and the ERP team. I was told that joining the ERP team would give me the best insight into how the business operates (not in terms of technology, but in terms of business.) So I went with ERP.

I found a system with over 4 million lines of "Progress 4GL" code (since renamed to Openedge ABL, because "4GLs sound bad") code, spread out around about 11000 files. The best part is, no file exists more than one folder down. So you have about 50 folders, each with 300-400 files a piece in them. Luckily, many of the files haven't been touched in over 7 years, and many of them are deprecated (but we don't delete them from version control "just in case." Don't even get me started on that one.) So there's really only about 1.5 million lines of code we'd actually have to test. "Only."

I could go on and on about the poor practices of the system. The bottom line was for years, developers had no option to say no. It was "gimme gimme gimme," combined with a lot of contractors moving in and out. Now, they want to clean up their act.

One of the first things I suggested was testing. They said, basically "we've been wanting to do testing for years, but we don't really know where to start." The business logic and database access are baked into the UI. (I'd say GUI, but it's not graphical, it's on a mainframe. Even the order-entry people telnet in.)

So this is pretty much a worst case scenario. Millions of lines of code. Over a billion dollars in annual sales running through the system, so it's not going to get a re-write (obviously "it works as it is.") No object orientation. No formal or automated tests. Our testers are also the ones who write the specs (which biases them toward "pushing the project through.")

I'm getting pretty desperate. I'm even willing to write whatever testing frameworks (there isn't much for Openedge in open source frameworks) we need on my spare time. I'm convinced it will pay for itself quite quickly. Where can we start? Has anyone here come across a similar project (even if smaller in scale) and if so, how did you cope with and overcome this?

Update: I've had a chat with my manager, and I have approval to draw up a spec/timeline for creating a test harness. At first, I heard this: "What we really need to do is write some tests." to which I responded "Having tests won't really help us if we don't have a way to run them. I've been looking into ProUnit (the old name for OEUnit) and think it will work for us." "We should write some tests using it." It took about 10 minutes of repeating this for him to realize it was not as simple as downloading the library and "using it." But, approval to write a spec to create a test harness is a start! Thanks guys for all your input!

share|improve this question
    
sounds like a certain Walldorf company –  xster May 1 '12 at 18:44
2  
@xster To be honest, this is a very common issue. Most people I've talked with have shared similar experiences at at least one place they've worked. –  corsiKa May 1 '12 at 20:14

10 Answers 10

up vote 22 down vote accepted

You are about to embark on a journey without end, so it is the means, not the ends, that are important.

Deciding where to start

  1. Business first: As Laura said, make sure that your work directly improves areas important to the business
  2. Metrics: It is critical to get good metrics both before you start and as you progress. Keep your metrics current, preferably by automation, because this will be your constant guide and justification.
    • As Tangurena said, find areas of higher defect rate. This is a good code smell
    • Static analysis: Can you run any tools such as a sloccount, lint or those that compute complexity (eg McCabe), dependency graph? Even better if there is a Findbugs or equivalent :) Doing so may highlight areas of:
      • Particular problem density
      • Above average complexity
      • High dependency (eg, routines and library functions used by many other functions). The more dependencies or uses of a function, the more higher the impact it has on the system
    • Dynamic analysis: Can you check which subroutines are more commonly called, or even called at all? (maybe by examining patterns in log files or code coverage tool) Testing these will have a higher impact
  3. Decisions: develop, and keep current an analysis which ranks areas based on:
    • The most important modules/function to the business
    • Defect rate/ksloc (per thousand source lines of code) in each module (with different values for criticals, majors, minors, etc)
    • Design/code quality problems/ksloc in each file
    • Number of dependencies on each module/function

Unit test vs. automated GUI/CUI test

Sometimes, especially on code bases not written for testing, writing unit tests is nearly impossible without major refactoring - modules have so many dependencies on other modules, and all depend on running within a 4GL runtime.

When the cost (and risk) involved in refactoring for unit tests is too high, you may want to automate front-end tests rather than build unit tests. While you will not be able to get the code insight of a unit test, nor the very high frequency test-fix cycles, you get tests which are directly relevant to the end-user experience (and thus management happiness).

Automated GUI/CUI test development is also typically undertaken by a different team of developers, not core software engineers, so can actually progress without affecting the core engineering team.

share|improve this answer
    
While I happen to not work at this firm anymore, this is something I will be coming back to any time I need to implement testing on a system. I also like how this applies not only to the OE environment, but to be more useful to future readers, it applies to practically any environment. I like it so much I'm changing this to my accepted answer (not to slight Tangurena!). –  corsiKa May 3 '12 at 17:10

My first idea might be to break the system up into smaller components (not literally just from a planning perspective). From here you can prioritise them based on which are more important and work with those more important sections first. With such a big project this modular approach would help prevent a sense of your team getting no-where and perhaps keep people motivated.

Alternatively if this system has a specification written you can write and perform your tests against those to ensure the more critical components of the system have been tested.

share|improve this answer
1  
Specifications? Ah yes, I remember those. We don't use specs here. I do hope original poster does have specs. If not, document everything. Documentation is the key (to professionalism and maintaining sanity, in this situation, I mean). –  Laura Hensley May 4 '11 at 15:21
1  
I don't get the luxury of spec's here either. Am currently trying to write a set myself and pass them back to the customers to ensure we are doing to the right thing ... seems a little backwards doesn't it. –  Craig Pilgrim May 4 '11 at 15:52
    
We have what are called specs, but they're each one off specs (that often contradict eachother.) I once had a spec that was estimated at 24 hours with one line of "spec." It's test plan was "set up the data, run the function, and compare the results." To be fair, at least the test plan was on par with the spec! –  corsiKa May 6 '11 at 18:40

Yes! Me too! Unstructured you say? You have my deepest sympathies; even so, it sounds like you have more weight to enforce rules and compliance than yours truly in my small but passionate team.

Broadly speaking, this is what I did:

  • Picked a module and began writing test cases for it - shooting for the moon, but still accepting what I managed to document (there were zero test cases when I started so anything is better than nothing)

  • Began my own personal bug list to track and monitor myself, my bugs and customers' bugs (there was/is not a professional defect tracking system where I work; only a barebones homegrown one that does not produce reports => so, no decent defect tracking system=no one had a bug list....that is, until I created one for my own use in Excel)

  • Documenting my own bugs using clarity and as few steps & screenshots as possible since this team was not used to documentation

  • Documenting customer bug fixes and clarifying discrepancies with support personnel when the bugs they created were far from understandable

  • Documenting features/enhancements as described above

  • Sharing knowledge acquired with others so they could benefit (as they could handle it...not everyone wanted to be educated)

I can't stress enough: go to management and get clear directives as to what you should focus/spend time on, and do not deviate from that, no matter how badly you want to. You must earn their respect before trailblazing in this environment.

share|improve this answer

Where can we start?

If you have a bug tracking system, get some sort of report of bugs for the past few years, broken down by module (if that's possible). Then make a histogram/pareto chart to see which are the buggiest areas (each bug counts as +1, each re-opened bug counts as +1, each "that didn't fix it you developer you" counts as +3). Those areas are where you should put your biggest effort into adding tests. Files "untouched for 7 years" won't appear in such a report.

A quick check of the web shows that OEUnit is a unit testing framework for OpenEdge. That's a good start as it shows you don't have to invent such a beast.

Bugs that aren't fixed the first time can either mean sloppy coder, poor specs or issues more complicated than they first appear; that's why they should count a lot higher when trying to triage what area to start focusing on.

share|improve this answer
6  
+1, Determining which areas are the biggest pain point is a great place to start. However, for someone new to the codebase starting in a "simple" area where the code is less complex and the business rules as simple as can be is a good way to get a "feel" for the application, how its been written and what "unwritten conventions" exist throughoug the code. –  Rob May 4 '11 at 15:48
1  
@Rob I've been working with it for about 8 months now. I have acquried a lot of the 'tribal knowledge' so to speak. That said, there are no "simple" areas of this application. Honestly. –  corsiKa May 4 '11 at 19:54
    
I am going to accept this; I hadn't considered the histogram approach, but I think that's a great idea, and it's in line with some of the current policies we have. Also, I had been told by my team mates that OEUnit was insufficient to our task. I followed your link and did some reading on it on my own and believe it will work quite well for me, so thanks for that reminder. I also like the last piece of insight as well. I did give an upvote to all those that responded, because I thought they were all full of good information. Laura's list and bruce's book recommendation are on my todo list. :) –  corsiKa May 6 '11 at 18:39
    
+1 for good answer.. –  Soner Gönül Aug 22 '11 at 12:46

I seem to recall there is a book that specifically addresses this exact scenario. Just a sec while I search Amazon...

...here we go: Working Effectively with Legacy Code

I admit I haven't read this myself, but I've heard good things about it, and it has 4.5 stars.

share|improve this answer
2  
I've got and read the book and would most definitely recommend it –  Phil Kirkham May 7 '11 at 0:20
1  
+1 for this book, it's definitely worth it! –  stuartf May 11 '11 at 17:36
    
This is definitely worth reading and applying. –  Matthew Rodatus May 12 '11 at 11:43
1  
I noticed this book on a colleague's shelf (on another team.) I take it this book deals with more than just testing? –  corsiKa Jun 23 '11 at 17:10
    
This is a great book, but it is more about refactoring code that does not have tests and putting seams in place so that you can add tests. While the book pushes the notion that all code should have automated tests, I dont think this is a great place to start for this particular problem. This will help the developers refactor tightly coupled code, but will not guide you to where to start testing. –  Eric Hexter Jun 26 at 15:27

My first thought, reading your description was to go after where the largest chunck of sales run through the system. Keeping 80% of the dollars flowing correctly is more important than a workflow that maybe accounts for 3% of them.

Since you said the testers also write the specs I'm assuming they have a good idea what business flow/transactions represent that, so start there.

Security also popped into my mind. Since you say "sales" I'm thinking maybe credit cards or bank information is being handled? If so then making sure that is safe would be another area to start.

I also like Tangurena's answer. If you have history or open bugs see if you can find one area(s) that appear more problematic and focus in there first.

Remember... you can't test everything with the time/resources allocated so focus on areas where you get the best return for the investment of your time.

share|improve this answer

Not to negate one anybody else has already said because I think it is all good advice, but the first step I would take is to run one very basic test. That moves you from having a system with zero tests to having a system with an incomplete set of tests that can be improved upon. My experience is that it's easier to complete a job that's already under way than try to figure out where to start.

share|improve this answer
    
taking you answer one step further- "modules" were mentioned in other answers, I would first write down a list of modules and possible relationship between them, then run a simple test for each module, or even preferably a functional test, a simple stress test, a negative test and one performance test. –  Rsf May 8 '11 at 13:03

As well interrogating your bug cases, it might be worth speaking to the Business about which features provide them the most value / money.

What feature would cost them the most money / bad reputation should it stop working.

share|improve this answer

Step 0, Get the build automated when they check into the source control system. TeamCity pro could probably be made to checkout and run a command - it is and it's free and easy to set up.

Step 1, Write one test. Don't get hung up on unit versus intergrtion - just get one test.

Step 2, Think of it as an oppotunity:

  • What is it your business is about?
  • Try writing some tests in an ideal testing language for your domain.
  • Use gerkin style regular expression matching to turn this testing-dsl into reality. (See rspec / specflow for ideas)

I feel your pain, I've just inherited 750k of untested code. It's all about changing mind sets and sometimes it's only seeing that first test when people's jaw and penny drops as they realise how much time this is going to save them.

share|improve this answer
    
I'm working on step 1 right now. Unfortunately, not much of the system lends it self to testing: so much has the database tied directly in it, and the UI. Also, I'm trying to get OEUnit set up for it, but of course that's going to take a day or two of exploration which they've said they're willing to do but I'm still waiting for the actualization of that willingness. I really like your step 2 (+1 for that alone) - I'm trying so hard to stay positive in the situation :-) –  corsiKa Jul 7 '11 at 22:45
    
We've used SQLite to great effect for test databases. The other option is to persuade your application to not do as much work - can you append "TOP 1" in the SQL if you set an inTest flag which could speed up the database access dramatically. –  Squirrel Jul 8 '11 at 5:57
    
Perhaps, but we're using OpenEdge ABL - it's a DSL for tight database integration and it isn't going anywhere any time soon. :-{ –  corsiKa Jul 8 '11 at 15:58

Additionally to Tangurena approach I suggest writing several tests at the highest level possible. Essentially this will be a test that your software works as intended. If most sub-systems are involved in the test, then any error in any of the systems will show a malfunction in that test. It will not pinpoint your problem but will show if some refactoring introduced some bugs.

Basic high level test:

  1. Make a clean setup
  2. Add some users/reports
  3. Do some operations on the users/reports
  4. Save the results
  5. Check the results

Make it run on a server regularly as it will be probably too long for an unit test.

Also get rid of any deprecated file. If you need them just look them up from the version control. Those files add friction to development and make it harder to see what code is actually being executed.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.