142

I'm a Java developer by trade. I was 'brought-up' in what you could call best practices. Then I took my current job. I had a choice between the Java/SOA team, and the ERP team. I was told that joining the ERP team would give me the best insight into how the business operates (not in terms of technology, but in terms of business.) So I went with ERP.

I found a system with over 4 million lines of "Progress 4GL" code (since renamed to Openedge ABL, because "4GLs sound bad") code, spread out around about 11000 files. The best part is, no file exists more than one folder down. So you have about 50 folders, each with 300-400 files a piece in them. Luckily, many of the files haven't been touched in over 7 years, and many of them are deprecated (but we don't delete them from version control "just in case." Don't even get me started on that one.) So there's really only about 1.5 million lines of code we'd actually have to test. "Only."

I could go on and on about the poor practices of the system. The bottom line was for years, developers had no option to say no. It was "gimme gimme gimme," combined with a lot of contractors moving in and out. Now, they want to clean up their act.

One of the first things I suggested was testing. They said, basically "we've been wanting to do testing for years, but we don't really know where to start." The business logic and database access are baked into the UI. (I'd say GUI, but it's not graphical, it's on a mainframe. Even the order-entry people telnet in.)

So this is pretty much a worst case scenario. Millions of lines of code. Over a billion dollars in annual sales running through the system, so it's not going to get a re-write (obviously "it works as it is.") No object orientation. No formal or automated tests. Our testers are also the ones who write the specs (which biases them toward "pushing the project through.")

I'm getting pretty desperate. I'm even willing to write whatever testing frameworks (there isn't much for Openedge in open source frameworks) we need on my spare time. I'm convinced it will pay for itself quite quickly. Where can we start? Has anyone here come across a similar project (even if smaller in scale) and if so, how did you cope with and overcome this?

Update: I've had a chat with my manager, and I have approval to draw up a spec/timeline for creating a test harness. At first, I heard this: "What we really need to do is write some tests." to which I responded "Having tests won't really help us if we don't have a way to run them. I've been looking into ProUnit (the old name for OEUnit) and think it will work for us." "We should write some tests using it." It took about 10 minutes of repeating this for him to realize it was not as simple as downloading the library and "using it." But, approval to write a spec to create a test harness is a start! Thanks guys for all your input!

Improve this question
6
  • 2
    sounds like a certain Walldorf company
    – xster
    Commented May 1, 2012 at 18:44
  • 4
    @xster To be honest, this is a very common issue. Most people I've talked with have shared similar experiences at at least one place they've worked.
    – corsiKa ♦
    Commented May 1, 2012 at 20:14
  • 2
    For readers wondering why someone would use such bad language: Back in '90 Progress 4GL was the most productive environment for client-server programming, loved by developers (won productivity award every year).
    – Peter M. - stands for Monica
    Commented Aug 13, 2014 at 14:31
  • 1
    It was before object programming became all the rage. 4GL had (non-SQL) database access integrated in language, no need to cobble together SQL statements, execute them, and pry out data. It was before object-relational mappers. And app language was able to distinguish between (changed) value on the screen, in buffer, and stored on disk. Transaction processing integrated in language. Stuff like ON ERROR UNDO, RETRY in batch. Very nice. Warm feelings.
    – Peter M. - stands for Monica
    Commented Aug 13, 2014 at 14:32
  • 1
    I was looking around and found this project that you can use to test a Telnet GUI in Python: sourceforge.net/projects/three-t-py . Although, I would lean towards avoiding GUI testing and favoring unit tests.
    – djangofan
    Commented Mar 14, 2015 at 15:55

13 Answers 13

Reset to default
74

You are about to embark on a journey without end, so it is the means, not the ends, that are important.

Deciding where to start

  1. Business first: As Laura said, make sure that your work directly improves areas important to the business
  2. Metrics: It is critical to get good metrics both before you start and as you progress. Keep your metrics current, preferably by automation, because this will be your constant guide and justification.
    • As Tangurena said, find areas of higher defect rate. This is a good code smell
    • Static analysis: Can you run any tools such as a sloccount, lint or those that compute complexity (eg McCabe), dependency graph? Even better if there is a Findbugs or equivalent :) Doing so may highlight areas of:
      • Particular problem density
      • Above average complexity
      • High dependency (eg, routines and library functions used by many other functions). The more dependencies or uses of a function, the more higher the impact it has on the system
    • Dynamic analysis: Can you check which subroutines are more commonly called, or even called at all? (maybe by examining patterns in log files or code coverage tool) Testing these will have a higher impact
  3. Decisions: develop, and keep current an analysis which ranks areas based on:
    • The most important modules/function to the business
    • Defect rate/ksloc (per thousand source lines of code) in each module (with different values for criticals, majors, minors, etc)
    • Design/code quality problems/ksloc in each file
    • Number of dependencies on each module/function

Unit test vs. automated GUI/CUI test

Sometimes, especially on code bases not written for testing, writing unit tests is nearly impossible without major refactoring - modules have so many dependencies on other modules, and all depend on running within a 4GL runtime.

When the cost (and risk) involved in refactoring for unit tests is too high, you may want to automate front-end tests rather than build unit tests. While you will not be able to get the code insight of a unit test, nor the very high frequency test-fix cycles, you get tests which are directly relevant to the end-user experience (and thus management happiness).

Automated GUI/CUI test development is also typically undertaken by a different team of developers, not core software engineers, so can actually progress without affecting the core engineering team.

Improve this answer
1
  • 1
    While I happen to not work at this firm anymore, this is something I will be coming back to any time I need to implement testing on a system. I also like how this applies not only to the OE environment, but to be more useful to future readers, it applies to practically any environment. I like it so much I'm changing this to my accepted answer (not to slight Tangurena!).
    – corsiKa ♦
    Commented May 3, 2012 at 17:10
73

Where can we start?

If you have a bug tracking system, get some sort of report of bugs for the past few years, broken down by module (if that's possible). Then make a histogram/pareto chart to see which are the buggiest areas (each bug counts as +1, each re-opened bug counts as +1, each "that didn't fix it you developer you" counts as +3). Those areas are where you should put your biggest effort into adding tests. Files "untouched for 7 years" won't appear in such a report.

A quick check of the web shows that OEUnit is a unit testing framework for OpenEdge. That's a good start as it shows you don't have to invent such a beast.

Bugs that aren't fixed the first time can either mean sloppy coder, poor specs or issues more complicated than they first appear; that's why they should count a lot higher when trying to triage what area to start focusing on.

Improve this answer
3
  • 9
    +1, Determining which areas are the biggest pain point is a great place to start. However, for someone new to the codebase starting in a "simple" area where the code is less complex and the business rules as simple as can be is a good way to get a "feel" for the application, how its been written and what "unwritten conventions" exist throughoug the code.
    – Rob
    Commented May 4, 2011 at 15:48
  • 1
    @Rob I've been working with it for about 8 months now. I have acquried a lot of the 'tribal knowledge' so to speak. That said, there are no "simple" areas of this application. Honestly.
    – corsiKa ♦
    Commented May 4, 2011 at 19:54
  • I am going to accept this; I hadn't considered the histogram approach, but I think that's a great idea, and it's in line with some of the current policies we have. Also, I had been told by my team mates that OEUnit was insufficient to our task. I followed your link and did some reading on it on my own and believe it will work quite well for me, so thanks for that reminder. I also like the last piece of insight as well. I did give an upvote to all those that responded, because I thought they were all full of good information. Laura's list and bruce's book recommendation are on my todo list. :)
    – corsiKa ♦
    Commented May 6, 2011 at 18:39
37

I seem to recall there is a book that specifically addresses this exact scenario. Just a sec while I search Amazon...

...here we go: Working Effectively with Legacy Code

I admit I haven't read this myself, but I've heard good things about it, and it has 4.5 stars.

Improve this answer
4
  • 3
    I've got and read the book and would most definitely recommend it
    – Phil Kirkham
    Commented May 7, 2011 at 0:20
  • This is definitely worth reading and applying.
    – Matthew Rodatus
    Commented May 12, 2011 at 11:43
  • 1
    I noticed this book on a colleague's shelf (on another team.) I take it this book deals with more than just testing?
    – corsiKa ♦
    Commented Jun 23, 2011 at 17:10
  • 1
    This is a great book, but it is more about refactoring code that does not have tests and putting seams in place so that you can add tests. While the book pushes the notion that all code should have automated tests, I dont think this is a great place to start for this particular problem. This will help the developers refactor tightly coupled code, but will not guide you to where to start testing.
    – Eric Hexter
    Commented Jun 26, 2014 at 15:27
18

Yes! Me too! Unstructured you say? You have my deepest sympathies; even so, it sounds like you have more weight to enforce rules and compliance than yours truly in my small but passionate team.

Broadly speaking, this is what I did:

  • Picked a module and began writing test cases for it - shooting for the moon, but still accepting what I managed to document (there were zero test cases when I started so anything is better than nothing)

  • Began my own personal bug list to track and monitor myself, my bugs and customers' bugs (there was/is not a professional defect tracking system where I work; only a barebones homegrown one that does not produce reports => so, no decent defect tracking system=no one had a bug list....that is, until I created one for my own use in Excel)

  • Documenting my own bugs using clarity and as few steps & screenshots as possible since this team was not used to documentation

  • Documenting customer bug fixes and clarifying discrepancies with support personnel when the bugs they created were far from understandable

  • Documenting features/enhancements as described above

  • Sharing knowledge acquired with others so they could benefit (as they could handle it...not everyone wanted to be educated)

I can't stress enough: go to management and get clear directives as to what you should focus/spend time on, and do not deviate from that, no matter how badly you want to. You must earn their respect before trailblazing in this environment.

Improve this answer
11

My first idea might be to break the system up into smaller components (not literally just from a planning perspective). From here you can prioritise them based on which are more important and work with those more important sections first. With such a big project this modular approach would help prevent a sense of your team getting no-where and perhaps keep people motivated.

Alternatively if this system has a specification written you can write and perform your tests against those to ensure the more critical components of the system have been tested.

Improve this answer
3
  • 1
    Specifications? Ah yes, I remember those. We don't use specs here. I do hope original poster does have specs. If not, document everything. Documentation is the key (to professionalism and maintaining sanity, in this situation, I mean).
    – Laura Hensley
    Commented May 4, 2011 at 15:21
  • 1
    I don't get the luxury of spec's here either. Am currently trying to write a set myself and pass them back to the customers to ensure we are doing to the right thing ... seems a little backwards doesn't it.
    – Craig Pilgrim
    Commented May 4, 2011 at 15:52
  • We have what are called specs, but they're each one off specs (that often contradict eachother.) I once had a spec that was estimated at 24 hours with one line of "spec." It's test plan was "set up the data, run the function, and compare the results." To be fair, at least the test plan was on par with the spec!
    – corsiKa ♦
    Commented May 6, 2011 at 18:40
10

My first thought, reading your description was to go after where the largest chunck of sales run through the system. Keeping 80% of the dollars flowing correctly is more important than a workflow that maybe accounts for 3% of them.

Since you said the testers also write the specs I'm assuming they have a good idea what business flow/transactions represent that, so start there.

Security also popped into my mind. Since you say "sales" I'm thinking maybe credit cards or bank information is being handled? If so then making sure that is safe would be another area to start.

I also like Tangurena's answer. If you have history or open bugs see if you can find one area(s) that appear more problematic and focus in there first.

Remember... you can't test everything with the time/resources allocated so focus on areas where you get the best return for the investment of your time.

Improve this answer
8

Step 0, Get the build automated when they check into the source control system. TeamCity pro could probably be made to checkout and run a command - it is and it's free and easy to set up.

Step 1, Write one test. Don't get hung up on unit versus intergrtion - just get one test.

Step 2, Think of it as an oppotunity:

  • What is it your business is about?
  • Try writing some tests in an ideal testing language for your domain.
  • Use gerkin style regular expression matching to turn this testing-dsl into reality. (See rspec / specflow for ideas)

I feel your pain, I've just inherited 750k of untested code. It's all about changing mind sets and sometimes it's only seeing that first test when people's jaw and penny drops as they realise how much time this is going to save them.

Improve this answer
3
  • I'm working on step 1 right now. Unfortunately, not much of the system lends it self to testing: so much has the database tied directly in it, and the UI. Also, I'm trying to get OEUnit set up for it, but of course that's going to take a day or two of exploration which they've said they're willing to do but I'm still waiting for the actualization of that willingness. I really like your step 2 (+1 for that alone) - I'm trying so hard to stay positive in the situation :-)
    – corsiKa ♦
    Commented Jul 7, 2011 at 22:45
  • We've used SQLite to great effect for test databases. The other option is to persuade your application to not do as much work - can you append "TOP 1" in the SQL if you set an inTest flag which could speed up the database access dramatically.
    – Squirrel
    Commented Jul 8, 2011 at 5:57
  • Perhaps, but we're using OpenEdge ABL - it's a DSL for tight database integration and it isn't going anywhere any time soon. :-{
    – corsiKa ♦
    Commented Jul 8, 2011 at 15:58
6

Not to negate one anybody else has already said because I think it is all good advice, but the first step I would take is to run one very basic test. That moves you from having a system with zero tests to having a system with an incomplete set of tests that can be improved upon. My experience is that it's easier to complete a job that's already under way than try to figure out where to start.

Improve this answer
1
  • taking you answer one step further- "modules" were mentioned in other answers, I would first write down a list of modules and possible relationship between them, then run a simple test for each module, or even preferably a functional test, a simple stress test, a negative test and one performance test.
    – Rsf
    Commented May 8, 2011 at 13:03
6

As well interrogating your bug cases, it might be worth speaking to the Business about which features provide them the most value / money.

What feature would cost them the most money / bad reputation should it stop working.

Improve this answer
2

Additionally to Tangurena approach I suggest writing several tests at the highest level possible. Essentially this will be a test that your software works as intended. If most sub-systems are involved in the test, then any error in any of the systems will show a malfunction in that test. It will not pinpoint your problem but will show if some refactoring introduced some bugs.

Basic high level test:

  1. Make a clean setup
  2. Add some users/reports
  3. Do some operations on the users/reports
  4. Save the results
  5. Check the results

Make it run on a server regularly as it will be probably too long for an unit test.

Also get rid of any deprecated file. If you need them just look them up from the version control. Those files add friction to development and make it harder to see what code is actually being executed.

Improve this answer
1

Make it clear: Unit testing is no longer an optional task.

Period

All code changes now get tests, new or updated as appropriate. The first ones will be REALLY hard as there are so many hurdles at the beginning and the learning curve is steep. So maybe allow a month or six to get everyone ready for this with training and presentations and coaching and tutoring.

To be this severe (all code now requiring tests) you'll first have to work with several levels and parts of the organization to get folks ready and trained with the resources and knowledge they need to make such a big change.

This sort of change can easily take several months to get going and one to three years to pick up steam, so management need to be in it for the long haul and have their expectations set. The result is a modern company that competes well and wins. The alternative is going out of business.

Phase 1 - Start introducing the testing

Step 1) Write a true is true test.
Step 2) Write a 'calling a simple method call in app code works' test
Step 3) Write a unit test for a simple method that mocks and stubs any dependencies

Phase 2 - Apply the testing to actual development

For all code changes going forward:

Step 1) Write tests that are committed with the code
Step 2) Write failing tests before writing app code
Step 3) Write tests for the backlog of missing tests

For all of this make the pain obvious along with the advantages. Use a CI server to run your branches and tests in. Like circleCI for example. Expose the innards openly to improve them.

"Having tests won't really help us if we don't have a way to run them. " is really just one of dozens of excuses by people who have not been convinced that testing is a truly great thing - for developers!

Also, lets be honest, a developer MUST be running their code somehow to make sure the happy path is actually working without syntax errors. No-one I've ever seen writes code 'blind'. testing alreadys takes place, it just hasn't been formalized and turned into automation yet is likely the case.

I myself used to write code without automated tests (you can best I tested it a lot mind you), millions of lines of it. Changing it was terrifying! Tests are a developers best friend. I mean seriously, how can you confidently refactor code - the secret to quality - if you don't have tests to protect you? The incentive to improve something, but risk breaking something else means crap left everywhere in my experience. Once one window is broken, the house gets smashed up fast (broken window theory).
Changing peoples mind by persuading them with arguments often fails so setting the right example and getting supporting collaborators is key.

Improve this answer
0

The good thing to start with is with the requirement document which will help first getting the manual test done then other types regression, load and other types

Which will dedicatedly leads towards better quality of product

Improve this answer
1
  • 3
    Your answer - while accurate - doesn't really add anything to the answers that are already here. The other answers have mentioned requirements documents, manual tests, and potential strategies for this problem. As a rule, a new answer to an old question should be adding something to the answers already present, especially if there is an accepted answer
    – Kate Paulk
    Commented Oct 31, 2014 at 14:19
0

Adding to craig-pilgrim's answer: if you cannot fracture your monolith into smaller functions or microservices, consider the option of each new feature being added as a new function or microservice. Obviously, each new function would be fully tested in whatever test framework you end up using.

After a few months of that new paradigm, you can reassess the possibility of splitting up the monolith.

It is not a quick solution, but it will eventually get you to a more testable setup.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.