Does simply running the code during development mean 'Testing'?

Question

I'm a reasonably inexperienced developer, and I recently got into a debate with the manager of our current project.

It's a concept demonstration for trial funded internally by our company. After discussing with the project manager while going over the project plan I'd put together he said we didn't want to put any time into testing. It's not a safety critical system, it's just simply reading and analyzing data, and no one will be making any decisions on it that might affect anything because it will just be a trial concept demonstration, no one is buying it yet. Fair enough.

However, I think my manager (not prior project manager) was getting pressure to prove we're doing what we say we're doing for project review etc, because he started asking us about what testing we were doing and to record "our testing" in a spreadsheet.

There was some confusion. Me and my fellow developer were not comfortable saying we were doing any testing, formally or informally. Our manager seemed to think no testing meant we'd never run any code. We'd used the word 'test' when describing we run the code. You know what it's like, you spend 20% of your time writing some code and 80% of the time trying to work out why it wasn't running properly.

I was arguing this is not testing. My manager was saying this was testing, and if we wrote it down in a spreadsheet with a date we can prove we've done testing in reviews etc.

Cue 2 hours of argument/discussion on what 'testing' means. At the end, I was unwilling to budge and our fundamental disagreement was: When asked if we were doing testing, my answer would be no, and his yes.

I feel very uncomfortable moving from this position, my understanding is that testing has more of a formal definition in software development and saying that simply running the code was 'testing' would be misleading.

It came down to the guy questioning my experience (I've not got a CS degree) and asking if I would benefit of going on a 'course', which makes me even more uncomfortable. We've had fundamental disagreements in the past that we haven't been able to overcome.

Would you consider running code during development testing, or would saying that be misleading?

Edit 09/05/2016

Well, what an overwhelming response, I wasn't expecting that. There seems to be more of a divide than I originally considered, with some agreeing that it shouldn't be called testing and is merely development and some saying it is just very limited testing (smoke testing coming up a lot).

Most agree that it needs to be qualified with at least some more definition, like "smoke testing" or "development testing" to avoid miscommunication.

Thanks everyone for the advice, I think the "Definition of done" that came up will be very valuable in setting expectations. I'm really not sure what answer to mark as "correct", I think I'll go with the consensus and pick corsiKa's, but I'd like to thank Kate Paulk for perhaps the most helpful regarding advice for my situation.

Answer 1

A test is an experiment. You have a hypothesis, which is normally governed by a specification, such as "When I enter a username and password that I know to be valid and click login, I am brought past the login screen and to the dashboard." or "When I log in to the program with 10,000 simultaneous users, performance is no less than 95% of that when there are 1,000 simultaneous users".

All tests start with this hypothesis. The scope of the hypothesis may be limited to a small portion of the program, like a single function or component. It may involve multiple components. It may contain known good data, known bad data, or random data. There's a lot of different things you can shove into this hypothesis. But whatever it is, the hypothesis comes from a part of the product specification.

The next part of the test is the experiment itself. You perform the action and collect the result. This might be a Selenium or JUnit test. It might be a human clicking buttons. It might be a batch job that does routine checks. But someone has to perform the action and collect observations.

Then the observation is compared to the original hypothesis to determine if it passes. If you can't tell whether or not it passes, then you need to refine the criteria. This is why things like "fun" are so hard to test for.

So if you don't have a formal hypothesis and a formal comparison of observations to that hypothesis, then sorry, in my books you're not doing testing.

As a former developer, I can tell you a good rule of thumb:

If you're trying to make it work, you're developing.
If you're trying to make it break, you're testing.

Answer 2

That is really up to your organization and your relationship with your stakeholders.

In most cases, testing is just a label with no specific meaning. The ultimate goal is to deliver a high-quality product. "Testing" is just a label for a kind of activity that might help you reach that goal. You can define that label however you want, but it is your responsibility to deliver something that meets your quality goals.

That said, your arrangement with your stakeholders (the people who will interact with your software) may stipulate how and/or when testing should take place. If you have that kind of arrangement, you will need to use those stipulations to decide whether running software on your desktop meets their definition of testing.

Answer 3

TLDR; The specific terminology that probably describes what he is asking for is smoke testing.

The long version: You are arguing with your manger instead of working towards a solution. Terminology is important, but so is communication and getting the job done.

You haven't done unit or integration testing, you probably haven't done any performance or usability testing, there are all kinds of specific test that you haven't done -- forget about all of them.

Focus on what is being asked, and not the terminology used. Your manager wants you to document at least some of what you have verified about what you have built. What have you verified that can usefully be reported?

Write something up that gives him what he needs, if there is a specific terminology for what he is asking for, introduce him to it, if there's not, and you are uncomfortable using the word "tested", then use another word that means what he wants. Call it verified, examined, checked, inspected, scrutinized or Fred. Whatever works and makes it clear that you didn't just bang on the keyboard until your hands got tired and then call that done.

Finally, get a clear statement of what "done" means, both for the overall project and for the documentation that he wants from you.

Answer 4

Without testing, how would you know it works right? And if it does not have to work right, why would you spend resources on it? When it passed tests, it means is is "DONE" - and you need to agree what "DONE" means.

Testing is the difference between Wally (from Dilbert cartoon) saying "it worked for me, my test file in my browser did not crashed" and saying "we can expect it will produce correct results for any input by any of our customers".

But the "testing" you did was developing. Code might work for other people processing different input files, or might not. Testing (by other people using different inputs and possibly different workflow) will determine if code is "done".

Multiple Levels of Done - you need to find out what is your boss' definition of done.

Good testing costs resources and time. If you can release "beta version", your customers will test it for you, give you input files which will break your assumption on input, etc (for no cost to your boss), but there is price: your customers will see failures of your code. It is business decision if it makes business sense to spend resources to present better products to your customers. I.e. first version of Windows which was usable was 3.1, previous versions were junk for beta testers.

It could be a perfectly valid approach to release beta version (without spending too much resources on it), so customers can see the product and try it in "beta" - if they have no expectations for it to work, produce valid input etc ("release early, release often"). It allows you to face customer's expectations before lots of resources were spent on something what customers don't care about. Talk to your boss, we have no idea what s/he wants.

As a side note, I would never argue with a boss for two hours like that. Instead, I would ask for time-out to research the differences. It might be that you have different definitions of "done", and your boss' definition might be perfectly fine for the product you work on. Getting some training in "agile development" would be beneficial for you - google is your friend.

Answer 5

Testing is validating a situation on a set of conditions. In your case you validate if it runs, you could say "I am going to test if it builds". This would mean you are actual testing something.

From a SDLC perspective you are actual just coding and checking if you think it works and is good enough, this it not testing. Testing is actually a bit more structured and hopefully a repeatable way of validating the same. Often it is also a different phase in the life-cycle, unless it is test-driven development or an other automated testing effort, which would be exercised parallel with developing code. Test (case) design could be done in parallel, but not by one person.

In your-case if management wants to really minimize on testing efforts describe which steps have to be taking to release a feature to a production environment in a definition of done. This definition should include that it builds and that the happy path of feature is executed to verify it is complete. I guess some developers could build systems without ever running it, but the time of punch-cards is far behind us. You could skip testing efforts like automated, exploratory and or usability testing.

Agree on a definition of done! Minimize it for now and extend it in the future when you run into trouble.

I get the feeling you are prototyping as you say the following in a comment:

the point of the project is to develop something and see if it's a viable product

Be aware that because you are taking shortcuts (e.g. skipping testing) you are creating technical debt. If the product is viable it might need a clean and well tested reimplementation, because taking shortcuts now will mostly lead to a product that is unmaintainable in the long run. Discuss technical debt with your stakeholders so they have a honest view on the future. Selling your minimal testing efforts as testing by placing them in a sheet with a timestamp sounds as lying to yourself and to your stakeholders and clients. Take technical debt as a serious issue. Be warned! :)

Answer 6

The TL;DR answer: No.

The explanation:

Running code during development, on the developer's machine, is simply iterative development.

Running it on another system (one that does not have the development environment and does not compile the code first) can be considered smoke-testing.

That said, here's what I'd suggest you do, given that your application is a proof-of-concept trial (and my experience with these is that a distressing number of them become production code far too quickly):

• Before you do anything else, ask your manager what level of testing he expects. You want to make sure that both of you are using the same definitions or you'll be talking past each other. At this point, it matters less that the definitions are correct than that you are both communicating (which is a bad position to be in, but that's life).
• You should also ask your manager why there has been a change from the original "no need to test because this is a proof of concept". That will give you a better idea of why your manager is insisting you document some form of testing.
• If (as I suspect) there is a requirement from further up the food chain that proof of testing be shown, I'd recommend that your proof of testing be something like the following:
  • List installation version (i.e. you're running the application on a different machine) and the actions performed. Call this a smoke test.
  • For each listing in your spreadsheet, enter the modules you covered. So, for instance, you logged in and added a user, then logged out and logged in as that user. Your smoke tests covered login and user management.
  • Have a separate listing of all modules in the application and all functions within each module. Yes, it's work, but I suspect you're going to need a certain amount of covering your anatomy here.
  • Include a list of what you have not tested, and why (the "why" in your case is going to be "no time"). This list would cover the functions you expect to receive little use, the error handling (which is out of scope for smoke tests), exotic inputs, and so forth. If you're using anything resembling an agile process, make sure to be explicit that your tests cover only the happy path or steel thread - the minimum set of functionality that makes the application viable with correct inputs
  • Also include in your documentation that you're making no guarantees about what will happen if the application is used in any way outside your happy path testing. In my experience caveats like this get ignored, but they're useful to point to when the blamestorming starts.
• Finally, good luck. What I'm seeing in your question suggests that the politics in your workplace aren't the best. That's never fun to deal with.

Answer 7

Your problem is very simply a confusion of semantics. The word "testing" is too broad, hence allowing wildly different interpretations. It just makes no sense to talk about "testing" without saying what you actually mean.

Someone mentioned "definition of done". This is what you are looking for; it is a contract between you and your customer. The DoD can be as fuzzy or specific as you (and your customer, even if it is internal to your company) like. Part of the DoD is a metric of when the software is considered to be "correct". As long as you and your customer both agree to it, and your code fulfills it, you are in the clear.

In your scenario, the metric of your DoD could be: "the compiled program can be started manually with a trivial input on the developers laptop". For a DoD, that is perfectly acceptable. Everybody will know that the software will likely fail immediately when confronted with the real world, but for a prototype/proof of concept that may just be fine.

Answer 8

Developing without doing unit testing is, in my experience, similar to taking a used car for a test ride, without taking a look at the engine or any other components. Yes, you might make a long ride and experience no issues, but are you confident enough that this car is worth paying for?

If this is how you do your development, it's entirely possible that iteratively running code during your development cycle might be enough to cover a great portion of issues, without writing tests explicitly, for a smaller project. However, in my experience, doing this alone is not enough to ensure the lowest amount of production issues you can ensure as a developer.

So, again, in my personal experience:

1. Not doing unit testing while coding (not necessarily TDD, but just testing tiny chunks of code while you code) has a couple of issues.

   By "running and seeing what happens" you observe the system as a "black box", evaluating its quality through the UI (or whatever results you get from the program).

   Perhaps you write a chunk of code, then try to build, and repeat until you get rid of all the syntax errors. Then you try some inputs and it looks promising. But then you try some other, and it doesn't work - it's time to debug!

   Depending on your skills, this might be fun, it's like solving puzzles throughout the day. So, this tiny bug you found in a certain function just happened to manifest itself under right conditions, but the underlying problem is that this tiny function doesn't do what it's supposed to do.

   Unit testing tries to catch these errors in functions while you write them, even if those errors would be very hard to provoke in very rare conditions, and that's what adds confidence in your code to your manager and stakeholders.

   It's also efficient - it's very easy to prove a function is doing what it's supposed to do using a couple of quick tests. It also promotes writing code which is easily testable, which is generally as purely functional as possible, with as little state as possible. These functions are easy to test, and you can rest assured they will work in all conditions.

2. What you are doing while developing is a sort of a manual integration testing. But not doing automated unit + integration testing means you are risking regression bugs in your commits, and making it harder to refactor your code or safely introduce changes.

   If your CI server is running a bunch of tests on each commit, and they are all green, it adds confidence in your code.

   Refactoring the code is much easier too. Needing to rewrite a part of an app is not so scary if you know there are automated integration tests which will tell you something is broken.

Other types of testing (functional tests, like testing user stories, or performance testing) seem to me like obviously separate phases of the development, and you can hardly argue they can be done implicitly while programming.

Answer 9

I run into this exact problem a surprising amount. People in various corners of the business use terms like "data" and "testing" ambiguously without themselves really understanding what they're asking when they utter these terms.

So when we're constructing some throw-away proof of concept, our development plan may not include any period of formal functional testing (because it's not necessary), or regression testing (because there's no prior version to compare with) or even writing regression tests (because there's not going to be a future version, either^†).

But of course this doesn't mean we never actually ran the code. Basic developer-level tests are an inherent part of programming. If you'd never "tested" what you wrote, you would literally be programming blind. At its most extreme, this approach would mean not even performing a check that your code compiles, and in my experience getting something to compile first time is very rare. So obviously you are going to be doing some amount of testing of your code before you announce that it works and is ready for demonstration.

But managers don't always get this. Your project manager will say "why did you spend time on testing? I said not to". And then when you decide "okay I won't count that as 'testing' because it was just basic development sanity checks" your line manager comes to you wondering why — and how — you haven't done any testing at all.

At this point I find it's best to qualify the word "testing" by replying with a sentence. Something like:

We haven't done a formal test run (and aren't planning to because it's a Proof of Concept), but my basic development-level tests show that it works well enough for the demo tomorrow.

It's worth noting that since your line manager asked you for a spreadsheet of test results, that suggests to me that you actually are expected to do some formal testing. In that case, he is actually disagreeing with the project manager and you should refer them to each other to figure out what they want your team to do.

^{† Hahaha yeah right. Proofs of concept always end up evolving into real products. Management always ruins things by making that happen. Nowadays I try not to cut corners even on a proof of concept, because that's the best way to give yourself a tremendous ton of technical debt on the first version of your product.}

Answer 10

Manual Testing is a large area, I'd break it down into:

• Unit testing
  Do basic functions achieve their purpose when used by the user?

• Integrated Testing
  Are related services and datastores updated and communicated with correctly?

• Performance Testing
  How long do responses take? How many users can be supported simultaneously (load)

• Exploratory Testing
  Does the system work from a real users perspective?

These are similar to the 4 quadrants of Agile testing, note that the tests could be manual or automated, the areas and need are similar.

Answer 11

the question is: does running code equal testing?

if it means launching the code and to observe it runs and loads the page or initial screen, yes, it would be considered a test - a smoke test if you will.

but there's more to testing than just a smoke test. as others have mentioned, you would do justice to the project by performing other types of test e.g. functional test or even non-functional tests. There's UI or API testing. Testing against the sw requirements or user stories would be a good place.

from what I gather, the project needs at least a Test Lead, snr test analyst or better yet a Test Manager! Then you will have a test process, defect management, test design, test analysis, test strategy & test closure.

Answer 12

You'd have to be nuts to go along with your "manager"'s approach to "testing". I write lots of complex code, and I can certainly go along with holding off on testing for a while -- at a prototype stage, clarifying requirements, data formats, type specifications, and so forth, are a more productive use of time (arguably). Testing is not magic and it doesn't prove code is well-designed or well-implemented -- but it does serve a purpose, especially when (for example) there are multiple projects pursuing a similar problem space, so passing most of a test suite becomes a goal for a project to aim for (it's a way of clarifying which projects are at a more experimental stage and which are close to being usable). When designing your own test suite (rather than using a pre-established one, say for a project porting something to a new platform, etc.), testing can send a similar signal -- its a sign that your code, classes, types, interfaces, and so forth, are clearly established enough that unit (and other kinds of) tests are worth doing. Your project may not be at that stage, probably by design. Testing may simply be unnecessary under the current circumstances.

But you can't use the fact that the English word "testing" is overloaded, as an excuse to misrepresent your actions. No responsible programmer would call "running code to make sure it works" testing in the professional sense, even though it sort of fits the word "testing". No one in a software development context thinks that "testing" would refer to, say, taking a quiz in school. Words have meanings in context, and "testing" means something specific in the context where you're using it. To call plugging info into a spreadsheet, as you've described it, a record of "tests", would be to exploit a linguistic accident for nefarious or at least irresponsible ends, which might be a necessary evil when, say, negotiating a peace treaty, but not in the genteel world of computer professionals. "Stakeholders" don't get to change the meanings of words.

As a programmer, I find the suggestion that you go "take a course" to be offensive -- if anyone needs a refresher course, it's your manager, sounds like. To misquote "Come Out Ye Black and Tans", "Get out of here and take your bloody spreadsheets". It seems he's slouching off on you his own responsibility to push back against the pressure to demonstrate testing being done in the first place. There are professional standards that trump the chain of command, and you can't avoid them by making stuff up. Either you to your manager, or him to his boss, or somewhere up the line, someone has to say that you as developers are bound by these standards, and no-one has the right to pressure you otherwise. I obviously don't know much about your workplace -- whether you're unionized, and so forth -- but you're describing abusive treatment which shall not be tolerated.

I hope you continue to use forums like this to describe the situation if it does not improve. There's been more and more stories about improper treatment of programmers in professional settings, often driven by supervisors who are not sufficiently well-informed technically (I hate to imply that that's somehow a failing, but lack of requisite knowledge does become an issue when it causes someone to behave unreasonably from a position of authority). This kind of thing belies the notion -- which is unfortunate, because it's probably not wholly wrong -- that the IT world is disproportionately civic and progressive. ("Anytime anyone upsets the status quo / You're stabbed in the back by the so-called liberal wh**es" -- Black 47 [thanks to a responder for observing that a direct quote of these lyrics involves an offensive term -- part of the point in the original context, but ...]) We as programmers need a better infrastructure to stand up to such treatment, and your case could symbolically serve a demonstration of a community of computer professionals becoming more attuned to collective mutual support, and more militant in requiring a rational, scientifically literate, and respectful workplace.

"Stakeholders" don't have an a priori right to their (hugely profitable) IT universe, and the privilege of commercial success is earned by respecting the scientific methods and knowledge which coders and researchers have achieved. Stakeholders (in tech settings, anyhow) only exist because developers agree to produce code, and we always do so provisionally. Sounds like you're dealing with some bureaucrats who need to be reminded of that.

Answer 13

When you use a very general and generic word like testing and try to tightly define it you'll have those sort of "discussions" ;) Everyones opinions are worthy of respect and consideration and usually have a basis from experience. So move 'down' to find out whether you are talking about:

• Testing bits of code you are writing
• Manual testing as you develop
• Writing unit tests as you develop
• Writing Integration tests
• Automated User Acceptance tests
• Manual User Acceptance tests

All of these may be part of your 'testing' but it will depend on the details of your situation as to which is important, of value to your company and included in your current process of 'testing'.

I would consider going over these categories with dev, QE and business to get everyone on the same page and avoid assumptions and needless arguments about the definition of [whatever].