10

I'm a computer science master's student and I need to write a research proposal for my software testing class.

I need to find a problem in the software testing field that hasn't been solved yet and one that prevents engineers or researchers from being as effective at testing than they otherwise could be.

As I don't work in this field, it's been tough finding a problem or challenge that no one has been able to solve. So I'm wondering, are there any problems in software testing in which testers or researchers have yet to come up with a solution?

  • 3
    Are you looking for an unsolved problem that you can solve this semester? – user246 Mar 17 '17 at 19:09
  • 3
    @PeterMasiar Random people come up with new idea's all the time that change the world. Not sure how your feedback is helping here at all. It is just negative. – Niels van Reijmersdal Mar 17 '17 at 19:34
  • 1
    @user246 that's basically what my professor is asking, and it seems ridiculous to me. – rjw0924 Mar 17 '17 at 20:09
  • 1
    @NielsvanReijmersdal - yes, but: In a semester? As condition of graduation? I think I am being realistic. Professor is asking for something else, or something what just very few Master-level students can deliver. – Peter M. - stands for Monica Mar 17 '17 at 21:50
  • 2
    I'm not sure if this is a valid question. You've marked an answer (from someone I personally find has wonderfully helpful answers in general) so it's nice that you got some help, but can you truly say that this answers the question? I believe this is too open ended of a question - it's almost akin to "what's the best color" if that makes sense. I'm not VTC'ing because it seems most of the community feels otherwise, but I am concerned about it. – corsiKa Mar 19 '17 at 1:20
7

In a world of continuous deployment test automation is becoming the most important form of testing.

The biggest problem with test automation is how do the engineers know they have covered all the important execution paths. There exists code coverage, but that just finds untested code, it is not a quality target.

If you combine code coverage with mutation testing it becomes better, but mutation testing is still too slow for large codebases. Also processing the results requires manual research. It is not repeatable continuously for these reasons.

There is something called formal verification, but this seems to be so time consuming that I wonder if it is realistic for most software projects.

For me the biggest challenge is how do my engineers know they have written enough tests, while spending their time balanced between testing and coding. There is missing a good standard and most good techniques are currently to slow or to complex for complex systems. Coming up with a process that is automatable, (relatively) fast, lowers risks of defects significantly and which is repeatable seems like a great win for our industry.

You could discuss that such a process already exists, haven't people been testing software for ages? Sure, but a good testing process is slow as hell. The future needs something fast and furious. Test-automation helps, but still it is hard to know if you covered everything. Some help here would be great.

The second biggest problem with software quality is schooling:

  • Engineers are not taught deep test-automation skills in school. Designing software that is testable and maintainable is nearly an art form.
  • Engineers are not taught quality principles like clean code.

I understand schools try to give students broad knowledge, but writing high quality code should be a standard that starts in your first weeks. So that you can practise it in your practical assignments over and over again. Unit-testing for example can not be a couple of classes to explain the basics. You need good understanding of the reasoning and how to structure your code. Then you also need a lot of practise and a larger project to make you see that test-automation is worth your while.

Maybe a new programming paradigm which includes code quality and testability as it primary focus.

  • 1
    One could argue that test driven development is the approach for this. I found that usually when development designs the test before coding with the architecture as a whole in mind and QA reviews it and then programming starts it cuts down on the number of issues later in the life cycle and drastically decreases the test cycle at the end. The issue I have found is most projects just don't want to invest in quality, but prefer speed and quantity to quality. With speed/quantity being the focus quality will always seem "cumbersome and slow" as it's not considered as important. – mutt Mar 17 '17 at 21:44
  • I have been practising and teaching TDD lately. It leads to near 100% code coverage and most intended functionality is safeguarded. Still some issues happen in production. The good thing is that it easy to write some extra tests and fix the issues found. Some crappy tests and shortcuts might drag you down. Mutation testing helps find these crappy or missing tests. Still I think there should be a better way of software development where it easier to be confident your code works. Also TDD is easy to learn, but very hard to master and get enthusiastic about. – Niels van Reijmersdal Mar 17 '17 at 21:59
  • Formal Verification gets wheeled out for for the core distributed system algorithms: Chubby, Zookeeper, etc. Look for "Zab High-performance broadcast for primary-backup systems" if you want to see something scary. Most people don't need to know that, all we care about it "somebody proved it worked and we know who to email about it" – Steve Loughran Mar 18 '17 at 12:46
  • Niels; TDD doesn't always measure the configuration space of real world deployment, e.g. "Does it work in Turkey?" (not a trick question, go look up "software not working turkey locale"). The other thing that is usually undertested is failure modes, though good mocking helps there. What TDD does do is get the developers to write testable apps and not put the tests off until a "later" which never comes – Steve Loughran Mar 18 '17 at 12:48
  • I get the impression that "mutatuon testing" is about testing the testing. And yes, tests can contain erors, the same type of logical (thinking) errors that developers make developing the software. But how far should you go? Add another level of testing the testing? and another? What is the added value (and what is the cost). And once our testing is "complete" we still have the errors in specifications, meannig an error has correctly been implemented. – Paul Ogilvie Mar 19 '17 at 9:21
5

It might be interesting for you to try to solve automated usability testing. This would include 508 testing and color recognition as well as trying to identify annoying things like having to click 3 times for something etc...

In general this is virtually impossible to automate as it's usually a user perception issue that is being tested for and a machine is not as capable at testing this.

  • yeah, anything intended for a human to interpret graphically is hard for a script to understand, e.g. facial recognition. – Yu Zhang Mar 17 '17 at 19:24
3

Distributed Testing is something we are still in the opening days of. Cloud computing makes it possible to innovate: you can bring up virtual clusters with different system configs, use SDN to create networks and even inject faults. At the same time, they make life hard, as the time you come in to work the VMs of a test run which have failed the test have already been recycled, and all you have are the logs of 8 services running on ten machines; you get to open up the likely ones in your text editor and using timestamps try and work out what was happening.

Some of the best work there comes from telecom

My 2015 thoughts on the matter.

My view is that reporting the results could be radically improved, diagnostics, and even automating the generation of complex configurations. A "configuration space" is one of those Hilbert Space thingys, where every time someone goes "hey, add a new config option!", you've added another dimension. The key point is that only some regions in the configuration space are valid for successful system behavior. Some of the work of a good tester is to identify those areas where the system fails to work, yet it is within the space of areas where customers expect to run the system.

Some reading from a list of a local directory

1999_ulrich_siemens_Test Architectures for Testing Distributed Systems.pdf
2001_long_DOA_ A case study in testing distributed systems.pdf
2006_rutherford_phd_Adequate System-Level Testing of Distributed Systems.pdf
2009_Hierons-_brunel_TAROT09_Testing Distributed Systems.pdf
2014_suny_IST_Model-Based Testing of Global Properties on Large-Scale Distributed Systems.pdf
2015_riesco_madrid_sparkTest_A Lightweight Tool for Random Testing of Stream Processing Systems.pdf

Ulrichs is a good intro from telecoms; focused on state machines, and building the aggregate test from smaller stes. The others follow on.

In my current work I've been exploring a concept which doesn't appear to have anything written up from an academic perspective: Metrics first testing. Here we add more system metrics to the application, essentially exposing some of the internal state. I can then make assertions about the state of parts of the system in unit tests. In larger system tests: collect those metrics and use them to help understand what happened.

Early days there, for unit/integration tests it's good, albeit brittle to changes in what is now a "grey box" system under test: optimise performance and tests counting the number of times something happened fail, you get to decide whether its a false negative or legit.

Regarding Formal Specs and verification, I have actually written the Formal Filesystem Specification for the Hadoop FS API, albeit working backwards from what was implemented in HDFS. Here the specification language used is Python, which works well for one key reason: it is understood by developers, who can grasp the declarative filesystem model as some tuples, lists and hash tables. It all maps 1:1 to the set theoretic calculus of Spivey's Z language, it's just disguised as python. In doing so, we have a spec which is broadly understood and maintained —and from there, we derive the Filesystem contract tests.

The limitations we have there is that it doesn't do temporal logic/concurrency, and there's no automated parsing/validation of the spec. I've been playing with TLA+, but that language just scares everyone off.

Anyway, hope I haven't scared you off, but I can assure you that there are PhDs to be had in this area. And, as so much of this work is being done as Open Source, you can even get to see how some of it works development communities,

  • This is above my skill level, but I really do appreciate the feedback. It's my job to research more on the these topics. Thank you! – rjw0924 Mar 18 '17 at 16:07
  • oh, I know it's not your skill level yet, but as its such an immature area, it'd be easy to make a difference – Steve Loughran Mar 18 '17 at 17:28
2

How about language interpretation?

There has been lots of research going into language interpretation, but it is extremely difficult for a script to understand a sentence, within or without a context.

Letting along if a sentence is spoken, its tone will play a vital role in its meaning. E.g. Thank you very much can be interpreted as a sincere thank you or a sarcastic comment, depends on the tone it is spoken in.

It is more of a software development hurdle, especially when you data mine keywords and try to summarize it.

  • How does it relate to sofware testing? – dzieciou Jun 1 '17 at 2:11
  • @dzieciou, language recognition is getting more and more common in a software application. It needs to be tested? – Yu Zhang Jun 1 '17 at 2:14
  • Right. But what is the actual research problem that is both about software testing and language recognition? Testing language recognition algorithms and systems? You mentioned challenges in building such algorithms and systems. But what are challenges in their testing? – dzieciou Jun 1 '17 at 2:21
2

Testing graphics API interactions with OpenGL, Vulkan, etc.

I am not sure if this fits your requirements, but while attempting to write unit tests for a graphical application I had to give up, because there were no solid methods to do this.

At least most of which I could find about were either extremely time-consuming and error-prone(mock interface took 5x the code needed for the tests than actual implementation) or worked only on complete systems that didn't want to change often. (Image comparison)

This is a practical problem that could be solved with clever engineering, so I am not sure if it fits a research subject.

2

Change best practices into language features.

Some examples:

  • Making sure that all references to HTML locators are done through Page Objects.
  • Making sure Page Objects entries are unique, aren't duplicated and are actually used
  • All code has tests or it is not accepted to the master branch
  • All tests cover happy, sad and optional happy and sad paths
  • Unit tests and acceptance tests don't duplicate each others tests
  • All features include some level of volume testing
  • Usability and accessibility is built-in and not optional
  • Treat QE engineeers as full members of the development team
  • Write one set of tests that work in all devices, browsers and versions

As you can see these various from easy to hard. Some technical, some human. Some might seem impossible. Then again you asked for unsolved problems or challenges.

0

Contribute a patch (fix a bug) in a popular open source software project.

Bug solved can be as ambitious as you can dare. There are many of them in any project.

  • 1
    How is this a research proposal? – Niels van Reijmersdal Mar 17 '17 at 19:35
  • For sure you know about a bug which, if solved, would help quite a few people and can easily take few months of hard work? – Peter M. - stands for Monica Mar 17 '17 at 21:53
  • OSS projects are place to experiment with methodologies. In the Hadoop stack, we will gladly welcome people who want to innovate in testing, especially if it integrates with the Yetus patch submission process, big integration tests, and failure diagnostics. Getting a simple patch in first will show today's workflow. For the curious, this is my life]: github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/… – Steve Loughran Mar 18 '17 at 12:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.