5

As part of a larger project, we are creating (or rather updating an existing) compiler that translates a Domain Specific Language (DSL) into a proprietary bytecode.
The bytecode then gets loaded into and interpreted by the appliance that we are creating.

Due to time constraints and the fact that testability was not taken into account in the design of the compiler, we will be doing almost exclusively end-to-end (sub-)product tests on the compiler, meaning that the compiler will be tested as a single, black-box, unit.

The question that keeps nagging me is how we can get some confidence that our tests are covering the likely areas of problems to the extent that we deem necessary.
What (formal) testing techniques can be employed in the testing of a compiler to gain confidence that we haven't overlooked an important area in designing our test cases?

Improve this question
12
  • Please elaborate on "end-to-end (sub-)product tests on the compiler."
    – user246
    Mar 28, 2016 at 15:29
  • @user246: Is my edit the kind of elaboration you were looking for, or is there some additional information that is missing?
    – Bart van Ingen Schenau
    Mar 28, 2016 at 15:32
  • So in other words you have a set of sample files written in your DSL, and you compile them and run them to check whether they're correct?
    – user246
    Mar 28, 2016 at 15:56
  • @user246: The testcases will most likely consist of sample files written in the DSL and then comparing the bytecode output with a known-good sample. Comparing the bytecode rather than running the sample code is to avoid depending on the interpreter (which can contain errors as well. It will be tested separately).
    – Bart van Ingen Schenau
    Mar 28, 2016 at 16:02
  • 1
    And are there any optimizing changes being made?
    – Kevin McKenzie
    Mar 28, 2016 at 18:06

1 Answer 1

Reset to default
3

(Edit to add answer here instead of just linking to a blog entry, thanks for the feedback on my original)

Here are some ideas for a test plan for your complier. Many of these ideas were derived from a blog about how Microsoft tests their C++ complier.

Functional Tests:

  • In short, you should write many programs in your DSL and test the resulting programs. You can use the syntax of your DSL as a baseline for coverage for the happy paths.

  • If your complier is deterministic in the bytecodes that are generated, you could short cut regression testing by comparing bytecodes.

  • Use erroneous code to test the error handling of your compilier.

  • Don't forget to get coverage on any build switches for your compiler (optimization levels, output format, etc.)

  • Test with your real world code, programs already created with the complier.

Non-functional tests:

  • Performance of the compiled output code. Especially if you have speed optimization switches.

  • Size of compiled object code

  • Time to compile

-Resiliency with model-based techniques: automatically generate valid (and invalid) random programs based on the DSL syntax - and make sure the complier behaves as expected (successful build for valid, error generated for invalid)

Good luck,

John

Improve this answer
2
  • 1
    This is just a link, but not an answer
    – Nick Volynkin
    Apr 2, 2016 at 6:41
  • John, generally an answer that's mostly a link should have its primary points brought into the post, and send people back to the blog for more info. Could you bring the key points into this post?
    – corsiKa ♦
    Apr 2, 2016 at 12:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.