4

As part of a larger project, we are creating (or rather updating an existing) compiler that translates a Domain Specific Language (DSL) into a proprietary bytecode.
The bytecode then gets loaded into and interpreted by the appliance that we are creating.

Due to time constraints and the fact that testability was not taken into account in the design of the compiler, we will be doing almost exclusively end-to-end (sub-)product tests on the compiler, meaning that the compiler will be tested as a single, black-box, unit.

The question that keeps nagging me is how we can get some confidence that our tests are covering the likely areas of problems to the extent that we deem necessary.
What (formal) testing techniques can be employed in the testing of a compiler to gain confidence that we haven't overlooked an important area in designing our test cases?

  • Please elaborate on "end-to-end (sub-)product tests on the compiler." – user246 Mar 28 '16 at 15:29
  • @user246: Is my edit the kind of elaboration you were looking for, or is there some additional information that is missing? – Bart van Ingen Schenau Mar 28 '16 at 15:32
  • So in other words you have a set of sample files written in your DSL, and you compile them and run them to check whether they're correct? – user246 Mar 28 '16 at 15:56
  • @user246: The testcases will most likely consist of sample files written in the DSL and then comparing the bytecode output with a known-good sample. Comparing the bytecode rather than running the sample code is to avoid depending on the interpreter (which can contain errors as well. It will be tested separately). – Bart van Ingen Schenau Mar 28 '16 at 16:02
  • 1
    And are there any optimizing changes being made? – Kevin McKenzie Mar 28 '16 at 18:06
2

(Edit to add answer here instead of just linking to a blog entry, thanks for the feedback on my original)

Here are some ideas for a test plan for your complier. Many of these ideas were derived from a blog about how Microsoft tests their C++ complier.

Functional Tests:

  • In short, you should write many programs in your DSL and test the resulting programs. You can use the syntax of your DSL as a baseline for coverage for the happy paths.

  • If your complier is deterministic in the bytecodes that are generated, you could short cut regression testing by comparing bytecodes.

  • Use erroneous code to test the error handling of your compilier.

  • Don't forget to get coverage on any build switches for your compiler (optimization levels, output format, etc.)

  • Test with your real world code, programs already created with the complier.

Non-functional tests:

  • Performance of the compiled output code. Especially if you have speed optimization switches.

  • Size of compiled object code

  • Time to compile

-Resiliency with model-based techniques: automatically generate valid (and invalid) random programs based on the DSL syntax - and make sure the complier behaves as expected (successful build for valid, error generated for invalid)

Good luck,

John

  • 1
    This is just a link, but not an answer – Nick Volynkin Apr 2 '16 at 6:41
  • John, generally an answer that's mostly a link should have its primary points brought into the post, and send people back to the blog for more info. Could you bring the key points into this post? – corsiKa Apr 2 '16 at 12:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.