5

Currently, we do a lot of things when there is a application build job triggered (the trigger can be manual, or there is a commit to the develop/trunk branch, or there is a Pull Request created or updated).

The build itself is a multi-step process which aside from other things does, as a build verification process, the whole test run, static code analysis check and a coverage report analysis. A build would be marked as failed if:

  • any of the tests failed ("fail first" mode)
  • there is a static code analysis error or warning
  • coverage dropped lower than N% (in our case it's 90%)

Do you see any problems in this approach? Is it too strict and can be improved?


For instance, recently we've got a failed build just because someone did not put a space after a # comment character in a Python module (which is a PEP8 Python code style violation):

#some_code = "here"
^ no space here - PEP8 code style violation
  • Just wondering - who decraled that build as failed? – Embedded Sep 20 '17 at 16:56
  • @Alex.S we use jenkins+ansible scripts. – alecxe Sep 20 '17 at 17:08
  • I assume you're using feature branches when you say you have pull requests? – beatngu13 Sep 20 '17 at 19:33
  • @beatngu13 yes, that's right. Thanks. – alecxe Sep 20 '17 at 19:45
2

Personally, I think this is absolutely fine. Since you said you are using feature branches, only builds on top of these branches are failing. Pull requests are only merged when the corresponding build passes, hence, trunk/master is not affected.

The example with the missing space may sound a bit pettifogging, but I think having many different comment styles is much worse:

# Lorem ipsum.
#Lorem ipsum
#lorem ipsum.
# lorem ispum
#Lorem ipsum

This is what you can easily avoid with static code analysis, which is what you guys are already doing. If your team/department/organization thinks the rules are too strict, adapt them (and/or your DoD). But having a failing build due to a code style violation is not a problem—IMHO it is actually a really good thing.

However, what you could do is adapt the build pipeline, particularly the build steps. For instance, move the fast steps to the beginning so that the developers get fast feedback. With feature branches, you can simply commit/push micro commits or an intermediate result because only the build of the corresponding branch will fail fast (you can clean up later, in case you are using Git).

  • 1
    Great ideas, thanks so much! We're thinking about applying static code analysis to diffs only, run it before the tests just to get a faster feedback. – alecxe Sep 21 '17 at 2:57
2

Perfect, fail fast(er).

When the extreme programmers came up with continuous integration the idea was when someone checked in their code a couple of times per day and if that failed their automated build, it should be fixed within 10 minutes. Making sure that all code combines would keep working at all times.

Ideally noone should check in anything that will break the build ever, certainly not a "style/linting error" like your example, that is just lazy and should be caught by the IDE.

I think developers should be using a process somewhat like this to prevent build fails like that:

  • Write some code
  • Run all unit-tests and linters locally with a build-script (should take max some seconds)
  • Merge locally with integration branch (e.g. recent code from others)
  • Run all unit-tests and linters locally again
  • Run fast integration tests locally
  • Commit & push code
  • Build server runs same fast-tests on another machine
  • Build server runs static code analysis (e.g. code coverage, sonarqube rules)

Up till here it should be not taking longer than 10 minutes and the developer should have a pretty fail-fast cycle if something is wrong. Next:

  • Build server deploys to staging environment
  • Build server runs slow-integration and functional tests

When a build breaks the whole team should stop working and fix it as soon as possible. Being able to run the full build locally is key here, if you are depended on a build-server your builds will become slower and slower as noone cares. Be sure to read the 10 minute build.

  • 1
    "When a build breaks the whole team should stop working and fix it as soon as possible." I know, this is somewhat of a religious war, but that's not necessary with feature branches, only when you do trunk-based development. – beatngu13 Sep 21 '17 at 8:26
  • 1
    @beatngu13 True, but I would expect multiple people to be working in a feature branch, so the issue returns. In our teams ~4 developers swarm a feature: infoq.com/news/2013/02/swarming-agile-teams-deliver Personally I think single developers working in a single feature branches is a big anti-pattern, but that could be come another religious war ;-) – Niels van Reijmersdal Sep 21 '17 at 8:28
  • 1
    Indeed, we better leave it here. ;-) And thanks for pointing to the two articles, nice reads. – beatngu13 Sep 21 '17 at 8:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.