[Tails-dev] Adding "look at test suite results" to our revie…

Delete this message

Reply to this message
Author: intrigeri
Date:  
To: tails-dev
Subject: [Tails-dev] Adding "look at test suite results" to our reviewing guidelines
Hi!

Recently a branch was merged, that introduced a regression in our test
suite, which Jenkins was able to spot. But for some reason, we noticed
that regression only *after* the branch was merged into stable & devel.

I think that one of the key features of our test suite, by design [1],
is to allow us to notice such issues *before* we merge topic branches.

[1] https://tails.boum.org/blueprint/automated_builds_and_tests/automated_tests_specs/

So, until we have Jenkins results integrate with Redmine somehow,
IMO we should add to our reviewing guidelines [2] something like
"look at automated tests results on Jenkins for the proposed branch,
and make sure it introduces no regression; but our Jenkins setup does
not run 100% of the test suite, so bonus points if you feel like doing
a complete test suite run locally".

[2] https://tails.boum.org/contribute/merge_policy/review/

Thoughts?

Cheers,
--
intrigeri