Re: [Tails-dev] Automated tests specification

Delete this message

Reply to this message
Author: intrigeri
Date:  
To: The Tails public development discussion list
New-Topics: Re: [Tails-dev] Automated tests specification
Subject: Re: [Tails-dev] Automated tests specification
Hi,

bertagaz wrote (09 Jul 2015 13:28:23 GMT) :
> On Wed, Jul 01, 2015 at 07:19:04PM +0200, intrigeri wrote:
>> bertagaz wrote (25 Jun 2015 09:41:23 GMT) :


> I've tried to sum this up in a 'current proposal' subsection.


Seen this, yay. And pushed a few commits on top.

>> It also needs to know what commit that ISO was built from, in order to
>> run the test suite from the same commit. Surely we can dynamically get
>> this information by inspecting the ISO (maybe even in the iso9660
>> metadata), if passing through the info via Jenkins is too painful.
>> Maybe that's worth a research ticket?


> Yes, that's what the phrase means when it says "a given automated build
> corresponding to a given commit", but maybe that's too fuzzy?


Indeed I find it too vague so I've rephrased this paragraph (414e4f3),
and added (414e4f3) another requirement:

* if applicable, the commit at which the base branch was at, when it
was merged into the branch being built;

(because we need to run the test suite not from the commit on the
topic branch, but on the result of merging its base branch into the
topic branch at that commit)

IIRC I've made it so the build log contains that information, and if
the build scripts need adjustements to make it easier to retrieve from
Jenkins (e.g. logging it in a machine-parsable format), let me know,
I'll be happy to follow-up on this.

> With most of the solution out there in Jenkins to chain build and test
> jobs, it doesn't seem complicated to pass a parameter to the test job
> containing the value of the commit used in the previous (upstream in
> Jenkins) build ISO job.


Cool. Seems like we'll need to pass at least two such pieces of info.

>> > The automated test suite MUST be run in a clean environment.

[...]
> I've seen your commits on this slave reboot between jobs idea and made
> some research myself, and it sure looks quite scary. I've updated the
> blueprint to detail a bit what a "clean environment" means and include
> your comment.


Good.

Now, it would be good to have feedback from other contributors, in
particular those who will be directly affected. Let's schedule
a session to look into this at the summit, shall we?

Cheers,
--
intrigeri