-----Original Message-----
From: Sean Brogan <sean.brogan@...>
Sent: Friday, September 20, 2019 2:30 PM
To: Kinney, Michael D <michael.d.kinney@...>;
devel@edk2.groups.io; rfc@edk2.groups.io; Kinney,
Michael D <michael.d.kinney@...>
Cc: Bret Barkelew <Bret.Barkelew@...>
Subject: RE: [edk2-devel] [RFC] EDK II Continuous
Integration Phase 1

Currently it is building on Windows with VS2019.
VS2017 would be trivial but not worth it in my
perspective given how aligned the two are. If you
really wanted to do a weekly or nightly build it could
be added to that but I have been focused on a PR build.
I have a pipeline for GCC on linux. It doesn't'
support host based unit tests but I am working to get
the rest of the tests running.
I do not have a pipeline for Mac or LLVM/Clang. That
would be next and I think your files below should help.

To date we use PYTOOLs features to install our extra
tools. This makes it super easy and works for both
local and server based builds. It gives strong
versioning and management of that. It also tracks
those versions and you can see them on every build in
the Built Tool Report artifact. It can be downloaded
here for html version.
https://dev.azure.com/tianocore/edk2-ci-
play/_build/results?buildId=803&view=artifacts.
BUILD_TOOL_REPORT.html

Here it is pasted as plain text below. You can see the
iasl and nasm versions here.

Key Value Type
TOOL_CHAIN_TAG VS2019 TOOL
VC Version 14.22.27905 TOOL
d:\a\1\s\Conf\build_rule.txt 1.04 INFO
d:\a\1\s\Conf\target.txt 1.03 INFO
d:\a\1\s\Conf\tools_def.txt 1.22 INFO
iasl 20190215.0.0 INFO
Mu-Basetools 2019.03.1 INFO
mu_nasm 2.14.02 INFO


Hope that helps.

I would be happy to queue up this topic for next design
meeting and open it up to questions? I hope others
would take a look prior to meeting.

Thanks
Sean



-----Original Message-----
From: Kinney, Michael D <michael.d.kinney@...>
Sent: Thursday, September 19, 2019 2:56 PM
To: devel@edk2.groups.io; Sean Brogan
<sean.brogan@...>; rfc@edk2.groups.io;
Kinney, Michael D <michael.d.kinney@...>
Cc: Bret Barkelew <Bret.Barkelew@...>
Subject: RE: [edk2-devel] [RFC] EDK II Continuous
Integration Phase 1

Hi Sean,

Which OS/Compiler configurations are currently enabled
for the Code Compilation Test?

I have been working on enabling multiple OS/Compiler
configurations in Azure Pipelines. There are some
tools that need to be installed for each of these
environments.
Examples include NASM, iASL, Python.

For the work you have done, how are these extra tools
installed? Is it in the YML files or in the Python
scripts.

One critical task is to identify the tools and their
specific versions that the CI system is configured to
use.
These configurations should be documented in a Wiki
page and updated as new tools are released and adopted
by EDK II.
The inventory of tools used to validate a release
should Also be documented in a release notes for a
stable tag.

Here are the YML files that install the additional
tools required to support EDK II builds. I need the
source and versions of these tools to be reviewed and
approved.

https://nam06.safelinks.protection.outlook.com/?url=htt
ps%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-
ci%2Fblob%2Fmaster%2FAzurePipelines%2FWindowsPrerequisi
tes.yml&amp;data=02%7C01%7Csean.brogan%40microsoft.com%
7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91
ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=h
M4AKoMutISI5Oc%2FeVMevx%2FVRUCjJHuhEwqom0R30Ak%3D&amp;r
eserved=0
https://nam06.safelinks.protection.outlook.com/?url=htt
ps%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-
ci%2Fblob%2Fmaster%2FAzurePipelines%2FUbuntuPrerequisit
es.yml&amp;data=02%7C01%7Csean.brogan%40microsoft.com%7
C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91a
b2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=KQ
2zas2bJ%2FCjSRWGyMrxq5Rk4cW5lOgXQNR99QJbEKY%3D&amp;rese
rved=0
https://nam06.safelinks.protection.outlook.com/?url=htt
ps%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-
ci%2Fblob%2Fmaster%2FAzurePipelines%2FMacOsPrerequisite
s.yml&amp;data=02%7C01%7Csean.brogan%40microsoft.com%7C
9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91ab
2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=OzA
mzYkRJ3rQOHEDgKTREz%2BNp7acOWUACh1s%2Fb4UPGk%3D&amp;res
erved=0

Thanks,

Mike

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On

Behalf Of Sean

via Groups.Io
Sent: Thursday, August 29, 2019 7:22 PM
To: rfc@edk2.groups.io; Kinney, Michael D
<michael.d.kinney@...>; devel@edk2.groups.io
Cc: Bret Barkelew <Bret.Barkelew@...>
Subject: Re: [edk2-devel] [RFC] EDK II Continuous

Integration Phase 1

Mike, as you mentioned we have been working towards

enabling a

practical and extensible CI for Edk2 using Azure dev

ops and the

recently added edk2-pytool infrastructure. We have

been using similar

CI for Project Mu for the last few years.

Our approach is a little different in that we focus

on validating the

whole code base rather than just the incoming patch.

We do this

because we have found unexpected consequences of

patches and overall

we want all code to be compliant not just new

additions. We have

found the time to test the whole tree is not much

longer than only the

parts impacted by a code change (except maybe when

running the entire

compile test on every package). This obviously comes

with an initial

tax of needing to get the codebase into compliant

form.

Anyway we have prepared an RFC in addition to yours

and would like to

see these two efforts merged together.

We are still working on making a few optimizations.
Currently if the full set of tests are run we take

about 20 minutes.

This is because compiling MdeModulePkg for debug,

release, and host

based tests take a while. Most other packages are in

the 10 minute

range. We do have easy ways to disable or limit

certain tests as well

as expand the matrix to leverage more cloud resources

(more parallel

builds).


Content is best viewed online with links to helpful

content but is

also attached below:

https://nam06.safelinks.protection.outlook.com/?url=htt
ps%3A%2F%2Fgith

ub.com%2Fspbrogan%2Fedk2-staging%2Fblob%2Fedk2-

&amp;data=02%7C01%7Csea

n.brogan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c
1e1b%7C72f988bf

86f141af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&a
mp;sdata=xOiPHH

VbMhFBsIpj%2F9mhOhrkhdsF2Gbehd6%2Frk0XYgM%3D&amp;reserv
ed=0

stuart-ci-latest/Readme-CI-RFC.md

# CI and PR Gates

## Background

Historically, while the TianoCore maintainers and

stewards have done a

fantastic job of keeping contribution policies

consistent and

contributions clean and well-documented, there have

been few processes

that ran to verify the sanity, cleanliness, and

efficacy of the

codebase, and even fewer that publicly published

their results for the

community at large. This has caused inconsistancies

and issues within

the codebase from time to time.

Adding continuous integration (and potentially PR
gates) to the checkin process ensures that simple

errors like these

are caught and can be fixed on a regular basis.

## Strategy

While a number of CI solutions exist, this proposal

will focus on the

usage of Azure Dev Ops and Build Pipelines. For

demonstration, a

sample [TianoCore

repo](https://nam06.safelinks.protection.outlook.com/?u
rl=https%3A%2F%

2Fgithub.com%2Fspbrogan%2Fedk2-

staging.git&amp;data=02%7C01%7Csean.bro

gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%
7C72f988bf86f14

1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sd
ata=QmzQhemLUNB

7FGJwoGeeewExThrUjUArxnkmtro1b8A%3D&amp;reserved=0)
(branch edk2-stuart-ci-latest) and [Dev Ops

Pipeline](https://nam06.safelinks.protection.outlook.co
m/?url=https%3A

%2F%2Fdev.azure.com%2Ftianocore%2Fedk2-ci-

&amp;data=02%7C01%7Csean.bro

gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%
7C72f988bf86f14

1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sd
ata=rS%2FrjIFkS

uEhI%2FDWyslf34i711%2FbY8Gw%2Byexq%2FwydjU%3D&amp;reser
ved=0

play/_build?definitionId=12) have been set up.

Furthermore, this proposal will leverage the

TianoCore python tools

PIP modules:

[library](https://nam06.safelinks.protection.outlook.co
m/?url=https%3A

%2F%2Fpypi.org%2Fproject%2Fedk2-pytool-

&amp;data=02%7C01%7Csean.brogan

%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C7
2f988bf86f141af

91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata
=jzM6MKZFFAEvyR

4h0%2Bkf5pUcBSsoVzkIHFFi1Bd6Il4%3D&amp;reserved=0
library/) and

[extensions](https://nam06.safelinks.protection.outlook
.com/?url=https

%3A%2F%2Fpypi.org%2Fproject%2Fedk2-pytool-

&amp;data=02%7C01%7Csean.bro

gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%
7C72f988bf86f14

1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sd
ata=jzM6MKZFFAE

vyR4h0%2Bkf5pUcBSsoVzkIHFFi1Bd6Il4%3D&amp;reserved=0
extensions/) (with repos located

[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F

%2Fgithub.com%2Ftianocore%2Fedk2-pytool-

&amp;data=02%7C01%7Csean.broga

n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C
72f988bf86f141a

f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdat
a=OwT%2BiNTN9Rv

r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
library) and

[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F

%2Fgithub.com%2Ftianocore%2Fedk2-

&amp;data=02%7C01%7Csean.brogan%40mic

rosoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988b
f86f141af91ab2d

7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=tInVw
GwCXXGBgcmHQ%2B

0UwRaEbtZWRy8hTp5wpRtN0R0%3D&amp;reserved=0
pytool-extensions)).

The primary execution flows can be found in the

`azure-

pipelines-pr-gate.yml` and `azure-pipelines-pr-gate-

linux.yml` files.

These YAML files are consumed by the Azure Dev Ops

Build Pipeline and

dictate what server resources should be used, how

they should be

configured, and what processes should be run on them.
An overview of this schema can be found

[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F

%2Fdocs.microsoft.com%2Fen-

&amp;data=02%7C01%7Csean.brogan%40microsoft

.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f14
1af91ab2d7cd011

db47%7C1%7C0%7C637045269452466989&amp;sdata=j8nyYG3Vniz
wKFZ8l1BWoLcPy%

2BuaSoVT0jN2Esi7wH8%3D&amp;reserved=0
us/azure/devops/pipelines/yaml-schema?view=azure-
devops&tabs=schema).

Inspection of these files reveals the EDKII Tools

commands that make

up the primary processes for the CI
build: 'stuart_setup', 'stuart_update', and

'stuart_ci_build'. These

commands come from the EDKII Tools PIP modules and

are configured as

described below. More documentation on the stuart

tools can be found

[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F

%2Fgithub.com%2Ftianocore%2Fedk2-pytool-

&amp;data=02%7C01%7Csean.broga

n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C
72f988bf86f141a

f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdat
a=OwT%2BiNTN9Rv

r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
extensions/blob/master/docs/using.md) and

[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F

%2Fgithub.com%2Ftianocore%2Fedk2-pytool-

&amp;data=02%7C01%7Csean.broga

n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C
72f988bf86f141a

f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdat
a=OwT%2BiNTN9Rv

r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0

extensions/blob/master/docs/features/feature_invocables

.md).

## Configuration

Configuration of the CI process consists of (in order

of precedence):

* command-line arguments passed in via the Pipeline

YAML

* a per-package configuration file (e.g. `<package-
name>.mu.yaml`) that is detected by the CI system in
EDKII Tools.
* a global configuration Python module (e.g.
`CISetting.py`) passed in via the command-line

The global configuration file is described in [this

readme](https://nam06.safelinks.protection.outlook.com/
?url=https%3A%2

F%2Fgithub.com%2Ftianocore%2Fedk2-pytool-

&amp;data=02%7C01%7Csean.brog

an%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7
C72f988bf86f141

af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sda
ta=OwT%2BiNTN9R

vr14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0

extensions/blob/master/docs/usability/using_settings_ma

nager.md) from the EDKII Tools documentation. This

configuration is

written as a Python module so that decisions can be

made dynamically

based on command line parameters and codebase state.

The per-package configuration file can override most

settings in the

global configuration file, but is not dynamic. This

file can be used

to skip or customize tests that may be incompatible

with a specific

package.
By default, the global configuration will try to run

all tests on all

packages.

## CI Test Types

All CI tests are instances of EDKII Tools plugins.
Documentation on the plugin system can be found

[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F

%2Fgithub.com%2Ftianocore%2Fedk2-pytool-

&amp;data=02%7C01%7Csean.broga

n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C
72f988bf86f141a

f91ab2d7cd011db47%7C1%7C0%7C637045269452476994&amp;sdat
a=1AM4geY%2BEFh

ekun7oQOYyVVt%2Bwn6CAF2hhtkS0sgwPU%3D&amp;reserved=0

extensions/blob/master/docs/usability/using_plugin_mana

ger.md) and

[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F

%2Fgithub.com%2Ftianocore%2Fedk2-

&amp;data=02%7C01%7Csean.brogan%40mic

rosoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988b
f86f141af91ab2d

7cd011db47%7C1%7C0%7C637045269452476994&amp;sdata=3vyPM
vrfG3V4doNM0%2B

cKTpDSHEEEvkfxKAtw3eiJWsY%3D&amp;reserved=0
pytool-

extensions/blob/master/docs/features/feature_plugin_man

ager.md). Upon invocation, each plugin will be passed

the path to the

current package under test and a dictionary

containing its targeted

configuration, as assembled from the command line,

per-package

configuration, and global configuration.

Note: CI plugins are considered unique from build

plugins and helper

plugins, even though some CI plugins may execute

steps of a build.

In the example, these plugins live alongside the code

under test (in

the `BaseTools` directory), but may be moved to the

'edk2-test' repo

if that location makes more sense for the community.

### Module Inclusion Test - DscCompleteCheck

This test scans all available modules (via INF files)

and compares

them to the package-level DSC file for the package

each module is

contained within. The test considers it an error if

any module does

not appear in the `Components` section of at least

one package-level

DSC (indicating that it would not be built if the

package were built).

### Code Compilation Test - CompilerPlugin

Once the Module Inclusion Test has verified that all

modules would be

built if all package-level DSCs were built, the Code

Compilation Test

simply runs through and builds every package-level

DSC on every

toolchain and for every architecture that is

supported. Any module

that fails to build is considered an error.

### Host-Based UnitTests - HostUnitTestCompilerPlugin

and

HostUnitTestDscCompleteCheck

The [Testing RFC doc](Readme-Testing-RFC.md) has much

more detail on

this, but the basic idea is that host- based unit

tests can be

compiled against individual modules and libraries and

run on the build

agent (in this case, the Dev Ops build server). The

successful and

failing test case results are collected and included

in the final

build report.

### GUID Uniqueness Test - GuidCheck

This test works on the collection of all packages

rather than an

individual package. It looks at all FILE_GUIDs and

GUIDs declared in

DEC files and ensures that they are unique for the

codebase. This

prevents, for example, accidental duplication of

GUIDs when using an

existing INF as a template for a new module.

### Cross-Package Dependency Test - DependencyCheck

This test compares the list of all packages used in

INFs files for a

given package against a list of "allowed

dependencies" in plugin

configuration for that package. Any module that

depends on a

disallowed package will cause a test failure.

### Library Declaration Test - LibraryClassCheck

This test looks at all library header files found in

a package's

`Include/Library` directory and ensures that all

files have a matching

LibraryClass declaration in the DEC file for the

package. Any missing

declarations will cause a failure.

### Invalid Character Test - CharEncodingCheck

This test scans all files in a package to make sure

that there are no

invalid Unicode characters that may cause build

errors in some

character sets/localizations.

## Next Steps

* Receive community feedback on RFC.
* Determine where this phase makes sense given

existing RFCs from

other TianoCore contributors.
* Optimize testing beharior.
* Only run a subset of tests on PRs or individual

commits.

* Run full testing either once per day or once

every several

commits.
* Add more tests/capabilities.
* Continue to improve results formatting.
* Continue to improve CI documentation.
* Much of this documentation effort is pending

community feedback on

which parts are needed and what phases are

priorities.

Thanks


-----Original Message-----
From: rfc@edk2.groups.io <rfc@edk2.groups.io> On

Behalf Of Michael D

Kinney via Groups.Io
Sent: Thursday, August 29, 2019 1:23 PM
To: devel@edk2.groups.io; rfc@edk2.groups.io
Subject: [edk2-rfc] [RFC] EDK II Continuous

Integration Phase 1

Hello,

This is a proposal for a first step towards

continuous integration for

all TianoCore repositories to help improve to quality

of commits and

automate testing and release processes for all EDK II

packages and

platforms.

This is based on work from a number of EDK II

community members that

have provide valuable input and evaluations.

* Rebecca Cran <mailto:rebecca@...> Jenkins

evaluation

* Laszlo Ersek <mailto:lersek@...> GitLab

evaluation

* Philippe Mathieu-Daudé <mailto:philmd@...>

GitLab evaluation

* Sean Brogan <mailto:sean.brogan@...>

Azure Pipelines and

HBFA
* Bret Barkelew <mailto:Bret.Barkelew@...>
Azure Pipelines and HBFA
* Jiewen Yao <mailto:jiewen.yao@...> HBFA

The following link is a link to an EDK II WIKI page

that contains a

summary of the work to date. Please provide feedback

in the EDK II

mailing lists. The WIKI pages will be updated with

input from the

entire EDK II community.

https://nam06.safelinks.protection.outlook.com/?url=htt

ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io

%2Fwiki%2FEDK-II-Continuous-

Integration&amp;data=02%7C01%7Csean.brogan%40microsoft.

com%7C6f67f169a6c746b4288608d72cbea7b6%7C72f988bf86f141

af91ab2d7cd011db47%7C1%7C0%7C637027069686644659&amp;sda

ta=GR9wN6gP3mJlCTopAaQ2rlzhby1nuF%2BwDVsfFIQAZjA%3D&amp

;reserved=0

Proposal
========
Phase 1 of adding continuous integration is limited

to the

edk2 repository. Additional repositories will be

added later.

The following changes are proposed:
* Remove EDK II Maintainers write access to edk2

repository.

Only EDK II Administrators will continue to have

write

access, and that should only be used to handle

extraordinary

events.
* EDK II Maintainers use a GitHub Pull Request

instead of push

to commit a patch series to the edk2 repository.
There are
no other changes to the development and review

process. The

patch series is prepared in an EDK II maintainer

branch with

all commit message requirements met on each patch

in the series.

* The edk2 repository only accepts Pull Requests from

members

of the EDK II Maintainers team. Pull Requests from

anyone else

are rejected.
* Run pre-commit checks using Azure Pipelines
* If all pre-commit checks pass, then the patch

series is auto

committed. The result of this commit must match

the contents

and commit history that would have occurred using

the previous

push operation.
* If any pre-commit checks fail, then notify the

submitter.

A typical reason for a failure would be a merge

conflict with

another pull request that was just processed.
* Limit pre-commit checks execution time to 10

minutes.

* Provide on-demand builds to EDK II Maintainers that

to allow

EDK II Maintainers to submit a branch through for

the same

set of pre-commit checks without submitting a pull

request.

## Pre-Commit Checks in Phase 1
* Run and pass PatchCheck.py with no errors

=====================================================

The following are some additional pre-commit check

ideas that could be

quickly added once the initial version using

PatchCheck.py is fully

functional.
Please provide feedback on the ones you like and

additional ones you

think may improve the quality of the commits to the

edk2 repository.

## Proposed Pre-Commit Checks in Phase 2
* Verify Reviewed-by and Acked-by tags are present

with

correct maintainer email addresses
* Verify no non-ASCII characters in modified files
* Verify no binary files in set of modified files
* Verify package dependency rules in modified files

## Proposed Pre-Commit Checks in Phase 3
* Run ECC on modified files
* Verify modified modules/libs build
* Run available host based tests (HBFA) against

modified

modules/libs

Best regards,

Mike