PYTOOLs
--------------
Sean from MSFT has updates on this thread:
https://edk2.groups.io/g/rfc/topic/edk2_devel_rfc_edk_ii/34204771?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,34204771

We are looking for folks with opinions on this topic to comment as the structure is a bit different from what we've done in the past. PYTOOLs has not been ported to macOS, but once everyone agrees we are going in the right direction, we can make that happen. We will be finishing review of this topic thread by *October 11th*, with the implementation (Phase 1) going live on October 25th.

Maintainer PRs
-----------------
There is a very interesting command line utility called "hub". This allows command line github interface. (https://github.com/github/hub) We want to be sure the PRs are from maintainers only. We're currently testing that out, and will move to this process if testing goes well.

Hardware Testing
-------------------
From the Open Source Firmware Conference last month:
Facebook is working on a project called "Contest" for as their continuous integration framework for firmware. It includes a very modular, plugin-based way to test hardware. 9elements has been working on a hardware testing jig (Rasp Pi based) for automated flashing of firmware. We will be looking at integration these frameworks with our current CI efforts in order to keep in line with what the firmware community on the whole is doing in terms of CI.

If you have any questions here please contact Stephano and he will connect you with folks at 9elements / Facebook.

Whitespace / CRLF
---------------------
We are working on cleaning up whitespace and line ending so that we are all working in the same way. There are historical reasons why this was difficult in the past, however those issues have been addressed. Eventually, we'd like everyone to use "--global core.autocrlf" as our default process, so we are exploring the side effects that might have. We will be looking on feedback here shortly. Look for that RFC coming soon from Leif. Obviously this cleanup will cause some conflicts and git blame messiness, however we think the benefits are worth those inconveniences.

Package Dependencies
--------------------------
We'd like to update the wiki with the rules behind dependencies. Also, we need to address circular dependencies. We should also look at the UART class and see if there are parts that can be cleaned up or shared better (specifically the Libclass interface). We will be looking for those maintainers of packages with circular dependencies to make some changes. Please keep an eye out on that RFC as well.

Automation Scripts
------------------
There is currently a python script for ensuring the correct CC is included, taking advantage of our move to a machine readable maintainers.txt. The edk2-platforms maintainers file will be changing to this format soon. There are also tools like PatchCheck and SetupGit which we will likely integrate into PYTOOLs in the near future. Please have a look at these scripts and send any comments / concerns to the list.

See:
BaseTools/Scripts/GetMaintainer.py
BaseTools/Scripts/PatchCheck.py
BaseTools/Scripts/SetupGit.py

On 2019-10-04 07:30, Stephano Cetola wrote:

PYTOOLs
--------------
Sean from MSFT has updates on this thread:
https://edk2.groups.io/g/rfc/topic/edk2_devel_rfc_edk_ii/34204771?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,34204771


We are looking for folks with opinions on this topic to comment as the
structure is a bit different from what we've done in the past. PYTOOLs
has not been ported to macOS, but once everyone agrees we are going in
the right direction, we can make that happen. We will be finishing
review of this topic thread by *October 11th*, with the implementation
(Phase 1) going live on October 25th.

I've worked on CI for a number of years, and from what I've seen we're
going in the wrong direction. It looks like it's really difficult to get
full build logs from the DevOps interface (which will be useful for
maintainers tracking down build problems). I'm not sure many people are
going to want to use pytools on non-Windows systems since it uses NuGet
etc., and wants to pull in loads of other stuff too. It looks like
building OVMF is on the todo list, but since it's something we discussed
several times I'm surprised to see it's not implemented yet.


I don't know how much of this is just my knowledge of CI technologies
having fallen behind a bit, since my work still uses Jenkins, and before
that I used CruiseControl.NET which I'm not sure is still around. I'm
not familiar with other systems like Travis CI, Azure DevOps, Cirrus etc.


--
Rebecca Cran

Full build logs are available for every step and plugin that is run during the process.
You can see an example here .
https://dev.azure.com/tianocore/edk2-ci-play/_build/results?buildId=1000&view=artifacts
CI_BuildLog.txt is probably the log you are looking for as that will have all the "compile" process output.



Pytools does support NuGet as one source of dependencies management. It also supports git repos and web dependencies (download from URL). You can see that here.
https://github.com/tianocore/edk2-pytool-extensions/tree/master/edk2toolext/environment/extdeptypes

Other methods of getting dependencies can be extended easily by subclassing the ExternalDependency class.

For the CI build I have chosen to use NuGet to collect my binary dependencies. I have chosen this because NuGet is fast, easy, and we already use it internally. It also has security features that are on par or ahead of most public package management systems.

For example:
https://github.com/spbrogan/edk2/blob/master/BaseTools/Bin/basetools_ext_dep.json
https://github.com/spbrogan/edk2/blob/master/BaseTools/Bin/iasl_ext_dep.json
https://github.com/spbrogan/edk2/blob/master/BaseTools/Bin/nasm_ext_dep.json

These don't have to be managed as nuget packages but again we already have them and support this today.

You can see we use a web dependency here
https://github.com/spbrogan/edk2/blob/master/BaseTools/Bin/gcc_aarch64_linux_ext_dep.json


Can you help me with the " loads of other stuff too" comment? What is in there that you don't like? It is all tailored for UEFI development.


Ovmf is a platform. Nothing in our CI is about building a platform. That is another thing entirely and our PyTools supports building platforms but it is distinct from building CI. I can imagine potentially adding platform builds to our Pull Request process but that is another topic.


All of the above things could be used on any CI system (not tied to azure pipelines). This is how we manage our dependencies at scale and this allows local developers to use the same system as our CI.

Happy to talk more and thanks for looking.

Thanks
Sean

On 2019-10-04 16:09, Sean via Groups.Io wrote:

Full build logs are available for every step and plugin that is run during the process.
You can see an example here .
https://dev.azure.com/tianocore/edk2-ci-play/_build/results?buildId=1000&view=artifacts
CI_BuildLog.txt is probably the log you are looking for as that will have all the "compile" process output.

Thanks. I'm more used to a web interface for viewing log files instead
of having to download them, so that's probably what confused me.

Can you help me with the " loads of other stuff too" comment? What is in there that you don't like? It is all tailored for UEFI development.

I tried to run it and it had packages it wanted to download. I tried to
chase the dependencies and get it working on my KUbuntu system, but gave
up. I should probably use the repository or web method instead.

Ovmf is a platform. Nothing in our CI is about building a platform. That is another thing entirely and our PyTools supports building platforms but it is distinct from building CI. I can imagine potentially adding platform builds to our Pull Request process but that is another topic.

I think this is the critical thing I was misunderstanding. To me, a CI
system is mostly about building platforms.


--
Rebecca Cran

Hi Rebecca,

I the POC I did for EDK II CI using Azure Pipelines,
I generated artifacts that are the build log and build
report files from each package and platform build.

All the artifacts from a single build can be downloaded
as a ZIP file:

https://dev.azure.com/mikekinney/_apis/resources/Containers/1265024?itemPath=drop&$format=zip

Each log/report file can also be accessed as a link

https://dev.azure.com/mikekinney/_apis/resources/Containers/1265024?itemPath=drop%2FMde_DEBUG_GCC5_X64.log

https://dev.azure.com/mikekinney/_apis/resources/Containers/1265024?itemPath=drop%2FMde_DEBUG_GCC5_X64.report

I agree that an HTML dashboard view of these results
would be much more useful, and I plan on working on
this for the Daily/Weekly/Release views and then make
it available for pre-commit builds.

I will work with Sean to merge some of the concepts
from my POC with the CI work he has done. The initial
focus is on package builds, but adding platform builds
is also very important.

In focus on the first phase is to improve the quality
of patch submissions and catch obvious errors and build
breaks before commits. Platform builds and boots will
be addressed in a later phase.

Mike

On 10/04/19 23:44, Rebecca Cran wrote:

On 2019-10-04 07:30, Stephano Cetola wrote:

PYTOOLs
--------------
Sean from MSFT has updates on this thread:
https://edk2.groups.io/g/rfc/topic/edk2_devel_rfc_edk_ii/34204771?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,34204771


We are looking for folks with opinions on this topic to comment as the
structure is a bit different from what we've done in the past. PYTOOLs
has not been ported to macOS, but once everyone agrees we are going in
the right direction, we can make that happen. We will be finishing
review of this topic thread by *October 11th*, with the implementation
(Phase 1) going live on October 25th.

I've worked on CI for a number of years, and from what I've seen we're
going in the wrong direction. It looks like it's really difficult to get
full build logs from the DevOps interface (which will be useful for
maintainers tracking down build problems). I'm not sure many people are
going to want to use pytools on non-Windows systems since it uses NuGet
etc., and wants to pull in loads of other stuff too. It looks like
building OVMF is on the todo list, but since it's something we discussed
several times I'm surprised to see it's not implemented yet.


I don't know how much of this is just my knowledge of CI technologies
having fallen behind a bit, since my work still uses Jenkins, and before
that I used CruiseControl.NET which I'm not sure is still around. I'm
not familiar with other systems like Travis CI, Azure DevOps, Cirrus etc.

I'm terribly starved for cycles on this topic, plus (worse) I have
basically zero experience with CI on the server side.

Regarding the client side... I guess I'll have to set up an "untrusted"
virtual machine even for submitting builds :( I'd prefer if the client
side dependencies were minimal and could be satisfied from a small
number of trusted repositories. But, I don't want to block progress on
this topic, especially because I cannot offer code contributions here.

Thanks
Laszlo

On Oct 7, 2019, at 12:10 PM, Laszlo Ersek <lersek@...> wrote:

Regarding the client side... I guess I'll have to set up an "untrusted"
virtual machine even for submitting builds :( I'd prefer if the client
side dependencies were minimal and could be satisfied from a small
number of trusted repositories. But, I don't want to block progress on
this topic, especially because I cannot offer code contributions here.

To submit a build you’ll create a pull request in Github. The dependencies are only needed if you want to replicate what the server is doing.

Rebecca

On 10/7/19 10:58 AM, Kinney, Michael D wrote:

Each log/report file can also be accessed as a link

https://dev.azure.com/mikekinney/_apis/resources/Containers/1265024?itemPath=drop%2FMde_DEBUG_GCC5_X64.log

https://dev.azure.com/mikekinney/_apis/resources/Containers/1265024?itemPath=drop%2FMde_DEBUG_GCC5_X64.report

Unfortunately I get a message that I'm not authorized to access those files:


TF400813: The user '<redacted>' is not authorized to access this resource."

I did eventually find CI_Buildlog.txt that Sean referred to.

I agree that an HTML dashboard view of these results
would be much more useful, and I plan on working on
this for the Daily/Weekly/Release views and then make
it available for pre-commit builds.

What I was expecting was something similar to https://ci.freebsd.org/job/FreeBSD-head-amd64-build/14822/console . Which is a 83MB log file (by default the first 82.5MB are skipped in the display!) which shows all the output from a "make buildworld" for FreeBSD amd64. It seems unfortunate that users have to go through the process of downloading artifacts, unzipping them and then opening the log file. Hopefully this can be improved on with the HTML dashboard view.

In focus on the first phase is to improve the quality
of patch submissions and catch obvious errors and build
breaks before commits. Platform builds and boots will
be addressed in a later phase.

I feel the comments on the RFC have been ignored in favor of a pre-determined process. We discussed that we wanted to build OVMF in various configurations, but for some reason that's being delayed to future work.


--
Rebecca Cran

Hi Rebecca,

I have that project set to “Public” so anyone can look at it. The ZIP file appears to be available from the Public view, but not the links to individual files within the ZIP file. I can only access those links if I am logged into the account. That is an unexpected limitation. I will do some more research.

We can work on OVMF platform builds as the next step.

Thanks,

Mike

From: Rebecca Cran <rebecca@...>
Sent: Monday, October 7, 2019 12:57 PM
To: Kinney, Michael D <michael.d.kinney@...>; sean.brogan@...; Stephano Cetola <stephano.cetola@...>; announce@edk2.groups.io
Subject: Re: [edk2-announce] TianoCore Community Meeting Minutes - October


On 10/7/19 10:58 AM, Kinney, Michael D wrote:

Each log/report file can also be accessed as a link



https://dev.azure.com/mikekinney/_apis/resources/Containers/1265024?itemPath=drop%2FMde_DEBUG_GCC5_X64.log



https://dev.azure.com/mikekinney/_apis/resources/Containers/1265024?itemPath=drop%2FMde_DEBUG_GCC5_X64.report

Unfortunately I get a message that I'm not authorized to access those files:



TF400813: The user '<redacted>' is not authorized to access this resource."

I did eventually find CI_Buildlog.txt that Sean referred to.

I agree that an HTML dashboard view of these results

would be much more useful, and I plan on working on

this for the Daily/Weekly/Release views and then make

it available for pre-commit builds.

What I was expecting was something similar to https://ci.freebsd.org/job/FreeBSD-head-amd64-build/14822/console . Which is a 83MB log file (by default the first 82.5MB are skipped in the display!) which shows all the output from a "make buildworld" for FreeBSD amd64. It seems unfortunate that users have to go through the process of downloading artifacts, unzipping them and then opening the log file. Hopefully this can be improved on with the HTML dashboard view.

In focus on the first phase is to improve the quality

of patch submissions and catch obvious errors and build

breaks before commits. Platform builds and boots will

be addressed in a later phase.

I feel the comments on the RFC have been ignored in favor of a pre-determined process. We discussed that we wanted to build OVMF in various configurations, but for some reason that's being delayed to future work.



--
Rebecca Cran

On 10/07/19 20:23, Rebecca Cran wrote:

On Oct 7, 2019, at 12:10 PM, Laszlo Ersek <lersek@...> wrote:

Regarding the client side... I guess I'll have to set up an "untrusted"
virtual machine even for submitting builds :( I'd prefer if the client
side dependencies were minimal and could be satisfied from a small
number of trusted repositories. But, I don't want to block progress on
this topic, especially because I cannot offer code contributions here.

To submit a build you’ll create a pull request in Github. The dependencies are only needed if you want to replicate what the server is doing.

Thank you for clarifying that!
Laszlo