[edk2-devel] Basetools as a pip module


Andrew Fish <afish@...>
 

On Sep 1, 2020, at 4:35 PM, Matthew Carlson <matthewfcarlson@gmail.com> wrote:

Hello all,

A recent topic on the RFC mailing list went out and the work on moving Basetools/Sources/Python to a separate repo has started. See the RFC conversation here: https://edk2.groups.io/g/rfc/topic/74009714#270

The repo in question is here: https://github.com/tianocore/edk2-basetools

The current plan is shortly after the stable tag is created, a series of patches will come into edk2 that redirects the build system into the new python module as well as adds additional documentation. You can see a sample of this work here: https://github.com/matthewfcarlson/edk2 as this has a branch that has the work required to use the basetools pip module. The patches won't delete the Basetools/Sources/Python folder but will allow users to select between them. After a certain grace period, the python folder will be deleted and the pip module will be the de facto way of using basetools.

Three questions need to be answered:

1. After the patches that enable the pip module land, how long should the grace period be?
2. During the grace period, should basetools commits land in both places or just in the edk2-basetools directory?
3. How should the user be able to select which basetools to use (the one in EDK2 or the pip module)? Currently the approach being considered is a simple environmental variable? One of the key considerations is transparency since it won't be apparent what is being used for a particular build without some sort of mechanism to notify the developer. With two seperate versions of Basetools, it becomes very easy for the version of basetools you're using to not be the one you expect.
Matthew,

I’ll throw out some current developer centric ideas.

1) If you `source edksetup.sh` (edksetup.bat) you get the current behavior, and you add an argument you get the pip flavor? So maybe `edksetup.bat pip-basetools`?
2) We have similar issues to this with env variables and the build command dumps them out when it runs. Can we use the current EDK_TOOL_PATH? Or maybe add an extra print to show that the pip module is being used?

Thanks,

Andrew Fish


Thank you.
-Matthew Carlson


Laszlo Ersek
 

On 09/02/20 02:49, Andrew Fish via groups.io wrote:


On Sep 1, 2020, at 4:35 PM, Matthew Carlson <matthewfcarlson@gmail.com> wrote:

Hello all,

A recent topic on the RFC mailing list went out and the work on moving Basetools/Sources/Python to a separate repo has started. See the RFC conversation here: https://edk2.groups.io/g/rfc/topic/74009714#270

The repo in question is here: https://github.com/tianocore/edk2-basetools

The current plan is shortly after the stable tag is created, a series of patches will come into edk2 that redirects the build system into the new python module as well as adds additional documentation. You can see a sample of this work here: https://github.com/matthewfcarlson/edk2 as this has a branch that has the work required to use the basetools pip module. The patches won't delete the Basetools/Sources/Python folder but will allow users to select between them. After a certain grace period, the python folder will be deleted and the pip module will be the de facto way of using basetools.

Three questions need to be answered:

1. After the patches that enable the pip module land, how long should the grace period be?
2. During the grace period, should basetools commits land in both places or just in the edk2-basetools directory?
3. How should the user be able to select which basetools to use (the one in EDK2 or the pip module)? Currently the approach being considered is a simple environmental variable? One of the key considerations is transparency since it won't be apparent what is being used for a particular build without some sort of mechanism to notify the developer. With two seperate versions of Basetools, it becomes very easy for the version of basetools you're using to not be the one you expect.
Matthew,

I’ll throw out some current developer centric ideas.

1) If you `source edksetup.sh` (edksetup.bat) you get the current behavior, and you add an argument you get the pip flavor? So maybe `edksetup.bat pip-basetools`?
2) We have similar issues to this with env variables and the build command dumps them out when it runs. Can we use the current EDK_TOOL_PATH? Or maybe add an extra print to show that the pip module is being used?
I've skimmed:

- the earlier discussion linked above (rooted at <https://edk2.groups.io/g/rfc/message/270>),

- the even earlier comments in the "Discussion: Basetools a separate repo" thread on edk2-devel:

https://edk2.groups.io/g/devel/message/57482
http://mid.mail-archive.com/734D49CCEBEEF84792F5B80ED585239D5C5019CE@SHSMSX104.ccr.corp.intel.com

If I still understand / remember correctly, the way at least *I* would use the new feature is the following:

- set up a new virtual python environment,

- either install the new pip module "permanently" in the virtual environment, or else install it in "editable mode" from a git checkout (but *still* in the virtual environment)

- build edk2 with the virtual environment active

That is, for me anyway, the key distinguishing factor would be that I'd be in a python virtual environment where this particular python module existed / were installed.

Does this answer question (3)? Because, in my case anyway, I wouldn't have to be *notified* about using the separate basetools repo vs. using the one resident in edk2 -- instead, I'd have to *activate* the separate basetools repo myself, as first step. So if that activation brings some queriable feature into the environment (sets a new environment variable or makes a new python package appear in the environment), then I think it's good enough -- the usual tools that I run then can query these artifacts.

In short (I guess): commands should use the in-tree variant, unless I activate the virtual environment that has the basetools PIP module installed.

I think it would be fine to require that, *if* someone intends to activate such a python virtual environment, *then* they do so *before* running "edksetup.sh". So "edksetup.sh" could check for the python env having the external basetools repo / module active. Hopefully that would be "early enough".


Regarding the grace period -- questions (1) and (2):

- The patches introducing the new feature to edk2 should be posted to the list. These patches should also add a warning to "edksetup.sh" that urges the developer to use the out-of-tree basetools repo / PIP module, in case "edksetup.sh" determines the current choice is the in-tree variant (that is, the virtual env is inactive, or does not contain the new PIP module)

- While the patches are pending approval, BaseTools development is put on hold (no fixes, no features).

- For every package (subsystem) listed in Maintainers.txt, in *both* edk2 *and* edk2-platfomrs, at least one "M" person is required to report back with a Tested-by, meaning that they built said package successfully with the new PIP module.

- When this feedback is complete, the patch is merged, and the new PIP module becomes the default build system (see the edksetup.sh warning described above).

- Optimally, the above (= comprehensive testing feedback, and merging) would occur *early* in the development cycle (just after the last stable tag).

- Going forward, bug reports and feature requests are only addressed in the new (out-of-tree) module. If someone reports that they have to switch back, *temporarily*, for whatever reason, to the in-tree variant, and the in-tree variant no longer builds edk2 for them, then such issues can be resolved on a case-by-case basis, *after* the issue is reported. Point being, we make the out-of-tree system the new default because of the comprehensive and strict initial testing requirements (see above); after which the old system is preserved for a while only as a fallback. If the fallback proves lacking later on (but still during the grace period), then the community works to resolve the issue in one of two ways: either help the issue reporter eliminate their need to use the fallback in the first place, or backport the subject bugfix/feature to the fallback.

- After the *next* stable release (which still contains both the fallback and the support for the out-of-tree PIP module), the fallback is removed.

Ultimately this would make the grace period almost one full development cycle, in which cycle the new system should be tested comprehensively, and become the default, near the beginning of the period.

This is just my proposal. Some of the other stewards are temporarily away; I'd suggest waiting for their feedback too.

To finish up, I would like to highlight something from the earlier RFC:

"""
Contribution/Dev Process:
Since this is a separate repo, it will follow a slightly different contribution and code review process.
1. Github PR process will be used for contributions and code review feedback
a. The yet to be released “Tianocore PR archiver” will be used to send to a dedicated list for basetools patch review archive
2. PRs will only be committed if they keep linear history (no merge commits)
3. The PR review must be approved by at least 2 members of the basetools team (not including the author)
4. The PR must pass all automated checks
a. Formatting/style
b. Unit tests
c. Code coverage (can’t commit change that would decrease overall %)
d. DCO enforcement - https://probot.github.io/apps/dco/
e. See other python requirements from the Python coding standard
5. Github Issues will be used for non-security sensitive bugs/issues/feature requests
"""

Point (1a) is a pre-requisite for merging the edk2 patches!

We cannot make the new system the default unless its development process is integrated with the github-to-email gateway (webhook).

Thanks!
Laszlo


Matthew Carlson <matthewfcarlson@...>
 

Andrew:

I think leveraging the existing edksetup is a great idea. Using the
existing EDK_TOOL_PATH variable could work but it seems clunky. Since the
pip module wouldn't be a path, it seems strange to put a boolean value in a
variable meant to hold a path. I definitely think that the scripts could
print whether they're using the pip modules or the in-source tools. Since
Lazlo suggested that pip will be the default, we could have the in-source
modules notify of the fact that you're using the in-source modules. An
additional feature for the pip module could be printing the version that
they are (since you can use the pip infrastructure to query the version of
a given module within a python script). Another option would be simply
trying the pip module first and then falling back to the in-source module.
There would be a slight speed penalty (likely around 10ms) but since this
would only apply to trim and build, it should have relatively low impact.

Lazlo:
Thank you for the excellent summary of the different pieces of the
discussion along with the links. To answer your first point, I think what a
developer does with their pip module is largely up to them. They could do a
virtual environment, they could just do what the requirements state, or pip
install from a checked out basetools.I do think there are some variables
that the virtual environment sets up that would be a good signal whether
you're in a virtual environment or not. I agree with your approach of
basetools development going into the out of edk2 repo and the importance of
making sure package maintainers test and validate their areas with the new
setup. I would personally try to get this early into the development cycle,
(just after this next stable tag this week) to give the community and
developers the most amount of time to get used to things. A trial period of
one release makes sense.

I also agree that the gateway is important in maintaining cohesion between
the new and the old. Hopefully that's nearing completion.

Hopefully other stewards will weigh in but otherwise we'll move ahead with
a proposed solution in patches next week.

-Matthew Carlson

On Wed, Sep 2, 2020 at 1:49 AM Laszlo Ersek <lersek@redhat.com> wrote:

On 09/02/20 02:49, Andrew Fish via groups.io wrote:


On Sep 1, 2020, at 4:35 PM, Matthew Carlson <matthewfcarlson@gmail.com>
wrote:

Hello all,

A recent topic on the RFC mailing list went out and the work on moving
Basetools/Sources/Python to a separate repo has started. See the RFC
conversation here: https://edk2.groups.io/g/rfc/topic/74009714#270 <
https://edk2.groups.io/g/rfc/topic/74009714#270>

The repo in question is here:
https://github.com/tianocore/edk2-basetools <
https://github.com/tianocore/edk2-basetools>

The current plan is shortly after the stable tag is created, a series
of patches will come into edk2 that redirects the build system into the new
python module as well as adds additional documentation. You can see a
sample of this work here: https://github.com/matthewfcarlson/edk2 <
https://github.com/matthewfcarlson/edk2> as this has a branch that has
the work required to use the basetools pip module. The patches won't delete
the Basetools/Sources/Python folder but will allow users to select between
them. After a certain grace period, the python folder will be deleted and
the pip module will be the de facto way of using basetools.

Three questions need to be answered:

1. After the patches that enable the pip module land, how long should
the grace period be?
2. During the grace period, should basetools commits land in both
places or just in the edk2-basetools directory?
3. How should the user be able to select which basetools to use (the
one in EDK2 or the pip module)? Currently the approach being considered is
a simple environmental variable? One of the key considerations is
transparency since it won't be apparent what is being used for a particular
build without some sort of mechanism to notify the developer. With two
seperate versions of Basetools, it becomes very easy for the version of
basetools you're using to not be the one you expect.
Matthew,

I’ll throw out some current developer centric ideas.

1) If you `source edksetup.sh` (edksetup.bat) you get the current
behavior, and you add an argument you get the pip flavor? So maybe
`edksetup.bat pip-basetools`?
2) We have similar issues to this with env variables and the build
command dumps them out when it runs. Can we use the current EDK_TOOL_PATH?
Or maybe add an extra print to show that the pip module is being used?

I've skimmed:

- the earlier discussion linked above (rooted at <
https://edk2.groups.io/g/rfc/message/270>),

- the even earlier comments in the "Discussion: Basetools a separate repo"
thread on edk2-devel:

https://edk2.groups.io/g/devel/message/57482

http://mid.mail-archive.com/734D49CCEBEEF84792F5B80ED585239D5C5019CE@SHSMSX104.ccr.corp.intel.com

If I still understand / remember correctly, the way at least *I* would use
the new feature is the following:

- set up a new virtual python environment,

- either install the new pip module "permanently" in the virtual
environment, or else install it in "editable mode" from a git checkout (but
*still* in the virtual environment)

- build edk2 with the virtual environment active

That is, for me anyway, the key distinguishing factor would be that I'd be
in a python virtual environment where this particular python module existed
/ were installed.

Does this answer question (3)? Because, in my case anyway, I wouldn't have
to be *notified* about using the separate basetools repo vs. using the one
resident in edk2 -- instead, I'd have to *activate* the separate basetools
repo myself, as first step. So if that activation brings some queriable
feature into the environment (sets a new environment variable or makes a
new python package appear in the environment), then I think it's good
enough -- the usual tools that I run then can query these artifacts.

In short (I guess): commands should use the in-tree variant, unless I
activate the virtual environment that has the basetools PIP module
installed.

I think it would be fine to require that, *if* someone intends to activate
such a python virtual environment, *then* they do so *before* running
"edksetup.sh". So "edksetup.sh" could check for the python env having the
external basetools repo / module active. Hopefully that would be "early
enough".


Regarding the grace period -- questions (1) and (2):

- The patches introducing the new feature to edk2 should be posted to the
list. These patches should also add a warning to "edksetup.sh" that urges
the developer to use the out-of-tree basetools repo / PIP module, in case
"edksetup.sh" determines the current choice is the in-tree variant (that
is, the virtual env is inactive, or does not contain the new PIP module)

- While the patches are pending approval, BaseTools development is put on
hold (no fixes, no features).

- For every package (subsystem) listed in Maintainers.txt, in *both* edk2
*and* edk2-platfomrs, at least one "M" person is required to report back
with a Tested-by, meaning that they built said package successfully with
the new PIP module.

- When this feedback is complete, the patch is merged, and the new PIP
module becomes the default build system (see the edksetup.sh warning
described above).

- Optimally, the above (= comprehensive testing feedback, and merging)
would occur *early* in the development cycle (just after the last stable
tag).

- Going forward, bug reports and feature requests are only addressed in
the new (out-of-tree) module. If someone reports that they have to switch
back, *temporarily*, for whatever reason, to the in-tree variant, and the
in-tree variant no longer builds edk2 for them, then such issues can be
resolved on a case-by-case basis, *after* the issue is reported. Point
being, we make the out-of-tree system the new default because of the
comprehensive and strict initial testing requirements (see above); after
which the old system is preserved for a while only as a fallback. If the
fallback proves lacking later on (but still during the grace period), then
the community works to resolve the issue in one of two ways: either help
the issue reporter eliminate their need to use the fallback in the first
place, or backport the subject bugfix/feature to the fallback.

- After the *next* stable release (which still contains both the fallback
and the support for the out-of-tree PIP module), the fallback is removed.

Ultimately this would make the grace period almost one full development
cycle, in which cycle the new system should be tested comprehensively, and
become the default, near the beginning of the period.

This is just my proposal. Some of the other stewards are temporarily away;
I'd suggest waiting for their feedback too.

To finish up, I would like to highlight something from the earlier RFC:

"""
Contribution/Dev Process:
Since this is a separate repo, it will follow a slightly different
contribution and code review process.
1. Github PR process will be used for contributions and code review
feedback
a. The yet to be released “Tianocore PR archiver” will be used to
send to a dedicated list for basetools patch review archive
2. PRs will only be committed if they keep linear history (no merge
commits)
3. The PR review must be approved by at least 2 members of the
basetools team (not including the author)
4. The PR must pass all automated checks
a. Formatting/style
b. Unit tests
c. Code coverage (can’t commit change that would decrease overall %)
d. DCO enforcement - https://probot.github.io/apps/dco/
e. See other python requirements from the Python coding standard
5. Github Issues will be used for non-security sensitive
bugs/issues/feature requests
"""

Point (1a) is a pre-requisite for merging the edk2 patches!

We cannot make the new system the default unless its development process
is integrated with the github-to-email gateway (webhook).

Thanks!
Laszlo


Andrew Fish <afish@...>
 

Matthew,

I did not meant to imply we should optimize for the current installed base, I just think it is useful to think about them. I think Lazlo is pointing out what is best for the project and that should have more weight than the installed base, but it is always good to think how things impact different groups.

In terms of the UI I was thinking of pointing to the pip install location vs. your git repo not so much a boolean. I actually don’t quite understand what EDK_TOOL_PATH actually means if BaseTools is a pip module. If that is an obsolete concept then we should remove it, and replace with some kind of statement that the pip installed BaseTools are being used. Another question if I run build from the command line how did my path get set? For example my user account has 5 different versions of edk2 in it how do I configure different versions of BaseTools? How do I pip multiple versions on to my system? I guess my repo could have a yml file that points to the version of the tools that I use, but that seems like a per user, not per repo config? It seems to me we could have developers that want to contribute to edk2 and work on their own code base and that could rehire two different Basetool versions installed on the system, and I think we need a story for that.

For the macOS Xcode compiler you can install as many versions as you want and there is a command line tool to let you set the current version of Xcode, and to show you the currently select versions. Basically the tools in the magic location in your path are just redirectors to the currently selected tools.

Thanks,

Andrew Fish

On Sep 2, 2020, at 12:06 PM, Matthew Carlson <matthewfcarlson@gmail.com> wrote:

Andrew:

I think leveraging the existing edksetup is a great idea. Using the existing EDK_TOOL_PATH variable could work but it seems clunky. Since the pip module wouldn't be a path, it seems strange to put a boolean value in a variable meant to hold a path. I definitely think that the scripts could print whether they're using the pip modules or the in-source tools. Since Lazlo suggested that pip will be the default, we could have the in-source modules notify of the fact that you're using the in-source modules. An additional feature for the pip module could be printing the version that they are (since you can use the pip infrastructure to query the version of a given module within a python script). Another option would be simply trying the pip module first and then falling back to the in-source module. There would be a slight speed penalty (likely around 10ms) but since this would only apply to trim and build, it should have relatively low impact.

Lazlo:
Thank you for the excellent summary of the different pieces of the discussion along with the links. To answer your first point, I think what a developer does with their pip module is largely up to them. They could do a virtual environment, they could just do what the requirements state, or pip install from a checked out basetools.I do think there are some variables that the virtual environment sets up that would be a good signal whether you're in a virtual environment or not. I agree with your approach of basetools development going into the out of edk2 repo and the importance of making sure package maintainers test and validate their areas with the new setup. I would personally try to get this early into the development cycle, (just after this next stable tag this week) to give the community and developers the most amount of time to get used to things. A trial period of one release makes sense.

I also agree that the gateway is important in maintaining cohesion between the new and the old. Hopefully that's nearing completion.

Hopefully other stewards will weigh in but otherwise we'll move ahead with a proposed solution in patches next week.

-Matthew Carlson


On Wed, Sep 2, 2020 at 1:49 AM Laszlo Ersek <lersek@redhat.com <mailto:lersek@redhat.com>> wrote:
On 09/02/20 02:49, Andrew Fish via groups.io <http://groups.io/> wrote:


On Sep 1, 2020, at 4:35 PM, Matthew Carlson <matthewfcarlson@gmail.com <mailto:matthewfcarlson@gmail.com>> wrote:

Hello all,

A recent topic on the RFC mailing list went out and the work on moving Basetools/Sources/Python to a separate repo has started. See the RFC conversation here: https://edk2.groups.io/g/rfc/topic/74009714#270

The repo in question is here: https://github.com/tianocore/edk2-basetools

The current plan is shortly after the stable tag is created, a series of patches will come into edk2 that redirects the build system into the new python module as well as adds additional documentation. You can see a sample of this work here: https://github.com/matthewfcarlson/edk2 as this has a branch that has the work required to use the basetools pip module. The patches won't delete the Basetools/Sources/Python folder but will allow users to select between them. After a certain grace period, the python folder will be deleted and the pip module will be the de facto way of using basetools.

Three questions need to be answered:

1. After the patches that enable the pip module land, how long should the grace period be?
2. During the grace period, should basetools commits land in both places or just in the edk2-basetools directory?
3. How should the user be able to select which basetools to use (the one in EDK2 or the pip module)? Currently the approach being considered is a simple environmental variable? One of the key considerations is transparency since it won't be apparent what is being used for a particular build without some sort of mechanism to notify the developer. With two seperate versions of Basetools, it becomes very easy for the version of basetools you're using to not be the one you expect.
Matthew,

I’ll throw out some current developer centric ideas.

1) If you `source edksetup.sh` (edksetup.bat) you get the current behavior, and you add an argument you get the pip flavor? So maybe `edksetup.bat pip-basetools`?
2) We have similar issues to this with env variables and the build command dumps them out when it runs. Can we use the current EDK_TOOL_PATH? Or maybe add an extra print to show that the pip module is being used?
I've skimmed:

- the earlier discussion linked above (rooted at <https://edk2.groups.io/g/rfc/message/270>),

- the even earlier comments in the "Discussion: Basetools a separate repo" thread on edk2-devel:

https://edk2.groups.io/g/devel/message/57482
http://mid.mail-archive.com/734D49CCEBEEF84792F5B80ED585239D5C5019CE@SHSMSX104.ccr.corp.intel.com

If I still understand / remember correctly, the way at least *I* would use the new feature is the following:

- set up a new virtual python environment,

- either install the new pip module "permanently" in the virtual environment, or else install it in "editable mode" from a git checkout (but *still* in the virtual environment)

- build edk2 with the virtual environment active

That is, for me anyway, the key distinguishing factor would be that I'd be in a python virtual environment where this particular python module existed / were installed.

Does this answer question (3)? Because, in my case anyway, I wouldn't have to be *notified* about using the separate basetools repo vs. using the one resident in edk2 -- instead, I'd have to *activate* the separate basetools repo myself, as first step. So if that activation brings some queriable feature into the environment (sets a new environment variable or makes a new python package appear in the environment), then I think it's good enough -- the usual tools that I run then can query these artifacts.

In short (I guess): commands should use the in-tree variant, unless I activate the virtual environment that has the basetools PIP module installed.

I think it would be fine to require that, *if* someone intends to activate such a python virtual environment, *then* they do so *before* running "edksetup.sh". So "edksetup.sh" could check for the python env having the external basetools repo / module active. Hopefully that would be "early enough".


Regarding the grace period -- questions (1) and (2):

- The patches introducing the new feature to edk2 should be posted to the list. These patches should also add a warning to "edksetup.sh" that urges the developer to use the out-of-tree basetools repo / PIP module, in case "edksetup.sh" determines the current choice is the in-tree variant (that is, the virtual env is inactive, or does not contain the new PIP module)

- While the patches are pending approval, BaseTools development is put on hold (no fixes, no features).

- For every package (subsystem) listed in Maintainers.txt, in *both* edk2 *and* edk2-platfomrs, at least one "M" person is required to report back with a Tested-by, meaning that they built said package successfully with the new PIP module.

- When this feedback is complete, the patch is merged, and the new PIP module becomes the default build system (see the edksetup.sh warning described above).

- Optimally, the above (= comprehensive testing feedback, and merging) would occur *early* in the development cycle (just after the last stable tag).

- Going forward, bug reports and feature requests are only addressed in the new (out-of-tree) module. If someone reports that they have to switch back, *temporarily*, for whatever reason, to the in-tree variant, and the in-tree variant no longer builds edk2 for them, then such issues can be resolved on a case-by-case basis, *after* the issue is reported. Point being, we make the out-of-tree system the new default because of the comprehensive and strict initial testing requirements (see above); after which the old system is preserved for a while only as a fallback. If the fallback proves lacking later on (but still during the grace period), then the community works to resolve the issue in one of two ways: either help the issue reporter eliminate their need to use the fallback in the first place, or backport the subject bugfix/feature to the fallback.

- After the *next* stable release (which still contains both the fallback and the support for the out-of-tree PIP module), the fallback is removed.

Ultimately this would make the grace period almost one full development cycle, in which cycle the new system should be tested comprehensively, and become the default, near the beginning of the period.

This is just my proposal. Some of the other stewards are temporarily away; I'd suggest waiting for their feedback too.

To finish up, I would like to highlight something from the earlier RFC:

"""
Contribution/Dev Process:
Since this is a separate repo, it will follow a slightly different contribution and code review process.
1. Github PR process will be used for contributions and code review feedback
a. The yet to be released “Tianocore PR archiver” will be used to send to a dedicated list for basetools patch review archive
2. PRs will only be committed if they keep linear history (no merge commits)
3. The PR review must be approved by at least 2 members of the basetools team (not including the author)
4. The PR must pass all automated checks
a. Formatting/style
b. Unit tests
c. Code coverage (can’t commit change that would decrease overall %)
d. DCO enforcement - https://probot.github.io/apps/dco/
e. See other python requirements from the Python coding standard
5. Github Issues will be used for non-security sensitive bugs/issues/feature requests
"""

Point (1a) is a pre-requisite for merging the edk2 patches!

We cannot make the new system the default unless its development process is integrated with the github-to-email gateway (webhook).

Thanks!
Laszlo


Matthew Carlson <matthewfcarlson@...>
 

Hey Andrew,

Your comments were helpful and valid 😊

Part of the appeal of making it into a pip module is that it doesn’t have a specific file system path. Python itself take cares of it and it can be addressed by the module name rather than adding it to the path. So you hit the nail on the head, EDK_TOOL_PATH doesn’t really make sense in an all pip module world. It will probably be deprecated. You can see here: https://github.com/matthewfcarlson/edk2/blob/feature/pip-basetools/BaseTools/BinWrappers/WindowsLike/build.bat that we call directly into the module without having to setup the python path. The file here doesn’t use the BaseTools inside EDK2, that still need to built.

You have some great questions and I’ll do my best to answer with what has been discussed on the original RFC.

* if I run build from the command line how did my path get set?
* The pip system adds it to the python patch for you where ever you have it installed (locally, user, system, etc). In addition, pip module offers console scripts, which are small hooks that basically do what the build.bat script does: start the python interpreter and call the build module. So in the current pip module there are a few new commands, one of them being edk2_build which is available anywhere (within the scope of your pip environment). Alternatively, you could use the BinWrappers and let the edksetup do the heavy lifting for you. It should all just work out of the box (fingers crossed).
* How do I configure different versions of BaseTools?
* So Lazlo pointed out that you could use virtual environments, one for each edk2 clone on your system. It has the added benefit of making your system state much simpler and more reproducible since the pip module are for that repo rather than your whole system. Another option is just to install a local copy of the pip module using pip install -e . which will stay in sync with whatever you have checked out. This does mean an extra step of making sure you have the right commit checked out and probably isn’t advisable since it’s easy to forget. But since I only am ever have one clone open at a time, it’s not bad for me personally.
* I guess my repo could have a yml file that points to the version of the tools that I use, but that seems like a per user, not per repo config
* There’s a pip requirement file that keeps track of the exact version of the pip module you need (EDK2 already has one for CI). For a given commit in EDK2, you’d have the exact version of basetools that you know works.
* It seems to me we could have developers that want to contribute to edk2 and work on their own code base and that could rehire (require?) two different Basetool versions installed on the system, and I think we need a story for that?
* Yeah- I agree that story isn’t as fleshed out. In some way it’s better than EDK2’s current story as the basetools can be consumed separately from EDK2 itself. You could setup your own codebase to leverage the basetools as is or you could point it at your own fork of the BaseTools. But I can see folks who have forked EDK2 and have tweaks in BaseTools having to push those changes back into BaseTools. I believe that moving it into it’s own project will make it easier for fixes and other projects to be built off the BaseTools as it makes it easy to include without needing the rest of EDK2 with it. For example, easy to use and easy to install capsule generation tools for FMP devices.
* For the macOS Xcode compiler you can install as many versions as you want and there is a command line tool to let you set the current version of Xcode, and to show you the currently select versions. Basically the tools in the magic location in your path are just redirectors to the currently selected tools.
* I think that’s possible inside pip to have sort of the functionality you describe but generally it would be through virtual environments and perhaps installing it in different places (system vs user for example). With pip, you can also do a pip freeze to see the state of your pip system in the given environment. It’s likely not as robust/fully-featured as the XCode system, though it’s been a few years since I was a serious XCode user and my recollection of exactly the experience is a little fuzzy.

Thanks again for your feedback and hopefully that addresses your questions.

*From:* Andrew Fish ( afish@apple.com )
*Sent:* Wednesday, September 2, 2020 7:14 PM
*To:* Matthew Carlson ( matthewfcarlson@gmail.com )
*Cc:* Laszlo Ersek ( lersek@redhat.com ) ; edk2-devel-groups-io ( devel@edk2.groups.io ) ; rfc@edk2.groups.io
*Subject:* Re: [edk2-devel] Basetools as a pip module

Matthew,

I did not meant to imply we should optimize for the current installed base, I just think it is useful to think about them. I think Lazlo is pointing out what is best for the project and that should have more weight than the installed base, but it is always good to think how things impact different groups.

In terms of the UI I was thinking of pointing to the pip install location vs. your git repo not so much a boolean. I actually don’t quite understand what EDK_TOOL_PATH actually means if BaseTools is a pip module. If that is an obsolete concept then we should remove it, and replace with some kind of statement that the pip installed BaseTools are being used. Another question if I run build from the command line how did my path get set? For example my user account has 5 different versions of edk2 in it how do I configure different versions of BaseTools? How do I pip multiple versions on to my system? I guess my repo could have a yml file that points to the version of the tools that I use, but that seems like a per user, not per repo config? It seems to me we could have developers that want to contribute to edk2 and work on their own code base and that could rehire two different Basetool versions installed on the system, and I think we need a story for that.

For the macOS Xcode compiler you can install as many versions as you want and there is a command line tool to let you set the current version of Xcode, and to show you the currently select versions. Basically the tools in the magic location in your path are just redirectors to the currently selected tools.

Thanks,

Andrew Fish




On Sep 2, 2020, at 12:06 PM, Matthew Carlson < matthewfcarlson@gmail.com >
wrote:








Andrew:









I think leveraging the existing edksetup is a great idea. Using the
existing EDK_TOOL_PATH variable could work but it seems clunky. Since the
pip module wouldn't be a path, it seems strange to put a boolean value in
a variable meant to hold a path. I definitely think that the scripts could
print whether they're using the pip modules or the in-source tools. Since
Lazlo suggested that pip will be the default, we could have the in-source
modules notify of the fact that you're using the in-source modules. An
additional feature for the pip module could be printing the version that
they are (since you can use the pip infrastructure to query the version of
a given module within a python script). Another option would be simply
trying the pip module first and then falling back to the in-source module.
There would be a slight speed penalty (likely around 10ms) but since this
would only apply to trim and build, it should have relatively low impact.









Lazlo:




Thank you for the excellent summary of the different pieces of the
discussion along with the links. To answer your first point, I think what
a developer does with their pip module is largely up to them. They could
do a virtual environment, they could just do what the requirements state,
or pip install from a checked out basetools.I do think there are some
variables that the virtual environment sets up that would be a good signal
whether you're in a virtual environment or not. I agree with your approach
of basetools development going into the out of edk2 repo and the
importance of making sure package maintainers test and validate their
areas with the new setup. I would personally try to get this early into
the development cycle, (just after this next stable tag this week) to give
the community and developers the most amount of time to get used to
things. A trial period of one release makes sense.









I also agree that the gateway is important in maintaining cohesion between
the new and the old. Hopefully that's nearing completion.









Hopefully other stewards will weigh in but otherwise we'll move ahead with
a proposed solution in patches next week.









-Matthew Carlson













On Wed, Sep 2, 2020 at 1:49 AM Laszlo Ersek < lersek@redhat.com > wrote:





On 09/02/20 02:49, Andrew Fish via groups.io ( http://groups.io/ ) wrote:


On Sep 1, 2020, at 4:35 PM, Matthew Carlson < matthewfcarlson@gmail.com
wrote:

Hello all,

A recent topic on the RFC mailing list went out and the work on moving
Basetools/Sources/Python to a separate repo has started. See the RFC
conversation here: https://edk2.groups.io/g/rfc/topic/74009714#270 < https://edk2.groups.io/g/rfc/topic/74009714#270


The repo in question is here: https://github.com/tianocore/edk2-basetools
< https://github.com/tianocore/edk2-basetools >

The current plan is shortly after the stable tag is created, a series
of patches will come into edk2 that redirects the build system into the
new python module as well as adds additional documentation. You can see a
sample of this work here: https://github.com/matthewfcarlson/edk2 < https://github.com/matthewfcarlson/edk2
as this has a branch that has the work required to use the basetools pip
module. The patches won't delete the Basetools/Sources/Python folder but
will allow users to select between them. After a certain grace period, the
python folder will be deleted and the pip module will be the de facto way
of using basetools.

Three questions need to be answered:

1. After the patches that enable the pip module land, how long should
the grace period be?
2. During the grace period, should basetools commits land in both
places or just in the edk2-basetools directory?
3. How should the user be able to select which basetools to use (the
one in EDK2 or the pip module)? Currently the approach being considered is
a simple environmental variable? One of the key considerations is
transparency since it won't be apparent what is being used for a
particular build without some sort of mechanism to notify the developer.
With two seperate versions of Basetools, it becomes very easy for the
version of basetools you're using to not be the one you expect.
Matthew,

I’ll throw out some current developer centric ideas.

1) If you `source edksetup.sh` (edksetup.bat) you get the current
behavior, and you add an argument you get the pip flavor? So maybe
`edksetup.bat pip-basetools`?
2) We have similar issues to this with env variables and the build
command dumps them out when it runs. Can we use the current EDK_TOOL_PATH?
Or maybe add an extra print to show that the pip module is being used?

I've skimmed:

- the earlier discussion linked above (rooted at < https://edk2.groups.io/g/rfc/message/270
),
- the even earlier comments in the "Discussion: Basetools a separate repo"
thread on edk2-devel:

https://edk2.groups.io/g/devel/message/57482
http://mid.mail-archive.com/734D49CCEBEEF84792F5B80ED585239D5C5019CE@SHSMSX104.ccr.corp.intel.com


If I still understand / remember correctly, the way at least *I* would use
the new feature is the following:

- set up a new virtual python environment,

- either install the new pip module "permanently" in the virtual
environment, or else install it in "editable mode" from a git checkout
(but *still* in the virtual environment)

- build edk2 with the virtual environment active

That is, for me anyway, the key distinguishing factor would be that I'd be
in a python virtual environment where this particular python module
existed / were installed.

Does this answer question (3)? Because, in my case anyway, I wouldn't have
to be *notified* about using the separate basetools repo vs. using the one
resident in edk2 -- instead, I'd have to *activate* the separate basetools
repo myself, as first step. So if that activation brings some queriable
feature into the environment (sets a new environment variable or makes a
new python package appear in the environment), then I think it's good
enough -- the usual tools that I run then can query these artifacts.

In short (I guess): commands should use the in-tree variant, unless I
activate the virtual environment that has the basetools PIP module
installed.

I think it would be fine to require that, *if* someone intends to activate
such a python virtual environment, *then* they do so *before* running
"edksetup.sh". So "edksetup.sh" could check for the python env having the
external basetools repo / module active. Hopefully that would be "early
enough".


Regarding the grace period -- questions (1) and (2):

- The patches introducing the new feature to edk2 should be posted to the
list. These patches should also add a warning to "edksetup.sh" that urges
the developer to use the out-of-tree basetools repo / PIP module, in case
"edksetup.sh" determines the current choice is the in-tree variant (that
is, the virtual env is inactive, or does not contain the new PIP module)

- While the patches are pending approval, BaseTools development is put on
hold (no fixes, no features).

- For every package (subsystem) listed in Maintainers.txt, in *both* edk2
*and* edk2-platfomrs, at least one "M" person is required to report back
with a Tested-by, meaning that they built said package successfully with
the new PIP module.

- When this feedback is complete, the patch is merged, and the new PIP
module becomes the default build system (see the edksetup.sh warning
described above).

- Optimally, the above (= comprehensive testing feedback, and merging)
would occur *early* in the development cycle (just after the last stable
tag).

- Going forward, bug reports and feature requests are only addressed in
the new (out-of-tree) module. If someone reports that they have to switch
back, *temporarily*, for whatever reason, to the in-tree variant, and the
in-tree variant no longer builds edk2 for them, then such issues can be
resolved on a case-by-case basis, *after* the issue is reported. Point
being, we make the out-of-tree system the new default because of the
comprehensive and strict initial testing requirements (see above); after
which the old system is preserved for a while only as a fallback. If the
fallback proves lacking later on (but still during the grace period), then
the community works to resolve the issue in one of two ways: either help
the issue reporter eliminate their need to use the fallback in the first
place, or backport the subject bugfix/feature to the fallback.

- After the *next* stable release (which still contains both the fallback
and the support for the out-of-tree PIP module), the fallback is removed.

Ultimately this would make the grace period almost one full development
cycle, in which cycle the new system should be tested comprehensively, and
become the default, near the beginning of the period.

This is just my proposal. Some of the other stewards are temporarily away;
I'd suggest waiting for their feedback too.

To finish up, I would like to highlight something from the earlier RFC:

"""
Contribution/Dev Process:
Since this is a separate repo, it will follow a slightly different
contribution and code review process.
1.      Github PR process will be used for contributions and code review
feedback
a.      The yet to be released “Tianocore PR archiver” will be used to
send to a dedicated list for basetools patch review archive
2.      PRs will only be committed if they keep linear history (no merge
commits)
3.      The PR review must be approved by at least 2 members of the
basetools team (not including the author)
4.      The PR must pass all automated checks
a.      Formatting/style
b.      Unit tests
c.      Code coverage (can’t commit change that would decrease overall %)
d.      DCO enforcement - https://probot.github.io/apps/dco/
e.      See other python requirements from the Python coding standard
5.      Github Issues will be used for non-security sensitive
bugs/issues/feature requests
"""

Point (1a) is a pre-requisite for merging the edk2 patches!

We cannot make the new system the default unless its development process
is integrated with the github-to-email gateway (webhook).

Thanks!
Laszlo