I am looking at converting our build from using a bunch of custom scripts/makefiles around the UEFI build process to the edk2toolext based python builder and am running into an Issue. Our overall build automation system whats to keep the source directory unchanged with all build output in another directory. I am supporting this currently by having WORKSPACE point to the specified out directory and PACKAGES_PATH pointing to our source. However if I have workspace point somewhere else with edk2toolext it seems like it requires the workspace to be with the sources. Is this something that can be done or is just a limitation/design feature of this build system?

Thanks,
Jeff

Can you elaborate more on what you have tried. What files are causing problems? The build output should be handled the same as any edk2 build

Thanks
Sean
________________________________
From: discuss@edk2.groups.io <discuss@edk2.groups.io> on behalf of Jeff Brasen <jbrasen@...>
Sent: Tuesday, July 6, 2021 9:14:14 AM
To: discuss@edk2.groups.io <discuss@edk2.groups.io>
Subject: [edk2-discuss] edk2toolext based build workspace question

I am looking at converting our build from using a bunch of custom scripts/makefiles around the UEFI build process to the edk2toolext based python builder and am running into an Issue. Our overall build automation system whats to keep the source directory unchanged with all build output in another directory. I am supporting this currently by having WORKSPACE point to the specified out directory and PACKAGES_PATH pointing to our source. However if I have workspace point somewhere else with edk2toolext it seems like it requires the workspace to be with the sources. Is this something that can be done or is just a limitation/design feature of this build system?

Thanks,
Jeff

Basically, the workspace directory that is used in the scripts seem to assume that there are files under there already, as opposed to just an out directory as we use it and use packages path to search everything. It seems to search for a bunch of things based on that prior to calling the edk2 build subsystem. I think we found another way to support that with our CI system so I think I can work with the current behavior.


Thanks,

Jeff

________________________________
From: discuss@edk2.groups.io <discuss@edk2.groups.io> on behalf of Sean via groups.io <spbrogan@...>
Sent: Tuesday, July 6, 2021 8:55 PM
To: discuss@edk2.groups.io <discuss@edk2.groups.io>; Jeff Brasen <jbrasen@...>
Subject: Re: [edk2-discuss] edk2toolext based build workspace question

External email: Use caution opening links or attachments


Can you elaborate more on what you have tried. What files are causing problems? The build output should be handled the same as any edk2 build

Thanks
Sean
________________________________
From: discuss@edk2.groups.io <discuss@edk2.groups.io> on behalf of Jeff Brasen <jbrasen@...>
Sent: Tuesday, July 6, 2021 9:14:14 AM
To: discuss@edk2.groups.io <discuss@edk2.groups.io>
Subject: [edk2-discuss] edk2toolext based build workspace question

I am looking at converting our build from using a bunch of custom scripts/makefiles around the UEFI build process to the edk2toolext based python builder and am running into an Issue. Our overall build automation system whats to keep the source directory unchanged with all build output in another directory. I am supporting this currently by having WORKSPACE point to the specified out directory and PACKAGES_PATH pointing to our source. However if I have workspace point somewhere else with edk2toolext it seems like it requires the workspace to be with the sources. Is this something that can be done or is just a limitation/design feature of this build system?

Thanks,
Jeff

Jeff,

All the stuart tools (edk2toolext) require a configuration file. For edk2 the file is

https://github.com/tianocore/edk2/blob/master/.pytool/CISettings.py

In this file you can see the workspace is controlled here:
https://github.com/tianocore/edk2/blob/master/.pytool/CISettings.py#L205
as well as the packages path here:
https://github.com/tianocore/edk2/blob/master/.pytool/CISettings.py#L202

The intent is for your repo you would have your own CISettings.py where you would configure your workspace and packages path.

The log files are created in <workspace>/Build/<filename>

<workspace>: is controlled by your CISettings as mentioned above.
<filename>: Invokable invoked controls the name. Example for CI build -
https://github.com/tianocore/edk2-pytool-extensions/blob/master/edk2toolext/invocables/edk2_ci_build.py#L46

The edk2 conf is created at <workspace>/Conf as you can see here:
https://github.com/tianocore/edk2-pytool-extensions/blob/9400bfd0162a0da40f14f7f7154ac6097de43302/edk2toolext/environment/uefi_build.py#L379

And build binary output is controlled by edk2 build process using the OUTPUT_DIRECTORY variable generally defined in your DSC:
https://github.com/tianocore/edk2-pytool-extensions/blob/9400bfd0162a0da40f14f7f7154ac6097de43302/edk2toolext/environment/uefi_build.py#L406

You could also set that in your command line.


Just for information:

I tried to do this with edk2 so I had a tree like

/test1 <workspace>
/edk2 <packages path>

I then updated edk2 CI settings to use test1 as the workspace root and edk2 as a packages path. This moved the output into /test1/Build and /test1/Conf but it doesn't completely work because if you are using git submodules (and stuart_setup) there is an assumption that the repo you are testing is the workspace. This makes sense because your repo is what is in your control. You can't control a folder above your repo. I have seen many projects setup that use a readme file to require you to configure a workspace differently but to be honest I don't think that is valid. That leads to a fragile repo configuration that doesn't scale and/or can't be included in other projects. pytools took the approach that repos can only control what is in them and not the larger environment they may be a part of.

So the above breaks that rule and doesn't work. Now I am confident that if you were building a platform and included edk2 in your repository as a submodule it would work fine. Also if you updated edk2 to have a new root folder and updated all the git submodules correctly the above change would work too. For platform building this mainly impacts "stuart_setup" because stuart_setup uses submodules. In stuart_ci_build I do see that some of the edk2 plugins (ecccheck and patchcheck) which try to use git commands fail. Again this is because the workspace is assumed to be the git repo root. If there is a desire to change the assumption between workspace being a repo root that would a new conversation but as mentioned above I don't think that is scalable design.

Anyway, lots of options. I am on the discord channel (https://discord.gg/WAzZbrF7) if you would like to have a faster back and forth discussion. If you find an issue or even a question in the pytools please file it here https://github.com/tianocore/edk2-pytool-extensions/issues but it sounds like you might have a solution.

Thanks
Sean