Hi everyone,

I am looking for a good way to keep stuart_update from downloading
(some) external dependencies. I want to make use of the tools
provided by the system instead. Until now I just deleted the
corresponding ext_dep.yaml files, but that is not a viable general
solution.

Is there a mechanism to ignore specific ext_dep files? If not, what
about a command line option for that?

Background: I am working on a containerized CI setup and there the
container image shall provide tool chain (gcc, ...) and I want to
avoid downloading everything over and over.

Any comments and ideas are very welcome!

Thanks,
Oliver

Oliver,

This would be better posted here: Discussions · tianocore/edk2-pytool-extensions (github.com) <https://github.com/tianocore/edk2-pytool-extensions/discussions> as then it will be searchable and can be referenced by all (and overtime).

In short.  All ext_deps and stuart files use a scope.  The scope allows platform and ci builds to opt-in or opt-out.  We generally don't expect callers to change scopes directly (and don't want that) but instead the build files detect the environment and the caller's feature request and then set those scope appropriately. For example in edk2 core CI if the caller wants to build AARCH64 and is on Linux then it adds the scope to get a known version of the compiler.

Stuart and ext_deps are intended to enable local and server based development consistently and handle changes over time (meaning I can build a 2 year old branch/release now just as i could 2 years ago).    I think with the work you are doing now around containers we can probably solve this problems more easily using that.

To resolve your current question i think we would need to update CiSettings.py (edk2/CISettings.py at master · tianocore/edk2 (github.com) <https://github.com/tianocore/edk2/blob/master/.pytool/CISettings.py#L146>) to know if it is containerized and then not add the scopes for compiler download and instead leverage local compilers. If we want to drop support for non-containerized linux builds from Edk2 CI I would be fine with that too but would be curious to know what others think and find useful.


Thanks

Sean

Sean,

thanks for the info.

On Mon, Oct 10, 2022 at 9:28 PM Sean Brogan <spbrogan@...> wrote:

Oliver,

This would be better posted here: Discussions · tianocore/edk2-pytool-extensions (github.com) as then it will be searchable and can be referenced by all (and overtime).

Then let me re-post my question there.
Is it ok if I include your answer too?

In short. All ext_deps and stuart files use a scope. The scope allows platform and ci builds to opt-in or opt-out. We generally don't expect callers to change scopes directly (and don't want that) but instead the build files detect the environment and the caller's feature request and then set those scope appropriately. For example in edk2 core CI if the caller wants to build AARCH64 and is on Linux then it adds the scope to get a known version of the compiler.

Stuart and ext_deps are intended to enable local and server based development consistently and handle changes over time (meaning I can build a 2 year old branch/release now just as i could 2 years ago). I think with the work you are doing now around containers we can probably solve this problems more easily using that.

To resolve your current question i think we would need to update CiSettings.py (edk2/CISettings.py at master · tianocore/edk2 (github.com)) to know if it is containerized and then not add the scopes for compiler download and instead leverage local compilers. If we want to drop support for non-containerized linux builds from Edk2 CI I would be fine with that too but would be curious to know what others think and find useful.

Thanks. I'll take a look.

Thanks

Sean


On 10/10/2022 5:56 AM, Oliver Steffen wrote:

Hi everyone,

I am looking for a good way to keep stuart_update from downloading
(some) external dependencies. I want to make use of the tools
provided by the system instead. Until now I just deleted the
corresponding ext_dep.yaml files, but that is not a viable general
solution.

Is there a mechanism to ignore specific ext_dep files? If not, what
about a command line option for that?

Background: I am working on a containerized CI setup and there the
container image shall provide tool chain (gcc, ...) and I want to
avoid downloading everything over and over.

Any comments and ideas are very welcome!

Thanks,
Oliver