devel@edk2.groups.io

Re: [PATCH v2 1/2] OvmfPkg/Bhyve: add USB support

Michael Brown

#77447

On 02/07/2021 00:31, Peter Grehan wrote:
  bhyve doesn't (and will likely never) support Uhci/Ehci controllers so those lines can be removed.Is it possible to pass such a controller by PCI-Passthrough to a VM?
If it's possible, I would keep these lines.  Uhci/Ehci only support legacy interrupts and that isn't supported by bhyve PCI passthru.Is that a detail of the current implementation, or a fundamental limitation in the bhyve architecture? Only two choices ? :) Maybe half way between those points.When there is no fundamental limitation, i.e. when a future version of the hypervisor may be able to support the feature with no changes to the firmware, then it would be good practice to leave the drivers enabled. Doing so avoids creating an unnecessarily tight coupling between the hypervisor and firmware versions.

More importantly: does it even matter that the hypervisor doesn't support passthrough of PCI legacy interrupts?  UEFI operates on a polling basis, with the only active interrupt being some kind of periodic timer.  Where do you see any requirement for legacy interrupts in the UHCI/EHCI drivers?

Thanks,

Michael

Re: EFI_AUDIO_OUTPUT_PROTOCOL: assistance with VirtIO initialization

Michael Brown

#77446

On 02/07/2021 10:41, Michael Brown wrote:
UsbIo->UsbControlTransfer(UsbIo, &Req, EfiUsbDataIn,
                           PcdGet32 (PcdUsbTransferTimeoutValue),
                           &Header, sizeof(Header), &Status);
(Error handling etc omitted for brevity)
That would get you the first 8 bytes of the class-specific AC interface header descriptor.  You would then need to extract the TotalLength field, allocate that length of memory, and repeat the UsbControlTransfer() call to fetch the full-length descriptor into the newly allocated block.
Hope that helps,BTW, in case you aren't already aware of this: wireshark is pretty good at dissecting USB traffic.  You can capture it by doing a "modprobe usbmon", after which you'll see a number of usbmonN devices show up in the wireshark interface list.  Try them each in turn until you find which one corresponds to the host controller to which your device is attached.

My normal method for developing or debugging iPXE USB drivers will typically involve using wireshark to capture the USB traffic that happens when the device is being used by a known-working driver (e.g. the Linux driver for that device) and comparing it to the traffic that happens when I'm using my own driver (via USB pass-through in a VM). This is often a lot faster than trying to pull together all of the information from the multiple USB spec documents.

Good luck!

Michael

Re: EFI_AUDIO_OUTPUT_PROTOCOL: assistance with VirtIO initialization

Michael Brown

#77445

On 01/07/2021 19:10, Ethin Probst wrote:
The first: I don't know how to retrieve the descriptors on the UEFI
API. I've looked at the Chaos Key DXE, if memory serves, or the
display port one -- one of the two. One of them contained a
ReadDescriptor() function, and so I've borrowed that, but that doesn't
really help much.I have no idea what higher-level utility library functions exist within EDK2 that you might be able to use.  Based on skimming through the USB audio spec, at the raw protocol level you would need to use EFI_USB_IO_PROTOCOL.UsbControlTransfer() something like:

#define AUDIO_CS_INTERFACE 0x24

#pragma pack(1)
typedef struct {
  UINT8 Length;
  UINT8 DescriptorType;
  UINT8 DescriptorSubType;
  UINT16 BcdADC;
  UINT16 TotalLength;
  UINT8 InCollection;
} AUDIO_HEADER_DESCRIPTOR;
#pragma pack()

EFI_USB_INTERFACE_DESCRIPTOR InterfaceDescriptor;
EFI_USB_DEVICE_REQUEST Req;
AUDIO_CS_HEADER Header;
UINT32 Status;

UsbIo->UsbGetInterfaceDescriptor(UsbIo, &InterfaceDescriptor);

Req.RequestType = (USB_DEV_GET_DESCRIPTOR_REQ_TYPE |
                   USB_REQ_TYPE_CLASS);
Req.Request = USB_REQ_GET_DESCRIPTOR;
Req.Value = (AUDIO_CS_INTERFACE << 8);
Req.Index = InterfaceDescriptor.InterfaceNumber;
Req.Length = sizeof(Header);

UsbIo->UsbControlTransfer(UsbIo, &Req, EfiUsbDataIn,
                          PcdGet32 (PcdUsbTransferTimeoutValue),
                          &Header, sizeof(Header), &Status);

(Error handling etc omitted for brevity)

That would get you the first 8 bytes of the class-specific AC interface header descriptor.  You would then need to extract the TotalLength field, allocate that length of memory, and repeat the UsbControlTransfer() call to fetch the full-length descriptor into the newly allocated block.

Hope that helps,

Michael

Re: EFI_AUDIO_OUTPUT_PROTOCOL: assistance with VirtIO initialization

Laszlo Ersek

#77444

On 07/01/21 20:10, Ethin Probst wrote:
The first: I don't know how to retrieve the descriptors on the UEFI
API. I've looked at the Chaos Key DXE, if memory serves, or the
display port one -- one of the two. One of them contained a
ReadDescriptor() function, and so I've borrowed that, but that doesn't
really help much.Does the stuff in "MdeModulePkg/Bus/Usb/UsbBusDxe/UsbDesc.c" help?

(I don't know how that's exposed, if at all, through UEFI protocols, alas.)

I've added Hao and Ray to the address list (IIUC they maintain the USB
modules in edk2).

Thanks
Laszlo


On 7/1/21, Michael Brown <mcb30@...> wrote:
On 01/07/2021 00:01, Ethin Probst wrote:
So Leif and I have been working on USB Audio but we've run into a snag.
We've encountered a problem -- neither of us knows enough about USB to
figure out how to get the class-specific AC interface descriptors, and
those contain vital information that I need to be able to control the
audio device.Do you mean that you can't figure out how to retrieve the descriptors
via the UEFI APIs, or that you don't know how to interpret the
descriptors after retrieval?

Thanks,

Michael

Re: EFI_AUDIO_OUTPUT_PROTOCOL: assistance with VirtIO initialization

Laszlo Ersek

#77443

On 07/01/21 01:01, Ethin Probst wrote:
Hi all,
So Leif and I have been working on USB Audio but we've run into a snag. We've encountered a problem -- neither of us knows enough about USB to figure out how to get the class-specific AC interface descriptors, and those contain vital information that I need to be able to control the audio device. The audio specification was quite useless. So I have two projects running in parallel: the USB audio one and the VirtIO audio (We managed to get the virtio-snd patchset in qemu and I've got a built version of it over here and, though its not a fully working implementation, as long as I get something -- no matter how bad it sounds -- I'll be happy). Whichever project we get working first is the one we move forward with.
The VirtIO sound device specification contains four virtqueues: the control queue for controlling the device; the event queue for receiving notifications from the device; the tx queue for transmitting audio data to the device; and the rx queue for receiving audio data from the device. Thus far I've been following the VirtioRngDxe code as a guide on how to get a VirtIO device initialized using the VirtioLib in OVMF, but I've reached an impasse regarding the queues and descriptors.

The VirtioRngDxe DXE uses a single descriptor and a single queue for operation. However, the RNG device works vastly differently to the audio device and is far simpler. The virtio-snd specification says that, to initialize the device, I must (copied from the spec):

* Configure the control, event, tx and rx queues.
* Read the jacks field and send a control request to query information about the available jacks.
* Read the streams field and send a control request to query information about the available PCM streams.
* Read the chmaps field and send a control request to query information about the available channel maps.
* Populate the event queue with empty buffers.

Steps 2, 3, 4, and 5 are a bit confusing and raise some questions:

1. Does the device automatically send notifications in the event queue about the jacks, streams, and channel maps, or do I have to explicitly ask for them?
2. How many buffers would we need to populate the event queue with, and what VirtioLib function (if any) specifically accomplishes that?What you describe here makes the virtio-sound device the most complex
virtio device that OVMF has ever had any interest in. The only driver
you can use as an example is VirtioNetDxe. VirtioRngDxe is about the
simplest virtio device / driver, but for an example here, you need to be
looking at the opposite end of the spectrum.

For the duplex communication, please refer in particular to
"OvmfPkg/VirtioNetDxe/TechNotes.txt".

In virtio, requests are always created by the driver (= the guest), and
requests are always completed by the device (= the host). Therefore, for
asynchronous host-to-guest communication, the guest has to fully
pre-populate the affected queues with requests, and when host actions
arrive (technically presented as "request completions"), the guest has
to process those "completions" and re-populate the affected queue at
once to the brim.

VirtioLib provides a number of utility / convenience functions, for
various areas / actions. However, the VirtioPrepare(),
VirtioAppendDesc(), VirtioFlush() functions in particular are
inappropriate for your use case. Those helper functions implement a
simple, synchronous request-response pattern, building and submitting a
single descriptor chain, and blocking until the host reports that chain
processed. These functions exist because this simple pattern is suitable
for most of the virtio drivers (VirtioBlkDxe, VirtioFsDxe, VirtioGpuDxe,
VirtioRngDxe, VirtioScsiDxe). The reason being that the UEFI protocols
implemented all support the same, simple, synchronous pattern. These
functions are however not helpful for actual async communication, so
they are not used in VirtioNetDxe -- VirtioNetDxe does its own queue and
descriptor chain management. (It's documented in detail in "TechNotes.txt".)

To be honest, I would recommend abandoning the virtio-sound path for
now. The virtio programming will eat up a huge amount of your time, and
that's not what you want to be concentrating on.

I had had no previous experience with either audio programming or the
virtio-sound device itself; TBH I didn't expect this level of complexity
(in particular, based on your description, that *all four* queues would
have to be initialized, no questions asked). The virtio drivers in OVMF
are always kept as simple as possible, only the inevitably necessary
queues are initialized (and the devices are configured accordingly), and
only the simplest communication patterns are used. These drivers are not
OS drivers; what they need to do is "facilitate booting".

Example: the BlockIo protocol only supports synchronous requests,
whereas the BlockIo2 protocol supports asynchronous requests as well.
Technically speaking, virtio is fully capable of implementing the
asynchrony of BlockIo2, so (e.g.) a BlockIo2 protocol implementation on
top of virtio-blk would be possible. But why bother? It just complicates
the code a whole lot, and booting an OS with just BlockIo is
sufficiently fast already. VirtioNetDxe *must* be different however,
because the UEFI-level abstraction, SimpleNetworkProtocol (SNP), is
truly asynchronous itself.

If you can design a UEFI protocol for just sound *output* that lets you
get away with synchronous (blocking) UEFI protocol member function
calls, then you *might* be able to ignore most of the virtio-sound
complexity, and use the existent descriptor chain management functions
in VirtioLib. You could perhaps ignore the "event" and "rx" queues
altogether, and use the synchronous VirtioLib functions for both
"control" requests and actual audio output ("tx"). I can't tell.

Thanks
Laszlo

[PATCH 2/2] UefiPayloadPkg: Assign the length of UniversalPayload ExtraData

duntan

#77442

Assign the length and revision of UniversalPayload ExtraData

Cc: Guo Dong <guo.dong@...>
Cc: Ray Ni <ray.ni@...>
Cc: Maurice Ma <maurice.ma@...>
Cc: Benjamin You <benjamin.you@...>

Signed-off-by: DunTan <dun.tan@...>
---
 UefiPayloadPkg/PayloadLoaderPeim/PayloadLoaderPeim.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/UefiPayloadPkg/PayloadLoaderPeim/PayloadLoaderPeim.c b/UefiPayloadPkg/PayloadLoaderPeim/PayloadLoaderPeim.c
index c619470dbb..3de601025e 100644
--- a/UefiPayloadPkg/PayloadLoaderPeim/PayloadLoaderPeim.c
+++ b/UefiPayloadPkg/PayloadLoaderPeim/PayloadLoaderPeim.c
@@ -110,6 +110,8 @@ PeiLoadFileLoadPayload (
                sizeof (UNIVERSAL_PAYLOAD_EXTRA_DATA) + ExtraDataCount * sizeof (UNIVERSAL_PAYLOAD_EXTRA_DATA_ENTRY)
                );
   ExtraData->Count = ExtraDataCount;
+  ExtraData->Header.Revision = UNIVERSAL_PAYLOAD_EXTRA_DATA_REVISION;
+  ExtraData->Header.Length = sizeof (UNIVERSAL_PAYLOAD_EXTRA_DATA) + ExtraDataCount * sizeof (UNIVERSAL_PAYLOAD_EXTRA_DATA_ENTRY);
   if (ExtraDataCount != 0) {
     for (ExtraDataIndex = 0, Index = 0; Index < Context.ShNum; Index++) {
       Status = GetElfSectionName (&Context, Index, &SectionName);
-- 
2.31.1.windows.1

[PATCH 1/2] MdeModulePkg: Change the PldHeader to Header in ExtraData.h

duntan

#77441

Change the PldHeader to Header in UNIVERSAL_PAYLOAD_EXTRA_DATA
The meaning of Pld is too general

Cc: Zhiguang Liu <zhiguang.liu@...>
Cc: Ray Ni <ray.ni@...>

Signed-off-by: DunTan <dun.tan@...>
---
 MdeModulePkg/Include/UniversalPayload/ExtraData.h | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/MdeModulePkg/Include/UniversalPayload/ExtraData.h b/MdeModulePkg/Include/UniversalPayload/ExtraData.h
index 146ec845f6..1128f5f47b 100644
--- a/MdeModulePkg/Include/UniversalPayload/ExtraData.h
+++ b/MdeModulePkg/Include/UniversalPayload/ExtraData.h
@@ -18,11 +18,13 @@ typedef struct {
 } UNIVERSAL_PAYLOAD_EXTRA_DATA_ENTRY;
 
 typedef struct {
-  UNIVERSAL_PAYLOAD_GENERIC_HEADER   PldHeader;
+  UNIVERSAL_PAYLOAD_GENERIC_HEADER   Header;
   UINT32                             Count;
   UNIVERSAL_PAYLOAD_EXTRA_DATA_ENTRY Entry[0];
 } UNIVERSAL_PAYLOAD_EXTRA_DATA;
 
 #pragma pack()
 
+#define UNIVERSAL_PAYLOAD_EXTRA_DATA_REVISION 1
+
 #endif
-- 
2.31.1.windows.1

[PATCH 0/2] Fix the length and PldHeader for UniversalPayload ExtraData

duntan

#77440

Change PldHeader to Header in ExtraData.h
Assign the length and revision of UniversalPayload ExtraData

duntan (2):
  MdeModulePkg: Change the PldHeader to Header in ExtraData.h
  UefiPayloadPkg: Assign the length of UniversalPayload ExtraData

 MdeModulePkg/Include/UniversalPayload/ExtraData.h    | 4 +++-
 UefiPayloadPkg/PayloadLoaderPeim/PayloadLoaderPeim.c | 2 ++
 2 files changed, 5 insertions(+), 1 deletion(-)

-- 
2.31.1.windows.1

Re: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Steven Shi

#77439

Bret,

For my failure of “Unable to load the service index for source…”, I see it might be the Nuget proxy setting issue: https://stackoverflow.com/questions/41185443/nuget-connection-attempt-failed-unable-to-load-the-service-index-for-source. I tried to set the Nuget https_proxy and http_proxy as below, but it still does not work. Do you know how to set the Nuget Proxy in Pytool?

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe config -set https_proxy=http://xxx:xxxx@...:xxx

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe config -set http_proxy=http://xxx:xxxx@...:xxx

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe install edk2-acpica-iasl -Source https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json -ExcludeVersion -NonInteractive -Version 20200717.0.0 -Verbosity detailed -OutputDirectory "/home/jshi19/wksp_efi/edk2-2/BaseTools/Bin/edk2-acpica-iasl_extdep_temp"

NuGet Version: 5.3.1.6268

Feeds used:

https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json

Attempting to gather dependency information for package 'edk2-acpica-iasl.20200717.0.0' with respect to project '/home/jshi19/wksp_efi/edk2-2/BaseTools/Bin/edk2-acpica-iasl_extdep_temp', targeting 'Any,Version=v0.0'

Unable to load the service index for source https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json.

GSSAPI operation failed with error - Unspecified GSS failure. Minor code may provide more information (SPNEGO cannot find mechanisms to negotiate).

System.InvalidOperationException: Unable to load the service index for source https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json. ---> NuGet.Protocol.Core.Types.FatalProtocolException: Unable to load the service index for source https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json. ---> System.ComponentModel.Win32Exception: GSSAPI operation failed with error - Unspecified GSS failure. Minor code may provide more information (SPNEGO cannot find mechanisms to negotiate).

at System.Net.NTAuthentication.GetOutgoingBlob (System.Byte[] incomingBlob, System.Boolean throwOnError, System.Net.SecurityStatusPal& statusCode) [0x00258] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0

at System.Net.NTAuthentication.GetOutgoingBlob (System.String incomingBlob) [0x00045] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0

Thanks

Steven Shi

Intel\IAGS\SFP\FIA (Firmware Infrastructure Automation)

Show quoted text

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Steven Shi
Sent: Friday, July 2, 2021 10:44 AM
To: Rebecca Cran <rebecca@...>; devel@edk2.groups.io; Bret Barkelew <Bret.Barkelew@...>; rebecca@...
Subject: Re: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Thank you to Rebecca. After install the libmono-system-net-http4.0-cil and mono-devel pkg as below, the ‘mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list’ pass.

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ sudo apt-get install libmono-system-net-http4.0-cil

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ sudo apt-get install mono-devel

But now there are new failures show up when install edk2-acpica-iasl:

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5 -v

… …

INFO - Cmd to run is: mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list

INFO - ------------------------------------------------

INFO - --------------Cmd Output Starting---------------

INFO - ------------------------------------------------

...INFO - global-packages: /home/jshi19/.nuget/packages/

INFO - global-packages: /home/jshi19/.nuget/packages/

INFO - ------------------------------------------------

INFO - --------------Cmd Output Finished---------------

INFO - ------------------------------------------------

INFO - --------- Running Time (mm:ss): 00:00 ----------

INFO - --------------Cmd Output Finished---------------

INFO - ----------- Return Code: 0x00000000 ------------

INFO - --------- Running Time (mm:ss): 00:00 ----------

INFO - ------------------------------------------------

INFO - ----------- Return Code: 0x00000000 ------------

INFO - Could not determine Nuget global packages cache location.

INFO - ------------------------------------------------

INFO - Cmd to run is: mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe install edk2-acpica-iasl -Source https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json -ExcludeVersion -NonInteractive -Version 20200717.0.0 -Verbosity detailed -OutputDirectory "/home/jshi19/wksp_efi/edk2-2/BaseTools/Bin/edk2-acpica-iasl_extdep_temp"

INFO - Could not determine Nuget global packages cache location.

INFO - ------------------------------------------------

INFO - --------------Cmd Output Starting---------------

INFO - Cmd to run is: mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe install mu_nasm -Source https://api.nuget.org/v3/index.json -ExcludeVersion -NonInteractive -Version 2.15.05 -Verbosity detailed -OutputDirectory "/home/jshi19/wksp_efi/edk2-2/BaseTools/Bin/mu_nasm_extdep_temp"

INFO - ------------------------------------------------

INFO - --------------Cmd Output Starting---------------

INFO - ------------------------------------------------

INFO - NuGet Version: 5.3.1.6268

INFO - Feeds used:

INFO - https://api.nuget.org/v3/index.json

INFO -

INFO - Feeds used:

INFO - https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json

INFO -

INFO - Attempting to gather dependency information for package 'mu_nasm.2.15.5' with respect to project '/home/jshi19/wksp_efi/edk2-2/BaseTools/Bin/mu_nasm_extdep_temp', targeting 'Any,Version=v0.0'

INFO -

INFO - Attempting to gather dependency information for package 'edk2-acpica-iasl.20200717.0.0' with respect to project '/home/jshi19/wksp_efi/edk2-2/BaseTools/Bin/edk2-acpica-iasl_extdep_temp', targeting 'Any,Version=v0.0'

INFO - Unable to load the service index for source https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json.

INFO - GSSAPI operation failed with error - An invalid name was supplied (Configuration file does not specify default realm).

INFO - System.InvalidOperationException: Unable to load the service index for source https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json. ---> NuGet.Protocol.Core.Types.FatalProtocolException: Unable to load the service index for source https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json. ---> System.ComponentModel.Win32Exception: GSSAPI operation failed with error - An invalid name was supplied (Configuration file does not specify default realm).

INFO - at System.Net.Security.NegotiateStreamPal.AcquireCredentialsHandle (System.String package, System.Boolean isServer, System.Net.NetworkCredential credential) [0x000a3] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0

… …

Thanks

Steven Shi

Intel\IAGS\SFP\FIA (Firmware Infrastructure Automation)

From: Rebecca Cran <rebecca@...>
Sent: Friday, July 2, 2021 10:00 AM
To: devel@edk2.groups.io; Shi, Steven <steven.shi@...>; Bret Barkelew <Bret.Barkelew@...>; rebecca@...
Subject: Re: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

That means you need to install the various mono assemblies, such as the libmono-system-net-http4.0-cil package in Ubuntu.

I spent some time chasing down the dependencies last year which got very frustrating and tedious, but have forgotten which were needed. All I remember is there were quite a lot.

--
Rebecca Cran

On 7/1/21 7:36 PM, Steven Shi wrote:

I already installed mono, but NuGet.exe still fails to work in my side. I’m behind a proxy but don’t know how to configure the NuGet.exe proxy in .pytool. Below is my verbose error message.

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5 -v

… …

SECTION - Start Invocable Tool

SECTION - Initial update of environment

DEBUG - --- self_describing_environment.update_extdeps()

DEBUG - Verify 'mu_nasm' returning 'False'.

DEBUG - Verify 'edk2-acpica-iasl' returning 'False'.

DEBUG - Creating 5 threads for the SDE update

UpdatingDEBUG - Verify 'gcc_aarch64_linux' returning 'True'.

DEBUG - Verify 'gcc_riscv64_unknown' returning 'True'.

DEBUG - Verify 'mu_nasm' returning 'False'.

DEBUG - Verify 'gcc_arm_linux' returning 'True'.

DEBUG - Verify 'edk2-acpica-iasl' returning 'False'.

DEBUG - Cleaning dependency directory for 'mu_nasm'...

DEBUG - Cleaning dependency directory for 'edk2-acpica-iasl'...

INFO - Cmd to run is: mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list

INFO - Cmd to run is: mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list

INFO - ------------------------------------------------

INFO - ------------------------------------------------

INFO - --------------Cmd Output Starting---------------

INFO - --------------Cmd Output Starting---------------

INFO - ------------------------------------------------

INFO - ------------------------------------------------

...INFO - Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO -   Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO -   Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO -   Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO -   Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO - System.AggregateException: One or more errors occurred. (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO -    --- End of inner exception stack trace ---

… …

If I run the Cmd manually, I get below error:

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

System.AggregateException: One or more errors occurred. (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

   --- End of inner exception stack trace ---

at NuGet.CommandLine.Program.Initialize (NuGet.IFileSystem fileSystem, NuGet.CommandLine.IConsole console) [0x000ce] in <ff502667b3fe4a85827635c6fe6b0d0d>:0

at NuGet.CommandLine.Program.MainCore (System.String workingDirectory, System.String[] args) [0x000d8] in <ff502667b3fe4a85827635c6fe6b0d0d>:0

---> (Inner Exception #0) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #1) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #2) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #3) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #4) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

Thanks

Steven Shi

From: Bret Barkelew <Bret.Barkelew@...>
Sent: Friday, July 2, 2021 9:04 AM
To: Shi, Steven <steven.shi@...>; devel@edk2.groups.io; rebecca@...
Subject: RE: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

I believe the Azure DevOps build agents that are used by the EDK2 CI pipeline already have mono installed.

- Bret

From: Shi, Steven
Sent: Thursday, July 1, 2021 5:54 PM
To: devel@edk2.groups.io; Bret Barkelew; rebecca@...
Subject: RE: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Hi Bret, Rebecca,

Thanks for the info. Does edk2 CI run the .pytool unit tests in Linux? If yes, how does edk2 CI solve the nuget dependency problem in Linux right now?

Thanks

Steven Shi

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Bret Barkelew via groups.io
Sent: Wednesday, June 30, 2021 1:02 AM
To: devel@edk2.groups.io; rebecca@...; Shi, Steven <steven.shi@...>
Subject: Re: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Hmmm. If that works well, we might be able to investigate carrying both and picking based on the host OS.

- Bret

From: devel@edk2.groups.io <devel@edk2.groups.io> on behalf of Rebecca Cran via groups.io <rebecca@...>
Sent: Tuesday, June 29, 2021 9:42:41 AM
To: devel@edk2.groups.io <devel@edk2.groups.io>; steven.shi@... <steven.shi@...>
Subject: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Your Linux distribution should have a nuget package available (e.g. https://packages.ubuntu.com/hirsute/nuget) but the EDK2 CI is designed to use the copy of nuget installed with the edk2toolext package.

--
Rebecca Cran

On 6/29/21 9:44 AM, Steven Shi wrote:

Cran, Thanks. I’m curious whether the .pytool has native Linux executable binary to run? It looks the edk2 CI framework has more dependency in Linux than Windows.

Thanks

Steven Shi

Intel\IAGS\SFP\FIA (Firmware Infrastructure Automation)

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Rebecca Cran
Sent: Tuesday, June 29, 2021 8:13 PM
To: devel@edk2.groups.io; Shi, Steven <steven.shi@...>
Subject: Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

NuGet.exe is a .NET assembly/executable, so on Linux you'll need to install Mono (https://www.mono-project.com/).

--

Rebecca Cran

On 6/28/21 7:45 PM, Steven Shi wrote:

Hello,

I’m interested in the edk2 CI unit tests(https://github.com/tianocore/edk2/tree/master/.pytool). I tested it in my local. It works well in Windows but does not work in Linux. It looks the NuGet in pytool only has windows version (C:\steven\unit_test_env\Lib\site-packages\edk2toolext\bin\NuGet.exe) but not support the Linux. Below is my detail test steps. Do the edk2 CI unit tests work in Linux?

Windows :

c:\steven>py -m venv unit_test_env

c:\steven>cd c:\steven\edk2

c:\steven\edk2>c:\steven\unit_test_env\Scripts\activate.bat

(unit_test_env) c:\steven\edk2>pip install --upgrade -r pip-requirements.txt

(unit_test_env) c:\steven\edk2>stuart_setup -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2019

(unit_test_env) c:\steven\edk2>stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2019

(unit_test_env) c:\steven\edk2>edksetup.bat

(unit_test_env) c:\steven\edk2>edksetup.bat Rebuild

(unit_test_env) c:\steven\edk2>stuart_ci_build -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2019 -a X64,IA32 -t DEBUG --verbose

…

PROGRESS - Overall Build Status: Success

SECTION - Summary

PROGRESS - Success

Linux:

jshi19@ub2-uefi-b01:~/wksp_efi$ python3 -m venv unit_test_env

jshi19@ub2-uefi-b01:~/wksp_efi$ source unit_test_env/bin/activate

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ pip install --upgrade -r pip-requirements.txt

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ make -C BaseTools/

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ stuart_setup -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5

SECTION - Init SDE

WARNING - Using Pip Tools based BaseTools

SECTION - Loading Plugins

SECTION - Start Invocable Tool

SECTION - Initial update of environment

UpdatingWARNING - [SDE] Failed to fetch NugetDependecy: edk2-acpica-iasl@....0: [Nuget] We failed to install this version 20200717.0.0 of edk2-acpica-iasl

WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.15.05: [Nuget] We failed to install this version 2.15.05 of mu_nasm

. Done

SECTION -       Updated/Verified 3 dependencies

SECTION - Second pass update of environment

UpdatingWARNING - [SDE] Failed to fetch NugetDependecy: edk2-acpica-iasl@....0: [Nuget] We failed to install this version 20200717.0.0 of edk2-acpica-iasl

WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.15.05: [Nuget] We failed to install this version 2.15.05 of mu_nasm

. Done

SECTION -       Updated/Verified 3 dependencies

ERROR - We were unable to successfully update 2 dependencies in environment

SECTION - Summary

ERROR - Error

Thanks

Steven Shi

Re: [edk2-platforms PATCH v5 3/4] RISC-V Platforms: add SecureBootVariableLib class resolution

Grzegorz Bernacki

#77438

Hi,

No, not yet.
Thanks,
Grzegorz

pt., 2 lip 2021 o 06:12 Chang, Abner (HPS SW/FW Technologist)
<abner.chang@...> napisał(a):

Show quoted text

Hi Grzegorz,
Was the entire series of patches got reviewed-by?

Regards,
Abner

-----Original Message-----
From: Grzegorz Bernacki [mailto:gjb@...]
Sent: Thursday, July 1, 2021 5:21 PM
To: devel@edk2.groups.io
Cc: leif@...; ardb+tianocore@...; Samer.El-Haj-
Mahmoud@...; sunny.Wang@...; mw@...;
upstream@...; jiewen.yao@...; jian.j.wang@...;
min.m.xu@...; lersek@...; sami.mujawar@...;
afish@...; ray.ni@...; jordan.l.justen@...;
rebecca@...; grehan@...; thomas.abraham@...;
chasel.chiu@...; nathaniel.l.desimone@...;
gaoliming@...; eric.dong@...;
michael.d.kinney@...; zailiang.sun@...; yi.qian@...;
graeme@...; rad@...; pete@...; Grzegorz
Bernacki <gjb@...>; Chang, Abner (HPS SW/FW Technologist)
<abner.chang@...>; Schaefer, Daniel <daniel.schaefer@...>
Subject: [edk2-platforms PATCH v5 3/4] RISC-V Platforms: add
SecureBootVariableLib class resolution

The edk2 patch
  SecurityPkg: Create library for setting Secure Boot variables.

removes generic functions from SecureBootConfigDxe and places
them into SecureBootVariableLib. This patch adds SecureBootVariableLib
mapping for each RICS-V platform which uses SecureBootConfigDxe.

Signed-off-by: Grzegorz Bernacki <gjb@...>
Reviewed-by: Abner Chang <abner.chang@...>
Reviewed-by: Daniel Schaefer <daniel.schaefer@...>
---
 Platform/SiFive/U5SeriesPkg/FreedomU500VC707Board/U500.dsc           | 1 +

Platform/SiFive/U5SeriesPkg/FreedomU540HiFiveUnleashedBoard/U540.dsc
| 1 +
 2 files changed, 2 insertions(+)

diff --git a/Platform/SiFive/U5SeriesPkg/FreedomU500VC707Board/U500.dsc
b/Platform/SiFive/U5SeriesPkg/FreedomU500VC707Board/U500.dsc
index b91823ceeb..fc5ba2a07f 100644
--- a/Platform/SiFive/U5SeriesPkg/FreedomU500VC707Board/U500.dsc
+++ b/Platform/SiFive/U5SeriesPkg/FreedomU500VC707Board/U500.dsc
@@ -122,6 +122,7 @@
   OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf

TpmMeasurementLib|SecurityPkg/Library/DxeTpmMeasurementLib/DxeTp
mMeasurementLib.inf
   AuthVariableLib|SecurityPkg/Library/AuthVariableLib/AuthVariableLib.inf
+
SecureBootVariableLib|SecurityPkg/Library/SecureBootVariableLib/SecureB
ootVariableLib.inf
 !else

TpmMeasurementLib|MdeModulePkg/Library/TpmMeasurementLibNull/Tp
mMeasurementLibNull.inf

AuthVariableLib|MdeModulePkg/Library/AuthVariableLibNull/AuthVariableL
ibNull.inf
diff --git
a/Platform/SiFive/U5SeriesPkg/FreedomU540HiFiveUnleashedBoard/U540.d
sc
b/Platform/SiFive/U5SeriesPkg/FreedomU540HiFiveUnleashedBoard/U540.
dsc
index 0eafe29880..71add8ff9a 100644
---
a/Platform/SiFive/U5SeriesPkg/FreedomU540HiFiveUnleashedBoard/U540.d
sc
+++
b/Platform/SiFive/U5SeriesPkg/FreedomU540HiFiveUnleashedBoard/U540.
dsc
@@ -122,6 +122,7 @@
   OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf

TpmMeasurementLib|SecurityPkg/Library/DxeTpmMeasurementLib/DxeTp
mMeasurementLib.inf
   AuthVariableLib|SecurityPkg/Library/AuthVariableLib/AuthVariableLib.inf
+
SecureBootVariableLib|SecurityPkg/Library/SecureBootVariableLib/SecureB
ootVariableLib.inf
 !else

TpmMeasurementLib|MdeModulePkg/Library/TpmMeasurementLibNull/Tp
mMeasurementLibNull.inf

AuthVariableLib|MdeModulePkg/Library/AuthVariableLibNull/AuthVariableL
ibNull.inf
--
2.25.1

[PATCH] UefiCpuPkg/ExceptionLib: Conditionally clear shadow stack token busy bit

Sheng Wei

#77437

When enter SMM exception, there will be a stack switch only if the IST
field of the interrupt gate is set. When CET shadow stack feature is
enabled, if there is a stack switch between SMM exception and SMM, the
shadow stack token busy bit needs to be cleared when return from SMM
exception to SMM. In UEFI BIOS, only page fault exception does the stack
swith when SMM shack guard feature is enabled. The condition of clear
shadow stack token busy bit should be SMM stack guard enabled, CET shadows
stack feature enabled and page fault exception.
The shadow stack token should be initialized by UINT64.

REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3462

Signed-off-by: Sheng Wei <w.sheng@...>
Cc: Eric Dong <eric.dong@...>
Cc: Ray Ni <ray.ni@...>
Cc: Laszlo Ersek <lersek@...>
Cc: Rahul Kumar <rahul1.kumar@...>
Cc: Jiewen Yao <jiewen.yao@...>
Cc: Qihua Zhuang <qihua.zhuang@...>
Cc: Daquan Dong <daquan.dong@...>
Cc: Justin Tong <justin.tong@...>
Cc: Tom Xu <tom.xu@...>
---
 .../X64/Xcode5ExceptionHandlerAsm.nasm             | 83 +++++++++++-----------
 UefiCpuPkg/PiSmmCpuDxeSmm/X64/SmmFuncsArch.c       |  2 +-
 2 files changed, 43 insertions(+), 42 deletions(-)

diff --git a/UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/Xcode5ExceptionHandlerAsm.nasm b/UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/Xcode5ExceptionHandlerAsm.nasm
index ebe0eec874..4881a02848 100644
--- a/UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/Xcode5ExceptionHandlerAsm.nasm
+++ b/UefiCpuPkg/Library/CpuExceptionHandlerLib/X64/Xcode5ExceptionHandlerAsm.nasm
@@ -20,6 +20,7 @@
 ;
 
 %define VC_EXCEPTION 29
+%define PF_EXCEPTION 14
 
 extern ASM_PFX(mErrorCodeFlag)    ; Error code flags for exceptions
 extern ASM_PFX(mDoFarReturnFlag)  ; Do far return flag
@@ -279,6 +280,46 @@ DrFinish:
     call    ASM_PFX(CommonExceptionHandler)
     add     rsp, 4 * 8 + 8
 
+    ; The follow algorithm is used for clear shadow stack token busy bit.
+    ; The comment is based on the sample shadow stack.
+    ; The sample shadow stack layout :
+    ; Address | Context
+    ;         +-------------------------+
+    ;  0xFD0  |   FREE                  | it is 0xFD8|0x02|(LMA & CS.L), after SAVEPREVSSP.
+    ;         +-------------------------+
+    ;  0xFD8  |  Prev SSP               |
+    ;         +-------------------------+
+    ;  0xFE0  |   RIP                   |
+    ;         +-------------------------+
+    ;  0xFE8  |   CS                    |
+    ;         +-------------------------+
+    ;  0xFF0  |  0xFF0 | BUSY           | BUSY flag cleared after CLRSSBSY
+    ;         +-------------------------+
+    ;  0xFF8  | 0xFD8|0x02|(LMA & CS.L) |
+    ;         +-------------------------+
+    ; Instructions for Intel Control Flow Enforcement Technology (CET) are supported since NASM version 2.15.01.
+    cmp     qword [ASM_PFX(mDoFarReturnFlag)], 0
+    jz      CetDone
+    cmp     qword [rbp + 8], PF_EXCEPTION   ; check if it is a Page Fault
+    jnz     CetDone
+    cmp     byte [dword ASM_PFX(FeaturePcdGet (PcdCpuSmmStackGuard))], 0
+    jz      CetDone
+    mov     rax, cr4
+    and     rax, 0x800000       ; check if CET is enabled
+    jz      CetDone
+                                ; SSP should be 0xFD8 at this point
+    mov     rax, 0x04           ; advance past cs:lip:prevssp;supervisor shadow stack token
+    INCSSP_RAX                  ; After this SSP should be 0xFF8
+    SAVEPREVSSP                 ; now the shadow stack restore token will be created at 0xFD0
+    READSSP_RAX                 ; Read new SSP, SSP should be 0x1000
+    sub     rax, 0x10
+    CLRSSBSY_RAX                ; Clear token at 0xFF0, SSP should be 0 after this
+    sub     rax, 0x20
+    RSTORSSP_RAX                ; Restore to token at 0xFD0, new SSP will be 0xFD0
+    mov     rax, 0x01           ; Pop off the new save token created
+    INCSSP_RAX                  ; SSP should be 0xFD8 now
+CetDone:
+
     cli
 ;; UINT64  ExceptionData;
     add     rsp, 8
@@ -373,47 +414,7 @@ DoReturn:
     push    qword [rax + 0x18]       ; save EFLAGS in new location
     mov     rax, [rax]        ; restore rax
     popfq                     ; restore EFLAGS
-
-    ; The follow algorithm is used for clear shadow stack token busy bit.
-    ; The comment is based on the sample shadow stack.
-    ; The sample shadow stack layout :
-    ; Address | Context
-    ;         +-------------------------+
-    ;  0xFD0  |   FREE                  | it is 0xFD8|0x02|(LMA & CS.L), after SAVEPREVSSP.
-    ;         +-------------------------+
-    ;  0xFD8  |  Prev SSP               |
-    ;         +-------------------------+
-    ;  0xFE0  |   RIP                   |
-    ;         +-------------------------+
-    ;  0xFE8  |   CS                    |
-    ;         +-------------------------+
-    ;  0xFF0  |  0xFF0 | BUSY           | BUSY flag cleared after CLRSSBSY
-    ;         +-------------------------+
-    ;  0xFF8  | 0xFD8|0x02|(LMA & CS.L) |
-    ;         +-------------------------+
-    ; Instructions for Intel Control Flow Enforcement Technology (CET) are supported since NASM version 2.15.01.
-    push     rax                ; SSP should be 0xFD8 at this point
-    cmp      byte [dword ASM_PFX(FeaturePcdGet (PcdCpuSmmStackGuard))], 0
-    jz       CetDone
-    mov      rax, cr4
-    and      rax, 0x800000      ; check if CET is enabled
-    jz       CetDone
-    mov      rax, 0x04          ; advance past cs:lip:prevssp;supervisor shadow stack token
-    INCSSP_RAX                  ; After this SSP should be 0xFF8
-    SAVEPREVSSP                 ; now the shadow stack restore token will be created at 0xFD0
-    READSSP_RAX                 ; Read new SSP, SSP should be 0x1000
-    push     rax
-    sub      rax, 0x10
-    CLRSSBSY_RAX                ; Clear token at 0xFF0, SSP should be 0 after this
-    sub      rax, 0x20
-    RSTORSSP_RAX                ; Restore to token at 0xFD0, new SSP will be 0xFD0
-    pop      rax
-    mov      rax, 0x01          ; Pop off the new save token created
-    INCSSP_RAX                  ; SSP should be 0xFD8 now
-CetDone:
-    pop      rax                ; restore rax
-
-    DB       0x48               ; prefix to composite "retq" with next "retf"
+    DB      0x48                ; prefix to composite "retq" with next "retf"
     retf                        ; far return
 DoIret:
     iretq
diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/X64/SmmFuncsArch.c b/UefiCpuPkg/PiSmmCpuDxeSmm/X64/SmmFuncsArch.c
index 661c1ba294..ca3f5ff91a 100644
--- a/UefiCpuPkg/PiSmmCpuDxeSmm/X64/SmmFuncsArch.c
+++ b/UefiCpuPkg/PiSmmCpuDxeSmm/X64/SmmFuncsArch.c
@@ -202,7 +202,7 @@ InitShadowStack (
       // Please refer to UefiCpuPkg/Library/CpuExceptionHandlerLib/X64 for the full stack frame at runtime.
       //
       InterruptSsp = (UINT32)((UINTN)ShadowStack + EFI_PAGES_TO_SIZE(1) - sizeof(UINT64));
-      *(UINT32 *)(UINTN)InterruptSsp = (InterruptSsp - sizeof(UINT64) * 4) | 0x2;
+      *(UINT64 *)(UINTN)InterruptSsp = (InterruptSsp - sizeof(UINT64) * 4) | 0x2;
       mCetInterruptSsp = InterruptSsp - sizeof(UINT64);
 
       mCetInterruptSspTable = (UINT32)(UINTN)(mSmmInterruptSspTables + sizeof(UINT64) * 8 * CpuIndex);
-- 
2.16.2.windows.1

Re: [edk2-platforms PATCH v5 3/4] RISC-V Platforms: add SecureBootVariableLib class resolution

Abner Chang

#77436

Hi Grzegorz,
Was the entire series of patches got reviewed-by?

Regards,
Abner

Show quoted text

-----Original Message-----
From: Grzegorz Bernacki [mailto:gjb@...]
Sent: Thursday, July 1, 2021 5:21 PM
To: devel@edk2.groups.io
Cc: leif@...; ardb+tianocore@...; Samer.El-Haj-
Mahmoud@...; sunny.Wang@...; mw@...;
upstream@...; jiewen.yao@...; jian.j.wang@...;
min.m.xu@...; lersek@...; sami.mujawar@...;
afish@...; ray.ni@...; jordan.l.justen@...;
rebecca@...; grehan@...; thomas.abraham@...;
chasel.chiu@...; nathaniel.l.desimone@...;
gaoliming@...; eric.dong@...;
michael.d.kinney@...; zailiang.sun@...; yi.qian@...;
graeme@...; rad@...; pete@...; Grzegorz
Bernacki <gjb@...>; Chang, Abner (HPS SW/FW Technologist)
<abner.chang@...>; Schaefer, Daniel <daniel.schaefer@...>
Subject: [edk2-platforms PATCH v5 3/4] RISC-V Platforms: add
SecureBootVariableLib class resolution

The edk2 patch
  SecurityPkg: Create library for setting Secure Boot variables.

removes generic functions from SecureBootConfigDxe and places
them into SecureBootVariableLib. This patch adds SecureBootVariableLib
mapping for each RICS-V platform which uses SecureBootConfigDxe.

Signed-off-by: Grzegorz Bernacki <gjb@...>
Reviewed-by: Abner Chang <abner.chang@...>
Reviewed-by: Daniel Schaefer <daniel.schaefer@...>
---
 Platform/SiFive/U5SeriesPkg/FreedomU500VC707Board/U500.dsc           | 1 +

Platform/SiFive/U5SeriesPkg/FreedomU540HiFiveUnleashedBoard/U540.dsc
| 1 +
 2 files changed, 2 insertions(+)

diff --git a/Platform/SiFive/U5SeriesPkg/FreedomU500VC707Board/U500.dsc
b/Platform/SiFive/U5SeriesPkg/FreedomU500VC707Board/U500.dsc
index b91823ceeb..fc5ba2a07f 100644
--- a/Platform/SiFive/U5SeriesPkg/FreedomU500VC707Board/U500.dsc
+++ b/Platform/SiFive/U5SeriesPkg/FreedomU500VC707Board/U500.dsc
@@ -122,6 +122,7 @@
   OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf

TpmMeasurementLib|SecurityPkg/Library/DxeTpmMeasurementLib/DxeTp
mMeasurementLib.inf
   AuthVariableLib|SecurityPkg/Library/AuthVariableLib/AuthVariableLib.inf
+
SecureBootVariableLib|SecurityPkg/Library/SecureBootVariableLib/SecureB
ootVariableLib.inf
 !else

TpmMeasurementLib|MdeModulePkg/Library/TpmMeasurementLibNull/Tp
mMeasurementLibNull.inf

AuthVariableLib|MdeModulePkg/Library/AuthVariableLibNull/AuthVariableL
ibNull.inf
diff --git
a/Platform/SiFive/U5SeriesPkg/FreedomU540HiFiveUnleashedBoard/U540.d
sc
b/Platform/SiFive/U5SeriesPkg/FreedomU540HiFiveUnleashedBoard/U540.
dsc
index 0eafe29880..71add8ff9a 100644
---
a/Platform/SiFive/U5SeriesPkg/FreedomU540HiFiveUnleashedBoard/U540.d
sc
+++
b/Platform/SiFive/U5SeriesPkg/FreedomU540HiFiveUnleashedBoard/U540.
dsc
@@ -122,6 +122,7 @@
   OpensslLib|CryptoPkg/Library/OpensslLib/OpensslLib.inf

TpmMeasurementLib|SecurityPkg/Library/DxeTpmMeasurementLib/DxeTp
mMeasurementLib.inf
   AuthVariableLib|SecurityPkg/Library/AuthVariableLib/AuthVariableLib.inf
+
SecureBootVariableLib|SecurityPkg/Library/SecureBootVariableLib/SecureB
ootVariableLib.inf
 !else

TpmMeasurementLib|MdeModulePkg/Library/TpmMeasurementLibNull/Tp
mMeasurementLibNull.inf

AuthVariableLib|MdeModulePkg/Library/AuthVariableLibNull/AuthVariableL
ibNull.inf
--
2.25.1

Re: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Bret Barkelew <bret.barkelew@...>

#77435

I swear I got mine working with just an ‘apt install mono-devel’. But YMMV.

- Bret

Show quoted text

From: Rebecca Cran
Sent: Thursday, July 1, 2021 7:00 PM
To: devel@edk2.groups.io; steven.shi@...; Bret Barkelew; rebecca@...
Subject: Re: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

That means you need to install the various mono assemblies, such as the libmono-system-net-http4.0-cil package in Ubuntu.

I spent some time chasing down the dependencies last year which got very frustrating and tedious, but have forgotten which were needed. All I remember is there were quite a lot.

--
Rebecca Cran

On 7/1/21 7:36 PM, Steven Shi wrote:

I already installed mono, but NuGet.exe still fails to work in my side. I’m behind a proxy but don’t know how to configure the NuGet.exe proxy in .pytool. Below is my verbose error message.

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5 -v

… …

SECTION - Start Invocable Tool

SECTION - Initial update of environment

DEBUG - --- self_describing_environment.update_extdeps()

DEBUG - Verify 'mu_nasm' returning 'False'.

DEBUG - Verify 'edk2-acpica-iasl' returning 'False'.

DEBUG - Creating 5 threads for the SDE update

UpdatingDEBUG - Verify 'gcc_aarch64_linux' returning 'True'.

DEBUG - Verify 'gcc_riscv64_unknown' returning 'True'.

DEBUG - Verify 'mu_nasm' returning 'False'.

DEBUG - Verify 'gcc_arm_linux' returning 'True'.

DEBUG - Verify 'edk2-acpica-iasl' returning 'False'.

DEBUG - Cleaning dependency directory for 'mu_nasm'...

DEBUG - Cleaning dependency directory for 'edk2-acpica-iasl'...

INFO - Cmd to run is: mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list

INFO - Cmd to run is: mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list

INFO - ------------------------------------------------

INFO - ------------------------------------------------

INFO - --------------Cmd Output Starting---------------

INFO - --------------Cmd Output Starting---------------

INFO - ------------------------------------------------

INFO - ------------------------------------------------

...INFO - Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO -   Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO -   Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO -   Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO -   Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO - System.AggregateException: One or more errors occurred. (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO -    --- End of inner exception stack trace ---

… …

If I run the Cmd manually, I get below error:

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

System.AggregateException: One or more errors occurred. (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

   --- End of inner exception stack trace ---

at NuGet.CommandLine.Program.Initialize (NuGet.IFileSystem fileSystem, NuGet.CommandLine.IConsole console) [0x000ce] in <ff502667b3fe4a85827635c6fe6b0d0d>:0

at NuGet.CommandLine.Program.MainCore (System.String workingDirectory, System.String[] args) [0x000d8] in <ff502667b3fe4a85827635c6fe6b0d0d>:0

---> (Inner Exception #0) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #1) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #2) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #3) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #4) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

Thanks

Steven Shi

From: Bret Barkelew <Bret.Barkelew@...>
Sent: Friday, July 2, 2021 9:04 AM
To: Shi, Steven <steven.shi@...>; devel@edk2.groups.io; rebecca@...
Subject: RE: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

I believe the Azure DevOps build agents that are used by the EDK2 CI pipeline already have mono installed.

- Bret

From: Shi, Steven
Sent: Thursday, July 1, 2021 5:54 PM
To: devel@edk2.groups.io; Bret Barkelew; rebecca@...
Subject: RE: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Hi Bret, Rebecca,

Thanks for the info. Does edk2 CI run the .pytool unit tests in Linux? If yes, how does edk2 CI solve the nuget dependency problem in Linux right now?

Thanks

Steven Shi

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Bret Barkelew via groups.io
Sent: Wednesday, June 30, 2021 1:02 AM
To: devel@edk2.groups.io; rebecca@...; Shi, Steven <steven.shi@...>
Subject: Re: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Hmmm. If that works well, we might be able to investigate carrying both and picking based on the host OS.

- Bret

From: devel@edk2.groups.io <devel@edk2.groups.io> on behalf of Rebecca Cran via groups.io <rebecca@...>
Sent: Tuesday, June 29, 2021 9:42:41 AM
To: devel@edk2.groups.io <devel@edk2.groups.io>; steven.shi@... <steven.shi@...>
Subject: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Your Linux distribution should have a nuget package available (e.g. https://packages.ubuntu.com/hirsute/nuget) but the EDK2 CI is designed to use the copy of nuget installed with the edk2toolext package.

--
Rebecca Cran

On 6/29/21 9:44 AM, Steven Shi wrote:

Cran, Thanks. I’m curious whether the .pytool has native Linux executable binary to run? It looks the edk2 CI framework has more dependency in Linux than Windows.

Thanks

Steven Shi

Intel\IAGS\SFP\FIA (Firmware Infrastructure Automation)

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Rebecca Cran
Sent: Tuesday, June 29, 2021 8:13 PM
To: devel@edk2.groups.io; Shi, Steven <steven.shi@...>
Subject: Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

NuGet.exe is a .NET assembly/executable, so on Linux you'll need to install Mono (https://www.mono-project.com/).

--

Rebecca Cran

On 6/28/21 7:45 PM, Steven Shi wrote:

Hello,

I’m interested in the edk2 CI unit tests(https://github.com/tianocore/edk2/tree/master/.pytool). I tested it in my local. It works well in Windows but does not work in Linux. It looks the NuGet in pytool only has windows version (C:\steven\unit_test_env\Lib\site-packages\edk2toolext\bin\NuGet.exe) but not support the Linux. Below is my detail test steps. Do the edk2 CI unit tests work in Linux?

Windows :

c:\steven>py -m venv unit_test_env

c:\steven>cd c:\steven\edk2

c:\steven\edk2>c:\steven\unit_test_env\Scripts\activate.bat

(unit_test_env) c:\steven\edk2>pip install --upgrade -r pip-requirements.txt

(unit_test_env) c:\steven\edk2>stuart_setup -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2019

(unit_test_env) c:\steven\edk2>stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2019

(unit_test_env) c:\steven\edk2>edksetup.bat

(unit_test_env) c:\steven\edk2>edksetup.bat Rebuild

(unit_test_env) c:\steven\edk2>stuart_ci_build -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2019 -a X64,IA32 -t DEBUG --verbose

…

PROGRESS - Overall Build Status: Success

SECTION - Summary

PROGRESS - Success

Linux:

jshi19@ub2-uefi-b01:~/wksp_efi$ python3 -m venv unit_test_env

jshi19@ub2-uefi-b01:~/wksp_efi$ source unit_test_env/bin/activate

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ pip install --upgrade -r pip-requirements.txt

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ make -C BaseTools/

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ stuart_setup -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5

SECTION - Init SDE

WARNING - Using Pip Tools based BaseTools

SECTION - Loading Plugins

SECTION - Start Invocable Tool

SECTION - Initial update of environment

UpdatingWARNING - [SDE] Failed to fetch NugetDependecy: edk2-acpica-iasl@....0: [Nuget] We failed to install this version 20200717.0.0 of edk2-acpica-iasl

WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.15.05: [Nuget] We failed to install this version 2.15.05 of mu_nasm

. Done

SECTION -       Updated/Verified 3 dependencies

SECTION - Second pass update of environment

UpdatingWARNING - [SDE] Failed to fetch NugetDependecy: edk2-acpica-iasl@....0: [Nuget] We failed to install this version 20200717.0.0 of edk2-acpica-iasl

WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.15.05: [Nuget] We failed to install this version 2.15.05 of mu_nasm

. Done

SECTION -       Updated/Verified 3 dependencies

ERROR - We were unable to successfully update 2 dependencies in environment

SECTION - Summary

ERROR - Error

Thanks

Steven Shi

Re: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Steven Shi

#77434

Thank you to Rebecca. After install the libmono-system-net-http4.0-cil and mono-devel pkg as below, the ‘mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list’ pass.

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ sudo apt-get install libmono-system-net-http4.0-cil

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ sudo apt-get install mono-devel

But now there are new failures show up when install edk2-acpica-iasl:

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5 -v

… …

INFO - Cmd to run is: mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list

INFO - ------------------------------------------------

INFO - --------------Cmd Output Starting---------------

INFO - ------------------------------------------------

...INFO - global-packages: /home/jshi19/.nuget/packages/

INFO - global-packages: /home/jshi19/.nuget/packages/

INFO - ------------------------------------------------

INFO - --------------Cmd Output Finished---------------

INFO - ------------------------------------------------

INFO - --------- Running Time (mm:ss): 00:00 ----------

INFO - --------------Cmd Output Finished---------------

INFO - ----------- Return Code: 0x00000000 ------------

INFO - --------- Running Time (mm:ss): 00:00 ----------

INFO - ------------------------------------------------

INFO - ----------- Return Code: 0x00000000 ------------

INFO - Could not determine Nuget global packages cache location.

INFO - ------------------------------------------------

INFO - Cmd to run is: mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe install edk2-acpica-iasl -Source https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json -ExcludeVersion -NonInteractive -Version 20200717.0.0 -Verbosity detailed -OutputDirectory "/home/jshi19/wksp_efi/edk2-2/BaseTools/Bin/edk2-acpica-iasl_extdep_temp"

INFO - Could not determine Nuget global packages cache location.

INFO - ------------------------------------------------

INFO - --------------Cmd Output Starting---------------

INFO - Cmd to run is: mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe install mu_nasm -Source https://api.nuget.org/v3/index.json -ExcludeVersion -NonInteractive -Version 2.15.05 -Verbosity detailed -OutputDirectory "/home/jshi19/wksp_efi/edk2-2/BaseTools/Bin/mu_nasm_extdep_temp"

INFO - ------------------------------------------------

INFO - --------------Cmd Output Starting---------------

INFO - ------------------------------------------------

INFO - NuGet Version: 5.3.1.6268

INFO - Feeds used:

INFO - https://api.nuget.org/v3/index.json

INFO -

INFO - Feeds used:

INFO - https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json

INFO -

INFO - Attempting to gather dependency information for package 'mu_nasm.2.15.5' with respect to project '/home/jshi19/wksp_efi/edk2-2/BaseTools/Bin/mu_nasm_extdep_temp', targeting 'Any,Version=v0.0'

INFO -

INFO - Attempting to gather dependency information for package 'edk2-acpica-iasl.20200717.0.0' with respect to project '/home/jshi19/wksp_efi/edk2-2/BaseTools/Bin/edk2-acpica-iasl_extdep_temp', targeting 'Any,Version=v0.0'

INFO - Unable to load the service index for source https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json.

INFO - GSSAPI operation failed with error - An invalid name was supplied (Configuration file does not specify default realm).

INFO - System.InvalidOperationException: Unable to load the service index for source https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json. ---> NuGet.Protocol.Core.Types.FatalProtocolException: Unable to load the service index for source https://pkgs.dev.azure.com/projectmu/acpica/_packaging/mu_iasl/nuget/v3/index.json. ---> System.ComponentModel.Win32Exception: GSSAPI operation failed with error - An invalid name was supplied (Configuration file does not specify default realm).

INFO - at System.Net.Security.NegotiateStreamPal.AcquireCredentialsHandle (System.String package, System.Boolean isServer, System.Net.NetworkCredential credential) [0x000a3] in <a85c1a570f9a4f9f9c3d2cfa5504e34f>:0

… …

Thanks

Steven Shi

Intel\IAGS\SFP\FIA (Firmware Infrastructure Automation)

Show quoted text

From: Rebecca Cran <rebecca@...>
Sent: Friday, July 2, 2021 10:00 AM
To: devel@edk2.groups.io; Shi, Steven <steven.shi@...>; Bret Barkelew <Bret.Barkelew@...>; rebecca@...
Subject: Re: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

That means you need to install the various mono assemblies, such as the libmono-system-net-http4.0-cil package in Ubuntu.

I spent some time chasing down the dependencies last year which got very frustrating and tedious, but have forgotten which were needed. All I remember is there were quite a lot.

--
Rebecca Cran

On 7/1/21 7:36 PM, Steven Shi wrote:

I already installed mono, but NuGet.exe still fails to work in my side. I’m behind a proxy but don’t know how to configure the NuGet.exe proxy in .pytool. Below is my verbose error message.

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5 -v

… …

SECTION - Start Invocable Tool

SECTION - Initial update of environment

DEBUG - --- self_describing_environment.update_extdeps()

DEBUG - Verify 'mu_nasm' returning 'False'.

DEBUG - Verify 'edk2-acpica-iasl' returning 'False'.

DEBUG - Creating 5 threads for the SDE update

UpdatingDEBUG - Verify 'gcc_aarch64_linux' returning 'True'.

DEBUG - Verify 'gcc_riscv64_unknown' returning 'True'.

DEBUG - Verify 'mu_nasm' returning 'False'.

DEBUG - Verify 'gcc_arm_linux' returning 'True'.

DEBUG - Verify 'edk2-acpica-iasl' returning 'False'.

DEBUG - Cleaning dependency directory for 'mu_nasm'...

DEBUG - Cleaning dependency directory for 'edk2-acpica-iasl'...

INFO - Cmd to run is: mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list

INFO - Cmd to run is: mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list

INFO - ------------------------------------------------

INFO - ------------------------------------------------

INFO - --------------Cmd Output Starting---------------

INFO - --------------Cmd Output Starting---------------

INFO - ------------------------------------------------

INFO - ------------------------------------------------

...INFO - Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO -   Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO -   Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO -   Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO -   Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO - System.AggregateException: One or more errors occurred. (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO -    --- End of inner exception stack trace ---

… …

If I run the Cmd manually, I get below error:

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

System.AggregateException: One or more errors occurred. (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

   --- End of inner exception stack trace ---

at NuGet.CommandLine.Program.Initialize (NuGet.IFileSystem fileSystem, NuGet.CommandLine.IConsole console) [0x000ce] in <ff502667b3fe4a85827635c6fe6b0d0d>:0

at NuGet.CommandLine.Program.MainCore (System.String workingDirectory, System.String[] args) [0x000d8] in <ff502667b3fe4a85827635c6fe6b0d0d>:0

---> (Inner Exception #0) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #1) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #2) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #3) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #4) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

Thanks

Steven Shi

From: Bret Barkelew <Bret.Barkelew@...>
Sent: Friday, July 2, 2021 9:04 AM
To: Shi, Steven <steven.shi@...>; devel@edk2.groups.io; rebecca@...
Subject: RE: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

I believe the Azure DevOps build agents that are used by the EDK2 CI pipeline already have mono installed.

- Bret

From: Shi, Steven
Sent: Thursday, July 1, 2021 5:54 PM
To: devel@edk2.groups.io; Bret Barkelew; rebecca@...
Subject: RE: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Hi Bret, Rebecca,

Thanks for the info. Does edk2 CI run the .pytool unit tests in Linux? If yes, how does edk2 CI solve the nuget dependency problem in Linux right now?

Thanks

Steven Shi

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Bret Barkelew via groups.io
Sent: Wednesday, June 30, 2021 1:02 AM
To: devel@edk2.groups.io; rebecca@...; Shi, Steven <steven.shi@...>
Subject: Re: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Hmmm. If that works well, we might be able to investigate carrying both and picking based on the host OS.

- Bret

From: devel@edk2.groups.io <devel@edk2.groups.io> on behalf of Rebecca Cran via groups.io <rebecca@...>
Sent: Tuesday, June 29, 2021 9:42:41 AM
To: devel@edk2.groups.io <devel@edk2.groups.io>; steven.shi@... <steven.shi@...>
Subject: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Your Linux distribution should have a nuget package available (e.g. https://packages.ubuntu.com/hirsute/nuget) but the EDK2 CI is designed to use the copy of nuget installed with the edk2toolext package.

--
Rebecca Cran

On 6/29/21 9:44 AM, Steven Shi wrote:

Cran, Thanks. I’m curious whether the .pytool has native Linux executable binary to run? It looks the edk2 CI framework has more dependency in Linux than Windows.

Thanks

Steven Shi

Intel\IAGS\SFP\FIA (Firmware Infrastructure Automation)

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Rebecca Cran
Sent: Tuesday, June 29, 2021 8:13 PM
To: devel@edk2.groups.io; Shi, Steven <steven.shi@...>
Subject: Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

NuGet.exe is a .NET assembly/executable, so on Linux you'll need to install Mono (https://www.mono-project.com/).

--

Rebecca Cran

On 6/28/21 7:45 PM, Steven Shi wrote:

Hello,

I’m interested in the edk2 CI unit tests(https://github.com/tianocore/edk2/tree/master/.pytool). I tested it in my local. It works well in Windows but does not work in Linux. It looks the NuGet in pytool only has windows version (C:\steven\unit_test_env\Lib\site-packages\edk2toolext\bin\NuGet.exe) but not support the Linux. Below is my detail test steps. Do the edk2 CI unit tests work in Linux?

Windows :

c:\steven>py -m venv unit_test_env

c:\steven>cd c:\steven\edk2

c:\steven\edk2>c:\steven\unit_test_env\Scripts\activate.bat

(unit_test_env) c:\steven\edk2>pip install --upgrade -r pip-requirements.txt

(unit_test_env) c:\steven\edk2>stuart_setup -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2019

(unit_test_env) c:\steven\edk2>stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2019

(unit_test_env) c:\steven\edk2>edksetup.bat

(unit_test_env) c:\steven\edk2>edksetup.bat Rebuild

(unit_test_env) c:\steven\edk2>stuart_ci_build -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2019 -a X64,IA32 -t DEBUG --verbose

…

PROGRESS - Overall Build Status: Success

SECTION - Summary

PROGRESS - Success

Linux:

jshi19@ub2-uefi-b01:~/wksp_efi$ python3 -m venv unit_test_env

jshi19@ub2-uefi-b01:~/wksp_efi$ source unit_test_env/bin/activate

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ pip install --upgrade -r pip-requirements.txt

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ make -C BaseTools/

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ stuart_setup -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5

SECTION - Init SDE

WARNING - Using Pip Tools based BaseTools

SECTION - Loading Plugins

SECTION - Start Invocable Tool

SECTION - Initial update of environment

UpdatingWARNING - [SDE] Failed to fetch NugetDependecy: edk2-acpica-iasl@....0: [Nuget] We failed to install this version 20200717.0.0 of edk2-acpica-iasl

WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.15.05: [Nuget] We failed to install this version 2.15.05 of mu_nasm

. Done

SECTION -       Updated/Verified 3 dependencies

SECTION - Second pass update of environment

UpdatingWARNING - [SDE] Failed to fetch NugetDependecy: edk2-acpica-iasl@....0: [Nuget] We failed to install this version 20200717.0.0 of edk2-acpica-iasl

WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.15.05: [Nuget] We failed to install this version 2.15.05 of mu_nasm

. Done

SECTION -       Updated/Verified 3 dependencies

ERROR - We were unable to successfully update 2 dependencies in environment

SECTION - Summary

ERROR - Error

Thanks

Steven Shi

Re: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Rebecca Cran

#77433

That means you need to install the various mono assemblies, such as the libmono-system-net-http4.0-cil package in Ubuntu.

I spent some time chasing down the dependencies last year which got very frustrating and tedious, but have forgotten which were needed. All I remember is there were quite a lot.

--
Rebecca Cran

Show quoted text

On 7/1/21 7:36 PM, Steven Shi wrote:

I already installed mono, but NuGet.exe still fails to work in my side. I’m behind a proxy but don’t know how to configure the NuGet.exe proxy in .pytool. Below is my verbose error message.

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5 -v

… …

SECTION - Start Invocable Tool

SECTION - Initial update of environment

DEBUG - --- self_describing_environment.update_extdeps()

DEBUG - Verify 'mu_nasm' returning 'False'.

DEBUG - Verify 'edk2-acpica-iasl' returning 'False'.

DEBUG - Creating 5 threads for the SDE update

UpdatingDEBUG - Verify 'gcc_aarch64_linux' returning 'True'.

DEBUG - Verify 'gcc_riscv64_unknown' returning 'True'.

DEBUG - Verify 'mu_nasm' returning 'False'.

DEBUG - Verify 'gcc_arm_linux' returning 'True'.

DEBUG - Verify 'edk2-acpica-iasl' returning 'False'.

DEBUG - Cleaning dependency directory for 'mu_nasm'...

DEBUG - Cleaning dependency directory for 'edk2-acpica-iasl'...

INFO - Cmd to run is: mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list

INFO - Cmd to run is: mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list

INFO - ------------------------------------------------

INFO - ------------------------------------------------

INFO - --------------Cmd Output Starting---------------

INFO - --------------Cmd Output Starting---------------

INFO - ------------------------------------------------

INFO - ------------------------------------------------

...INFO - Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO -   Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO -   Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO -   Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO -   Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO - System.AggregateException: One or more errors occurred. (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO -    --- End of inner exception stack trace ---

… …

If I run the Cmd manually, I get below error:

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

System.AggregateException: One or more errors occurred. (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

   --- End of inner exception stack trace ---

at NuGet.CommandLine.Program.Initialize (NuGet.IFileSystem fileSystem, NuGet.CommandLine.IConsole console) [0x000ce] in <ff502667b3fe4a85827635c6fe6b0d0d>:0

at NuGet.CommandLine.Program.MainCore (System.String workingDirectory, System.String[] args) [0x000d8] in <ff502667b3fe4a85827635c6fe6b0d0d>:0

---> (Inner Exception #0) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #1) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #2) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #3) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #4) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

Thanks

Steven Shi

From: Bret Barkelew <Bret.Barkelew@...>
Sent: Friday, July 2, 2021 9:04 AM
To: Shi, Steven <steven.shi@...>; devel@edk2.groups.io; rebecca@...
Subject: RE: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

I believe the Azure DevOps build agents that are used by the EDK2 CI pipeline already have mono installed.

- Bret

From: Shi, Steven
Sent: Thursday, July 1, 2021 5:54 PM
To: devel@edk2.groups.io; Bret Barkelew; rebecca@...
Subject: RE: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Hi Bret, Rebecca,

Thanks for the info. Does edk2 CI run the .pytool unit tests in Linux? If yes, how does edk2 CI solve the nuget dependency problem in Linux right now?

Thanks

Steven Shi

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Bret Barkelew via groups.io
Sent: Wednesday, June 30, 2021 1:02 AM
To: devel@edk2.groups.io; rebecca@...; Shi, Steven <steven.shi@...>
Subject: Re: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Hmmm. If that works well, we might be able to investigate carrying both and picking based on the host OS.

- Bret

From: devel@edk2.groups.io <devel@edk2.groups.io> on behalf of Rebecca Cran via groups.io <rebecca@...>
Sent: Tuesday, June 29, 2021 9:42:41 AM
To: devel@edk2.groups.io <devel@edk2.groups.io>; steven.shi@... <steven.shi@...>
Subject: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Your Linux distribution should have a nuget package available (e.g. https://packages.ubuntu.com/hirsute/nuget) but the EDK2 CI is designed to use the copy of nuget installed with the edk2toolext package.

--
Rebecca Cran

On 6/29/21 9:44 AM, Steven Shi wrote:

Cran, Thanks. I’m curious whether the .pytool has native Linux executable binary to run? It looks the edk2 CI framework has more dependency in Linux than Windows.

Thanks

Steven Shi

Intel\IAGS\SFP\FIA (Firmware Infrastructure Automation)

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Rebecca Cran
Sent: Tuesday, June 29, 2021 8:13 PM
To: devel@edk2.groups.io; Shi, Steven <steven.shi@...>
Subject: Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

NuGet.exe is a .NET assembly/executable, so on Linux you'll need to install Mono (https://www.mono-project.com/).

--

Rebecca Cran

On 6/28/21 7:45 PM, Steven Shi wrote:

Hello,

I’m interested in the edk2 CI unit tests(https://github.com/tianocore/edk2/tree/master/.pytool). I tested it in my local. It works well in Windows but does not work in Linux. It looks the NuGet in pytool only has windows version (C:\steven\unit_test_env\Lib\site-packages\edk2toolext\bin\NuGet.exe) but not support the Linux. Below is my detail test steps. Do the edk2 CI unit tests work in Linux?

Windows :

c:\steven>py -m venv unit_test_env

c:\steven>cd c:\steven\edk2

c:\steven\edk2>c:\steven\unit_test_env\Scripts\activate.bat

(unit_test_env) c:\steven\edk2>pip install --upgrade -r pip-requirements.txt

(unit_test_env) c:\steven\edk2>stuart_setup -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2019

(unit_test_env) c:\steven\edk2>stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2019

(unit_test_env) c:\steven\edk2>edksetup.bat

(unit_test_env) c:\steven\edk2>edksetup.bat Rebuild

(unit_test_env) c:\steven\edk2>stuart_ci_build -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2019 -a X64,IA32 -t DEBUG --verbose

…

PROGRESS - Overall Build Status: Success

SECTION - Summary

PROGRESS - Success

Linux:

jshi19@ub2-uefi-b01:~/wksp_efi$ python3 -m venv unit_test_env

jshi19@ub2-uefi-b01:~/wksp_efi$ source unit_test_env/bin/activate

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ pip install --upgrade -r pip-requirements.txt

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ make -C BaseTools/

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ stuart_setup -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5

SECTION - Init SDE

WARNING - Using Pip Tools based BaseTools

SECTION - Loading Plugins

SECTION - Start Invocable Tool

SECTION - Initial update of environment

UpdatingWARNING - [SDE] Failed to fetch NugetDependecy: edk2-acpica-iasl@....0: [Nuget] We failed to install this version 20200717.0.0 of edk2-acpica-iasl

WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.15.05: [Nuget] We failed to install this version 2.15.05 of mu_nasm

. Done

SECTION -       Updated/Verified 3 dependencies

SECTION - Second pass update of environment

UpdatingWARNING - [SDE] Failed to fetch NugetDependecy: edk2-acpica-iasl@....0: [Nuget] We failed to install this version 20200717.0.0 of edk2-acpica-iasl

WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.15.05: [Nuget] We failed to install this version 2.15.05 of mu_nasm

. Done

SECTION -       Updated/Verified 3 dependencies

ERROR - We were unable to successfully update 2 dependencies in environment

SECTION - Summary

ERROR - Error

Thanks

Steven Shi

Re: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Bret Barkelew <bret.barkelew@...>

#77432

Good question! I don’t have much experience with that, but let me see what my Googling skills can turn up.

- Bret

Show quoted text

From: Shi, Steven <steven.shi@...>
Sent: Thursday, July 1, 2021 6:36:31 PM
To: Bret Barkelew <Bret.Barkelew@...>; devel@edk2.groups.io <devel@edk2.groups.io>; rebecca@... <rebecca@...>
Subject: RE: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

I already installed mono, but NuGet.exe still fails to work in my side. I’m behind a proxy but don’t know how to configure the NuGet.exe proxy in .pytool. Below is my verbose error message.

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5 -v

… …

SECTION - Start Invocable Tool

SECTION - Initial update of environment

DEBUG - --- self_describing_environment.update_extdeps()

DEBUG - Verify 'mu_nasm' returning 'False'.

DEBUG - Verify 'edk2-acpica-iasl' returning 'False'.

DEBUG - Creating 5 threads for the SDE update

UpdatingDEBUG - Verify 'gcc_aarch64_linux' returning 'True'.

DEBUG - Verify 'gcc_riscv64_unknown' returning 'True'.

DEBUG - Verify 'mu_nasm' returning 'False'.

DEBUG - Verify 'gcc_arm_linux' returning 'True'.

DEBUG - Verify 'edk2-acpica-iasl' returning 'False'.

DEBUG - Cleaning dependency directory for 'mu_nasm'...

DEBUG - Cleaning dependency directory for 'edk2-acpica-iasl'...

INFO - Cmd to run is: mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list

INFO - ------------------------------------------------

INFO - --------------Cmd Output Starting---------------

INFO - ------------------------------------------------

...INFO - Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO - Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO - System.AggregateException: One or more errors occurred. (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

INFO - --- End of inner exception stack trace ---

… …

If I run the Cmd manually, I get below error:

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ mono /home/jshi19/wksp_efi/unit_test_env/lib/python3.8/site-packages/edk2toolext/bin/NuGet.exe locals global-packages -list

Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

System.AggregateException: One or more errors occurred. (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) (Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.) ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

--- End of inner exception stack trace ---

at NuGet.CommandLine.Program.Initialize (NuGet.IFileSystem fileSystem, NuGet.CommandLine.IConsole console) [0x000ce] in <ff502667b3fe4a85827635c6fe6b0d0d>:0

at NuGet.CommandLine.Program.MainCore (System.String workingDirectory, System.String[] args) [0x000d8] in <ff502667b3fe4a85827635c6fe6b0d0d>:0

---> (Inner Exception #0) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #1) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #2) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #3) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

---> (Inner Exception #4) System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

File name: 'System.Net.Http, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'<---

Thanks

Steven Shi

From: Bret Barkelew <Bret.Barkelew@...>
Sent: Friday, July 2, 2021 9:04 AM
To: Shi, Steven <steven.shi@...>; devel@edk2.groups.io; rebecca@...
Subject: RE: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

I believe the Azure DevOps build agents that are used by the EDK2 CI pipeline already have mono installed.

- Bret

From: Shi, Steven
Sent: Thursday, July 1, 2021 5:54 PM
To: devel@edk2.groups.io; Bret Barkelew; rebecca@...
Subject: RE: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Hi Bret, Rebecca,

Thanks for the info. Does edk2 CI run the .pytool unit tests in Linux? If yes, how does edk2 CI solve the nuget dependency problem in Linux right now?

Thanks

Steven Shi

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Bret Barkelew via groups.io
Sent: Wednesday, June 30, 2021 1:02 AM
To: devel@edk2.groups.io; rebecca@...; Shi, Steven <steven.shi@...>
Subject: Re: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Hmmm. If that works well, we might be able to investigate carrying both and picking based on the host OS.

- Bret

From: devel@edk2.groups.io <devel@edk2.groups.io> on behalf of Rebecca Cran via groups.io <rebecca@...>
Sent: Tuesday, June 29, 2021 9:42:41 AM
To: devel@edk2.groups.io <devel@edk2.groups.io>; steven.shi@... <steven.shi@...>
Subject: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Your Linux distribution should have a nuget package available (e.g. https://packages.ubuntu.com/hirsute/nuget) but the EDK2 CI is designed to use the copy of nuget installed with the edk2toolext package.

--
Rebecca Cran

On 6/29/21 9:44 AM, Steven Shi wrote:

Cran, Thanks. I’m curious whether the .pytool has native Linux executable binary to run? It looks the edk2 CI framework has more dependency in Linux than Windows.

Thanks

Steven Shi

Intel\IAGS\SFP\FIA (Firmware Infrastructure Automation)

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Rebecca Cran
Sent: Tuesday, June 29, 2021 8:13 PM
To: devel@edk2.groups.io; Shi, Steven <steven.shi@...>
Subject: Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

NuGet.exe is a .NET assembly/executable, so on Linux you'll need to install Mono (https://www.mono-project.com/).

--

Rebecca Cran

On 6/28/21 7:45 PM, Steven Shi wrote:

Hello,

I’m interested in the edk2 CI unit tests(https://github.com/tianocore/edk2/tree/master/.pytool). I tested it in my local. It works well in Windows but does not work in Linux. It looks the NuGet in pytool only has windows version (C:\steven\unit_test_env\Lib\site-packages\edk2toolext\bin\NuGet.exe) but not support the Linux. Below is my detail test steps. Do the edk2 CI unit tests work in Linux?

Windows :

c:\steven>py -m venv unit_test_env

c:\steven>cd c:\steven\edk2

c:\steven\edk2>c:\steven\unit_test_env\Scripts\activate.bat

(unit_test_env) c:\steven\edk2>pip install --upgrade -r pip-requirements.txt

(unit_test_env) c:\steven\edk2>stuart_setup -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2019

(unit_test_env) c:\steven\edk2>stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2019

(unit_test_env) c:\steven\edk2>edksetup.bat

(unit_test_env) c:\steven\edk2>edksetup.bat Rebuild

(unit_test_env) c:\steven\edk2>stuart_ci_build -c .pytool/CISettings.py TOOL_CHAIN_TAG=VS2019 -a X64,IA32 -t DEBUG --verbose

…

PROGRESS - Overall Build Status: Success

SECTION - Summary

PROGRESS - Success

Linux:

jshi19@ub2-uefi-b01:~/wksp_efi$ python3 -m venv unit_test_env

jshi19@ub2-uefi-b01:~/wksp_efi$ source unit_test_env/bin/activate

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ pip install --upgrade -r pip-requirements.txt

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ make -C BaseTools/

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ stuart_setup -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5

(unit_test_env) jshi19@ub2-uefi-b01:~/wksp_efi/edk2-2$ stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5

SECTION - Init SDE

WARNING - Using Pip Tools based BaseTools

SECTION - Loading Plugins

SECTION - Start Invocable Tool

SECTION - Initial update of environment

UpdatingWARNING - [SDE] Failed to fetch NugetDependecy: edk2-acpica-iasl@....0: [Nuget] We failed to install this version 20200717.0.0 of edk2-acpica-iasl

WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.15.05: [Nuget] We failed to install this version 2.15.05 of mu_nasm

. Done

SECTION - Updated/Verified 3 dependencies

SECTION - Second pass update of environment

UpdatingWARNING - [SDE] Failed to fetch NugetDependecy: edk2-acpica-iasl@....0: [Nuget] We failed to install this version 20200717.0.0 of edk2-acpica-iasl

WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.15.05: [Nuget] We failed to install this version 2.15.05 of mu_nasm

. Done

SECTION - Updated/Verified 3 dependencies

ERROR - We were unable to successfully update 2 dependencies in environment

SECTION - Summary

ERROR - Error

Thanks

Steven Shi

Re: [EXTERNAL] Re: [edk2-devel] Do the edk2 CI unit tests work in Linux?

Steven Shi

#77431