Date   

Re: [edk2-devel] OVMF/QEMU shell based unit tests and writing to a virtual disk

Laszlo Ersek
 

On 10/27/20 14:26, Laszlo Ersek wrote:

Now, another option (on Linux anyway) is to loop-mount a "raw" virtual
disk image. This is not recommended, as it directly exposes the host
kernel's filesystem driver(s) to metadata produced by the guest. It
could trigger security issues in the host kernel.

(This is exactly what guestfish avoids, by running a separate Linux
guest -- called the "libguestfs appliance" -- on top of the virtual disk
image. The guestfish command interpreter on the host side exchanges
commands and data with the appliance over virtio-serial. If the metadata
on the disk image is malicious, it will break / exploit the *guest*
kernel in the appliance. The host-side component, the guestfish command
interpreter, only has to sanity-check the virtio-serial exchanges.)
If you trust your guest 100%, you *might* try the following:

- use a disk image file with "format=vhdx" on the QEMU command line,

- when QEMU is not running, I think you might be able to "loop-mount"
the VHDX file directly, on Windows.

Mutual exclusion remains just as important, of course.

... A reference about vhdx support:
<https://www.qemu.org/docs/master/system/images.html#cmdoption-image-formats-arg-vhdx>.

... A reference about the pitfalls with vvfat:
<https://www.qemu.org/docs/master/system/images.html#virtual-fat-disk-images>.

Thanks,
Laszlo


Re: [edk2-devel] OVMF/QEMU shell based unit tests and writing to a virtual disk

Laszlo Ersek
 

On 10/22/20 20:55, Sean wrote:
Laszlo and others familiar with QEMU,

I am trying to write automation that boots QEMU/OVMF and then runs EFI
applications and those efi applications then use the UEFI shell apis to
write back to the disk their state.  I am seeing very inconsistent
results with sometimes it working fine while other times the disk
contents are invalid.  If i run multiple tests it seems like the first
two work while the 3rd seems to start failing but overall it seems random.

Failing means:
    Disk contents corrupted but present.
    Disk contents wrong size (short).
    Files that show written in UEFI shell never show up on host.

I am trying to determine if this is a known limitation with QEMU or a
bug i need to track down in the unit test code.

My setup:

This is on a Windows 10 x64 host.  I am running current 5.1 version of
QEMU.

My script creates a folder in the Windows NTFS file system.  Copies the
EFI executables and startup.nsh files to it.  Then starts QEMU with the
following additional parameter.

 -drive file=fat:rw:{VirtualDrive},format=raw,media=disk
This is the problem. Don't use the fat / vvfat block driver for anything
meaningful.

I don't even have to look at particulars; as "fat" ("vvfat") is known to
be a hack. In particular write operations should not be relied on
(either guest->host or host->guest directions). Don't expect this QEMU
feature to work as a "semihosting" (esp. "live semihosting") solution.

What's important to understand about "vvfat" is that it attempts to
*re-compose* a filesystem view from block-level operations.
*De-composing* file operations into block operations is an everyday
occurrence (that's what filesystem drivers do everywhere). But the write
direction of vvfat attempts to do the *inverse* -- it seeks to recognize
block operations and to synthesize file operations from them. If you get
lucky, it sometimes even works.


Instead, please use a regular virtual disk image in the QEMU
configuration. This disk image should not be accessed concurrently by
QEMU (= the guest) and other host-side utilities. In other words, first
format / populate the disk image on the host side, then launch QEMU.
Then in the guest UEFI shell, terminate the guest with the "reset -s"
command. Finally, once QEMU has exited, use host-side utilities to fetch
the results from the virtual disk image.

On the host side (on a Linux installation anyway), I tend to use the
"mtools" package (such as "mcopy" etc), for manipulating the contents of
disk image files. Or, more frequently, the "guestfish" program (which is
extremely capable).

https://libguestfs.org/guestfish.1.html

I don't know if equivalents exist on Windows.

Now, another option (on Linux anyway) is to loop-mount a "raw" virtual
disk image. This is not recommended, as it directly exposes the host
kernel's filesystem driver(s) to metadata produced by the guest. It
could trigger security issues in the host kernel.

(This is exactly what guestfish avoids, by running a separate Linux
guest -- called the "libguestfs appliance" -- on top of the virtual disk
image. The guestfish command interpreter on the host side exchanges
commands and data with the appliance over virtio-serial. If the metadata
on the disk image is malicious, it will break / exploit the *guest*
kernel in the appliance. The host-side component, the guestfish command
interpreter, only has to sanity-check the virtio-serial exchanges.)


... Earlier I've looked into "virtio-fs" support for OVMF:

https://virtio-fs.gitlab.io/
https://www.redhat.com/archives/virtio-fs/2019-August/msg00349.html

however, it's very complex, and the wire format (the opcodes) are
extremely low-level and Linux specific. To the point where they directly
mirror Linux VFS system calls, and (due to lack of documentation) I
don't even understand what a big bunch of the opcodes *do*.

Earlier I had given some thought to a mapping between
EFI_SIMPLE_FILE_SYSTEM_PROTOCOL / EFI_FILE_PROTOCOL and the virtio-fs
opcodes, but when I did that, it looked like a bad fit. Virtio-fs seems
to aim at serializing *Linux guest* filesystem operations as tightly as
possble for host-side processing, and that seemed like a big obstacle
for a *UEFI guest* mapping.

In particular, EFI_SIMPLE_FILE_SYSTEM_PROTOCOL / EFI_FILE_PROTOCOL don't
really expect data and/or meta-data to change "under their feet" (-> due
to asynchronous host-side modifications). For a while I was hopeful to
expose such changes via the EFI_MEDIA_CHANGED return value. But -- alas,
I forget the details -- it seemed to turn out that the virtio-fs
interfaces wouldn't really let the EFI_SIMPLE_FILE_SYSTEM_PROTOCOL
driver, to be provided by OVMF, even *detect* the situations when it
should return EFI_SIMPLE_FILE_SYSTEM_PROTOCOL.

So, the virtio-fs driver for OVMF has been postponed indefinitely, and
the best I can recommend at the moment is to use a regular virtual disk
image file. Remember that QEMU (= guest), and the other (host-side)
utilities for manipulating the disk image, should be strictly serialized
(they should mutually exclude each other).

Thanks
Laszlo


VirtualDrive is the Windows file path of the said mentioned folder.


If interested you should be able to reproduce the results by pulling my
branch and/or you can review the above.

You can see the operations here:

PR: https://github.com/microsoft/mu_tiano_platforms/pull/1

My Branch:
https://github.com/spbrogan/mu_tiano_platforms/tree/personal/sebrogan/shellunittests


Or if interested you can reproduce it by following steps defined here:

https://github.com/spbrogan/mu_tiano_platforms/blob/personal/sebrogan/shellunittests/Platforms/QemuQ35Pkg/Docs/building.md


and more details here
https://github.com/spbrogan/mu_tiano_platforms/blob/personal/sebrogan/shellunittests/Platforms/QemuQ35Pkg/Plugins/QemuRunner/ReadMe.md


After building qemu with the right parameters for your environment you
can run <your stuart_build cmd> --flashonly MARK_STARTUP_NSH=TRUE
RUN_UNIT_TESTS=TRUE

For example in my environment it looks like
stuart_build -c Platforms\QemuQ35Pkg\PlatformBuild.py
TOOL_CHAIN_TAG=VS2019 --flashonly RUN_UNIT_TESTS=TRUE MAKE_STARTUP_NSH=TRUE


Anyway if i recall correctly last year when we talked briefly about
automation there was some concern that this would happen. Any
information and/or ideas would be greatly appreciated.

Thanks
Sean












Newbie question about EFI_BOOT_SERVICES_MEMORY

Vestigial Bidya <vestigialdev@...>
 

Hi, this question is about the UEFI PDFs rather than the Tianocore implementation. I'm writing my DXE Foundation, and the documentation states 
" Allocate the UEFI Boot Services Table from EFI_BOOT_SERVICES_MEMORY"

The way the term  EFI_BOOT_SERVICES_MEMORY is formatted makes it look like a defined Protocol or EFI_GUID, but its not mentioned anywhere else in the docs (or on Google). Is the author trying to convey just that these allocations should be from memory that will be released after calling EXIT_BOOT_SERVICES ? Or is  EFI_BOOT_SERVICES_MEMORY an actual term in the spec that I just can't find?


OVMF/QEMU shell based unit tests and writing to a virtual disk

Sean
 

Laszlo and others familiar with QEMU,

I am trying to write automation that boots QEMU/OVMF and then runs EFI applications and those efi applications then use the UEFI shell apis to write back to the disk their state. I am seeing very inconsistent results with sometimes it working fine while other times the disk contents are invalid. If i run multiple tests it seems like the first two work while the 3rd seems to start failing but overall it seems random.

Failing means:
Disk contents corrupted but present.
Disk contents wrong size (short).
Files that show written in UEFI shell never show up on host.

I am trying to determine if this is a known limitation with QEMU or a bug i need to track down in the unit test code.

My setup:

This is on a Windows 10 x64 host. I am running current 5.1 version of QEMU.

My script creates a folder in the Windows NTFS file system. Copies the EFI executables and startup.nsh files to it. Then starts QEMU with the following additional parameter.

-drive file=fat:rw:{VirtualDrive},format=raw,media=disk

VirtualDrive is the Windows file path of the said mentioned folder.


If interested you should be able to reproduce the results by pulling my branch and/or you can review the above.

You can see the operations here:

PR: https://github.com/microsoft/mu_tiano_platforms/pull/1

My Branch: https://github.com/spbrogan/mu_tiano_platforms/tree/personal/sebrogan/shellunittests

Or if interested you can reproduce it by following steps defined here:

https://github.com/spbrogan/mu_tiano_platforms/blob/personal/sebrogan/shellunittests/Platforms/QemuQ35Pkg/Docs/building.md

and more details here
https://github.com/spbrogan/mu_tiano_platforms/blob/personal/sebrogan/shellunittests/Platforms/QemuQ35Pkg/Plugins/QemuRunner/ReadMe.md

After building qemu with the right parameters for your environment you can run <your stuart_build cmd> --flashonly MARK_STARTUP_NSH=TRUE RUN_UNIT_TESTS=TRUE

For example in my environment it looks like
stuart_build -c Platforms\QemuQ35Pkg\PlatformBuild.py TOOL_CHAIN_TAG=VS2019 --flashonly RUN_UNIT_TESTS=TRUE MAKE_STARTUP_NSH=TRUE


Anyway if i recall correctly last year when we talked briefly about automation there was some concern that this would happen. Any information and/or ideas would be greatly appreciated.

Thanks
Sean


Re: Communicating with usb devices

Guomin Jiang
 

You can locate UsbIo protocol, the prototype is described in UEFI Spec.

You can check it first.

-----Original Message-----
From: discuss@edk2.groups.io <discuss@edk2.groups.io> On Behalf Of
alireza0101sadeghpour@gmail.com
Sent: Monday, October 19, 2020 7:57 PM
To: discuss@edk2.groups.io
Subject: [edk2-discuss] Communicating with usb devices

Hi All,
I am a newbie in EFI development. Currently I have a requirement of
enumerating attached USB devices, communicating with them and etc.

However, after spending a day, I could not able to crack this thing. i can see
that in MdePkg and MdeModulePkg exists some driver for communicating
with usb devices but i can't figure out how should i use them.

I appreciate it if anyone could give me an example code or something like
that for communicating with usb devcies.

thanks




Communicating with usb devices

alireza0101sadeghpour@...
 

Hi All,
I am a newbie in EFI development. Currently I have a requirement of enumerating attached USB devices, communicating with them and etc.

However, after spending a day, I could not able to crack this thing. i can see that in MdePkg and MdeModulePkg exists some driver for communicating with usb devices but i can't figure out how should i use them.

I appreciate it if anyone could give me an example code or something like that for communicating with usb devcies.

thanks


Re: Retrieving USB device vendorID and productID

Andrew Fish
 

What error did you get?

On Oct 15, 2020, at 4:07 AM, Ali Shirvani <aj.shirvani@gmail.com> wrote:

Regards,


Retrieving USB device vendorID and productID

aj.shirvani@...
 

Hi,

I've located a handle to the USB device with
`gEfiSimpleFileSystemProtocolGuid.` Now I want to retrieve vendorID and
productID with `gEfiUsbIoProtocolGuid,` and I got an error when trying to
get the appropriated handle with this function:

// UsbHandle is an EFI_HANDLE located by gEfiSimpleFileSystemProtocolGuid
gBS->HandleProtocol(UsbHandle, &gEfiUsbIoProtocolGuid, (VOID**)&UsbIo);

Would you please help me to get the vendorID and productID of that USB
device?

Regards,
Ali


Re: Come One, Come All to the Project Mu UEFI Talkbox

Bret Barkelew
 

Reminder that the first Talkbox Office Day will be happening today! Here’s a new link in case the one below has expired!

https://discord.gg/feYRkxz

- Bret

From: Bret Barkelew<mailto:Bret.Barkelew@microsoft.com>
Sent: Friday, October 9, 2020 11:26 AM
To: discuss@edk2.groups.io<mailto:discuss@edk2.groups.io>
Cc: Microsoft Core UEFI Team<mailto:MsCoreUefi@microsoft.com>
Subject: Come One, Come All to the Project Mu UEFI Talkbox

The Project Mu team is trying something a little new and different! In light of the limited involvement in UEFI and Tianocore community calls of late, we’re piloting a new idea for community engagement. We’ve set up a Discord server and are going to hosting “office hours” every Wednesday until Thanksgiving! This is an opportunity to connect, ask questions, bat around ideas, and just generally get to know the EDK2 community a little better. There is no planned content, and we can’t promise answers to every question, but we are highly knowledgeable and eager to see more energy in FW development.

This is the link to join the Discord:
https://discord.gg/DrmDQAv

This pilot is for the Wednesdays between 2020-10-14 and 2020-11-25. We’re a US-based, west-coast team, so expect the greatest availability between 10AM and 4PM Pacific (UTC-7). However, we’ll try to be around as much as possible those days (even in the off-hours), and may even be around on non-Wednesdays.

We should emphasize that this is NOT a Microsoft-sponsored event, or in any way related to Microsoft business. It’s not even REALLY a Project Mu event. It’s a bunch of interested folks attempting to be interesting. Who knows, maybe we’ll even figure out ways to engage with each other that don’t revolve around work.😉

- Bret


Re: Question about NonDiscoverablePciDeviceDxe

Jeff Brasen
 

-----Original Message-----
From: Laszlo Ersek <lersek@redhat.com>
Sent: Thursday, October 8, 2020 7:26 AM
To: Jeff Brasen <jbrasen@nvidia.com>; discuss@edk2.groups.io
Subject: Re: [edk2-discuss] Question about NonDiscoverablePciDeviceDxe

External email: Use caution opening links or attachments


On 10/07/20 17:53, Jeff Brasen wrote:


-----Original Message-----
From: discuss@edk2.groups.io <discuss@edk2.groups.io> On Behalf Of
Laszlo Ersek
Sent: Wednesday, October 7, 2020 8:01 AM
To: discuss@edk2.groups.io; Jeff Brasen <jbrasen@nvidia.com>
Subject: Re: [edk2-discuss] Question about
NonDiscoverablePciDeviceDxe

External email: Use caution opening links or attachments


On 10/06/20 23:52, Jeff Brasen wrote:
We have a use case where we would want to expose PciIO protocols for
a
device that doesn't match any of the guids that it supports.

What do you mean by "guids"? Did you mean "vendor id / device id"
perhaps?
[JMB] Sorry, should have been clearer

I was referring to the variable STATIC CONST EFI_GUID * CONST
SupportedNonDiscoverableDevices[] = {

That both guards the supported function in the binding protocol as well as
the selection of the right values in InitializePciIoProtocol.
The vendor/device ids are one of the things we will need to set based on
any extension to this.

Thanks for the explanation, I got it now.

Could you simply contribute the code to edk2 that you need in
MdeModulePkg/Bus/Pci/NonDiscoverablePciDeviceDxe?

If you add a new GUID and new branches in the driver that depend on that
GUID, that won't bother other consumers of the driver.

You mention "edk2-platforms area" below, so I assume you don't mind
open-sourcing the logic itself. My suggestion would be to simply extend
NonDiscoverablePciDeviceDxe right where it lives, in edk2.
[JMB] That works for me, just wasn't sure if there was a desire to keep the stuff in this driver to generic standard supported stuff.

Thanks,
Jeff

Thanks
Laszlo


Before we go and implement support for this wanted to get a feeling
from
the community on what to do.


1. Fork NonDiscoverablePciDeviceDxe to our edk2-platforms area
for
support for this (Doesn't seem ideal)
2. Add support for extending this
* Add a protocol that this driver consumes for overrides
* Add a library initializer that registers overrides in both the
supported
function and the configuration space setup code

Any thoughts?

Thanks,
Jeff









Come One, Come All to the Project Mu UEFI Talkbox

Bret Barkelew
 

The Project Mu team is trying something a little new and different! In light of the limited involvement in UEFI and Tianocore community calls of late, we’re piloting a new idea for community engagement. We’ve set up a Discord server and are going to hosting “office hours” every Wednesday until Thanksgiving! This is an opportunity to connect, ask questions, bat around ideas, and just generally get to know the EDK2 community a little better. There is no planned content, and we can’t promise answers to every question, but we are highly knowledgeable and eager to see more energy in FW development.

This is the link to join the Discord:
https://discord.gg/DrmDQAv

This pilot is for the Wednesdays between 2020-10-14 and 2020-11-25. We’re a US-based, west-coast team, so expect the greatest availability between 10AM and 4PM Pacific (UTC-7). However, we’ll try to be around as much as possible those days (even in the off-hours), and may even be around on non-Wednesdays.

We should emphasize that this is NOT a Microsoft-sponsored event, or in any way related to Microsoft business. It’s not even REALLY a Project Mu event. It’s a bunch of interested folks attempting to be interesting. Who knows, maybe we’ll even figure out ways to engage with each other that don’t revolve around work.😉

- Bret


Re: Question about NonDiscoverablePciDeviceDxe

Laszlo Ersek
 

On 10/07/20 17:53, Jeff Brasen wrote:


-----Original Message-----
From: discuss@edk2.groups.io <discuss@edk2.groups.io> On Behalf Of
Laszlo Ersek
Sent: Wednesday, October 7, 2020 8:01 AM
To: discuss@edk2.groups.io; Jeff Brasen <jbrasen@nvidia.com>
Subject: Re: [edk2-discuss] Question about NonDiscoverablePciDeviceDxe

External email: Use caution opening links or attachments


On 10/06/20 23:52, Jeff Brasen wrote:
We have a use case where we would want to expose PciIO protocols for a
device that doesn't match any of the guids that it supports.

What do you mean by "guids"? Did you mean "vendor id / device id"
perhaps?
[JMB] Sorry, should have been clearer

I was referring to the variable STATIC
CONST EFI_GUID * CONST
SupportedNonDiscoverableDevices[] = {

That both guards the supported function in the binding protocol as well as the selection of the right values in InitializePciIoProtocol.
The vendor/device ids are one of the things we will need to set based on any extension to this.
Thanks for the explanation, I got it now.

Could you simply contribute the code to edk2 that you need in
MdeModulePkg/Bus/Pci/NonDiscoverablePciDeviceDxe?

If you add a new GUID and new branches in the driver that depend on that
GUID, that won't bother other consumers of the driver.

You mention "edk2-platforms area" below, so I assume you don't mind
open-sourcing the logic itself. My suggestion would be to simply extend
NonDiscoverablePciDeviceDxe right where it lives, in edk2.

Thanks
Laszlo


Before we go and implement support for this wanted to get a feeling from
the community on what to do.


1. Fork NonDiscoverablePciDeviceDxe to our edk2-platforms area for
support for this (Doesn't seem ideal)
2. Add support for extending this
* Add a protocol that this driver consumes for overrides
* Add a library initializer that registers overrides in both the supported
function and the configuration space setup code

Any thoughts?

Thanks,
Jeff









Re: Question about NonDiscoverablePciDeviceDxe

Jeff Brasen
 

-----Original Message-----
From: discuss@edk2.groups.io <discuss@edk2.groups.io> On Behalf Of
Laszlo Ersek
Sent: Wednesday, October 7, 2020 8:01 AM
To: discuss@edk2.groups.io; Jeff Brasen <jbrasen@nvidia.com>
Subject: Re: [edk2-discuss] Question about NonDiscoverablePciDeviceDxe

External email: Use caution opening links or attachments


On 10/06/20 23:52, Jeff Brasen wrote:
We have a use case where we would want to expose PciIO protocols for a
device that doesn't match any of the guids that it supports.

What do you mean by "guids"? Did you mean "vendor id / device id"
perhaps?
[JMB] Sorry, should have been clearer

I was referring to the variable STATIC
CONST EFI_GUID * CONST
SupportedNonDiscoverableDevices[] = {

That both guards the supported function in the binding protocol as well as the selection of the right values in InitializePciIoProtocol.
The vendor/device ids are one of the things we will need to set based on any extension to this.

Thanks,
Jeff


Thanks
Laszlo

Before we go and implement support for this wanted to get a feeling from
the community on what to do.


1. Fork NonDiscoverablePciDeviceDxe to our edk2-platforms area for
support for this (Doesn't seem ideal)
2. Add support for extending this
* Add a protocol that this driver consumes for overrides
* Add a library initializer that registers overrides in both the supported
function and the configuration space setup code

Any thoughts?

Thanks,
Jeff









Re: Question about NonDiscoverablePciDeviceDxe

Laszlo Ersek
 

On 10/06/20 23:52, Jeff Brasen wrote:
We have a use case where we would want to expose PciIO protocols for a device that doesn't match any of the guids that it supports.
What do you mean by "guids"? Did you mean "vendor id / device id" perhaps?

Thanks
Laszlo

Before we go and implement support for this wanted to get a feeling from the community on what to do.


1. Fork NonDiscoverablePciDeviceDxe to our edk2-platforms area for support for this (Doesn't seem ideal)
2. Add support for extending this
* Add a protocol that this driver consumes for overrides
* Add a library initializer that registers overrides in both the supported function and the configuration space setup code

Any thoughts?

Thanks,
Jeff






Question about NonDiscoverablePciDeviceDxe

Jeff Brasen
 

We have a use case where we would want to expose PciIO protocols for a device that doesn't match any of the guids that it supports. Before we go and implement support for this wanted to get a feeling from the community on what to do.


1. Fork NonDiscoverablePciDeviceDxe to our edk2-platforms area for support for this (Doesn't seem ideal)
2. Add support for extending this
* Add a protocol that this driver consumes for overrides
* Add a library initializer that registers overrides in both the supported function and the configuration space setup code

Any thoughts?

Thanks,
Jeff


Re: GCD initialization and memory allocation HOBs

Jeff Brasen
 

Inline

-----Original Message-----
From: gaoliming <gaoliming@byosoft.com.cn>
Sent: Monday, September 28, 2020 8:40 AM
To: discuss@edk2.groups.io; Jeff Brasen <jbrasen@nvidia.com>;
lersek@redhat.com
Subject: 回复: [edk2-discuss] GCD initialization and memory allocation HOBs

External email: Use caution opening links or attachments


Jeff:

-----邮件原件-----
发件人: bounce+34241+372+4905953+8764802@groups.io
<bounce+34241+372+4905953+8764802@groups.io> 代表 Jeff Brasen
发送时间: 2020年9月26日 1:08
收件人: discuss@edk2.groups.io; lersek@redhat.com
主题: Re: [edk2-discuss] GCD initialization and memory allocation HOBs

#2 is the case we are in, we have data that is loaded by our pre-UEFI
bootloaders into memory that should behave as boot services memory (OS
can reclaim this) When we setup our HOB list we added a normal system
memory resource that covers both the HOB list and this other data. We
then add a memory allocation HOB to mark this memory as boot services
data. We are seeing cases where CoreInitializeMemoryServices Is
selecting the region that has this data and is both causing corruption
as well as preventing the memory allocation hob from being processed
correctly as the memory is not free when GCD parses it.
You mean the memory range specified by PHIT resource hob is selected by
CoreInitializeMemoryServices().
This memory range includes PEI memory and the allocated memory. So,
there is no free memory in this memory range. After
CoreInitializeMemoryServices(), do you meet with memory allocation
failure?
[JMB] So what we have is a system resource that includes both the PHIT memory area (From EfiMemoryBottom to EfiMemoryTop) as well as other data that is boot services memory that is marked by a memory allocation hob entry but is outside the PHIT region. In our case we see CoreInitializeMemoryServices selecting this region as it doesn't look for memory allocations in the case that it selections a region outside the PHIT region. We can split the system resource so that the memory isn't in question isn't in the same system resource which solves the issue for now (as we have enough memory in the EfiFreeMemoryBottom to EfiFreeMemoryTop region. However, if there isn't enough memory in this region (and assuming we don't have any memory before EfiMemoryBottom due to the same region) it will just go search other system resources which should have the same issue.

Note we don't build this with PEI as we don't use that phase as we just skip from SEC to DXE as our memory is already setup so there is less value to that phase.

From my reading of the PI spec there is no issue with have memory allocation entries that are outside phit areas so what we are setting up seems to be compliant with the spec.

I would suggest one of the two changes for this
1. Have CoreInitializeMemoryServices Handle memory allocation hobs
2. Update either the PI spec or have an EDK2 rule that makes sure that HOB lists are created in such a we aren't coding setup of the hob producer to the implementation of the dxe core.

Can you attach the failure boot log for further analysis?
[JMB] I don't think we actually see any logs related to this but it was causing our memory it be not marked as allocated which was causing some corruption, we can look at adding some prints to this case if this is useful.

Thanks
Liming
We can easily work around this particular issue by splitting the
system memory resource so that GCD then goes to look for space in the
free region in the hob or at the beginning of the resource that
contains the HOB list. But if we don't have enough space in those
regions then it will look for other memory regions and it doesn't seem
like the best practice to design the HOB producer with the specific
implementation of GCD services.

It seems that if we have memory allocation data that is in outside the
HOB list itself we could hit this problem, and this seems valid per my
reading of the PI spec.

-----Original Message-----
From: discuss@edk2.groups.io <discuss@edk2.groups.io> On Behalf Of
Laszlo Ersek
Sent: Friday, September 25, 2020 3:06 AM
To: discuss@edk2.groups.io; Jeff Brasen <jbrasen@nvidia.com>
Cc: Jan Bobek <jbobek@nvidia.com>; Ashish Singhal
<ashishsingha@nvidia.com>
Subject: Re: [edk2-discuss] GCD initialization and memory allocation
HOBs

External email: Use caution opening links or attachments


On 09/24/20 20:40, Jeff Brasen wrote:
It looks like CoreInitializeMemoryServices
(https://github.com/tianocore/edk2/blob/dd5c7e3c5282b084daa5bbf0ec229
c
ec699b2c17/MdeModulePkg/Core/Dxe/Gcd/Gcd.c#L2116)
does not honor any memory allocations outside of the HOB list itself.
Should we add code to this function to make sure the region this
function select does not end up over a reserved region that the HOB
producer phase marked as already allocated? Either that or codify
the specific requirements a bit more on to make sure cases like this
are invalid. (producer might have to split up system resource
entries,
etc)
Do you mean

(1) a physically reserved memory address range
(EFI_RESOURCE_MEMORY_RESERVED), or

(2) normal RAM (EFI_RESOURCE_SYSTEM_MEMORY) that is meant to be
"repurposed" (= pre-allocated) in PEI as a particular UEFI memory type?

EFI_HOB_RESOURCE_DESCRIPTOR "does not describe how memory is used
but
instead describes the attributes of the physical memory present".

CoreInitializeMemoryServices() honors (1).

CoreInitializeMemoryServices() does not honor (2); however, the later
function CoreInitializeGcdServices() does honor (2).


* Regarding (1):

My understanding is that the HOB producer phase should describe
"physically reserved" memory regions with
EFI_RESOURCE_MEMORY_RESERVED
resource descriptor HOBs, and these should not overlap normal memory
regions, which are described with EFI_RESOURCE_SYSTEM_MEMORY
descriptor HOBs.

Furthermore, whether (a part of) a memory resource is allocated or
not, is
*orthogonal* to whether the memory resource is "physically reserved
memory" or "normal RAM". In my opinion, the diagram at

PI 1.7
Volume 2 (DXE Core Interface)
7.2.2 GCD Memory Resources
Figure 2. GCD Memory State Transitions

applies.

So the HOB producer phase should firstly describe the area in question
as EFI_RESOURCE_MEMORY_RESERVED (without overlapping any other
resource descriptor in memory address space).

Secondly, if the HOB producer phase wants to prevent DXE modules from
allocating reserved memory out of this resource, then the HOB producer
phase should *also* cover the entire range with a memalloc HOB that
uses EfiReservedMemoryType.

The first step above (= describing the area with an
EFI_RESOURCE_MEMORY_RESERVED HOB) will activate the following
branch
in
CoreInitializeMemoryServices():

//
// Skip Resource Descriptor HOBs that do not describe tested
system
memory
//
ResourceHob = Hob.ResourceDescriptor;
if (ResourceHob->ResourceType !=
EFI_RESOURCE_SYSTEM_MEMORY) {
continue;
}

* Regarding (2):

Normal (= not physically reserved) RAM that is allocated in PEI --
using memalloc HOBs with various UEFI memory types, such as BS Data or
AcpiNVS
-- is set aside in the CoreInitializeGcdServices() function.

Is that too late for your purpose? If so, why?

AIUI, CoreInitializeGcdServices() does make sure that DXE-phase calls
to
gDS->AllocateMemorySpace(), gBS->AllocatePool(), and friends, will not
trample over the original memalloc HOBs.

Thanks,
Laszlo










Re: ESRT in OVMF

Tomas Pilar (tpilar)
 

I don't think you will be allowed to modify the DxeCapsuleLib to connect
all controllers when it's loading the FMP embedded driver. There is
probably a better way of doing it.

Also, you need to send the patches to the mailing list devel@ rather than
attaching them here. Here is another link describing the process:
https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Development-Process

The second patch looks okay but you'll need to expand on the commit message
a lot more with motivation, use-cases etc. If you search through the devel@
mailing list, you will see that a similar discussion was already had in the
past.

Cheers,
Tom

On Tue, Oct 6, 2020 at 1:06 PM Sandeep Dhanvada <sandeep.dhanvada@xilinx.com>
wrote:

Hi Tom,

Please see the patches attached and let me know in case of any issues.
I used all the "git config" options mentioned in the link you referred,
but, I am not able to remove the line breaks at EOL in the patch.

Thanks,
Sandeep


Re: ESRT in OVMF

Sandeep Dhanvada
 

Hi Tom,

Please see the patches attached and let me know in case of any issues.
I used all the "git config" options mentioned in the link you referred, but, I am not able to remove the line breaks at EOL in the patch.

Thanks,
Sandeep


Re: SmmSwDispatch2Protocol - Registering an SMI - Can't find protocol with OVMF

Laszlo Ersek
 

Hi,

(cross-posting to edk2-devel, comments below)

On 09/29/20 14:32, Simon McMillan wrote:
Trying to register a SMI in my custom SMM module I'm running into
issues.

Before registering the SMI, the code looks for the specific protocol
this way:
Status = gSmst->SmmLocateProtocol(
&gEfiSmmSwDispatch2ProtocolGuid,
NULL,
(VOID**)&SwDispatch
);
DEBUG ( (EFI_D_INFO, "SmmSwDispatch2Protocol Status: 0x%x\n", Status) );
ASSERT_EFI_ERROR(Status);

Unfortunately, while starting OVMF, boot hangs and the following
message appears:
SmmSwDispatch2Protocol Status: 0xE
ASSERT_EFI_ERROR (Status = Not Found)

Adding gEfiSmmSwDispatch2ProtocolGuid to the Depex section of my INF
file doesn't solve the issue, in fact it prevents the custom code from
being loaded at all.

For info in OVMF, I simply added a reference to the inf file in the
DSC file, so that my custom code is built along OVMF:

custom_code/custom_code.inf

OVMF is built with the following options:
build -DSMM_REQUIRE

Then the SMM module is converted to the right format and injected
manually with UEFITool.

Would anyone be of help with this situation? Or could give me an
advice of what would be wrong or missing?
OVMF does not contain an implementation of the
EFI_MM_SW_DISPATCH_PROTOCOL.

While OVMF does handle a particular software MMI, that handling occurs
via a root MMI handler (in "OvmfPkg/CpuHotplugSmm") that watches out for
a particular command port value.

When I reviewed using a root MMI handler vs. EFI_MM_SW_DISPATCH_PROTOCOL
in last October, the latter seemed like a needless complication to me.
Mike, Jiewen and others provided many helpful explanations, and I
decided that EFI_MM_SW_DISPATCH_PROTOCOL was unjustified for OVMF.
Below, I'll try to rehash and extend parts of those discussions.

Considering EFI_MM_SW_DISPATCH_PROTOCOL, OVMF would have had to produce
that protocol from a chipset (aka "silicon") driver. And for actually
handling the CPU hotplug MMI, OVMF would have to consume
EFI_MM_SW_DISPATCH_PROTOCOL in a different -- not silicon, but
"platform" -- driver, and register an MMI handler through
EFI_MM_SW_DISPATCH_PROTOCOL.

However, for OVMF and QEMU, there is no practical distinction between
"chipset" (= "silicon") and "platform". The EFI_MM_SW_DISPATCH_PROTOCOL
implementation would be a thick wrapper on top of IO port 0xB2,
providing a more or less useless software abstraction.

- The only small advantage would be that EFI_MM_SW_DISPATCH_PROTOCOL
would permit the caller to ask for a new (previously not agreed-upon)
"SwMmiInputValue" (via ((UINTN)-1)), eliminating the need for
coordination wrt. the 0xB2 IO port values.

This was not a compelling feature, as upstream OVMF was about to get
its first own MMI handler. Additionally, the command port value was
going to have to be agreed upon between the OS (QEMU would generate
the ACPI payload for the OS, for triggering the MMI) and the firmware.
For such "public" MMIs, the generation of "SwMmiInputValue" by
EFI_MM_SW_DISPATCH_PROTOCOL.Register() is not useful anyway.

- Furthermore, there was a *big* disadvantage -- the registered
"DispatchFunction" would have to be called back with
EFI_MM_SW_CONTEXT. And filling in EFI_MM_SW_CONTEXT.SwMmiCpuIndex --
"the 0-based index of the CPU which generated the software MMI" -- is
both quite useless [*], and very challenging on OVMF / QEMU [**].

[*] It might tell us what VCPU was executing the ACPI GPE handler in
the OS's AML interpreter (ultimately raising the VCPU hotplug
MMI), but what is that knowledge good for?

[**] To refer forward to the QncSmmDispatcher module as an example: in
that driver, "SwSmiCpuIndex" is laboriously determined in
SwGetBuffer(), by checking which CPU's save state map indicates
that *that* CPU was interrupted in the particular IO port write
instruction that triggered the SMI.

But on QEMU/KVM, this kind of information is not even available.
The SmmCpuFeaturesReadSaveStateRegister() function in
"OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c" returns
constant EFI_NOT_FOUND, for EFI_SMM_SAVE_STATE_REGISTER_IO.
QEMU/KVM does not describe the interrupted OUT instruction in the
save state map, and so the method seen in Quark's SwGetBuffer()
couldn't work.

So, quite a few complications, for little use.

In the edk2-platforms tree, the directory

Silicon/Intel/QuarkSocPkg/QuarkNorthCluster/Smm/DxeSmm/QncSmmDispatcher

contains a driver that produces EFI_MM_SW_DISPATCH_PROTOCOL (still using
the older / original name EFI_SMM_SW_DISPATCH2_PROTOCOL). When I
reviewed this driver, I formed the opinion that it wasn't really
"silicon-specific" -- I felt that it should be turned into a "universal"
IA32/X64 driver. The responsibilities of the driver (in relation to
EFI_MM_SW_DISPATCH_PROTOCOL) seem to be:

(1) It has to install a root MMI handler with
gMmst->MmiHandlerRegister().

(2) It must fetch the Command and Data port values from the *same*
chipset registers that the EFI_MM_CONTROL_PROTOCOL.Trigger()
function writes. The location and nature of this register block is
about the only "silicon-specific" bit, IMO.

(3) In the root MMI handler, the driver must look up the fetched Command
value against the dictionary that is managed by Register() /
UnRegister().

(4) Upon a match, the driver must call the callback associated with
Command, and also pass Data to it (coming from
EFI_MM_CONTROL_PROTOCOL.Trigger() via IO port 0xB3).

(5) EFI_MM_SW_DISPATCH_PROTOCOL.Register() must never hand out such a
Command value (a.k.a. "SwMmiInputValue"), for ((UINTN)-1), that
equals the value that EFI_MM_CONTROL_PROTOCOL.Trigger() stores for
(CommandPort==NULL).

Otherwise, a Communicate request could be mistaken for *that* SW
MMI, when SmmEntryPoint() [MdeModulePkg/Core/PiSmmCore/PiSmmCore.c]
invokes the root MMI handler registered in (1), after processing the
Communicate request.

This (ie., the agreement between EFI_MM_CONTROL_PROTOCOL and
EFI_MM_SW_DISPATCH_PROTOCOL) is maybe the 2nd silicon-specific point
that the EFI_MM_SW_DISPATCH_PROTOCOL driver has to take into
account.

If such a generic driver existed, then we could perhaps include it
verbatim in the OVMF platform DSC / FDF files. If someone really needed
EFI_MM_SW_DISPATCH_PROTOCOL specifically.

Until then, I suggest registering another root MMI handler with
gMmst->MmiHandlerRegister(), like "OvmfPkg/CpuHotplugSmm" does. In the
handler, filter for a command port value that is strictly greater than
ICH9_APM_CNT_CPU_HOTPLUG (4).

See commit 17efae27acaf ("OvmfPkg/CpuHotplugSmm: introduce skeleton for
CPU Hotplug SMM driver", 2020-03-04).

Manual arbitration between the various values for command port 0xB2 is
necessary, accordingly. In practice this should not be a problem, as
OVMF is open source; just be prepared that new Command values upstreamed
to OVMF in the future might require you to pick a different Command
value for your out-of-tree driver.

Picking a value greater than ICH9_APM_CNT_CPU_HOTPLUG (4) will also
ensure that your root MMI handler filter out the "default" command port
value (namely zero) that OvmfPkg/SmmControl2Dxe uses in the Trigger()
implementation, when SmmCommunicationCommunicate()
[MdeModulePkg/Core/PiSmmCore/PiSmmIpl.c] invokes Trigger() with
(Command==NULL), for submitting a Communicate request.

Thanks
Laszlo


Re: how to fix compile warning: cannot find entry symbol _ModuleEntryPoint

Andrew Fish
 

Wenyi,

You can pass -n 1 to the edk2 build command to force a single threaded build to help debug compile issues.

Thanks,

Andrew Fish

On Sep 30, 2020, at 12:00 AM, xiewenyi (A) <xiewenyi2@huawei.com> wrote:

Thanks, Andrew, Laszlo and liming, with your help I find the module which really own the warning.

As the compile is multithreading, the order in compile log is not right. The module which cause warning is written by ourselves.
In this module's inf file, MODULE_TYPE is set as USER_DEFINED, and UefiDriverEntryPoint is not set in [LibraryClasses].


On 2020/9/30 0:00, Andrew Fish wrote:
Wenyi,

If you look under the Build results <Platform>/<DEBUG/RELEASE>_<COMPILER>/AArch64/Edk2/ArmPkg/Drivers/ArmGic/ (hopefully I got that right) and go down a little bit you will see the build generated GNUmakefile. You might want to take a look at that to try and figure out what is going on?

Thanks,

Andrew Fish


On Sep 29, 2020, at 7:23 AM, wenyi,xie via groups.io <http://groups.io> <xiewenyi2=huawei.com@groups.io <mailto:xiewenyi2=huawei.com@groups.io>> wrote:

I checked my platform DSC and there's UefiDriverEntryPoint|MdePkg/Library/UefiDriverEntryPoint/UefiDriverEntryPoint.inf in the section [LibraryClasses.common].
I'm not sure why _ModuleEntryPoint can't be found when link the object.

Regards
Wenyi

On 2020/9/29 2:21, Laszlo Ersek wrote:
On 09/28/20 10:57, wenyi,xie via groups.io <http://groups.io> wrote:
Hi, all
I use edk2-stable201903 in my project and meet a compile warning like below,

/opt/buildtools/gcc-linaro-7.4.1-2019.02/aarch64-linux-gnu/bin/../lib/gcc/aarch64-linux-gnu/7.4.1/../../../../aarch64-linux-gnu/bin/ld: warning: cannot find entry symbol _ModuleEntryPoint; not setting start address

It seems like the warning happens when building EDKII/Edk2/ArmPkg/Drivers/ArmGic/ArmGicDxe.inf
does anyone know how to fix it?
If need more info, please contact me, thanks!

here's some of my build option,
Architecture(s) = AARCH64
Build target = RELEASE
Toolchain = GCC5
The UefiDriverEntryPoint lib class in the [Sources] section in
"ArmPkg/Drivers/ArmGic/ArmGicDxe.inf" should be resolved by your
platform DSC to
"MdePkg/Library/UefiDriverEntryPoint/UefiDriverEntryPoint.inf".

That's where the definition of _ModuleEntryPoint() should come from (see
"MdePkg/Library/UefiDriverEntryPoint/DriverEntryPoint.c").

Thanks
Laszlo

481 - 500 of 889