-------- Original Message --------
From: Albecki, Mateusz [mailto:mateusz.albecki@...]
Sent: Wednesday, November 10, 2021, 6:54 AM
To: Abner Chang <abner.chang@...>, rfc@edk2.groups.io
Subject: [edk2-rfc] Proposal to add support for PCIe enumeration protocols in PEI

Hi Abner,

The way I see it is that platform should be in control of starting the enumeration and which devices it wants enumerated(by enumerated I mean resources assigned and PPI installed). Right now our idea is to simply have our own PEIM implementation that will hold Intel specific logic inside of it and in the next step propose a common enumeration PEIM to EDK2 with several hooks that will allow platform configurability.

As for second part of your mail I see 2 points to unpack. First is the optimization that would eliminate the need for 2 resource assignment passes during boot(PEI + DXE). It should be possible to install HOB in enumeration PEIM that would later on be consumed by DXE PCI bus driver which would inform it that resources are assigned and it should only scan the hierarchy and install required protocols which would save us some boot time. The problem with this solution is that while it probably could work quite well on system on which both PEI and DXE are either 32-bit or 64-bit it is hard to achieve on what I think is right now the most popular configuration which is 32-bit PEI and 64-bit DXE. Due to limited system addresses in 32-bit PEI enumerator PEIM might be forced to make suboptimal decisions in resource assignment or might even decide to skip some devices altogether if it starts running out of memory.

Second point is about PCI device information that, if I understood you correctly, would contain PCI device information that we can get from PCI config of a device organized in a table and probably installed into a HOB. I can see some value in such table as it could be used by DXE code that runs before DXE enumeration however I don't really see why DXE code would prefer to use it over locating protocols. Note that protocols are required to be installed anyway to make DXE device drivers work so I don't really see how we could skip them.


Thanks,
Mateusz







--

Brian

--------------------------------------------------------------------

"As an adolescent I aspired to lasting fame, I craved factual
certainty, and I thirsted for a meaningful vision of human life - so
I became a scientist. This is like becoming an archbishop so you
can meet girls."
-- M. Cartmill, sociologist

-----Original Message-----
From: rfc@edk2.groups.io [mailto:rfc@edk2.groups.io] On Behalf Of Ni, Ray
Sent: Thursday, November 11, 2021 9:00 AM
To: rfc@edk2.groups.io; Johnson, Brian (EXL - Bloomington, MN)
<brian.johnson@...>; Albecki, Mateusz <mateusz.albecki@...>;
Chang, Abner (HPS SW/FW Technologist) <abner.chang@...>
Subject: Re: [edk2-rfc] Proposal to add support for PCIe enumeration
protocols in PEI

The CAR size concern caused two additional design requirements:
1. separate FV being loaded after mem is ready
2. pre-scan in pre-mem and full-scan in post-mem

So, I am wondering if pre-mem scan is really needed?
if not, let's just say: one full-scan in post-mem

-----Original Message-----
From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf Of Brian J.

Johnson

Sent: Thursday, November 11, 2021 7:22 AM
To: rfc@edk2.groups.io; Albecki, Mateusz <mateusz.albecki@...>;

Chang, Abner <abner.chang@...>

Subject: Re: [edk2-rfc] Proposal to add support for PCIe enumeration

protocols in PEI

Mateusz,

Unfortunately, the continued migration of DXE code into PEI
(specifically, into PEI code running before RAM is available) is putting
a huge crunch on BIOS size. Server processors have a limited amount of
cache-as-RAM (CAR) space available, and extremely complex code for
initializing memory and the system fabric. Space is at a premium today,
and adding PCIe initialization to CAR will make the problem worse.

Servers also tend to have very large, complex PCIe hierarchies, most of
which isn't needed for S3 or the other flows you mentioned. (Not that
servers do much with S3 anyway....) Spending CAR space on it doesn't
seem like a good tradeoff.

Agree that would be tough for server platform having PCI enum in CAR phase.

Can PCIe enumeration and the things that depend on it (S3, capsules,
recovery, etc.) isolate their code to a compressed FD placed outside of
the CAR region, and not run until RAM is available?

Or can you think of another way to reduce the amount of code and data in
CAR?

Thanks,

Brian J. Johnson
HPE

-------- Original Message --------
From: Albecki, Mateusz [mailto:mateusz.albecki@...]
Sent: Wednesday, November 10, 2021, 6:54 AM
To: Abner Chang <abner.chang@...>, rfc@edk2.groups.io
Subject: [edk2-rfc] Proposal to add support for PCIe enumeration
protocols in PEI

Hi Abner,

The way I see it is that platform should be in control of starting the
enumeration and which devices it wants enumerated(by enumerated I

mean

resources assigned and PPI installed). Right now our idea is to simply
have our own PEIM implementation that will hold Intel specific logic
inside of it and in the next step propose a common enumeration PEIM to
EDK2 with several hooks that will allow platform configurability.

As for second part of your mail I see 2 points to unpack. First is the
optimization that would eliminate the need for 2 resource assignment
passes during boot(PEI + DXE). It should be possible to install HOB in
enumeration PEIM that would later on be consumed by DXE PCI bus driver
which would inform it that resources are assigned and it should only
scan the hierarchy and install required protocols which would save us
some boot time. The problem with this solution is that while it probably
could work quite well on system on which both PEI and DXE are either
32-bit or 64-bit it is hard to achieve on what I think is right now the
most popular configuration which is 32-bit PEI and 64-bit DXE. Due to
limited system addresses in 32-bit PEI enumerator PEIM might be forced
to make suboptimal decisions in resource assignment or might even decide
to skip some devices altogether if it starts running out of memory.

I think this could be fixed if we can have the record in HOB for the 32-bit/64-bit mixed boot phases.
DXE PCI enumeration can reassign the resource for 64-bit boot phase if necessary.

Second point is about PCI device information that, if I understood you
correctly, would contain PCI device information that we can get from PCI
config of a device organized in a table and probably installed into a
HOB. I can see some value in such table as it could be used by DXE code
that runs before DXE enumeration however I don't really see why DXE

code

would prefer to use it over locating protocols. Note that protocols are
required to be installed anyway to make DXE device drivers work so I
don't really see how we could skip them.

I know this subject could be apart from your proposal. I add this suggestion to your proposal because I notice that we may need to carry some PCI information which discovered during PEI PCI enum to DXE phase in order to reduce the duplicate effort of PCI enum in DXE. We still can have that PCI device information even the PEI PCI enum is not enabled on platform. Some information is not able to be retrieved from the existing EFI protocol defined in UEFI spec, such as the PCI port number, slot number, embedded or slot, bifurcation, topology, device type and etc. We can either enhance the EFI device path to accommodate this information or install the additional protocol that points to the information on the PCI EFI handle. The use case could be in either DXE phase (with PEI PCI enum enabled) or in BDS phase.

Thanks
Abner

Thanks,
Mateusz







--

Brian

--------------------------------------------------------------------

"As an adolescent I aspired to lasting fame, I craved factual
certainty, and I thirsted for a meaningful vision of human life - so
I became a scientist. This is like becoming an archbishop so you
can meet girls."
-- M. Cartmill, sociologist

-------- Original Message --------
From: Albecki, Mateusz [mailto:mateusz.albecki@...]
Sent: Friday, November 12, 2021, 10:03 AM
To: Abner Chang <abner.chang@...>, rfc@edk2.groups.io
Subject: [edk2-rfc] Proposal to add support for PCIe enumeration protocols in PEI

Brian & Ray,

In general I think we can categorize platforms into 4 types for purpose of this discussion

1. Platform that doesn't care about PEI enumeration due to lack of use cases
2. Platform that needs devices enumerated in post mem(for instance desktops that want to support S3 opal unlock + capsule update)
3. Platform that needs devices enumerated in pre-mem and post-mem and can accommodate enumerator PEIM in pre-mem FV(for instance tablets or laptops that boot fw from block storage and need to support S3 opal unlock)
4. Platform that needs devices enumerated in pre-mem and post-mem and can't accommodate enumerator PEIM in pre-mem FV(same as 3 but smaller CAR)

For servers I think that majority of those platforms fall into 1 and 2. For 1 platform just doesn't include enumerator and device driver PEIMs. For 2 platform includes those PEIMs into post mem FVs. My initial description focused on 3 since that is the most complicated one from the enumerator PEIM perspective. 4 is something that each platform will have to decide on their own on how they want to support it. One possible solution for such platforms is to prepare a very simple PEIM that would only assign some hardcoded resources from a special address pool to the block IO device and skip over everything else. Later on platform would tell the enumerator PEIM to skip over such controller during full enumeration(this can be accomplished via platform hooks).

So to answer the question - "Can PCIe enumeration and the things that depend on it (S3, capsules, recovery, etc.) isolate their code to a compressed FD placed outside of the CAR region, and not run until RAM is available?"

Yes, on systems which do no require pre-mem enumeration this should be possible.


Abner,

Yes it should be possible to record in HOB that enumeration was done in 32-bit environment and should be redone in 64-bit DXE. I was simply rising this point to note that, right now at least, majority of the systems won't be improved by it unfortunately.

As for the additional info from PEI to DXE - I think we should maybe have a separate discussion to see what would be needed and why and how would PEI produce it. At its base enumerator PEIM won't have access to any special information about PCIe devices so I am not sure if we can demand that it produces it. Going over the list you have prepared I see that:

PCI port number, slot number - I think those are the same and can be retrieved from PCI Express Capabilities field Physical Slot Number
Embedded or Slot - Can be retrieved from Slot Implemented field in PCI Express Capabilities
Bifurcation - this should correspond to Maximum Link Width field in Link Capabilities
topology - I am not sure what you are referring to here. If this is just about PCIe topology(i.e. how many switches and other devices are there between root bus and endpoint) I think it is at least partially covered by device path
device type - can be retrieved from Device/Port Type field

So far it seems like all of the information can be retrieved from HW itself and device path protocol. Are you proposing to add an abstraction that would shield the user from the necessity to access HW to get this information?

Thanks,
Mateusz







--
Brian J. Johnson
Enterprise X86 Lab

Hewlett Packard Enterprise

brian.johnson@...
hpe.com

-------- Original Message --------
From: Albecki, Mateusz [mailto:mateusz.albecki@...]
Sent: Tuesday, November 16, 2021, 11:35 AM
To: Brian J. Johnson <brian.johnson@...>, rfc@edk2.groups.io
Subject: [edk2-rfc] Proposal to add support for PCIe enumeration protocols in PEI

Brian,

I fully agree that the PEI enumeration should be a self-contained(as in there shouldn't be a hard dependency between stuff produced by this stack, soft dependency should be allowed I think), optional feature. I am also not sure what is the best way to achieve it but I am leaning towards using feature PCDs. Given that device drivers PEIMs are available in EDK2 repo I think that both new PCI_DEVICE_PPI and the implementation of the enumerator PEIM should be in EDK2 repo as well. BTW did I understand correctly that you wouldn't like to see the optimization for platform types 2,3 and 4 which would skip the resources assignment in DXE when possible? While I also wouldn't like to introduce such dependency I am not sure I can guarantee that either myself or somebody else won't eventually propose such an optimization as the performance gain might be substantial on desktop systems(especially if it is running on 64-bit PEI). At the very least I understand that such dependency should be handled in a way that wouldn't break the DXE driver if enumerator PEIM is not present.

BTW, I agree that we shouldn't be doing enumeration in PEI. This proposition is just a band-aid that needs to be applied to fix several architectural mistakes that we have done in the past(S3 resume definition mostly...). From my point of view ideal solution for a large chunk of our pain points is to simply remove PEI and re-architect DXE. Having uniform execution environment across all Sx flows would help a lot in my opinion. Sadly such change is probably not possible in short term and even in long term I wouldn't know how to go about this.

Thanks,
Mateusz







--
Brian J. Johnson
Enterprise X86 Lab

Hewlett Packard Enterprise

brian.johnson@...
hpe.com

-----Original Message-----
From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf Of Brian J. Johnson
Sent: Tuesday, November 16, 2021 1:48 PM
To: rfc@edk2.groups.io; Albecki, Mateusz <mateusz.albecki@...>
Subject: Re: [edk2-rfc] Proposal to add support for PCIe enumeration protocols in PEI

Mateusz,

Your changes provide a nice cleanup to the driver mess in PEI. My main concern is that someone will later add a dependency which *requires* PCIe enumeration in PEI, without realizing that this is impossible on some platforms. The only ways to prevent that which I can think of are to document it clearly, in the code, and to choose not to introduce such dependencies now.

That's why I was suggesting not trying to optimize DXE resource assignment at this time. That may not be a reasonable request... code size and execution speed tend to trump other concerns in firmware.
I'd rather see us put our effort into removing things from PEI than adding things to it. But I'm not the only consumer of TianoCore....

I'm relieved you think that there have been some architectural mistakes made, particularly around S3 resume. I wanted to say so, but I'm glad you did so first. :) Putting device drivers in PEI at all seems to stray from the original design. But I don't know how to fix it either.
And I don't even work on platforms which use S3, so I know nothing about the requirements which led to the current design.

Maybe the stewards will have a suggestion. But as you said, this seems like it could easily become a very, very large effort.

Thanks,
Brian J. Johnson
HPE

-------- Original Message --------
From: Albecki, Mateusz [mailto:mateusz.albecki@...]
Sent: Tuesday, November 16, 2021, 11:35 AM
To: Brian J. Johnson <brian.johnson@...>, rfc@edk2.groups.io
Subject: [edk2-rfc] Proposal to add support for PCIe enumeration protocols in PEI

Brian,

I fully agree that the PEI enumeration should be a self-contained(as in there shouldn't be a hard dependency between stuff produced by this stack, soft dependency should be allowed I think), optional feature. I am also not sure what is the best way to achieve it but I am leaning towards using feature PCDs. Given that device drivers PEIMs are available in EDK2 repo I think that both new PCI_DEVICE_PPI and the implementation of the enumerator PEIM should be in EDK2 repo as well.
BTW did I understand correctly that you wouldn't like to see the optimization for platform types 2,3 and 4 which would skip the resources assignment in DXE when possible? While I also wouldn't like to introduce such dependency I am not sure I can guarantee that either myself or somebody else won't eventually propose such an optimization as the performance gain might be substantial on desktop systems(especially if it is running on 64-bit PEI). At the very least I understand that such dependency should be handled in a way that wouldn't break the DXE driver if enumerator PEIM is not present.

BTW, I agree that we shouldn't be doing enumeration in PEI. This proposition is just a band-aid that needs to be applied to fix several architectural mistakes that we have done in the past(S3 resume definition mostly...). From my point of view ideal solution for a large chunk of our pain points is to simply remove PEI and re-architect DXE.
Having uniform execution environment across all Sx flows would help a lot in my opinion. Sadly such change is probably not possible in short term and even in long term I wouldn't know how to go about this.

Thanks,
Mateusz







--
Brian J. Johnson
Enterprise X86 Lab

Hewlett Packard Enterprise

brian.johnson@...
hpe.com

-------- Original Message --------
From: Desimone, Nathaniel L [mailto:nathaniel.l.desimone@...]
Sent: Tuesday, November 30, 2021, 4:53 PM
To: rfc@edk2.groups.io <rfc@edk2.groups.io>, Johnson, Brian <brian.johnson@...>, Albecki, Mateusz <mateusz.albecki@...>
Cc: Ni, Ray <ray.ni@...>
Subject: [edk2-rfc] Proposal to add support for PCIe enumeration protocols in PEI

Hi All,

I would like to add a few points here.

First of all, allocating fixed BAR resources from a special address pool is not really scalable for any devices outside of Bus 0. Any device that is behind a PCI/PCI bridge needs to have its MMIO BARs fall within a single continuous region of physical address space that contains all MMIO BARs for all devices attached downstream from that bridge. If there is a chain of PCI/PCI bridges between the device and the root complex, then any PCI/PCI bridges that are not at the root need to have their MMIO base/limits programmed to a subset of the parent bridge's MMIO base/limit. Because of that, it may not be possible to set the MMIO base/limit for the PCI/PCI bridge to cover the MMIO needed by both the device with the special fixed BAR address and all of the other regular devices which do not have restrictions on where their BARs end up. For that reason, assigning a pre-determined fixed BAR to a PCI device really only makes sense for devices that happen to be on Bus 0. In a modern PCIe system the only devices that could ever be on Bus 0 are devices that are embedded into the chipset.

For this reason, any partial enumeration like what Mateusz is suggesting for special cases like platforms that store their UEFI firmware on block I/O devices would end up allocating BAR assignments that would need to be considered temporary. Certain block I/O devices like NVMe will always be devices behind a PCI/PCI bridge, so this issue is highly relevant to pre-memory PCI enumeration. Only after a full enumeration of the entire PCI topology is done would you be able to assume BAR assignments won't need to be reallocated. In general, a partial PCI enumeration has the potential to become very wasteful of MMIO address space if that address space is not reclaimed and reused during the full PCI enumeration. Additionally, we should aim to only do a single partial enumeration per boot. If more than one partial enumeration is performed (each time for separate devices), then it is possible for them to come up with conflicting BAR assignments.

Ray and I had a discussion on supporting BARs allocated from special MMIO ranges reserved by platform code on Bugzilla last year (https://bugzilla.tianocore.org/show_bug.cgi?id=2958 ) and we collectively came to the conclusion that a feature like this would be more trouble than it is worth. When you consider the continuing increase in SoC complexity, it is unlikely that we will be able to fit 100% of embedded devices in Bus 0 for very much longer. On server systems, we crossed that point back in Sandy Bridge 10 years ago. In the end, a feature like that will continue to become less useful over time and it is already close to the point of irrelevancy. Besides, a lot of the use cases for hardcoded BAR values would go away if we did a full PCI enumeration in PEI anyway. For example, SMM using a different BAR value for the SPI controller or the HECI controller than what was assigned by PCI enumeration would no longer be a problem if enumeration is done early.

Second, I would like to point out that the situation on server systems is a little more complicated than "the platform doesn't care about PCI enumeration during PEI". On a multi-socket server system, the UPI controllers need to know which PCI segment+bus numbers to route to which socket. Moreover, the MMIO ranges containing any BARs for PCIe devices that are connected to PCIe root ports that are off-socket need to also be routed through the UPI controller. So while you are right that it is unlikely that we will need to access PCIe devices outside the SoC during PEI on server systems... being able to count up those devices and enumerate the system topology is very useful as we would then be able to efficiently program which MMIO and PCI bus ranges to route to each UPI controller. Therefore I would like to propose that this PEI PCI enumeration feature have THREE modes:

MODE 1: Find all PCI devices, count the number of PCI bus numbers that would be needed make all PCI devices visible but do not assign PCI bus numbers. Count the amount of MMIO space each device needs but DO NOT assign actual BAR values. Detect and report any VGA devices found in the PCI topology.
MODE 2: Find all PCI devices, assign all PCI bus numbers (if > 256 PCI bus numbers is needed to expose the full system topology, assign PCI segment numbers as well), count the amount of MMIO space each device needs but only assign actual BAR values to a SUBSET of PCI devices. Detect and report any VGA devices found in the PCI topology.
MODE 3: Perform a full PCI enumeration, assign all PCI bus numbers (if > 256 PCI bus numbers is needed to expose the full system topology, assign PCI segment numbers as well), assign all BAR values. Detect and report any VGA devices found in the PCI topology.

MODE 1 is somewhat useful on client systems as well. MODE 1 could find all the VGA devices in the system which helps us decide which device to route the legacy VGA framebuffer BAR to (hardcoded to address 0xA0000 for legacy reasons.) In addition, if user wishes to boot a 32-bit OS, knowing the full amount of MMIO space required for PCI devices enables us to configure the system memory map such that the MMIO region below 4GB is large enough to contain all BARs, while still providing as much address space for RAM as possible. A feature like this is not as useful as it was in the past, since very few systems support 32-bit OSes anymore, and having <=3GB of MMIO BARs is less common now. But even on 64-bit systems there is a niche use for this feature as some older PCI devices have BARs that only support 32-bit addressing.

MODE 2 is useful for the block I/O use cases Mateusz mentions. MODE 3 makes a lot of sense for any systems that use 64-bit PEI, which I suspect we will start to see more of in the future. I think we should define a new PPI that allows platform code to request PCI enumeration. Whether to use MODE1/2/3 should be controllable via this PPI. In addition, for MODE 2 & 3 we would need the equivalent of the PCI host bridge driver to allow platform code to have a say in how PCI bus numbers and BARs are allocated.

Sorry for the long email but hopefully it helps!

Thanks,
Nate

-----Original Message-----
From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf Of Brian J. Johnson
Sent: Tuesday, November 16, 2021 1:48 PM
To: rfc@edk2.groups.io; Albecki, Mateusz <mateusz.albecki@...>
Subject: Re: [edk2-rfc] Proposal to add support for PCIe enumeration protocols in PEI

Mateusz,

Your changes provide a nice cleanup to the driver mess in PEI. My main concern is that someone will later add a dependency which *requires* PCIe enumeration in PEI, without realizing that this is impossible on some platforms. The only ways to prevent that which I can think of are to document it clearly, in the code, and to choose not to introduce such dependencies now.

That's why I was suggesting not trying to optimize DXE resource assignment at this time. That may not be a reasonable request... code size and execution speed tend to trump other concerns in firmware.
I'd rather see us put our effort into removing things from PEI than adding things to it. But I'm not the only consumer of TianoCore....

I'm relieved you think that there have been some architectural mistakes made, particularly around S3 resume. I wanted to say so, but I'm glad you did so first. :) Putting device drivers in PEI at all seems to stray from the original design. But I don't know how to fix it either.
And I don't even work on platforms which use S3, so I know nothing about the requirements which led to the current design.

Maybe the stewards will have a suggestion. But as you said, this seems like it could easily become a very, very large effort.

Thanks,
Brian J. Johnson
HPE

-------- Original Message --------
From: Albecki, Mateusz [mailto:mateusz.albecki@...]
Sent: Tuesday, November 16, 2021, 11:35 AM
To: Brian J. Johnson <brian.johnson@...>, rfc@edk2.groups.io
Subject: [edk2-rfc] Proposal to add support for PCIe enumeration protocols in PEI

Brian,

I fully agree that the PEI enumeration should be a self-contained(as in there shouldn't be a hard dependency between stuff produced by this stack, soft dependency should be allowed I think), optional feature. I am also not sure what is the best way to achieve it but I am leaning towards using feature PCDs. Given that device drivers PEIMs are available in EDK2 repo I think that both new PCI_DEVICE_PPI and the implementation of the enumerator PEIM should be in EDK2 repo as well.
BTW did I understand correctly that you wouldn't like to see the optimization for platform types 2,3 and 4 which would skip the resources assignment in DXE when possible? While I also wouldn't like to introduce such dependency I am not sure I can guarantee that either myself or somebody else won't eventually propose such an optimization as the performance gain might be substantial on desktop systems(especially if it is running on 64-bit PEI). At the very least I understand that such dependency should be handled in a way that wouldn't break the DXE driver if enumerator PEIM is not present.

BTW, I agree that we shouldn't be doing enumeration in PEI. This proposition is just a band-aid that needs to be applied to fix several architectural mistakes that we have done in the past(S3 resume definition mostly...). From my point of view ideal solution for a large chunk of our pain points is to simply remove PEI and re-architect DXE.
Having uniform execution environment across all Sx flows would help a lot in my opinion. Sadly such change is probably not possible in short term and even in long term I wouldn't know how to go about this.

Thanks,
Mateusz







--
Brian J. Johnson
Enterprise X86 Lab

Hewlett Packard Enterprise

brian.johnson@...
hpe.com








--

Brian

--------------------------------------------------------------------

Linux is sort of like the Three Stooges. There are people who "get
it" and think it's the greatest thing ever, and then there are
people who don't get it
-- Andy Ihnatko

-----Original Message-----
From: Brian J. Johnson <brian.johnson@...>
Sent: Wednesday, December 1, 2021 4:07 PM
To: Desimone, Nathaniel L <nathaniel.l.desimone@...>; rfc@edk2.groups.io; Albecki, Mateusz <mateusz.albecki@...>
Cc: Ni, Ray <ray.ni@...>
Subject: Re: [edk2-rfc] Proposal to add support for PCIe enumeration protocols in PEI

Nate,

I agree that MODE 3 mainly makes sense for systems with 64-bit PEI, since only a limited number of PCI segments and MMIO space can fit below the 4GB boundary. On most of our large systems, only the legacy socket and its devices fit there.

Plus, not all PCI root ports are visible in PEI on our systems, especially before memory initialization. Some can only be seen after all of the system interconnect is initialized, which requires memory and 64-bit addressing.

So whatever is done, we should recognize that some systems will need to (re)do PCI enumeration and BAR assignment in DXE, and will treat all early PCIe init as temporary.

Thanks,
Brian J. Johnson
HPE

-------- Original Message --------
From: Desimone, Nathaniel L [mailto:nathaniel.l.desimone@...]
Sent: Tuesday, November 30, 2021, 4:53 PM
To: rfc@edk2.groups.io <rfc@edk2.groups.io>, Johnson, Brian <brian.johnson@...>, Albecki, Mateusz <mateusz.albecki@...>
Cc: Ni, Ray <ray.ni@...>
Subject: [edk2-rfc] Proposal to add support for PCIe enumeration protocols in PEI

Hi All,

I would like to add a few points here.

First of all, allocating fixed BAR resources from a special address pool is not really scalable for any devices outside of Bus 0. Any device that is behind a PCI/PCI bridge needs to have its MMIO BARs fall within a single continuous region of physical address space that contains all MMIO BARs for all devices attached downstream from that bridge. If there is a chain of PCI/PCI bridges between the device and the root complex, then any PCI/PCI bridges that are not at the root need to have their MMIO base/limits programmed to a subset of the parent bridge's MMIO base/limit. Because of that, it may not be possible to set the MMIO base/limit for the PCI/PCI bridge to cover the MMIO needed by both the device with the special fixed BAR address and all of the other regular devices which do not have restrictions on where their BARs end up. For that reason, assigning a pre-determined fixed BAR to a PCI device really only makes sense for devices that happen to be on Bus 0. In a modern PCIe system the only devices that could ever be on Bus 0 are devices that are embedded into the chipset.

For this reason, any partial enumeration like what Mateusz is suggesting for special cases like platforms that store their UEFI firmware on block I/O devices would end up allocating BAR assignments that would need to be considered temporary. Certain block I/O devices like NVMe will always be devices behind a PCI/PCI bridge, so this issue is highly relevant to pre-memory PCI enumeration. Only after a full enumeration of the entire PCI topology is done would you be able to assume BAR assignments won't need to be reallocated. In general, a partial PCI enumeration has the potential to become very wasteful of MMIO address space if that address space is not reclaimed and reused during the full PCI enumeration.
Additionally, we should aim to only do a single partial enumeration per boot. If more than one partial enumeration is performed (each time for separate devices), then it is possible for them to come up with conflicting BAR assignments.

Ray and I had a discussion on supporting BARs allocated from special MMIO ranges reserved by platform code on Bugzilla last year
(https://bugzilla.tianocore.org/show_bug.cgi?id=2958
) and we collectively came to the conclusion that a feature like this would be more trouble than it is worth. When you consider the continuing increase in SoC complexity, it is unlikely that we will be able to fit 100% of embedded devices in Bus 0 for very much longer. On server systems, we crossed that point back in Sandy Bridge 10 years ago. In the end, a feature like that will continue to become less useful over time and it is already close to the point of irrelevancy. Besides, a lot of the use cases for hardcoded BAR values would go away if we did a full PCI enumeration in PEI anyway. For example, SMM using a different BAR value for the SPI controller or the HECI controller than what was assigned by PCI enumeration would no longer be a problem if enumeration is done early.

Second, I would like to point out that the situation on server systems is a little more complicated than "the platform doesn't care about PCI enumeration during PEI". On a multi-socket server system, the UPI controllers need to know which PCI segment+bus numbers to route to which socket. Moreover, the MMIO ranges containing any BARs for PCIe devices that are connected to PCIe root ports that are off-socket need to also be routed through the UPI controller. So while you are right that it is unlikely that we will need to access PCIe devices outside the SoC during PEI on server systems... being able to count up those devices and enumerate the system topology is very useful as we would then be able to efficiently program which MMIO and PCI bus ranges to route to each UPI controller. Therefore I would like to propose that this PEI PCI enumeration feature have THREE modes:

MODE 1: Find all PCI devices, count the number of PCI bus numbers that would be needed make all PCI devices visible but do not assign PCI bus numbers. Count the amount of MMIO space each device needs but DO NOT assign actual BAR values. Detect and report any VGA devices found in the PCI topology.
MODE 2: Find all PCI devices, assign all PCI bus numbers (if > 256 PCI bus numbers is needed to expose the full system topology, assign PCI segment numbers as well), count the amount of MMIO space each device needs but only assign actual BAR values to a SUBSET of PCI devices.
Detect and report any VGA devices found in the PCI topology.
MODE 3: Perform a full PCI enumeration, assign all PCI bus numbers (if >
256 PCI bus numbers is needed to expose the full system topology, assign PCI segment numbers as well), assign all BAR values. Detect and report any VGA devices found in the PCI topology.

MODE 1 is somewhat useful on client systems as well. MODE 1 could find all the VGA devices in the system which helps us decide which device to route the legacy VGA framebuffer BAR to (hardcoded to address 0xA0000 for legacy reasons.) In addition, if user wishes to boot a 32-bit OS, knowing the full amount of MMIO space required for PCI devices enables us to configure the system memory map such that the MMIO region below 4GB is large enough to contain all BARs, while still providing as much address space for RAM as possible. A feature like this is not as useful as it was in the past, since very few systems support 32-bit OSes anymore, and having <=3GB of MMIO BARs is less common now. But even on 64-bit systems there is a niche use for this feature as some older PCI devices have BARs that only support 32-bit addressing.

MODE 2 is useful for the block I/O use cases Mateusz mentions. MODE 3 makes a lot of sense for any systems that use 64-bit PEI, which I suspect we will start to see more of in the future. I think we should define a new PPI that allows platform code to request PCI enumeration.
Whether to use MODE1/2/3 should be controllable via this PPI. In addition, for MODE 2 & 3 we would need the equivalent of the PCI host bridge driver to allow platform code to have a say in how PCI bus numbers and BARs are allocated.

Sorry for the long email but hopefully it helps!

Thanks,
Nate

-----Original Message-----
From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf Of Brian J. Johnson
Sent: Tuesday, November 16, 2021 1:48 PM
To: rfc@edk2.groups.io; Albecki, Mateusz <mateusz.albecki@...>
Subject: Re: [edk2-rfc] Proposal to add support for PCIe enumeration protocols in PEI

Mateusz,

Your changes provide a nice cleanup to the driver mess in PEI. My main concern is that someone will later add a dependency which *requires* PCIe enumeration in PEI, without realizing that this is impossible on some platforms. The only ways to prevent that which I can think of are to document it clearly, in the code, and to choose not to introduce such dependencies now.

That's why I was suggesting not trying to optimize DXE resource assignment at this time. That may not be a reasonable request... code size and execution speed tend to trump other concerns in firmware.
I'd rather see us put our effort into removing things from PEI than adding things to it. But I'm not the only consumer of TianoCore....

I'm relieved you think that there have been some architectural mistakes made, particularly around S3 resume. I wanted to say so, but I'm glad you did so first. :) Putting device drivers in PEI at all seems to stray from the original design. But I don't know how to fix it either.
And I don't even work on platforms which use S3, so I know nothing about the requirements which led to the current design.

Maybe the stewards will have a suggestion. But as you said, this seems like it could easily become a very, very large effort.

Thanks,
Brian J. Johnson
HPE

-------- Original Message --------
From: Albecki, Mateusz [mailto:mateusz.albecki@...]
Sent: Tuesday, November 16, 2021, 11:35 AM
To: Brian J. Johnson <brian.johnson@...>, rfc@edk2.groups.io
Subject: [edk2-rfc] Proposal to add support for PCIe enumeration protocols in PEI

Brian,

I fully agree that the PEI enumeration should be a self-contained(as in there shouldn't be a hard dependency between stuff produced by this stack, soft dependency should be allowed I think), optional feature. I am also not sure what is the best way to achieve it but I am leaning towards using feature PCDs. Given that device drivers PEIMs are available in EDK2 repo I think that both new PCI_DEVICE_PPI and the implementation of the enumerator PEIM should be in EDK2 repo as well.
BTW did I understand correctly that you wouldn't like to see the optimization for platform types 2,3 and 4 which would skip the resources assignment in DXE when possible? While I also wouldn't like to introduce such dependency I am not sure I can guarantee that either myself or somebody else won't eventually propose such an optimization as the performance gain might be substantial on desktop systems(especially if it is running on 64-bit PEI). At the very least I understand that such dependency should be handled in a way that wouldn't break the DXE driver if enumerator PEIM is not present.

BTW, I agree that we shouldn't be doing enumeration in PEI. This proposition is just a band-aid that needs to be applied to fix several architectural mistakes that we have done in the past(S3 resume definition mostly...). From my point of view ideal solution for a large chunk of our pain points is to simply remove PEI and re-architect DXE.
Having uniform execution environment across all Sx flows would help a lot in my opinion. Sadly such change is probably not possible in short term and even in long term I wouldn't know how to go about this.

Thanks,
Mateusz







--
Brian J. Johnson
Enterprise X86 Lab

Hewlett Packard Enterprise

brian.johnson@...
hpe.com








--

Brian

--------------------------------------------------------------------

Linux is sort of like the Three Stooges. There are people who "get
it" and think it's the greatest thing ever, and then there are
people who don't get it
-- Andy Ihnatko

-----Original Message-----
From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf Of Albecki, Mateusz
Sent: Monday, December 6, 2021 8:11 AM
To: Johnson, Brian <brian.johnson@...>; rfc@edk2.groups.io
Subject: Re: [edk2-rfc] Proposal to add support for PCIe enumeration protocols in PEI

Hi Nate,

I see in your mail that you are referring to segment assignment and that is something that is firmly in the scope of the platform/silicon code. In general my assumption was that when the PEI enumeration runs root bridges are already configured which should take care of segment, bus range, mmio range and io range assignment. PEI enumerator would get the information about root bridges on the system via the PciHostBridgeLib the same way DXE does.

With that assumption in mind MODE3 and MODE2 are fairly straight forward to support since MODE3 is the same mode as the one we have in DXE while MODE2 is a reduced MODE3 which we can support by giving a platform control over which devices are needed via PPI. MODE1 is what is tricky as root bridges need to be in this half-configured state in which segments and bus numbers ranges are configured while MMIO and IO is not(or at least is not in its final state). This weird root bridge state aside the functionality itself would be useful even for client systems as you have mentioned so maybe we just need to accept that we need a code which works under such conditions. I think following algorithm for MODE1 would work:
1. Platform code assigns a wide range of bus resources to the root bridge to make sure enumerator will be able to access all PCI devices under that bridge. No memory or IO resources assigned.
2. Platform requests "enumeration"(or maybe resource discovery would be a better name?) from the enumerator PEIM via PPI or maybe via event(which I guess is also a PPI).
3. Enumerator PEIM scans the root bridge and reports the actual number of buses, mmio and IO needed by the root bridge to fully enumerate the bridge. We probably need a separate reporting for video bars as you mentioned.
4. Repeat for each root bridge once every root bridge is scanned balance the resources in the platform code.
5. Finally start the enumeration at whatever time suites the platform.

In the larger picture MODE1 resource scan would need to be executed before RAM is initialized if we want to use it for optimizing MEM32 usage as such:
Client systems that only need opal unlock:
Resource discovery(MODE1) -> Resource balancing -> RAM configuration -> Enumeration(resource assignment and PPI installation) Client systems with BIOS image on block io:
Resource discovery -> Resource balancing -> Resource assignment(full) -> Install PPI for bootable device -> RAM configuration -> Install PPI for remaining required devices Server systems:
Resource discovery -> Resource balancing -> RAM configuration -> Optional resource assignment and enumeration

Regarding the partial enumeration - partial resource assignment is probably not needed if we can do resource discovery and assignment on systems which boot from block io before we assign any resources to such device(which sounds strange but you can use one of the other agents running on your platform as a proxy). If we can't the only solution for such systems is special pools I am afraid. I think we only need to support several passes of the enumerator to install new PPIs as platform use cases demand. For instance for client system with boot from block IO we need to have PPI for bootable block io in pre-mem but we can wait with installation of PPIs for remaining storage devices until post-mem. In this model we do single resource assignment and during later passes simply install the PPIs without modifying resource map.

Thanks,
Mateusz

-----Original Message-----
From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf Of Nate DeSimone
Sent: Wednesday, December 22, 2021 4:21 PM
To: rfc@edk2.groups.io; Albecki, Mateusz <mateusz.albecki@...>; Johnson, Brian <brian.johnson@...>
Subject: Re: [edk2-rfc] Proposal to add support for PCIe enumeration protocols in PEI

Hi Mateusz,

I agree that deciding how many PCI segments worth of MMIO to allocate for PCI configuration space and what the BAR(s) are for those PCI segments is a platform decision. Where the PCI root ports end up in a multi-segment system should mostly be determined by PCI enumeration however. I agree that there are some corner cases like Thunderbolt where we will want to dedicate specific PCI segments for certain PCIe root ports, and we will need to enable platform code to specify those details somehow (maybe via PCI Host Bridge PPI?)

Keep in mind the server use case as well. On server there are so many "uncore" (aka System Agent) PCI devices that even just a single CPU north cluster can now easily consume the entirety of the 256 busses in the first PCI segment. Accordingly, a multi-socket system is almost always multi-segment... and the multiple segments are used for regular PCI devices... not special cases like Thunderbolt root ports. In such a system... it is more helpful to think of (Segment 0/Bus 255) as "bus 255" and (Segment 1/Bus 0) as "bus 256"... more or less an extension of the maximum available PCI bus numbers. With one important corner case... the downstream devices attached to a single PCI root port all need to fit within the same PCI segment. So that may force you to place a specific PCI root port into a different segment so there is enough PCI bus numbers to fully enumerate all downstream devices. On the server side almost all devices are fully capable of being mapped to a PCI segments other than Bus 0. For server, the PCH is considered "legacy" and is the only source of PCI devices that must be in Bus 0 on the system.

"Resource Discovery" seems like an good name for MODE1. Actually I don't think MODE1 will require a very large amount of bus resources in most cases. It only needs enough to expose the deepest part of the PCI topology. Once we have reached the leaf nodes of the tree we can reclaim all the bus numbers that where used to traverse down to the leaf(s) by disabling the PCI/PCI bridges and resetting the secondary/subordinate bus numbers back to zero.

I have actually implemented the algorithm for MODE1 a few years ago. It is in the closed source code right now, I'll send you a pointer to it.

For MODE2, yes I agree that it more or less requires "special pools" but my point is that we can reclaim the memory address space from those special pools once the full enumeration is done. And yes, I agree that for client systems it is probably feasible to just go through the full MODE3 in Pre-Memory PEI, which would make MODE2 rather redundant.

Thanks,
Nate

-----Original Message-----
From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf Of Albecki, Mateusz
Sent: Monday, December 6, 2021 8:11 AM
To: Johnson, Brian <brian.johnson@...>; rfc@edk2.groups.io
Subject: Re: [edk2-rfc] Proposal to add support for PCIe enumeration protocols in PEI

Hi Nate,

I see in your mail that you are referring to segment assignment and that is something that is firmly in the scope of the platform/silicon code. In general my assumption was that when the PEI enumeration runs root bridges are already configured which should take care of segment, bus range, mmio range and io range assignment. PEI enumerator would get the information about root bridges on the system via the PciHostBridgeLib the same way DXE does.

With that assumption in mind MODE3 and MODE2 are fairly straight forward to support since MODE3 is the same mode as the one we have in DXE while MODE2 is a reduced MODE3 which we can support by giving a platform control over which devices are needed via PPI. MODE1 is what is tricky as root bridges need to be in this half-configured state in which segments and bus numbers ranges are configured while MMIO and IO is not(or at least is not in its final state). This weird root bridge state aside the functionality itself would be useful even for client systems as you have mentioned so maybe we just need to accept that we need a code which works under such conditions. I think following algorithm for MODE1 would work:
1. Platform code assigns a wide range of bus resources to the root bridge to make sure enumerator will be able to access all PCI devices under that bridge. No memory or IO resources assigned.
2. Platform requests "enumeration"(or maybe resource discovery would be a better name?) from the enumerator PEIM via PPI or maybe via event(which I guess is also a PPI).
3. Enumerator PEIM scans the root bridge and reports the actual number of buses, mmio and IO needed by the root bridge to fully enumerate the bridge. We probably need a separate reporting for video bars as you mentioned.
4. Repeat for each root bridge once every root bridge is scanned balance the resources in the platform code.
5. Finally start the enumeration at whatever time suites the platform.

In the larger picture MODE1 resource scan would need to be executed before RAM is initialized if we want to use it for optimizing MEM32 usage as such:
Client systems that only need opal unlock:
Resource discovery(MODE1) -> Resource balancing -> RAM configuration -> Enumeration(resource assignment and PPI installation) Client systems with BIOS image on block io:
Resource discovery -> Resource balancing -> Resource assignment(full) -> Install PPI for bootable device -> RAM configuration -> Install PPI for remaining required devices Server systems:
Resource discovery -> Resource balancing -> RAM configuration -> Optional resource assignment and enumeration

Regarding the partial enumeration - partial resource assignment is probably not needed if we can do resource discovery and assignment on systems which boot from block io before we assign any resources to such device(which sounds strange but you can use one of the other agents running on your platform as a proxy). If we can't the only solution for such systems is special pools I am afraid. I think we only need to support several passes of the enumerator to install new PPIs as platform use cases demand. For instance for client system with boot from block IO we need to have PPI for bootable block io in pre-mem but we can wait with installation of PPIs for remaining storage devices until post-mem. In this model we do single resource assignment and during later passes simply install the PPIs without modifying resource map.

Thanks,
Mateusz