Runtime configuration methods for OVMF/AAVMF?


Laszlo Ersek
 

On 03/12/21 18:11, Laszlo Ersek wrote:
On 03/11/21 19:13, Aaron Young wrote:
[...]

DebugLogMask 0x8040004F
Nope.

First, there are multiple PCDs related to debug messages.

Second, those PCDs cannot be made dynamic-access; the DEBUG macro is
used in SEC modules too, and SEC does not have dynamic PCD access
(dynamic PCDs are implemented in the PEI phase via the PCD PPI, and in
the DXE phase via the PCD protocol).

Furthermore, if memory serves, the DXE Core can also not use dynamic
PCDs (precisely because the PEI phase is gone, but the DXE protocol is
not available yet), but at the same time, the DXE Core produces quite
relevant log messages.

So only Fixed-At-Build access is appropriate for these PCDs.
("Patchable" would be an option only if we didn't use LZMA compression
inside the firmware image, I think -- in that case, PCDs in the firmware
binary could be tweaked with external tools.)

Third, in a platform DSC file, Fixed-At-Build PCDs can be overridden at
module scope. You can enable special ("exceptional") debug masks on a
module-by-module basis -- and that's something we use heavily.

For example, in my ad-hoc personal OVMF builds, I enable DEBUG_VERBOSE
indiscriminately, at the platform level, but then clear DEBUG_VERBOSE
for NvmExpressDxe, QemuVideoDxe, QemuRamfbDxe, the DXE Core,
PiSmmCpuDxeSmm, IoMmuDxe, AmdSevDxe, VariableSmm, VariableRuntimeDxe.

I use somewhat different module-scope overrides in my ArmVirtQemu
builds. In official RHEL builds, we use yet different module-scope debug
mask overrides.
[...]

For our specific case, our users would like the ability to enable/disable DEBUG mask flag bits dynamically.
Won't work, sorry (see above).

(At least given the current scope of the debug-related PCDs. If you'd
like to propose a new set of PCDs, so that SEC use some Fixed-At-Build
PCDs, but PEI and DXE use a different set of Dynamic PCDs, you can try
that; they'd have to be proposed for MdePkg. Good luck with that.

... At least my understanding is that "access method" is
platform-global, for any given PCD. I've never seen some modules use a
PCD as dynamic, and other modules use the same PCD as fixed. I could be
wrong about that, of course.)

In case of OVMF, you have some dynamic configurability -- if you do not
enable the QEMU debug console (QEMU debug port), then OVMF will throw
away the debug messages, and you won't pay for the IO Port accesses at
least. (They are very expensive under SEV!)

In case of ArmVirtQemu, I can't recommend anything though. In RHEL, we
offer a verbose build (for debugging, with some module-scope overrides)
and a silent build (which only emits DEBUG_ERROR, again with some
module-scope overrides).
One more thought, regarding fw_cfg-controlled DebugLib instances:

- fw_cfg access is unavailable in ArmVirtQemu before the DXE phase,

- on both Ovmf and ArmVirtQemu, the QemuFwCfgLib instances themselves
use DEBUGs, as a part of the fw_cfg interface detection. This would
create a circular dependency.

DebugLib is one of the lowest-level abstractions in edk2; it is supposed
to work when there are basically zero services available. (It also
requires a very primitive output device. ARM's PL011 is serial port is
already a stretch.)

Thanks,
Laszlo


Laszlo Ersek
 

On 03/11/21 19:13, Aaron Young wrote:
Thanks Laszlo! This is very helpful.

From a user perspective, I was thinking perhaps OVMF/AAVMF could support a generic config file (on the host). Linux users are all too familiar with config files. ;-)

This config file could be passed into OVMF/AAVMF via the "-fw_cfg [name=]<item_name>,file=<path>" mechanism, such as: -fw_cfg name=opt/ovmf/ovmf_config,file=/usr/share/ovmf-x64/OVMF_CONFIG.txt

Contents could be simple name/value pairs (with comments) - each of which associates to values in the code. For example:
This is generally not a bad idea; we implemented a very simple version
of it (for setting dynamic boolean PCDs from fw_cfg) in
QemuFwCfgSimpleParserLib. See commit range 64ab457d1f21..cdc3fa54184a,
and TianoCore#2681.

*However*, regarding the specific firmware aspects, generalization is
not possible. Every setting has to be considered *very* carefully,
individually. For two reasons:

(1) Where and how exactly the different aspects of firmware behavior are
controlled varies *wildly*, inside the firmware. You absolutely cannot
start from a high-level (user-oriented) config file aspect, and then try
to push those things down into the firmware, somehow. The reverse
direction has a much better chance at working, one by one.

(2) Every setting you expose like this becomes a user-facing API, and
*even if* you declare a given configuration interface experimental
and/or unstable, users will *inevitably* build their scripts and whatnot
on top, and if they firmware internals change and the interface is
removed, users will show up with pitchforks. This applies especially
when the configurable behaviors do not exist in OvmfPkg platform code,
but in core edk2 code, such as NetworkPkg, MdePkg, MdeModulePkg,
UefiCpuPkg, ShellPkg, and so on.

It is a *hopeless* situation for OvmfPkg, for presenting a stable /
permanent bridge between OVMF users (who couldn't care less about edk2
core internals!) and edk2 core developers (who couldn't care less about
OVMF users). Unless those users repsent themselves *personally* on the
list, it's generally impossible for me to prevent regressions, or to
hold back core changes for the sake of OVMF users.

This is why I'm *extremely* reluctant to condone a large-scale approach
here.

Considering <https://bugzilla.tianocore.org/show_bug.cgi?id=2681>, that
was a lucky case, because the particular NetworkPkg PCDs had existed for
a long time before, and they looked both simple and very stable. That's
why I had the courage to expose them via fw_cfg, even without using an
"X-" prefix (experimental) in the fw_cfg filename.

For aspects that are controlled by standardized (specified) UEFI
variables, the prospects are better. In those cases, the UEFI spec
dictates the compatibility requirements, so exposing them may be safer.

----
#
# Language.
# Possible values:
# Eng: English
# Fr: Fran├žais
#
Language Eng
Maybe.

This could map to the standard "PlatformLang" variable. On the other
hand, upstream edk2 contains no support (messages or glyphs) for
anything other than English, so why bother?


#
# Boot Menu Wait Timeout (specified in seconds).
# Default is 3 seconds. 0 indicates to skip boot menu.
#
BootMenuTimeout 5
Nope. Already exposed via fw_cfg.


#
# Secure Boot Enable/Disable
# Possible Values: Enable, Disable
#
SecureBoot Disable
Changing Secure Boot status is way more involved than you might think.
Please scroll down to the bottom of
"OvmfPkg/EnrollDefaultKeys/EnrollDefaultKeys.c", to see the variables
that control the Secure Boot operational mode.

We could *perhaps* expose the "SecureBootEnable" (edk2-only) variable
over fw_cfg, but:

- it has stability risks: it's edk2-only,

- "SecureBootEnable" is good for temporarily disabling SB if you have
otherwise enabled it (with certificates enrolled etc), but per se it
does not suffice for *enabling* Secure Boot from zero, without further
actions (key enrollments into PK, KEK, db/dbx).

- flipping SB status requires a platform restart.

So, I would say "no" here.


#
# USB Device Enable/Disable
# Possible Values: Enable, Disable
# Default: Enable
#
# USBDevice Enable
Nah. I wouldn't even know where to start looking.

Plus, if you don't want USB devices, cold-unplug them from your domain
definition?


#
# Debug Log Mask.
#
# Possible bit mask values:
# DEBUG_INIT 0x00000001 // Initialization
# DEBUG_WARN 0x00000002 // Warnings
# DEBUG_LOAD 0x00000004 // Load events
# DEBUG_FS 0x00000008 // EFI File system
# DEBUG_POOL 0x00000010 // Alloc & Free (pool)
# DEBUG_PAGE 0x00000020 // Alloc & Free (page)
# DEBUG_INFO 0x00000040 // Informational debug messages
# DEBUG_DISPATCH 0x00000080 // PEI/DXE/SMM Dispatchers
# DEBUG_VARIABLE 0x00000100 // Variable
# DEBUG_BM 0x00000400 // Boot Manager
# DEBUG_BLKIO 0x00001000 // BlkIo Driver
# DEBUG_NET 0x00004000 // SNP Driver
# DEBUG_UNDI 0x00010000 // UNDI Driver
# DEBUG_LOADFILE 0x00020000 // LoadFile
# DEBUG_EVENT 0x00080000 // Event messages
# DEBUG_GCD 0x00100000 // Global Coherency Database changes
# DEBUG_CACHE 0x00200000 // Memory range cachability changes
# DEBUG_VERBOSE 0x00400000 // Detailed debug messages that may
# // significantly impact boot performance
# DEBUG_ERROR 0x80000000 // Error
#
DebugLogMask 0x8040004F
Nope.

First, there are multiple PCDs related to debug messages.

Second, those PCDs cannot be made dynamic-access; the DEBUG macro is
used in SEC modules too, and SEC does not have dynamic PCD access
(dynamic PCDs are implemented in the PEI phase via the PCD PPI, and in
the DXE phase via the PCD protocol).

Furthermore, if memory serves, the DXE Core can also not use dynamic
PCDs (precisely because the PEI phase is gone, but the DXE protocol is
not available yet), but at the same time, the DXE Core produces quite
relevant log messages.

So only Fixed-At-Build access is appropriate for these PCDs.
("Patchable" would be an option only if we didn't use LZMA compression
inside the firmware image, I think -- in that case, PCDs in the firmware
binary could be tweaked with external tools.)

Third, in a platform DSC file, Fixed-At-Build PCDs can be overridden at
module scope. You can enable special ("exceptional") debug masks on a
module-by-module basis -- and that's something we use heavily.

For example, in my ad-hoc personal OVMF builds, I enable DEBUG_VERBOSE
indiscriminately, at the platform level, but then clear DEBUG_VERBOSE
for NvmExpressDxe, QemuVideoDxe, QemuRamfbDxe, the DXE Core,
PiSmmCpuDxeSmm, IoMmuDxe, AmdSevDxe, VariableSmm, VariableRuntimeDxe.

I use somewhat different module-scope overrides in my ArmVirtQemu
builds. In official RHEL builds, we use yet different module-scope debug
mask overrides.


#
# 64-bit MMIO aperture size (specified in decimal in MB). Default is 32768.
#
PciMmio64Mb 65536
Haha, no :)

We absolutely cannot promise anything "stable" about this. It's
currently called "X-PciMmio64Mb" for a reason. The underlying logic is
not set in stone. (Related:
<https://bugzilla.tianocore.org/show_bug.cgi?id=2796>, as you likely now.)


----

Of course, these (above) config options are just a few possible/theoretical examples that a user may want to alter via a config file. Many more could be added with time as needs arise and could go well beyond the current possible current options supported via BootManager/HII/Variables/PCDs.

This config file would allow users to a simple way config OVMF/AAVMF prior to first boot and not require knowledge of EDK2 Variables/PCDs, etc.
Yes, and this config file would also make a promise to users that we
couldn't keep. The whole workflow / ecosystem is unsuitable for making
such a promise.

It's not random that, whenever you run a BIOS update on a physical
machine, your previous settings are (almost always) lost. Even if you
exported them / backed them up to a USB drive previously, there's no
guarantee you can just reimport them.


Perhaps OVMF RPMs could install a "template" config file (similar to the VARS template file) - which has all possible supported options by that version of OVMF/AAVMF commented out(?)

OVMF could parse this file early to populate the values into the code/PCDs/Variables as needed. Each config option would require updates to the code to properly handle the associated config value.

Of course, this brings about the question of whether these value should be transient startup-only values or stored permanently (into the VARS file/variables). Perhaps this could taken on a case-by-case basis what makes best sense for that config option? TBD.
What you describe here is very desirable from a user comfort
perspective, but it's not backed up by OVMF platform structure and the
development method.

With any such config interface, compatibility and deprecation strategy
become paramount. Worse, if you consider package *downgrades* (in case
of a regression, for example), treatment of undefined / invalid options
arises too.

In edk2, there are no people and there are no workflow elements that the
above would *map to*. The libvirt and QEMU projects have compatibility
promises and deprecation strategies, and I can tell you *first hand*
that they eat up immense amounts of resources, *despite* these projects
having complete control over the implementation of the config knobs.
(OVMF does not have "complete" control over core module configuration).

For some high profile use cases, such as CPU hot-(un)plug, we have
successfully proposed new dynamic PCDs (e.g. in UefiCpuPkg). The general
core subsystem maintainer preference remains to *not* introduce further
PCDs -- there are already way too many, and it's already difficult to
avoid regressions for corner cases.


For our specific case, our users would like the ability to enable/disable DEBUG mask flag bits dynamically.
Won't work, sorry (see above).

(At least given the current scope of the debug-related PCDs. If you'd
like to propose a new set of PCDs, so that SEC use some Fixed-At-Build
PCDs, but PEI and DXE use a different set of Dynamic PCDs, you can try
that; they'd have to be proposed for MdePkg. Good luck with that.

... At least my understanding is that "access method" is
platform-global, for any given PCD. I've never seen some modules use a
PCD as dynamic, and other modules use the same PCD as fixed. I could be
wrong about that, of course.)

In case of OVMF, you have some dynamic configurability -- if you do not
enable the QEMU debug console (QEMU debug port), then OVMF will throw
away the debug messages, and you won't pay for the IO Port accesses at
least. (They are very expensive under SEV!)

In case of ArmVirtQemu, I can't recommend anything though. In RHEL, we
offer a verbose build (for debugging, with some module-scope overrides)
and a silent build (which only emits DEBUG_ERROR, again with some
module-scope overrides).


Making an OVMF variant for every possible mask value of course is not tenable.
I agree; it doesn't scale to "every possible mask". It does scale to a
handful. (We've been doing it.)

How does this idea strike you?
Unpleasantly.

Thanks
Laszlo


Aaron Young
 

Thanks Laszlo! This is very helpful.

From a user perspective, I was thinking perhaps OVMF/AAVMF could support a generic config file (on the host). Linux users are all too familiar with config files. ;-)

This config file could be passed into OVMF/AAVMF via the "-fw_cfg [name=]<item_name>,file=<path>" mechanism, such as: -fw_cfg name=opt/ovmf/ovmf_config,file=/usr/share/ovmf-x64/OVMF_CONFIG.txt

Contents could be simple name/value pairs (with comments) - each of which associates to values in the code. For example:

----
#
# Language.
# Possible values:
# Eng: English
# Fr: Fran├žais
#
Language Eng

#
# Boot Menu Wait Timeout (specified in seconds).
# Default is 3 seconds. 0 indicates to skip boot menu.
#
BootMenuTimeout 5

#
# Secure Boot Enable/Disable
# Possible Values: Enable, Disable
#
SecureBoot Disable

#
# USB Device Enable/Disable
# Possible Values: Enable, Disable
# Default: Enable
#
# USBDevice Enable

#
# Debug Log Mask.
#
# Possible bit mask values:
# DEBUG_INIT 0x00000001 // Initialization
# DEBUG_WARN 0x00000002 // Warnings
# DEBUG_LOAD 0x00000004 // Load events
# DEBUG_FS 0x00000008 // EFI File system
# DEBUG_POOL 0x00000010 // Alloc & Free (pool)
# DEBUG_PAGE 0x00000020 // Alloc & Free (page)
# DEBUG_INFO 0x00000040 // Informational debug messages
# DEBUG_DISPATCH 0x00000080 // PEI/DXE/SMM Dispatchers
# DEBUG_VARIABLE 0x00000100 // Variable
# DEBUG_BM 0x00000400 // Boot Manager
# DEBUG_BLKIO 0x00001000 // BlkIo Driver
# DEBUG_NET 0x00004000 // SNP Driver
# DEBUG_UNDI 0x00010000 // UNDI Driver
# DEBUG_LOADFILE 0x00020000 // LoadFile
# DEBUG_EVENT 0x00080000 // Event messages
# DEBUG_GCD 0x00100000 // Global Coherency Database changes
# DEBUG_CACHE 0x00200000 // Memory range cachability changes
# DEBUG_VERBOSE 0x00400000 // Detailed debug messages that may
# // significantly impact boot performance
# DEBUG_ERROR 0x80000000 // Error
#
DebugLogMask 0x8040004F

#
# 64-bit MMIO aperture size (specified in decimal in MB). Default is 32768.
#
PciMmio64Mb 65536
----

Of course, these (above) config options are just a few possible/theoretical examples that a user may want to alter via a config file. Many more could be added with time as needs arise and could go well beyond the current possible current options supported via BootManager/HII/Variables/PCDs.

This config file would allow users to a simple way config OVMF/AAVMF prior to first boot and not require knowledge of EDK2 Variables/PCDs, etc.

Perhaps OVMF RPMs could install a "template" config file (similar to the VARS template file) - which has all possible supported options by that version of OVMF/AAVMF commented out(?)

OVMF could parse this file early to populate the values into the code/PCDs/Variables as needed. Each config option would require updates to the code to properly handle the associated config value.

Of course, this brings about the question of whether these value should be transient startup-only values or stored permanently (into the VARS file/variables). Perhaps this could taken on a case-by-case basis what makes best sense for that config option? TBD.

For our specific case, our users would like the ability to enable/disable DEBUG mask flag bits dynamically. Making an OVMF variant for every possible mask value of course is not tenable.

How does this idea strike you?

Thanks again,

-Aaron


Laszlo Ersek
 

On 03/10/21 16:16, Laszlo Ersek wrote:

One telling example is "efibootmgr". It lets you tweak some stuff (boot
order, boot options, Timeout variable) on physical platforms at least.
However, when using OVMF, those settings are overridden dynamically on
every boot, when using the "bootindex" device properties, and/or the
"-boot menu=on" / "-boot splash-time=N" options.
two other examples (not an exhaustive list by far), for fw_cfg-based
configuration:

- The cipher suites, and the CA certificates, that are to be trusted by
the firmware for UEFI HTTPS Boot (TLS), are propagated from the host
side to the guest via fw_cfg.

The platform-independent TLS machinery in edk2 uses a UEFI variable for
each of these two aspects (two variables in total). And, at least for
the CA certificates, the base assumption is that the variable is
non-volatile, and can be configured through the Setup TUI (adding
trusted CA certs one by one).

However, when using OVMF, both variables are deleted and re-created as
non-volatile, and they are populated from fw_cfg, from the host side.
(See "OvmfPkg/Library/TlsAuthConfigLib".) In turn, on the host side,
p11-kit and QEMU features were implemented, in order to generate /
expose the necessary content in the right format for OVMF.

- whether IPv4 and/or IPv6 is attempted in UEFI PXE boot is also
controlled via fw_cfg.

... Over the years I've observed a strong discrepancy between user
wishes for OVMF configurability. Some users want to control behaviors
from the inside (that is, running from within the guest), exluding even
the OVMF Setup TUI from that -- basically an "efibootmgr-like OS
utility. Other users want to control behaviors from as far outside as
possible (a management application layered even higher than libvirt, for
example). Yet other users feel OVMF should be as close to physical
platforms as possible, and everything that's configurable should be
exposed via the setup TUI (example: the preferred boot-time graphics
resolution; it was really difficult (and over-engineered) to implement
that with HII). Finally, the Confidential (remote-attested, encrypted)
Boot scenario requires OVMF to ignore as many firmware settings as
possible, from both the guest side and the host side. One can only hope
that users realize at some point that it's impossible to satisfy all
these desires at the same time. We've been creating dedicated OVMF
platforms for some of these (e.g. new DSC / FDF files, and a new
PlatformBootManagerLib instance, for the attested boot scenario), but it
remains a struggle.

Thanks
Laszlo


Laszlo Ersek
 

On 03/09/21 19:39, Aaron Young wrote:
Hello. What methods currently exist to configure OVMF/AAVMF dynamically at runtime?

For example, modify the preferred Language, set boot menu timeout or enable SecureBoot.

To my knowledge, the only way to configure OVMF/AAVMF is via the boot manager or perhaps via qemu args (i.e. fw_cfg, etc.).

Are there any other out-of-band methods or utilities out there that can be used to configure runtime settings (stored in the VARS file)?

Reason - Some users may wish to configure OVMF/AAVMF prior to first boot (i.e. can't use boot manager) and modification of qemu args is not always an easy (or even possible) option.

Thanks for any help/info/comments,
Sorry, can't give you anything.

The general approach is:

- UEFI variables are used for configuring firmware aspects, usually the
same as on physical platforms (meaning volatile/non-volatile, boot time
only access / runtime access as well) -- consult any individual edk2
UEFI variables for details

- for a number of behaviors / configuration knobs, dynamic PCDs are
used. Dynamic PCDs are boot time only. They are set from various
sources; sometimes they are exposed on the Setup TUI, sometimes they
come from UEFI variables (or vice versa).

- in a number of cases, these variables and these PCDs are set by OVMF
Platform Code (overriding defautls or previously set values) in
accordance with QEMU platform specifics -- for example (frequently),
fw_cfg. The most common "host integration" interface is indeed fw_cfg.
(As the name says: "firmware configuration".)

One telling example is "efibootmgr". It lets you tweak some stuff (boot
order, boot options, Timeout variable) on physical platforms at least.
However, when using OVMF, those settings are overridden dynamically on
every boot, when using the "bootindex" device properties, and/or the
"-boot menu=on" / "-boot splash-time=N" options.

If you name specifics, I can perhaps respond with more details.

Modifying the varstore file from the host side is not supported. The
wire format of that file is the functional composition of three OVMF /
edk2 drivers, namely the OVMF flash driver, the edk2 Fault Tolerant
Write (FTW) driver, and the edk2 Variable driver. Especially due to the
FTW abstraction, the varstore is a kind of "journaled storage", which is
supposed to protect against (and recover from) a power failure during a
flash write. In order to mess with the varstore from the host side,
you'd have to maintain a userspace utility in parallel with the edk2
drivers, and with the OVMF platform settings (flash size, internal
layout), to implement the same feature set (e.g., recognizing and
recovering from a power outage during a flash write).

"modification of qemu args is not always an easy (or even possible)
option" -- that's a feature gap in the virtualization management
application that drives QEMU. fw_cfg is the preferred configuration
method, and using QEMU's generic "-fw_cfg" switch, some firmware
tuneables don't even need a dedicated QEMU option. The important options
are then exposed through the libvirt domain XML format, for example. (A
case can always be made for exposing more.)

Regarding secure boot enablement: the
<https://github.com/rhuefi/qemu-ovmf-secureboot/> project provides a
utility that non-interactively enrolls some certificates in the variable
store (driving the guest's UEFI shell via the serial console), and
enables SB. The resultant variable store file can be used as a varstore
*template* for other (newly defined) domains.

An OVMF package on a Linux distro can offer multiple varstore templates.
To deal with these varstore templates, it is recommended that the OVMF
package also install a number of "firmware use case descriptors"
(firmware metadata files), which follow the JSON format defined in
QEMU's "docs/interop/firmware.json". A management application can then
parse the descriptors, and pick the best one for a particular use case
(for example, certificates enrolled vs. not enrolled). The descriptor
then exposes the firmware binary and the matching varstore template that
should be used for domain definition.

Libvirt has support for this, although more work is necessary (for
example it currently cannot filter for the "enrolled-keys" feature flag).

Thanks
Laszlo


Aaron Young
 

Hello. What methods currently exist to configure OVMF/AAVMF dynamically at runtime?

For example, modify the preferred Language, set boot menu timeout or enable SecureBoot.

To my knowledge, the only way to configure OVMF/AAVMF is via the boot manager or perhaps via qemu args (i.e. fw_cfg, etc.).

Are there any other out-of-band methods or utilities out there that can be used to configure runtime settings (stored in the VARS file)?

Reason - Some users may wish to configure OVMF/AAVMF prior to first boot (i.e. can't use boot manager) and modification of qemu args is not always an easy (or even possible) option.

Thanks for any help/info/comments,

-Aaron Young