Date   

Re: [edk2-devel] A problem with live migration of UEFI virtual machines

Laszlo Ersek
 

On 02/28/20 04:20, Zhoujian (jay) wrote:
Hi Laszlo,

-----Original Message-----
From: Qemu-devel
[mailto:qemu-devel-bounces+jianjay.zhou=huawei.com@nongnu.org] On Behalf
Of Laszlo Ersek
Sent: Wednesday, February 26, 2020 5:42 PM
To: Andrew Fish <afish@apple.com>; devel@edk2.groups.io
Cc: berrange@redhat.com; qemu-devel@nongnu.org; Dr. David Alan Gilbert
<dgilbert@redhat.com>; zhoujianjay <zhoujianjay@gmail.com>; discuss
<discuss@edk2.groups.io>; Alex Bennée <alex.bennee@linaro.org>;
wuchenye1995 <wuchenye1995@gmail.com>
Subject: Re: [edk2-devel] A problem with live migration of UEFI virtual machines

Hi Andrew,

On 02/25/20 22:35, Andrew Fish wrote:

Laszlo,

The FLASH offsets changing breaking things makes sense.

I now realize this is like updating the EFI ROM without rebooting the
system. Thus changes in how the new EFI code works is not the issue.

Is this migration event visible to the firmware? Traditionally the
NVRAM is a region in the FD so if you update the FD you have to skip
NVRAM region or save and restore it. Is that activity happening in
this case? Even if the ROM layout does not change how do you not lose
the contents of the NVRAM store when the live migration happens? Sorry
if this is a remedial question but I'm trying to learn how this
migration works.
With live migration, the running guest doesn't notice anything. This is a general
requirement for live migration (regardless of UEFI or flash).

You are very correct to ask about "skipping" the NVRAM region. With the
approach that OvmfPkg originally supported, live migration would simply be
unfeasible. The "build" utility would produce a single (unified) OVMF.fd file, which
would contain both NVRAM and executable regions, and the guest's variable
updates would modify the one file that would exist.
This is inappropriate even without considering live migration, because OVMF
binary upgrades (package updates) on the virtualization host would force guests
to lose their private variable stores (NVRAMs).

Therefore, the "build" utility produces "split" files too, in addition to the unified
OVMF.fd file. Namely, OVMF_CODE.fd and OVMF_VARS.fd.
OVMF.fd is simply the concatenation of the latter two.

$ cat OVMF_VARS.fd OVMF_CODE.fd | cmp - OVMF.fd [prints nothing]

When you define a new domain (VM) on a virtualization host, the domain
definition saves a reference (pathname) to the OVMF_CODE.fd file.
However, the OVMF_VARS.fd file (the variable store *template*) is not directly
referenced; instead, it is *copied* into a separate (private) file for the domain.

Furthermore, once booted, guest has two flash chips, one that maps the
firmware executable OVMF_CODE.fd read-only, and another pflash chip that
maps its private varstore file read-write.

This makes it possible to upgrade OVMF_CODE.fd and OVMF_VARS.fd (via
package upgrades on the virt host) without messing with varstores that were
earlier instantiated from OVMF_VARS.fd. What's important here is that the
various constants in the new (upgraded) OVMF_CODE.fd file remain compatible
with the *old* OVMF_VARS.fd structure, across package upgrades.

If that's not possible for introducing e.g. a new feature, then the package
upgrade must not overwrite the OVMF_CODE.fd file in place, but must provide an
additional firmware binary. This firmware binary can then only be used by freshly
defined domains (old domains cannot be switched over). Old domains can be
switched over manually -- and only if the sysadmin decides it is OK to lose the
current variable store contents. Then the old varstore file for the domain is
deleted (manually), the domain definition is updated, and then a new (logically
empty, pristine) varstore can be created from the *new* OVMF_2_VARS.fd that
matches the *new* OVMF_2_CODE.fd.


During live migration, the "RAM-like" contents of both pflash chips are migrated
(the guest-side view of both chips remains the same, including the case when the
writeable chip happens to be in "programming mode", i.e., during a UEFI variable
write through the Fault Tolerant Write and Firmware Volume Block(2) protocols).

Once live migration completes, QEMU dumps the full contents of the writeable
chip to the backing file (on the destination host). Going forward, flash writes from
within the guest are reflected to said host-side file on-line, just like it happened
on the source host before live migration. If the file backing the r/w pflash chip is
on NFS (shared by both src and dst hosts), then this one-time dumping when the
migration completes is superfluous, but it's also harmless.

The interesting question is, what happens when you power down the VM on the
destination host (= post migration), and launch it again there, from zero. In that
case, the firmware executable file comes from the *destination host* (it was
never persistently migrated from the source host, i.e. never written out on the
dst). It simply comes from the OVMF package that had been installed on the
destination host, by the sysadmin. However, the varstore pflash does reflect the
permanent result of the previous migration. So this is where things can fall apart,
if both firmware binaries (on the src host and on the dst host) don't agree about
the internal structure of the varstore pflash.
Hi Laszlo,

I found an ealier thread that you said there're 4 options to use ovmf:

https://lists.gnu.org/archive/html/qemu-discuss/2018-04/msg00045.html

Excerpt:
"(1) If you map the unified image with -bios, all of that becomes ROM --
read-only memory.
(2) If you map the unified image with -pflash, all of that becomes
read-write MMIO.
(3) If you use the split images (OVMF_CODE.fd and a copy of
OVMF_VARS.fd), and map then as flash chips, then the top part
(OVMF_CODE.fd, consisting of SECFV and FVMAIN_COMPACT) becomes
read-only flash (MMIO), and the bottom part (copy of OVMF_VARS.fd,
consisting of FTW Spare, FTW Work, Event log, and NV store) becomes
read-write flash (MMIO).
(4) If you use -bios with OVMF_CODE.fd only, then the top part will be
ROM, and the bottom part will be "black hole" MMIO."

I think you're talking about the option (2)(acceptable) and option (3)
(best solution) in this thread, and I agree.
Yes, exactly.


I'm wondering will it be different about ancient option (1) with live
migration. You tried add -DMEM_VARSTORE_EMU_ENABLE=FALSE
build flag to disable -bios support, but Option (1) may be used for the
old VMs started several years ago running on the cloud...
I'm unaware of any VMs running in clouds that use "-bios" with OVMF. It
certainly seems a terrible idea, regardless of live migration.


With developing new features, the size of OVMF.fd is becoming larger
and larger, that seems to be the trend. It would be nice if it could be
hot-updated to the new version. As Daniel said, could it feasible to add
zero-padding to the firmware images?
You're mixing up small details. OVMF_CODE.fd is already heavily padded,
internally. We've grown the *internal* DXEFV firmware volume repeatedly
over *years*, without *any* disruption to users. Please see:

- da78c88f4535 ("OvmfPkg: raise DXEFV size to 8 MB", 2014-03-05)

- 08df58ec3043 ("OvmfPkg: raise DXEFV size to 9 MB", 2015-10-07)

- 2f7b34b20842 ("OvmfPkg: raise DXEFV size to 10 MB", 2016-05-31)

- d272449d9e1e ("OvmfPkg: raise DXEFV size to 11 MB", 2018-05-29)

To this day, i.e., with edk2 master @ edfe16a6d9f8, you can build OVMF
in the default feature configuration [*] for -D FD_SIZE_2MB.

[*]
DEFINE SECURE_BOOT_ENABLE = FALSE
DEFINE SMM_REQUIRE = FALSE
DEFINE SOURCE_DEBUG_ENABLE = FALSE
DEFINE TPM2_ENABLE = FALSE
DEFINE TPM2_CONFIG_ENABLE = FALSE

DEFINE NETWORK_TLS_ENABLE = FALSE
DEFINE NETWORK_IP6_ENABLE = FALSE
DEFINE NETWORK_HTTP_BOOT_ENABLE = FALSE

For example:

$ build \
-a IA32 -a X64 \
-b DEBUG \
-p OvmfPkg/OvmfPkgIa32X64.dsc \
-t GCC48 \
-D FD_SIZE_2MB

Note that this build will contain DEBUG messages (at least DEBUG_INFO
level ones) and ASSERT()s too.

The final usage report at the end of the command is:

SECFV [14%Full] 212992 total, 31648 used, 181344 free
PEIFV [31%Full] 917504 total, 284584 used, 632920 free
DXEFV [44%Full] 11534336 total, 5113688 used, 6420648 free
FVMAIN_COMPACT [73%Full] 1753088 total, 1284216 used, 468872 free

What does that mean? It means that largest firmware volume, DXEFV, uses
just 44% of the 11MB allotted size.

And FVMAIN_COMPACT, which embeds (among other things) DXEFV in
LZMA-compressed format, only uses 73% of its allotted size, which is
1712 KB.

All this means that in the default feature config, there's still a bunch
of room free in the 2MB build, even with DEBUGs and ASSERT()s enabled,
and with an old compiler that does not do link-time optimization.

I think you must have misunderstood the purpose of the 4MB build. The
4MB build was solely introduced for enlarging the *varstore*. That was
motivated by passing an SVVP check. This is described in detail in the
relevant commit, which I may have linked earlier.

https://github.com/tianocore/edk2/commit/b24fca05751f

(Please consult the diagram in the commit message carefully. It shows
you how the various firmware volumes / flash devices are nested; it will
help you understand where the 1712 KB FVMAIN_COMPACT firmware volume is
placed in the final image, and how FVMAIN_COMPACT embeds / compresses
DXEFV.)

And *given that* we had to introduce an incompatible change (for
enlarging the varstore, for SVVP's sake), it made sense to *also*
enlarge the other parts of the flash content. But the motivation was
strictly the varstore change, and that was inevitably an incompatible
change. In fact, you can see in the commit message that the *outer*
container FVMAIN_COMPACT was enlarged from 1712 to 3360 kilobytes, the
embedded PEIFV and DXEFV firmware volumes didn't put that extra space to
use. The SECFV firmware volume runs directly from flash, so it's not
compressed, but even that firmware volume got no "space injection". So
basically all the size increase that *could* have been exploited for
executable code size was spent on padding.

As far as I can tell, we have never broken compatibility due to
executable code size increases.

Sorry if I over-explained this; I simply don't know how to express this
any better.


Things are a little different here,
i.e. the size of src and dest are 2M and 4M respectively, copy the source
2M to the dest side, and then add zero-padding to the end of the image
to round it upto 4 MB at the dest side (With some modification of
qemu_ram_resize in QEMU to avoid length mismatch error report)?
No, this doesn't make any sense.

On both the source host and the destination host, the same pathname (for
example, "/usr/share/OVMF/OVMF_CODE.fd") must point to same-size
(compatible) firmware binaries. Both must be built with the same -D
FD_SIZE_2MB flag, or with the same -D FD_SIZE_4MB flag. Then you can
migrate.

You can offer a 4MB build too on the destination host, but it must be
under a different pathname. So that after the domain has been migrated
in from the source host, and then re-launched against the firmware
binary that's on the destination host, there is an incompatibility
between the domain's *original* varstore, and the domain's *new*
firmware binary.


The physical address assigned to ovmf region will change from
0xffe00000 - 0xffffffff to 0xffc00000 - 0xffffffff, after the OS has
started I see this range will be recycled and assigned to other PCI
devices(using the command "cat /proc/iomem") by guest OS. So,
this range change seems that will not affect the guest I think.
But if the code of OVMF is running when paused at the src side
then will it be continued to run at the dest side, I'm not sure...

So, may I ask that would it be feasible or compatible for option (1)
when live migration between different ovmf sizes? Thanks.
Sorry, my brain just cannot cope with the idea of even *running* OVMF in
production with "-bios" -- let alone migrate it.

But anyway... if you are dead set on this, you can try the following:

- On the destination host, rename the 4MB build to a different filename.

- On the destination host, update all your domain definitions to refer
to the renamed filename with "-bios"

- on the destination host, rebuild your current (more modern) firmware
package, using the -D FD_SIZE_2MB flag. If you have not enabled a bunch
of features meanwhile, it will actually succeed.

- on the destination host, put this fresh build (with unified size 2MB)
in the original place (using the original pathname)

- now you can migrate domains from your source host. The pathname they
refer to with "-bios" will exist, and it will be a 2MB build. And the
contents of that build will be more modern (presumably) than what you
are migrating away from.

Please understand this: when you *allowed* OVMF to build with 4MB size,
and installed it under the exact same pathname (on the destination host)
where you previously used to keep a 2MB binary, *that* is when you broke
compatibility.

What's quite unfathomable to me is that the 2MB->4MB change in upstream
was *solely* motivated by varstore enlargement (for passing SVVP with
*flash*-based variables), but you're still using the ancient and
non-conformant \NvVars emulation that comes with "-bios".

Please, flash based variables with OVMF and QEMU have been supported
since QEMU v1.6.

I've attempted to remove -bios support from OVMF multiple times, I've
always been prevented from doing that, and the damage is obvious only now.

Laszlo


Re: [edk2-devel] A problem with live migration of UEFI virtual machines

Laszlo Ersek
 

Hi Andrew,

On 02/25/20 22:35, Andrew Fish wrote:

Laszlo,

The FLASH offsets changing breaking things makes sense.

I now realize this is like updating the EFI ROM without rebooting the
system. Thus changes in how the new EFI code works is not the issue.

Is this migration event visible to the firmware? Traditionally the
NVRAM is a region in the FD so if you update the FD you have to skip
NVRAM region or save and restore it. Is that activity happening in
this case? Even if the ROM layout does not change how do you not lose
the contents of the NVRAM store when the live migration happens? Sorry
if this is a remedial question but I'm trying to learn how this
migration works.
With live migration, the running guest doesn't notice anything. This is
a general requirement for live migration (regardless of UEFI or flash).

You are very correct to ask about "skipping" the NVRAM region. With the
approach that OvmfPkg originally supported, live migration would simply
be unfeasible. The "build" utility would produce a single (unified)
OVMF.fd file, which would contain both NVRAM and executable regions, and
the guest's variable updates would modify the one file that would exist.
This is inappropriate even without considering live migration, because
OVMF binary upgrades (package updates) on the virtualization host would
force guests to lose their private variable stores (NVRAMs).

Therefore, the "build" utility produces "split" files too, in addition
to the unified OVMF.fd file. Namely, OVMF_CODE.fd and OVMF_VARS.fd.
OVMF.fd is simply the concatenation of the latter two.

$ cat OVMF_VARS.fd OVMF_CODE.fd | cmp - OVMF.fd
[prints nothing]

When you define a new domain (VM) on a virtualization host, the domain
definition saves a reference (pathname) to the OVMF_CODE.fd file.
However, the OVMF_VARS.fd file (the variable store *template*) is not
directly referenced; instead, it is *copied* into a separate (private)
file for the domain.

Furthermore, once booted, guest has two flash chips, one that maps the
firmware executable OVMF_CODE.fd read-only, and another pflash chip that
maps its private varstore file read-write.

This makes it possible to upgrade OVMF_CODE.fd and OVMF_VARS.fd (via
package upgrades on the virt host) without messing with varstores that
were earlier instantiated from OVMF_VARS.fd. What's important here is
that the various constants in the new (upgraded) OVMF_CODE.fd file
remain compatible with the *old* OVMF_VARS.fd structure, across package
upgrades.

If that's not possible for introducing e.g. a new feature, then the
package upgrade must not overwrite the OVMF_CODE.fd file in place, but
must provide an additional firmware binary. This firmware binary can
then only be used by freshly defined domains (old domains cannot be
switched over). Old domains can be switched over manually -- and only if
the sysadmin decides it is OK to lose the current variable store
contents. Then the old varstore file for the domain is deleted
(manually), the domain definition is updated, and then a new (logically
empty, pristine) varstore can be created from the *new* OVMF_2_VARS.fd
that matches the *new* OVMF_2_CODE.fd.


During live migration, the "RAM-like" contents of both pflash chips are
migrated (the guest-side view of both chips remains the same, including
the case when the writeable chip happens to be in "programming mode",
i.e., during a UEFI variable write through the Fault Tolerant Write and
Firmware Volume Block(2) protocols).

Once live migration completes, QEMU dumps the full contents of the
writeable chip to the backing file (on the destination host). Going
forward, flash writes from within the guest are reflected to said
host-side file on-line, just like it happened on the source host before
live migration. If the file backing the r/w pflash chip is on NFS
(shared by both src and dst hosts), then this one-time dumping when the
migration completes is superfluous, but it's also harmless.

The interesting question is, what happens when you power down the VM on
the destination host (= post migration), and launch it again there, from
zero. In that case, the firmware executable file comes from the
*destination host* (it was never persistently migrated from the source
host, i.e. never written out on the dst). It simply comes from the OVMF
package that had been installed on the destination host, by the
sysadmin. However, the varstore pflash does reflect the permanent result
of the previous migration. So this is where things can fall apart, if
both firmware binaries (on the src host and on the dst host) don't agree
about the internal structure of the varstore pflash.

Thanks
Laszlo


Re: [edk2-devel] A problem with live migration of UEFI virtual machines

Laszlo Ersek
 

Hi Andrew,

On 02/25/20 19:56, Andrew Fish wrote:
Laszlo,

If I understand this correctly is it not more complicated than just size. It also assumes the memory layout is the same?
Yes.

The legacy BIOS used fixed magic address ranges, but UEFI uses dynamically allocated memory so addresses are not fixed. While the UEFI firmware does try to keep S3 and S4 layouts consistent between boots, I'm not aware of any mechanism to keep the memory map address the same between versions of the firmware?
It's not about RAM, but platform MMIO.

The core of the issue here is that the -D FD_SIZE_4MB and -D FD_SIZE_2MB
build options (or more directly, the different FD_SIZE_IN_KB macro
settings) set a bunch of flash-related build-time constant macros, and
PCDs, differently, in the following files:

- OvmfPkg/OvmfPkg.fdf.inc
- OvmfPkg/VarStore.fdf.inc
- OvmfPkg/OvmfPkg*.dsc

As a result, the OVMF_CODE.fd firmware binary will have different
hard-coded references to the variable store pflash addresses.
(Guest-physical MMIO addresses that point into the pflash range.)

If someone tries to combine an OVMF_CODE.fd firmware binary from e.g.
the 4MB build, with a variable store file that was originally
instantiated from an OVMF_VARS.fd varstore template from the 2MB build,
then the firmware binary's physical address references and various size
references will not match the contents / layout of the varstore pflash
chip, which maps an incompatibly structured varstore file.

For example, "OvmfPkg/VarStore.fdf.inc" describes two incompatible
EFI_FIRMWARE_VOLUME_HEADER structures (which "build" generates for the
OVMF_VARS.fd template) between the 4MB (total size) build, and the
1MB/2MB (total size) build.

The commit message below summarizes the internal layout differences,
from 1MB/2MB -> 4MB:

https://github.com/tianocore/edk2/commit/b24fca05751f

Excerpt (relevant for OVMF_VARS.fd):

Description Compression type Size [KB]
------------------------- ----------------- ----------------------
Non-volatile data storage open-coded binary 128 -> 528 ( +400)
data
Variable store 56 -> 256 ( +200)
Event log 4 -> 4 ( +0)
Working block 4 -> 4 ( +0)
Spare area 64 -> 264 ( +200)

Thanks
Laszlo


On Feb 25, 2020, at 9:53 AM, Laszlo Ersek <lersek@redhat.com> wrote:

On 02/24/20 16:28, Daniel P. Berrangé wrote:
On Tue, Feb 11, 2020 at 05:39:59PM +0000, Alex Bennée wrote:

wuchenye1995 <wuchenye1995@gmail.com> writes:

Hi all,
We found a problem with live migration of UEFI virtual machines
due to size of OVMF.fd changes.
Specifically, the size of OVMF.fd in edk with low version such as
edk-2.0-25 is 2MB while the size of it in higher version such as
edk-2.0-30 is 4MB.
When we migrate a UEFI virtual machine from the host with low
version of edk2 to the host with higher one, qemu component will
report an error in function qemu_ram_resize while
checking size of ovmf_pcbios: Length mismatch: pc.bios: 0x200000 in
!= 0x400000: Invalid argument.
We want to know how to solve this problem after updating the
version of edk2.
You can only migrate a machine that is identical - so instantiating a
empty machine with a different EDK image is bound to cause a problem
because the machines don't match.
I don't believe we are that strict for firmware in general. The
firmware is loaded when QEMU starts, but that only matters for the
original source host QEMU. During migration, the memory content of the
original firmware will be copied during live migration, overwriting
whatever the target QEMU loaded off disk. This works....provided the
memory region is the same size on source & target host, which is where
the problem arises in this case.

If there's a risk that newer firmware will be larger than old firmware
there's only really two options:

- Keep all firmware images forever, each with a unique versioned
filename. This ensures target QEMU will always load the original
smaller firmware

- Add padding to the firmware images. IOW, if the firmware is 2 MB,
add zero-padding to the end of the image to round it upto 4 MB
(whatever you anticipate the largest size wil be in future).

Distros have often taken the latter approach for QEMU firmware in the
past. The main issue is that you have to plan ahead of time and get
this padding right from the very start. You can't add the padding
after the fact on an existing VM.
Following up here *too*, just for completeness.

The query in this thread has been posted three times now (and I have
zero idea why). Each time it generated a different set of responses. For
completes, I'm now going to link the other two threads here (because the
present thread seems to have gotten the most feedback).

To the OP:

- please do *NOT* repost the same question once you get an answer. It
only fragments the discussion and creates confusion. It also doesn't
hurt if you *confirm* that you understood the answer.

- Yet further, if your email address has @gmail.com for domain, but your
msgids contain "tencent", that raises some eyebrows (mine for sure).
You say "we" in the query, but never identify the organization behind
the plural pronoun.

(I've been fuming about the triple-posting of the question for a while
now, but it's only now that, upon seeing how much work Dan has put into
his answer, I've decided that dishing out a bit of netiquette would be
in order.)

* First posting:
- msgid: <tencent_F1295F826E46EDFF3D77812B@qq.com <mailto:tencent_F1295F826E46EDFF3D77812B@qq.com>>
- edk2-devel: https://edk2.groups.io/g/devel/message/54146
- qemu-devel: https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg02419.html

* my response:
- msgid: <12553.1581366059422195003@groups.io <mailto:12553.1581366059422195003@groups.io>>
- edk2-devel: https://edk2.groups.io/g/devel/message/54161
- qemu-devel: none, because (as an exception) I used the stupid
groups.io <http://groups.io/> web interface to respond, and so my response
never reached qemu-devel

* Second posting (~4 hours after the first)
- msgid: <tencent_3CD8845EC159F0161725898B@qq.com <mailto:tencent_3CD8845EC159F0161725898B@qq.com>>
- edk2-devel: https://edk2.groups.io/g/devel/message/54147
- qemu-devel: https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg02415.html

* Dave's response:
- msgid: <20200220154742.GC2882@work-vm>
- edk2-devel: https://edk2.groups.io/g/devel/message/54681
- qemu-devel: https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg05632.html

* Third posting (next day, present thread) -- cross posted to yet
another list (!), because apparently Dave's feedback and mine had not
been enough:
- msgid: <tencent_BC7FD00363690990994E90F8@qq.com <mailto:tencent_BC7FD00363690990994E90F8@qq.com>>
- edk2-devel: https://edk2.groups.io/g/devel/message/54220
- edk2-discuss: https://edk2.groups.io/g/discuss/message/135
- qemu-devel: https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg02735.html

Back on topic: see my response again. The answer is, you can't solve the
problem (specifically with OVMF), and QEMU in fact does you service by
preventing the migration.

Laszlo



Re: [edk2-devel] A problem with live migration of UEFI virtual machines

Laszlo Ersek
 

On 02/24/20 16:28, Daniel P. Berrangé wrote:
On Tue, Feb 11, 2020 at 05:39:59PM +0000, Alex Bennée wrote:

wuchenye1995 <wuchenye1995@gmail.com> writes:

Hi all,
We found a problem with live migration of UEFI virtual machines
due to size of OVMF.fd changes.
Specifically, the size of OVMF.fd in edk with low version such as
edk-2.0-25 is 2MB while the size of it in higher version such as
edk-2.0-30 is 4MB.
When we migrate a UEFI virtual machine from the host with low
version of edk2 to the host with higher one, qemu component will
report an error in function qemu_ram_resize while
checking size of ovmf_pcbios: Length mismatch: pc.bios: 0x200000 in
!= 0x400000: Invalid argument.
We want to know how to solve this problem after updating the
version of edk2.
You can only migrate a machine that is identical - so instantiating a
empty machine with a different EDK image is bound to cause a problem
because the machines don't match.
I don't believe we are that strict for firmware in general. The
firmware is loaded when QEMU starts, but that only matters for the
original source host QEMU. During migration, the memory content of the
original firmware will be copied during live migration, overwriting
whatever the target QEMU loaded off disk. This works....provided the
memory region is the same size on source & target host, which is where
the problem arises in this case.

If there's a risk that newer firmware will be larger than old firmware
there's only really two options:

- Keep all firmware images forever, each with a unique versioned
filename. This ensures target QEMU will always load the original
smaller firmware

- Add padding to the firmware images. IOW, if the firmware is 2 MB,
add zero-padding to the end of the image to round it upto 4 MB
(whatever you anticipate the largest size wil be in future).

Distros have often taken the latter approach for QEMU firmware in the
past. The main issue is that you have to plan ahead of time and get
this padding right from the very start. You can't add the padding
after the fact on an existing VM.
Following up here *too*, just for completeness.

The query in this thread has been posted three times now (and I have
zero idea why). Each time it generated a different set of responses. For
completes, I'm now going to link the other two threads here (because the
present thread seems to have gotten the most feedback).

To the OP:

- please do *NOT* repost the same question once you get an answer. It
only fragments the discussion and creates confusion. It also doesn't
hurt if you *confirm* that you understood the answer.

- Yet further, if your email address has @gmail.com for domain, but your
msgids contain "tencent", that raises some eyebrows (mine for sure).
You say "we" in the query, but never identify the organization behind
the plural pronoun.

(I've been fuming about the triple-posting of the question for a while
now, but it's only now that, upon seeing how much work Dan has put into
his answer, I've decided that dishing out a bit of netiquette would be
in order.)

* First posting:
- msgid: <tencent_F1295F826E46EDFF3D77812B@qq.com>
- edk2-devel: https://edk2.groups.io/g/devel/message/54146
- qemu-devel: https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg02419.html

* my response:
- msgid: <12553.1581366059422195003@groups.io>
- edk2-devel: https://edk2.groups.io/g/devel/message/54161
- qemu-devel: none, because (as an exception) I used the stupid
groups.io web interface to respond, and so my response
never reached qemu-devel

* Second posting (~4 hours after the first)
- msgid: <tencent_3CD8845EC159F0161725898B@qq.com>
- edk2-devel: https://edk2.groups.io/g/devel/message/54147
- qemu-devel: https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg02415.html

* Dave's response:
- msgid: <20200220154742.GC2882@work-vm>
- edk2-devel: https://edk2.groups.io/g/devel/message/54681
- qemu-devel: https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg05632.html

* Third posting (next day, present thread) -- cross posted to yet
another list (!), because apparently Dave's feedback and mine had not
been enough:
- msgid: <tencent_BC7FD00363690990994E90F8@qq.com>
- edk2-devel: https://edk2.groups.io/g/devel/message/54220
- edk2-discuss: https://edk2.groups.io/g/discuss/message/135
- qemu-devel: https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg02735.html

Back on topic: see my response again. The answer is, you can't solve the
problem (specifically with OVMF), and QEMU in fact does you service by
preventing the migration.

Laszlo


Re: HTTP boot failed on timeout

Laszlo Ersek
 

On 02/24/20 11:01, doron.bleiberg@ecitele.com wrote:
Hi Laszlo,

At first I thought some of the problems I'm facing were related to virtio-net-pci network type. So I've switched to e1000.
Eventually I've found the issues are not related to virtio-net-pci and reverted back to this configuration which works well.
So bottom line, no need for taking E3522X2.EFI driver nor using e1000 (at least in my case).
OK, thank you for explaining!

Note for completeness: -global virtio-net-pci.romfile="" is required for the VM to operate.

It was pleasant discussing with you on this thread, by no means I take it for granted your questions, answers and guidance which helped a-lot. Really appreciated.
Thank you -- I didn't try to imply I was "displeased" or whatever; I was
just genuinely curious why you liked e1000 more than virtio-net-pci. :)

Thanks!
Laszlo


Re: HTTP boot failed on timeout

doron.bleiberg@...
 

Hi Laszlo,

At first I thought some of the problems I'm facing were related to virtio-net-pci network type. So I've switched to e1000.
Eventually I've found the issues are not related to virtio-net-pci and reverted back to this configuration which works well.
So bottom line, no need for taking E3522X2.EFI driver nor using e1000 (at least in my case).

Note for completeness: -global virtio-net-pci.romfile="" is required for the VM to operate.

It was pleasant discussing with you on this thread, by no means I take it for granted your questions, answers and guidance which helped a-lot. Really appreciated.

Doron


Re: HTTP boot failed on timeout

Laszlo Ersek
 

On 02/22/20 20:00, doron.bleiberg@ecitele.com wrote:
Hi Community,

An update regarding my case - case solved.
I was able to boot my system using HTTP boot. What worked for me eventually is:
Copy E3522X2.EFI from Intel
compile OVMF with e1000 support: build -p OvmfPkg/OvmfPkgX64.dsc -D E1000_ENABLE
run QEMU with flag -global e1000.romfile=""
Thanks!

May I ask why you switched from virtio-net-pci to e1000? Are you
HTTP-booting a Windows guest perhaps?

Thanks
Laszlo


IPSec Support

doron.bleiberg@...
 

Hi Community,

I read that the project supports IPSec, but what exactly this support includes?
Does it mean the UEFI drivers will enable all my traffic to be encapsulated with IPSec once I enable the feature?
Is there any support for key rotation?

Where can I find more detailed and technical documentation?

Doron


Re: HTTP boot failed on timeout

doron.bleiberg@...
 

Hi Community,

An update regarding my case - case solved.
I was able to boot my system using HTTP boot. What worked for me eventually is:
Copy E3522X2.EFI from Intel
compile OVMF with e1000 support: build -p OvmfPkg/OvmfPkgX64.dsc -D E1000_ENABLE
run QEMU with flag -global e1000.romfile=""

Thank you all for the support.

Doron


Re: HTTP boot failed on timeout

Laszlo Ersek
 

On 02/17/20 11:52, doron.bleiberg@ecitele.com wrote:
Hi Laszlo,

Thank you for the quick and detailed response. Some answers to your questions:
- Since I'm running the VMs as part of GNS3 project I'm not having full control over the VM startup command. I will try to run the VM outside GNS3 context just to make sure I'm having cleaner and more controlled environment.
- I'm using qcow2 drive file which is the HDD on which the net installation should run. The drive is empty waiting for the installation after boot completion.


I've used your suggestion for adding a 'romfile=""' property. I did it this way as I can't edit the device properties (GNS3...):
-global virtio-net-pci.romfile=""

Can you explain what is the meaning of setting this property and why it is related to my problem?
It is not necessarily related to your problem; it *could* be related.

The romfile property tells QEMU what PCI expansion ROM to load into the
device's PCI option ROM BAR. By default the ROM in question is built
from the iPXE project. This means that the Simple Network Protocol
driver that talks to the virtio-net-pci device directly in OVMF comes
from the iPXE project.

OVMF has a built-in driver (VirtioNetDxe) for the same device however,
it just has lower priority. So if you want VirtioNetDxe to take control
of the device, you need to prevent the loading of the expansion ROM
described above. That's what romfile='' does.

And this could be relevant to your problem because the SNP driver lies
at the bottom of the edk2 network stack (in OVMF anyway). If there is a
problem in your iPXE SNP driver, then that could affect the dependent
TCP connection, and hang your HTTP boot. Switching in the VirtioNetDxe
driver might show a difference here.

Note: I'm not blindly blaming the iPXE driver; I'm just saying it
*could* be related. Seeing how your QEMU binary is ancient (4+ years
old), there could be old iPXE issues affecting your iPXE expansion ROM
too, that have been fixed since, up-stream.

Also, if not suggested to use -bios what are the alternatives?
You should use the split build of OVMF (OVMF_CODE.fd and OVMF_VARS.fd).
OVMF_CODE.fd is the firmware executable. OVMF_VARS.fd is a *template*
file for creating actual variable store files from, when defining a new
domain. Once you define a new domain, the varstore file that was
originally copied from OVMF_VARS.fd should be considered basically
another "data disk" for the domain. The varstore file is where
persistent (non-volatile) UEFI variables are stored, for the domain.

With "-bios", you get a varstore emulation that's not spec-conformant.
It suffers from various obscure problems. Don't use it.

The related (traditional) QEMU cmdline options are shown below. There is
a more recent, more modern, format for the same, but that format
requires a newer QEMU release. (For details, check out
<http://mid.mail-archive.com/146b553d-cbe7-ac87-9423-bd07602e3e01@redhat.com>.
But honestly, the best idea is to just use libvirt.)

So, the original options are:

-drive if=pflash,format=raw,unit=0,file=OVMF_CODE.fd,readonly=on \
-drive if=pflash,format=raw,unit=1,file=guest-vars.fd \

where "guest-vars.fd" is specific to the guest in question, and was
originally copied from OVMF_VARS.fd.

If your distro doesn't package OVMF_CODE.fd and OVMF_VARS.fd separately,
then it's too old.

Is there an option for me to also skip 'Start PXE over IPv4' part from happening in the boot process?
No, there's not. While you can influence the UEFI boot order from the
QEMU command line, for example with:

-device virtio-net-pci,[other options],bootindex=0

the QEMU <-> firmware interface that exposes this to the firmware --
specifically, the "bootorder" fw_cfg file -- is not expressive enough to
tell apart PXE boot on a given NIC from HTTPv4 boot on the same NIC. You
can specify a particular NIC, but given that NIC, you'll have to stick
with the edk2-default boot order for a NIC.

If you can go into the firmware setup TUI once, and manually reorder the
PXE vs. HTTPv4 boot options, then OVMF will generally stick with that
order for you (until / unless you instruct OVMF to drop netboot from the
boot order altogether). But that requires you to interact with the guest
firmware.


This suggestion worked well! and I was able to fully download the file. Thank you!
Oh wow. :) So, it *is* related to the iPXE SNP driver.

For reference, can you search your installed package set for packages
that have "ipxe" in the name? Can you list their names and versions?

Basically now I expect that you are using your distro's very outdated
iPXE package, whose issues have long been fixed up-stream.

The only problem now, is that I get kernel panic on the next step - who said life are simple..... ;-)

I'll continue to debug the kernel panic.

I've attached qemu.log just in case.
Thanks,
Laszlo


Re: HTTP boot failed on timeout

doron.bleiberg@...
 

Hi Laszlo,

Thank you for the quick and detailed response. Some answers to your questions:
- Since I'm running the VMs as part of GNS3 project I'm not having full control over the VM startup command. I will try to run the VM outside GNS3 context just to make sure I'm having cleaner and more controlled environment.
- I'm using qcow2 drive file which is the HDD on which the net installation should run. The drive is empty waiting for the installation after boot completion.


I've used your suggestion for adding a 'romfile=""' property. I did it this way as I can't edit the device properties (GNS3...):
-global virtio-net-pci.romfile=""

Can you explain what is the meaning of setting this property and why it is related to my problem?
Also, if not suggested to use -bios what are the alternatives?
Is there an option for me to also skip 'Start PXE over IPv4' part from happening in the boot process?

This suggestion worked well! and I was able to fully download the file. Thank you!
The only problem now, is that I get kernel panic on the next step - who said life are simple..... ;-)

I'll continue to debug the kernel panic.

I've attached qemu.log just in case.

Doron


Re: HTTP boot failed on timeout

Laszlo Ersek
 

On 02/17/20 10:45, doron.bleiberg@ecitele.com wrote:
Some inputs:
Qemu version: QEMU emulator version 2.5.0 (Debian
1:2.5+dfsg-5ubuntu10.34), Copyright (c) 2003-2008 Fabrice Bellard
Qemu CMD: /usr/bin/qemu-system-x86_64 \
-name HTTP-BOOT-VM-1 \
-m 8192M \
-smp cpus=1 \
-enable-kvm \
-machine smm=off \
-boot order=c \
-bios /opt/gns3/images/QEMU/OVMF-RELEASE.fd \
Ugh this makes my eyes bleed. :/ Please never use the "-bios" option
with OVMF.

Anyway that's not our current topic here.

-drive file=/opt/gns3/projects/1a83274a-c57f-4337-8a0d-1e68a9312e9a/project-files/qemu/d8f37f0b-2b63-455f-b536-b309b9020e36/hda_disk.qcow2,if=ide,index=0,media=disk \
-uuid d8f37f0b-2b63-455f-b536-b309b9020e36 \
-vnc 0.0.0.0:3 \
-monitor tcp:127.0.0.1:41898,server,nowait \
-net none \
Seems inconsistent with your intent to HTTP Boot... At least superfluous
with the below, I'd think

-device virtio-net-pci,mac=0c:2e:9a:0e:36:00,netdev=gns3-0 \
Seems OK; so you are using virtio-net-pci.

-netdev socket,id=gns3-0,udp=127.0.0.1:10017,localaddr=127.0.0.1:10016 \
Unfortunately, I'm entirely unused to "-netdev socket", especially with
"udp=...". I only use "-netdev tap" (via libvirt).

-nographic \
-debugcon file:debug.log \
-global isa-debugcon.iobase=0x402

The OVMF log is empty, all the logs appear in attached qemu.log file.
Hmmm... Ah you mention "OVMF-RELEASE.fd" above. So that must be a
RELEASE build of OVMF, which indeed does not produce debug messages. Can
you try with DEBUG please?


I've did some debugging myself and found out the offending line is
here:
File: NetworkPkg/HttpBootDxe/HttpBootSupport.c#L1012
The error is handled here:
NetworkPkg/HttpBootDxe/HttpBootSupport.c#L1018
Yes, I checked those lines after reading your earlier message.

I asked for your command line and the OVMF debug log because I wanted to
see if you were using the iPXE SNP driver for virtio-net (you most
likely are, from the info thus far). It could be interesting to try the
built-in virtio-net driver, for one data point. (This would be a
front-end check.)

Second, I have no idea what's happening on the udp socket netdev
back-end. It could explain the virtual network glitches you are seeing.
I'm not sure.

Laszlo


Platform Variable hook library for Supporting UEFI Variable Resiliency

Wang, Sunny (HPS SW)
 

Hi all,

As the discussion in the email thread below, we would like to add a Platform Variable hook library for IBV/OEM to support the UEFI Variable Resiliency (protection, detection, and recovery for critical UEFI variable).

For your quick reference, I summarized the reasons why we need to do this below:
1. There are some guidelines and requirements in NIST 800-193 and other security guidelines for critical UEFI variables like BootOrder, so we need to make some changes to comply with them.
2. The current variable protection mechanism in EDK2 called "Variable Policy (EDKII_VARIABLE_LOCK_PROTOCOL and VarCheckLib)" can't satisfy our needs. Using VariableLock would cause issues with OSes, and VarCheckLib can't check the data. Therefore, we need to add code to check and override the changes that were made at runtime.
3. Since each platform has its defined critical data and its protection and recovery policies/actions, some changes need to be platform-specific. However, there is no hook function or interface consumed by the EDK2 variable driver for IBV/OEM to add platform changes. Therefore, we need to add a Platform Variable hook library.

Moreover, I checked this with Ray and will bring the details to this week's design meeting on 2/21. I also uploaded my slides to the link below for your further reference. Of course, if you have any questions, feel free to bring it to me anytime.
https://edk2.groups.io/g/devel/files/Designs/2020/0221/Platform%20Libraries%20for%20Supporting%20UEFI%20Variable%20Resiliency.pdf

Regards,
Sunny Wang

-----Original Message-----
From: discuss@edk2.groups.io [mailto:discuss@edk2.groups.io] On Behalf Of Wang, Sunny (HPS SW)
Sent: Thursday, February 6, 2020 4:52 PM
To: discuss@edk2.groups.io; lersek@redhat.com; sean.brogan@microsoft.com; tim.lewis@insyde.com; phlamorim@riseup.net; Samer El-Haj-Mahmoud <Samer.El-Haj-Mahmoud@arm.com>; ray.ni@intel.com
Cc: Spottswood, Jason <jason.spottswood@hpe.com>; Haskell, Darrell <darrell.haskell@hpe.com>; Wiginton, Scott <scott.wiginton@hpe.com>; Javier Martinez Canillas <javierm@redhat.com>; Peter Jones <pjones@redhat.com>; Wang, Sunny (HPS SW) <sunnywang@hpe.com>
Subject: Re: [edk2-discuss] Lock BootOrder variable

Hi Lazlo,

Thanks for the comments and sorry for the delay. I was just back from my long vacation and was busy with some other stuff.

Yeah, locking BootOrder is not acceptable, so we would like to override or ignore the BootOrder change made by OS or non-trusted software at the next boot to protect and recover the BootOrder. In other words, locking variables will definitely not be used as a solution. We just want OS vendors to gracefully deal with all the error status from the runtime variable service. However, if you think RuntimeServicesSupported is good and clear enough for OS vendors to gracefully deal with this, I'm totally fine with not adding another description into UEFI specification.

As for the solution, you're also right about that it should be implemented as a platform code. However, since the system may have more UEFI variables (not only BootOrder) that are identified as critical data and needed to be protected as well, we would like to propose a platform variable hook library to add the hook function calls into the variable driver for IBV/OEM like us to implement the platform protection code.

Moreover, before submitting my patches, I think it would be better to further discuss this at the design meeting, so I will check this with Ray to find an available time slot and then present the details to you guys. The purpose of our proposal will be to have platform libraries for IBV/OEM to implement their code for complying with the NIST 800-193 and other security guidelines mentioned in the previous emails (Supporting the UEFI Variable Resiliency (protection, detection, and recovery)).

Regards,
Sunny Wang

-----Original Message-----
From: discuss@edk2.groups.io [mailto:discuss@edk2.groups.io] On Behalf Of Laszlo Ersek
Sent: Saturday, January 4, 2020 1:48 AM
To: discuss@edk2.groups.io; Wang, Sunny (HPS SW) <sunnywang@hpe.com>; sean.brogan@microsoft.com; tim.lewis@insyde.com; phlamorim@riseup.net; Samer El-Haj-Mahmoud <Samer.El-Haj-Mahmoud@arm.com>
Cc: Spottswood, Jason <jason.spottswood@hpe.com>; Haskell, Darrell <darrell.haskell@hpe.com>; Wiginton, Scott <scott.wiginton@hpe.com>; Javier Martinez Canillas <javierm@redhat.com>; Peter Jones <pjones@redhat.com>
Subject: Re: [edk2-discuss] Lock BootOrder variable

On 12/18/19 11:03, Wang, Sunny (HPS SW) wrote:
Hi Sean,

Thanks for checking this further and bringing this to the Microsoft OS team.

Yeah, your guess is part of the solution that I'm working on. I'm making a spec'd way to fix this and will bring it to USWG and EDK2 design meeting to further discuss with you guys.

Moreover, I tried RHEL and SLES as well, and ran into the same problem
(installation failure). Therefore, it will be good if some people from
Linux OS vendors in this email list can also help drive this for Linux
OS. Of course, I will still bring this to USWG for checking if we need
to add a description into UEFI specification for asking/reminding OS
to gracefully deal with the locked variable (and error
Locking the boot order by way of causing SetVariable to fail for the OS seems wrong to me.

BDS is platform policy. A platform BDS implementation can simply re-generate all Boot#### variables, and the BootOrder variable, from scratch, every time the platform boots. Why is that not sufficient? (For example, platform BDS could base that boot order re-generation on a set of boot-time only variables.)

Furthermore, we already have RuntimeServicesSupported, for exposing (among other things) that SetVariable doesn't work, "en bloc". Punching further holes into previously promised interfaces, piece-meal, is becoming quite baroque.

I don't understand how "security" is improved by locking down boot order. We already have Secure Boot for executing trusted binaries only, and we already have TCG2 / TPM2 for tying secrets to any and all aspects of the boot path (such as executables launched, the order they are launched in, their config files, and so on).

Anyway, I've CC'd Javier and Peter from the rhboot team.

Laszlo

-----Original Message-----
From: discuss@edk2.groups.io [mailto:discuss@edk2.groups.io] On Behalf
Of Sean via Groups.Io
Sent: Wednesday, December 18, 2019 4:55 AM
To: Wang, Sunny (HPS SW) <sunnywang@hpe.com>; tim.lewis@insyde.com;
discuss@edk2.groups.io; phlamorim@riseup.net; Samer El-Haj-Mahmoud
<Samer.El-Haj-Mahmoud@arm.com>
Cc: Spottswood, Jason <jason.spottswood@hpe.com>; Haskell, Darrell
<darrell.haskell@hpe.com>; Wiginton, Scott <scott.wiginton@hpe.com>
Subject: Re: [edk2-discuss] Lock BootOrder variable

Sunny,

There isn't UEFI code that is going to resolve the OS failure. I guess you could let the OS write the values and just not use those values but that is probably a bad pattern and would lead to new issues. We have talked with the Microsoft OS team about fixing this but it hasn't been a high priority. Our general guidance is don't lock the boot order until after your have deployed your OS image but there are some servicing issues that can occur.

I'll talk with the boot manager team and see if we can start to fix this in Windows. I'll update this thread if I get an answer.
Does anyone know what the different Linux loader/install process does?

Thanks
Sean


-----Original Message-----
From: Wang, Sunny (HPS SW) <sunnywang@hpe.com>
Sent: Friday, December 13, 2019 12:55 AM
To: tim.lewis@insyde.com; discuss@edk2.groups.io; Sean Brogan
<sean.brogan@microsoft.com>; phlamorim@riseup.net; Samer
El-Haj-Mahmoud <Samer.El-Haj-Mahmoud@arm.com>
Cc: Spottswood, Jason <jason.spottswood@hpe.com>; Haskell, Darrell
<darrell.haskell@hpe.com>; Wiginton, Scott <scott.wiginton@hpe.com>;
Wang, Sunny (HPS SW) <sunnywang@hpe.com>
Subject: [EXTERNAL] RE: [edk2-discuss] Lock BootOrder variable

Thanks, guys.

Hi Samer,
Thanks for checking this and giving great information. I just knew that NIST 800-193 has some guidelines for locking down the UEFI boot order. The requirements and guidelines you added further proved the need of locking boot order.

Hi Sean,
Actually, I did check the Project Mu VariablePolicy protocol when you guys proposed this in the design meeting. This is definitely a good enhancement because I also ran into the problems mentioned in slides.
-
INVALID URI REMOVED
rotection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fedk2.groups.io-252F
g-252Fdevel-252Ffiles-252FDesigns-252F2019-252F0516-26amp-3Bdata-3D02-
257C01-257Csean.brogan-2540microsoft.com-257C7a27e3cc7d754208afd108d77
faa2a64-257C72f988bf86f141af91ab2d7cd011db47-257C1-257C0-257C637118241
153980473-26amp-3Bsdata-3DxjJYipCUkrC19mz9D0eRqAJYAOB0DGBVK7kgn7fTXuQ-
253D-26amp-3Breserved-3D0&d=DwICaQ&c=C5b8zRQO1miGmBeVZ2LFWg&r=Z9cLgEMd
GZCI1_R0bW6KqOAGzCXLJUR24f8N3205AYw&m=uWkd1_Wtepp1js5IwHPeUBVeN78J71Ii
_NieuwsJZk8&s=6O-ZcjoIEhjvpzBGydrRy4RM6BCiLwpTVRmAlkphX-o&e=
However, It looks like we would still run into the OS installation failure with the current VariablePolicy protocol implementation. I didn't deeply look into the VariablePolicy protocol or give it a try, so I may overlook the solution in your implementation. Could you help point out where the code for solving OS installation failure is? The problem here is that OS doesn't gracefully deal with the locked UEFI variable, so we may not be able to return EFI_WRITE_PROTECTED to OS for the writes to BootOrder.


Hi All,
I think we all agree with the following points:
1. There is a need to lock BootOrder like the information brought by Samer and NIST 800-193 guidelines.
2. OS needs to gracefully deal with the locked BootOrder without causing failure during the installation and operations.
3. We will at least need to update the UEFI specification for asking OS to gracefully deal with the locked variable. If no one is currently working on this, I will bring it to USWG and work on it.

Regards,
Sunny Wang

-----Original Message-----
From: tim.lewis@insyde.com [mailto:tim.lewis@insyde.com]
Sent: Friday, December 13, 2019 10:26 AM
To: discuss@edk2.groups.io; sean.brogan@microsoft.com;
phlamorim@riseup.net; Wang, Sunny (HPS SW) <sunnywang@hpe.com>
Subject: RE: [edk2-discuss] Lock BootOrder variable

Sean --

Since you already have published code and it is already publicly available, then I suggest that you bring it to USWG now, rather than later.

Thanks,

Tim

-----Original Message-----
From: discuss@edk2.groups.io <discuss@edk2.groups.io> On Behalf Of
Sean via Groups.Io
Sent: Thursday, December 12, 2019 5:31 PM
To: discuss@edk2.groups.io; phlamorim@riseup.net; sunnywang@hpe.com
Subject: Re: [edk2-discuss] Lock BootOrder variable

Sunny,

There are two other public, non-uefi spec solutions I am aware of.

1. Edk2 VariableLock protocol:
INVALID URI REMOVED
rotection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgithub.com-252Ftian
ocore-252Fedk2-252Fblob-252Fmaster-252FMdeModulePkg-252FInclude-252FPr
otocol-252FVariableLock.h-26amp-3Bdata-3D02-257C01-257Csean.brogan-254
0microsoft.com-257C7a27e3cc7d754208afd108d77faa2a64-257C72f988bf86f141
af91ab2d7cd011db47-257C1-257C0-257C637118241153980473-26amp-3Bsdata-3D
avZwx8B1gRbULoRQuFhAldnvYAe20QFzyhG802LRzcg-253D-26amp-3Breserved-3D0&
d=DwICaQ&c=C5b8zRQO1miGmBeVZ2LFWg&r=Z9cLgEMdGZCI1_R0bW6KqOAGzCXLJUR24f
8N3205AYw&m=uWkd1_Wtepp1js5IwHPeUBVeN78J71Ii_NieuwsJZk8&s=ACauqOvz9A5G
8S3vH-bALwc4RPyfS1u8O2sza1_9Hbw&e=
A relatively limited solution with hard coded lock points tied to edk2 SMM variable store.

2. Project Mu VariablePolicy protocol:
INVALID URI REMOVED
rotection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgithub.com-252Fmicr
osoft-252Fmu-5Fbasecore-252Fblob-252Frelease-252F201911-252FMdeModuleP
kg-252FInclude-252FProtocol-252FVariablePolicy.h-26amp-3Bdata-3D02-257
C01-257Csean.brogan-2540microsoft.com-257C7a27e3cc7d754208afd108d77faa
2a64-257C72f988bf86f141af91ab2d7cd011db47-257C1-257C0-257C637118241153
980473-26amp-3Bsdata-3D14Wv-252B8VbDcwwAcuDQ2rG0V8g561YSDw9By26gIFIr8c
-253D-26amp-3Breserved-3D0&d=DwICaQ&c=C5b8zRQO1miGmBeVZ2LFWg&r=Z9cLgEM
dGZCI1_R0bW6KqOAGzCXLJUR24f8N3205AYw&m=uWkd1_Wtepp1js5IwHPeUBVeN78J71I
i_NieuwsJZk8&s=xjXKoZxYjZwyWJrEHGS5jayMp6HNXhombPuvq7NNM8g&e=
Flexible policy based locking that can be implemented in various hardware architectures.

My team will be proposing the VariablePolicy protocol (potentially as a "code-first" effort) in the coming months and working to upstream this feature into edk2. The reality is some users and use cases want higher assurance for their platform settings and this can include the boot order. Doing this thru a well-defined and auditable protocol is better than an ad-hoc solutions. As you know locking some variables may break assumptions (or spec definition) that other code may have but that tradeoff is best evaluated by the use case.

Thanks
Sean


-----Original Message-----
From: discuss@edk2.groups.io <discuss@edk2.groups.io> On Behalf Of
Paulo Henrique Lacerda de Amorim via Groups.Io
Sent: Thursday, December 12, 2019 12:53 PM
To: discuss@edk2.groups.io; sunnywang@hpe.com
Subject: [EXTERNAL] Re: [edk2-discuss] Lock BootOrder variable

The UEFI define the BootOrder variable with NV+BS+RT attributes, so its not possible to lock this variable, you can try to delete the BootOrder variable and then set the variable with AT attribute, which will probably will result in an undefined behavior. OVMF just recreates the the BootOrder with NV+BS+RT again, then any code which can call Runtime Services will be able to change BootOrder again.

A possible method is too use a signed loader which have your own 'BootOrder' hardcoded.

-----Original Message-----
From: Samer El-Haj-Mahmoud [mailto:Samer.El-Haj-Mahmoud@arm.com]
Sent: Friday, December 13, 2019 12:30 AM
To: discuss@edk2.groups.io; Wang, Sunny (HPS SW) <sunnywang@hpe.com>
Cc: Spottswood, Jason <jason.spottswood@hpe.com>; Wiginton, Scott
<scott.wiginton@hpe.com>; Bodner, James <james.bodner@hpe.com>;
Haskell, Darrell <darrell.haskell@hpe.com>
Subject: RE: [edk2-discuss] Lock BootOrder variable

Sunny,

Looks like this is a Windows Hardware Compatibility Specification
requirement: https://go.microsoft.com/fwlink/?linkid=2086856 , in
Systems.pdf, under System.Fundamentals.Security.DGCG.DeviceGuard -
Firmware BIOS lockdown

I am aware of some proprietary implementations of "Lock Boot Order" as a firmware/UEFI setting, but not a standard method defined in the spec.

Also, NSA has some guidelines on locking down UEFI boot order using whatever firmware settings to disable any undesired boot sources (such as externally available USB or network ports): https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__www.nsa.gov_Portals_70_documents_what-2Dwe-2Ddo_cybersecurity_professional-2Dresources_csi-2Duefi-2Dlockdown.pdf%26d%3DDwIFAg%26c%3DC5b8zRQO1miGmBeVZ2LFWg%26r%3DZ9cLgEMdGZCI1_R0bW6KqOAGzCXLJUR24f8N3205AYw%26m%3DasfixhAVUFNND_WEH4KyE0iVEY8HgOOvdPb6NrNwOUQ%26s%3D-rAbAKmK6iJFSGCadTldga_88zXzHJY2rz7Zmyh_lSM%26e&;data=02%7C01%7Csean.brogan%40microsoft.com%7C7a27e3cc7d754208afd108d77faa2a64%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637118241153980473&amp;sdata=3F6hFgqkkXYyeqS%2BiHp2TfYNGK9J82AZdWwX1mZNc5w%3D&amp;reserved=0= .

Thanks,

Em 12/12/2019 05:49, Wang, Sunny (HPS SW) escreveu:
Hi All,

Is there any spec'd way that we can use to lock some UEFI variables like BootOrder without breaking OS installation and OS functionalities?

For some security reasons and customer use cases, we need to let system firmware completely own some UEFI variables like BootOrder. In other words, we don't want some UEFI variables to be controlled by the OS using the UEFI runtime service SetVariable. In addition, we tried to lock the BootOrder variable, but it would break OS installation or some OS functionalities.

By the way, we will bring this need to USWG if there is no existing spec'd way for satisfying this need.

Regards,
Sunny Wang


Re: HTTP boot failed on timeout

doron.bleiberg@...
 

I also need to mention that I've enabled the HTTP boot myself and compiled the project this way:

git clone https://github.com/tianocore/edk2.git edk2-stable201911
cd edk2-stable201911
git submodule update --init
make -C BaseTools
. edksetup.sh BaseTools

vi Conf/target.txt
ACTIVE_PLATFORM = OvmfPkg/OvmfPkgX64.dsc
TARGET_ARCH = X64
TOOL_CHAIN_TAG = GCC48
TARGET = RELEASE

vi OvmfPkg/OvmfPkgX64.dsc
DEFINE NETWORK_HTTP_BOOT_ENABLE = TRUE
BUILD_TARGETS = RELEASE

export EDK_TOOLS_PATH=$PWD/BaseTools
build -p OvmfPkg/OvmfPkgX64.dsc

And I'm taking the file:
Build/OvmfX64/RELEASE_GCC48/FV/OVMF.fd

Doron


Re: HTTP boot failed on timeout

doron.bleiberg@...
 

Some inputs:
Qemu version: QEMU emulator version 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.34), Copyright (c) 2003-2008 Fabrice Bellard
Qemu CMD: /usr/bin/qemu-system-x86_64 -name HTTP-BOOT-VM-1 -m 8192M -smp cpus=1 -enable-kvm -machine smm=off -boot order=c -bios /opt/gns3/images/QEMU/OVMF-RELEASE.fd -drive file=/opt/gns3/projects/1a83274a-c57f-4337-8a0d-1e68a9312e9a/project-files/qemu/d8f37f0b-2b63-455f-b536-b309b9020e36/hda_disk.qcow2,if=ide,index=0,media=disk -uuid d8f37f0b-2b63-455f-b536-b309b9020e36 -vnc 0.0.0.0:3 -monitor tcp:127.0.0.1:41898,server,nowait -net none -device virtio-net-pci,mac=0c:2e:9a:0e:36:00,netdev=gns3-0 -netdev socket,id=gns3-0,udp=127.0.0.1:10017,localaddr=127.0.0.1:10016 -nographic -debugcon file:debug.log -global isa-debugcon.iobase=0x402

The OVMF log is empty, all the logs appear in attached qemu.log file.

I've did some debugging myself and found out the offending line is here:
File: NetworkPkg/HttpBootDxe/HttpBootSupport.c#L1012
The error is handled here: NetworkPkg/HttpBootDxe/HttpBootSupport.c#L1018

Though I'm using large file I didn't observed a problem in buffer or RAM size.

I've tried to add the requests "Connection: Keep-Alive" header with no change in result.

The boot download always terminates at the same spot.

10x,
Doron


Re: HTTP boot failed on timeout

Laszlo Ersek
 

On 02/15/20 22:24, doron.bleiberg@ecitele.com wrote:
Some additional inputs:
I'm running the boot from a QEMU VM. In the QEMU logs I see the following error during HTTP boot:
Error: Server response timeout.
BdsDxe: failed to load Boot0005 "UEFI HTTPv4 (MAC:0C2E9A0E3600)" from PciRoot(0x0)/Pci(0x3,0x0)/MAC(0C2E9A0E3600,0x1)/IPv4(0.0.0.0,0x0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0)/Uri(): Not Found

I've also raised httpd server logging to trace6 but could not find anything pointing to a problem on this side. I'm only able to observe an error after the UEFI terminated the session, which is expected.
* Please specify
- your QEMU version,
- your full QEMU cmdline.

* Please capture the OVMF log by appending the following QEMU flags:

-debugcon file:debug.log -global isa-debugcon.iobase=0x402

and attach the log.

Thanks
Laszlo


Re: HTTP boot failed on timeout

doron.bleiberg@...
 

Some additional inputs:
I'm running the boot from a QEMU VM. In the QEMU logs I see the following error during HTTP boot:
Error: Server response timeout.
BdsDxe: failed to load Boot0005 "UEFI HTTPv4 (MAC:0C2E9A0E3600)" from PciRoot(0x0)/Pci(0x3,0x0)/MAC(0C2E9A0E3600,0x1)/IPv4(0.0.0.0,0x0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0)/Uri(): Not Found

I've also raised httpd server logging to trace6 but could not find anything pointing to a problem on this side. I'm only able to observe an error after the UEFI terminated the session, which is expected.

Doron


Re: HTTP boot failed on timeout

doron.bleiberg@...
 

Hi Rebecca,

I'm running Apache httpd/2.4.6 (CentOS)
File size is: ~420MB

I did try to run Wireshark, however I was not able to see anything special. The traffic just stopped flowing at some point.

Doron


Re: HTTP boot failed on timeout

Rebecca Cran
 

What web server are you using? How large is the file you're trying to download?

You might be able to get more information about which is at fault by running a traffic analyzer application such as WireShark.


--
Rebecca Cran

On 2020-02-15 13:29, doron.bleiberg@ecitele.com wrote:
Hi Community,

I'm trying to use HTTP boot, the process starts and download is progressing until ~40% of the download is complete. However, every-time the download it terminated at the same spot.
I've tried to debug both the UEFI and the http server side and was not able to get into a conclusion on which side the problem is.
I did find out the the timeout happen at:
File - NetworkPkg/HttpBootDxe/HttpBootSupport.c
Line: 1012
Code:
//
// Poll the network until receive finish.
//
while (!HttpIo->IsRxDone && ((HttpIo->TimeoutEvent == NULL) || EFI_ERROR (gBS->CheckEvent (HttpIo->TimeoutEvent)))) {
Http->Poll (Http);
}

The while loop terminates due to timeout and the below code takes action:
if (!HttpIo->IsRxDone) {
//
// Timeout occurs, cancel the response token.
//
Http->Cancel (Http, &HttpIo->RspToken);
Status = EFI_TIMEOUT;

return Status;
}

Is there any timeout I can set to avoid it? Is the problem on UEFI or http server side?

Doron


HTTP boot failed on timeout

doron.bleiberg@...
 

Hi Community,

I'm trying to use HTTP boot, the process starts and download is progressing until ~40% of the download is complete. However, every-time the download it terminated at the same spot.
I've tried to debug both the UEFI and the http server side and was not able to get into a conclusion on which side the problem is.
I did find out the the timeout happen at:
File - NetworkPkg/HttpBootDxe/HttpBootSupport.c
Line: 1012
Code:
//
// Poll the network until receive finish.
//
while (!HttpIo->IsRxDone && ((HttpIo->TimeoutEvent == NULL) || EFI_ERROR (gBS->CheckEvent (HttpIo->TimeoutEvent)))) {
Http->Poll (Http);
}

The while loop terminates due to timeout and the below code takes action:
if (!HttpIo->IsRxDone) {
//
// Timeout occurs, cancel the response token.
//
Http->Cancel (Http, &HttpIo->RspToken);
Status = EFI_TIMEOUT;

return Status;
}

Is there any timeout I can set to avoid it? Is the problem on UEFI or http server side?

Doron

741 - 760 of 897