[PATCH v6 0/6] SEV Live Migration support for OVMF.


Ashish Kalra
 

From: Ashish Kalra <ashish.kalra@amd.com>

BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3467

By default all the SEV guest memory regions are considered encrypted,
if a guest changes the encryption attribute of the page (e.g mark a
page as decrypted) then notify hypervisor. Hypervisor will need to
track the unencrypted pages. The information will be used during
guest live migration, guest page migration and guest debugging.

The patch-set detects if it is running under KVM hypervisor and then
checks for SEV live migration feature support via KVM_FEATURE_CPUID,
if detected setup a new UEFI enviroment variable to indicate OVMF
support for SEV live migration.

A branch containing these patches is available here:
https://github.com/ashkalra/edk2-1/tree/sev_live_migration_v5_10

Changes since v5:
- Split first patch into three components, one patch for the
MemEncryptSevLiveMigrationIsEnabled() API, one patch for the
SetMemoryEncDecHypercall3() API, one patch to make use of the
SetMemoryEncDecHypercall3() API.
- Fix patch subject, in code and patch comments and
additionally add relevant comments.
- Replace SetMemoryEncDecHypercall3() API's Status argument
with a boolean IsEncrypted argument and corresponding fixes
to users of this API call.
- Fix AsciiStrCmp() usage in KVM hypervisor detection code.

Changes since v4:
- Remove MemEncryptHypercallLib Library and add support to issue
hypercall in the BaseMemEncryptSevLib library itself.
- For SEV-ES, make the VC handler hypercall aware by comparing
the hypercall number and add the additional register values
in the GHCB.
- Fix comments in the hypercall API interface.
- The encryption bit is set/clear on the smallest page size, hence
use the 4k page size in MAP_GPA_RANGE hypercall.
- Make the hypercall expect the guest physical address to be
page-aligned.
- Add KVM live migration feature flag check in BaseMemEncryptSevLib
library similar to how BaseMemEncryptSevLib does for the
MemEncryptSevIsEnabled() and check it before invoking HC. Also
export the MemEncryptSevLiveMigrationIsEnabled() function as
part of the library.
- Add error handling on hypercall return, on failure, return error
code to caller which potentially will cause an assert() and
terminate the boot.

Changes since v3:
- Fix all DSC files under OvmfPkg except X64 to add support for
BaseMemEncryptLib and add NULL instance of BaseMemEncryptLib
for 32 bit platforms.
- Add the MemEncryptHypercallLib-related files to Maintainers.txt,
in section "OvmfPkg: Confidential Computing".
- Add support for the new KVM_HC_MAP_GPA_RANGE hypercall interface.
- Add patch for SEV live migration support.

Changes since v2:
- GHCB_BASE setup during reset-vector as decrypted is marked explicitly
in the hypervisor page encryption bitmap after setting the
PcdSevEsIsEnabled PCD.

Changes since v1:
- Mark GHCB_BASE setup during reset-vector as decrypted explicitly in
the hypervisor page encryption bitmap.
- Resending the series with correct shallow threading.

Ashish Kalra (6):
OvmfPkg/BaseMemEncryptLib: Detect SEV live migration feature.
OvmfPkg/BaseMemEncryptLib: Hypercall API for page encryption state
change
OvmfPkg/BaseMemEncryptLib: Invoke page encryption state change
hypercall
OvmfPkg/VmgExitLib: Encryption state change hypercall support in VC
handler
OvmfPkg/PlatformPei: Mark SEC GHCB page as unencrypted via hypercall
OvmfPkg/AmdSevDxe: Add support for SEV live migration.

OvmfPkg/AmdSevDxe/AmdSevDxe.c | 64 +++++++++++++++++
OvmfPkg/AmdSevDxe/AmdSevDxe.inf | 4 ++
OvmfPkg/Include/Guid/AmdSevMemEncryptLib.h | 20 ++++++
OvmfPkg/Include/Library/MemEncryptSevLib.h | 70 +++++++++++++++++++
.../DxeMemEncryptSevLib.inf | 1 +
.../DxeMemEncryptSevLibInternal.c | 39 +++++++++++
.../Ia32/MemEncryptSevLib.c | 27 +++++++
.../PeiDxeMemEncryptSevLibInternal.c | 52 ++++++++++++++
.../PeiMemEncryptSevLib.inf | 1 +
.../PeiMemEncryptSevLibInternal.c | 39 +++++++++++
.../SecMemEncryptSevLibInternal.c | 38 ++++++++++
.../X64/AsmHelperStub.nasm | 33 +++++++++
.../X64/MemEncryptSevLib.c | 62 ++++++++++++++++
.../X64/PeiDxeVirtualMemory.c | 20 ++++++
OvmfPkg/Library/VmgExitLib/VmgExitVcHandler.c | 13 ++++
OvmfPkg/OvmfPkg.dec | 1 +
OvmfPkg/PlatformPei/AmdSev.c | 11 +++
17 files changed, 495 insertions(+)
create mode 100644 OvmfPkg/Include/Guid/AmdSevMemEncryptLib.h
create mode 100644 OvmfPkg/Library/BaseMemEncryptSevLib/X64/AsmHelperStub.nasm

--
2.17.1


Yao, Jiewen
 

Hi
I have some questions:

1) May I know what is the usage of this UEFI variable - SevLiveMigrationEnabled?
I only see it is created, but I do not see how it is consumed.

2) Is this a full live migration patch, or is this just a startup and there will be more on the way?

Thank you
Yao Jiewen

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Ashish Kalra
via groups.io
Sent: Monday, August 2, 2021 8:31 PM
To: devel@edk2.groups.io
Cc: dovmurik@linux.vnet.ibm.com; brijesh.singh@amd.com; tobin@ibm.com;
Thomas.Lendacky@amd.com; jejb@linux.ibm.com; Justen, Jordan L
<jordan.l.justen@intel.com>; ard.biesheuvel@arm.com;
erdemaktas@google.com; Yao, Jiewen <jiewen.yao@intel.com>; Xu, Min M
<min.m.xu@intel.com>
Subject: [edk2-devel] [PATCH v6 0/6] SEV Live Migration support for OVMF.

From: Ashish Kalra <ashish.kalra@amd.com>

BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3467

By default all the SEV guest memory regions are considered encrypted,
if a guest changes the encryption attribute of the page (e.g mark a
page as decrypted) then notify hypervisor. Hypervisor will need to
track the unencrypted pages. The information will be used during
guest live migration, guest page migration and guest debugging.

The patch-set detects if it is running under KVM hypervisor and then
checks for SEV live migration feature support via KVM_FEATURE_CPUID,
if detected setup a new UEFI enviroment variable to indicate OVMF
support for SEV live migration.

A branch containing these patches is available here:
https://github.com/ashkalra/edk2-1/tree/sev_live_migration_v5_10

Changes since v5:
- Split first patch into three components, one patch for the
MemEncryptSevLiveMigrationIsEnabled() API, one patch for the
SetMemoryEncDecHypercall3() API, one patch to make use of the
SetMemoryEncDecHypercall3() API.
- Fix patch subject, in code and patch comments and
additionally add relevant comments.
- Replace SetMemoryEncDecHypercall3() API's Status argument
with a boolean IsEncrypted argument and corresponding fixes
to users of this API call.
- Fix AsciiStrCmp() usage in KVM hypervisor detection code.

Changes since v4:
- Remove MemEncryptHypercallLib Library and add support to issue
hypercall in the BaseMemEncryptSevLib library itself.
- For SEV-ES, make the VC handler hypercall aware by comparing
the hypercall number and add the additional register values
in the GHCB.
- Fix comments in the hypercall API interface.
- The encryption bit is set/clear on the smallest page size, hence
use the 4k page size in MAP_GPA_RANGE hypercall.
- Make the hypercall expect the guest physical address to be
page-aligned.
- Add KVM live migration feature flag check in BaseMemEncryptSevLib
library similar to how BaseMemEncryptSevLib does for the
MemEncryptSevIsEnabled() and check it before invoking HC. Also
export the MemEncryptSevLiveMigrationIsEnabled() function as
part of the library.
- Add error handling on hypercall return, on failure, return error
code to caller which potentially will cause an assert() and
terminate the boot.

Changes since v3:
- Fix all DSC files under OvmfPkg except X64 to add support for
BaseMemEncryptLib and add NULL instance of BaseMemEncryptLib
for 32 bit platforms.
- Add the MemEncryptHypercallLib-related files to Maintainers.txt,
in section "OvmfPkg: Confidential Computing".
- Add support for the new KVM_HC_MAP_GPA_RANGE hypercall interface.
- Add patch for SEV live migration support.

Changes since v2:
- GHCB_BASE setup during reset-vector as decrypted is marked explicitly
in the hypervisor page encryption bitmap after setting the
PcdSevEsIsEnabled PCD.

Changes since v1:
- Mark GHCB_BASE setup during reset-vector as decrypted explicitly in
the hypervisor page encryption bitmap.
- Resending the series with correct shallow threading.

Ashish Kalra (6):
OvmfPkg/BaseMemEncryptLib: Detect SEV live migration feature.
OvmfPkg/BaseMemEncryptLib: Hypercall API for page encryption state
change
OvmfPkg/BaseMemEncryptLib: Invoke page encryption state change
hypercall
OvmfPkg/VmgExitLib: Encryption state change hypercall support in VC
handler
OvmfPkg/PlatformPei: Mark SEC GHCB page as unencrypted via hypercall
OvmfPkg/AmdSevDxe: Add support for SEV live migration.

OvmfPkg/AmdSevDxe/AmdSevDxe.c | 64 +++++++++++++++++
OvmfPkg/AmdSevDxe/AmdSevDxe.inf | 4 ++
OvmfPkg/Include/Guid/AmdSevMemEncryptLib.h | 20 ++++++
OvmfPkg/Include/Library/MemEncryptSevLib.h | 70 +++++++++++++++++++
.../DxeMemEncryptSevLib.inf | 1 +
.../DxeMemEncryptSevLibInternal.c | 39 +++++++++++
.../Ia32/MemEncryptSevLib.c | 27 +++++++
.../PeiDxeMemEncryptSevLibInternal.c | 52 ++++++++++++++
.../PeiMemEncryptSevLib.inf | 1 +
.../PeiMemEncryptSevLibInternal.c | 39 +++++++++++
.../SecMemEncryptSevLibInternal.c | 38 ++++++++++
.../X64/AsmHelperStub.nasm | 33 +++++++++
.../X64/MemEncryptSevLib.c | 62 ++++++++++++++++
.../X64/PeiDxeVirtualMemory.c | 20 ++++++
OvmfPkg/Library/VmgExitLib/VmgExitVcHandler.c | 13 ++++
OvmfPkg/OvmfPkg.dec | 1 +
OvmfPkg/PlatformPei/AmdSev.c | 11 +++
17 files changed, 495 insertions(+)
create mode 100644 OvmfPkg/Include/Guid/AmdSevMemEncryptLib.h
create mode 100644
OvmfPkg/Library/BaseMemEncryptSevLib/X64/AsmHelperStub.nasm

--
2.17.1





Ashish Kalra
 

On Thu, Aug 05, 2021 at 05:17:15AM +0000, Yao, Jiewen wrote:
Hi
I have some questions:

1) May I know what is the usage of this UEFI variable - SevLiveMigrationEnabled?
I only see it is created, but I do not see how it is consumed.
Yes, it is created by UEFI but consumed by the (guest) linux kernel.

This is actually part of a 3-way negotiation of the live migration
feature between hypervisor, guest OVMF and guest kernel. Host indicates
support for live migration, which is detected by OVMF and
correspondingly OVMF sets this SetLiveMigrationEnabled UEFI variable,
which is read by the guest kernel and it indicates to the guest kernel
that both host and OVMF support and have enabled the live migration
feature.

The guest flow for detecting live migration feature is captured below
(from the corresponding kernel patch):

The guest support for detecting and enabling SEV Live migration
feature uses the following logic :

- kvm_init_plaform() checks if its booted under the EFI

- If not EFI,

i) if kvm_para_has_feature(KVM_FEATURE_MIGRATION_CONTROL), issue a wrmsrl()
to enable the SEV live migration support

- If EFI,

i) If kvm_para_has_feature(KVM_FEATURE_MIGRATION_CONTROL), read
the UEFI variable which indicates OVMF support for live migration
(SetLiveMigrationEnabled)

ii) the variable indicates live migration is supported, issue a wrmsrl() to
enable the SEV live migration support


2) Is this a full live migration patch, or is this just a startup and there will be more on the way?
Yes. Actually this is hypervisor based live migration, so it relies on
host KVM and qemu support and it mainly requires OVMF support to make
hypercalls for any decrypted memory regions it sets up, for example,
MMIO and non-existant memory regions, so that hypervisor can keep track
of guest firmware's page encryption status.

So this is the full patch-set for UEFI for PSP based live migration.

Thanks,
Ashish

Thank you
Yao Jiewen


-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Ashish Kalra
via groups.io
Sent: Monday, August 2, 2021 8:31 PM
To: devel@edk2.groups.io
Cc: dovmurik@linux.vnet.ibm.com; brijesh.singh@amd.com; tobin@ibm.com;
Thomas.Lendacky@amd.com; jejb@linux.ibm.com; Justen, Jordan L
<jordan.l.justen@intel.com>; ard.biesheuvel@arm.com;
erdemaktas@google.com; Yao, Jiewen <jiewen.yao@intel.com>; Xu, Min M
<min.m.xu@intel.com>
Subject: [edk2-devel] [PATCH v6 0/6] SEV Live Migration support for OVMF.

From: Ashish Kalra <ashish.kalra@amd.com>

BZ: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D3467&;data=04%7C01%7Cashish.kalra%40amd.com%7Cd4bf7bc0c17840d8a25208d957d04dcd%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637637374441109760%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=238zqkur1zpuKDorzZLrzcbTNhlO6gM8aFoi7Zr%2BUao%3D&amp;reserved=0

By default all the SEV guest memory regions are considered encrypted,
if a guest changes the encryption attribute of the page (e.g mark a
page as decrypted) then notify hypervisor. Hypervisor will need to
track the unencrypted pages. The information will be used during
guest live migration, guest page migration and guest debugging.

The patch-set detects if it is running under KVM hypervisor and then
checks for SEV live migration feature support via KVM_FEATURE_CPUID,
if detected setup a new UEFI enviroment variable to indicate OVMF
support for SEV live migration.

A branch containing these patches is available here:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fashkalra%2Fedk2-1%2Ftree%2Fsev_live_migration_v5_10&;data=04%7C01%7Cashish.kalra%40amd.com%7Cd4bf7bc0c17840d8a25208d957d04dcd%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637637374441109760%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=PyChejF9mzIVLCvQiCW5olQ8LOFWM3%2FGGO938qL80f8%3D&amp;reserved=0

Changes since v5:
- Split first patch into three components, one patch for the
MemEncryptSevLiveMigrationIsEnabled() API, one patch for the
SetMemoryEncDecHypercall3() API, one patch to make use of the
SetMemoryEncDecHypercall3() API.
- Fix patch subject, in code and patch comments and
additionally add relevant comments.
- Replace SetMemoryEncDecHypercall3() API's Status argument
with a boolean IsEncrypted argument and corresponding fixes
to users of this API call.
- Fix AsciiStrCmp() usage in KVM hypervisor detection code.

Changes since v4:
- Remove MemEncryptHypercallLib Library and add support to issue
hypercall in the BaseMemEncryptSevLib library itself.
- For SEV-ES, make the VC handler hypercall aware by comparing
the hypercall number and add the additional register values
in the GHCB.
- Fix comments in the hypercall API interface.
- The encryption bit is set/clear on the smallest page size, hence
use the 4k page size in MAP_GPA_RANGE hypercall.
- Make the hypercall expect the guest physical address to be
page-aligned.
- Add KVM live migration feature flag check in BaseMemEncryptSevLib
library similar to how BaseMemEncryptSevLib does for the
MemEncryptSevIsEnabled() and check it before invoking HC. Also
export the MemEncryptSevLiveMigrationIsEnabled() function as
part of the library.
- Add error handling on hypercall return, on failure, return error
code to caller which potentially will cause an assert() and
terminate the boot.

Changes since v3:
- Fix all DSC files under OvmfPkg except X64 to add support for
BaseMemEncryptLib and add NULL instance of BaseMemEncryptLib
for 32 bit platforms.
- Add the MemEncryptHypercallLib-related files to Maintainers.txt,
in section "OvmfPkg: Confidential Computing".
- Add support for the new KVM_HC_MAP_GPA_RANGE hypercall interface.
- Add patch for SEV live migration support.

Changes since v2:
- GHCB_BASE setup during reset-vector as decrypted is marked explicitly
in the hypervisor page encryption bitmap after setting the
PcdSevEsIsEnabled PCD.

Changes since v1:
- Mark GHCB_BASE setup during reset-vector as decrypted explicitly in
the hypervisor page encryption bitmap.
- Resending the series with correct shallow threading.

Ashish Kalra (6):
OvmfPkg/BaseMemEncryptLib: Detect SEV live migration feature.
OvmfPkg/BaseMemEncryptLib: Hypercall API for page encryption state
change
OvmfPkg/BaseMemEncryptLib: Invoke page encryption state change
hypercall
OvmfPkg/VmgExitLib: Encryption state change hypercall support in VC
handler
OvmfPkg/PlatformPei: Mark SEC GHCB page as unencrypted via hypercall
OvmfPkg/AmdSevDxe: Add support for SEV live migration.

OvmfPkg/AmdSevDxe/AmdSevDxe.c | 64 +++++++++++++++++
OvmfPkg/AmdSevDxe/AmdSevDxe.inf | 4 ++
OvmfPkg/Include/Guid/AmdSevMemEncryptLib.h | 20 ++++++
OvmfPkg/Include/Library/MemEncryptSevLib.h | 70 +++++++++++++++++++
.../DxeMemEncryptSevLib.inf | 1 +
.../DxeMemEncryptSevLibInternal.c | 39 +++++++++++
.../Ia32/MemEncryptSevLib.c | 27 +++++++
.../PeiDxeMemEncryptSevLibInternal.c | 52 ++++++++++++++
.../PeiMemEncryptSevLib.inf | 1 +
.../PeiMemEncryptSevLibInternal.c | 39 +++++++++++
.../SecMemEncryptSevLibInternal.c | 38 ++++++++++
.../X64/AsmHelperStub.nasm | 33 +++++++++
.../X64/MemEncryptSevLib.c | 62 ++++++++++++++++
.../X64/PeiDxeVirtualMemory.c | 20 ++++++
OvmfPkg/Library/VmgExitLib/VmgExitVcHandler.c | 13 ++++
OvmfPkg/OvmfPkg.dec | 1 +
OvmfPkg/PlatformPei/AmdSev.c | 11 +++
17 files changed, 495 insertions(+)
create mode 100644 OvmfPkg/Include/Guid/AmdSevMemEncryptLib.h
create mode 100644
OvmfPkg/Library/BaseMemEncryptSevLib/X64/AsmHelperStub.nasm

--
2.17.1