On 9/18/21 10:14 PM, Xu, Min M wrote:

Hi, Brijesh
On September 18, 2021 7:30 PM, Brijesh Singh wrote:

Hi Min,

On 9/18/21 12:16 AM, Xu, Min M wrote:

Hi, Brijesh

On September 17, 2021 11:52 PM, Brijesh Singh wrote:

Hi Min,

On 9/17/21 7:55 AM, Xu, Min M wrote:
...

As I mentioned in my last mail, in the beginning I missed the
limitation of

smsw.

So I update the code (ResetVectorVtf0.asm) as below using mov CRx.
<1> BITS 16
176 00000800 0F20C0 <1> mov eax, cr0 <-- previously it

was smsw

177 00000803 A801 <1> test al, 1
178 00000805 7405 <1> jz .Real
179 <1> BITS 32
180 00000807 E951FFFFFF <1> jmp Main32
181 <1> BITS 16
182 <1> .Real:
183 0000080C E939FF <1> jmp EarlyBspInitReal16

I test the code in a AMD SEV server and try to launch a SEV guest.
This time

it stuck at the *mov eax, cr0*.

I am curious if *mov eax, cr0* works in real mode in a SEV guest?
I also test the code in a legacy vm guest and td guest, all passed.
Did I miss something?

Hmm, I am not aware of any limitation w.r.t encrypted VMs. I just
added the below code in my branch and I do not see any issues, my
SEV, SEV-ES and SEV-SNP all are able to boot fine. And KVM trace
confirms that code it read

diff --git a/OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm
b/OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm
index f0e509d0672e..98e34332b04c 100644
--- a/OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm
+++ b/OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm
@@ -175,9 +175,21 @@ resetVector:
;
; This is where the processor will begin execution
;
+%ifdef ARCH_IA32
nop
nop
jmp EarlyBspInitReal16
+%else
+ mov eax, cr0
+ test al, 1
+ jz .Real
+BITS 32
+ hlt
+ ;jmp Main32
+BITS 16
+.Real:
+ jmp EarlyBspInitReal16
+%endif

ALIGN 16


And KVM trace:

kvm_exit: vcpu 0 reason npf rip 0xfff0 info1 0x0000000500000014 info2
0x00000000fffff000 intr_info 0x00000000 error_code 0x00000000
kvm_page_fault: address fffff000 error_code 500000014
kvm_entry: vcpu 0, rip 0xfff0
kvm_exit: vcpu 0 reason read_cr0 rip 0xfff0 info1 0x8000000000000000
info2 0x0000000000000000 intr_info 0x00000000 error_code 0x00000000
kvm_cr: cr_read 0 = 0x60000010
kvm_entry: vcpu 0, rip 0xfff3

As we can see from the kvm trace, the first instruction here is the
Cr0 read and it was successfully intercepted and rip moved to next

instruction.

Can you please provide me KVM trace for your failure case ? Also,
provide me the output of "lscpu" and "dmesg" from the host.

The OVMF image you tested is built with GCC tool chain, right?

Yes, we have been using the GCC tool chain only.

Is VS Tool chain (VS2017, VS2019, etc) supported by AMD SEV in OVMF?
I am a little nervous when the Ovmf img failed to be launched in AMD SEV
server after my TDX patch is applied.

I usually do the development in windows and build the OVMF image with

VS2019.

If the new feature works, then I cherry-pick the patch-sets to code
base in ubuntu
18.04 and build/test the new feature.

The weird thing is that, with VS2019, even the OVMF image is built
from edk2-master, such image doesn't work on AMD SEV server either.
But if the image is built by Ubuntu 18.04, it does work on AMD SEV server.

This seems very strange that we are failing to execute the hand written
assembly code.

Actually even the OvmfPkg from edk2-master (without any changes) cannot be
launched on AMD SEV server if it is built with VS2019 tool chain.
This is the qemu-kvm used:
$/usr/libexec/qemu-kvm --version
QEMU emulator version 4.2.0 (qemu-kvm-4.2.0-48.module_el8.4.0+885+5e18b468.3)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers
This is the launch scripts.
QEMU=/usr/libexec/qemu-kvm
DRIVE=rhel-8.qcow2
${QEMU} \
-enable-kvm -cpu EPYC -machine q35 \
-smp 4,maxcpus=64 \
-m 4096M,slots=5,maxmem=30G \
-drive if=pflash,format=raw,unit=0,file=OVMF_CODE.fd,readonly \
-drive if=pflash,format=raw,unit=1,file=OVMF_VARS.fd \
-netdev user,id=vmnic \
-device e1000,netdev=vmnic,romfile= \
-drive file=${DRIVE},if=none,id=disk0,format=qcow2 \
-device virtio-scsi-pci,id=scsi,disable-legacy=on,iommu_platform=true \
-device scsi-hd,drive=disk0 \
-object sev-guest,id=sev0,cbitpos=47,reduced-phys-bits=1 \
-machine memory-encryption=sev0 \
-nographic

I am wondering if somehow the VS compiler is generating a
wrong byte code and thus causing a trap on KVM that requires emulation.
Since the guest memory is encrypted, so KVM emulation code will not be
able to decode the instruction bytes and thus leading in repetitive nested
fault. Only way I could verify my theory is if I can get a KVM trace or an OVMF
binary. If you have have KVM trace or OVMF_CODE.fd handy then please
share.

The OVMF_CODE.fd and OVMF_VARS.fd are attached.
The code base is :
ac6388add4 2021-09-15 (HEAD -> master, origin/master, origin/HEAD) ArmPkg/ProcessorSubClassDxe: Fix the format of ProcessorId
This is the build command:
build -p OvmfPkgX64.dsc -a X64 -t VS2019

Actually, I am not able to boot a non SEV guest with your attached binary. It appears that sometime during an easy boot, we are getting a triple fault (maybe guest is accessing invalid memory) and thus KVM issues a SHUTDOWN. The same is seen with the SEV guest.

I don't know what VS compiler is doing to cause this.

-Brijesh