-----Original Message-----
From: Laszlo Ersek [mailto:lersek@redhat.com]
Sent: Wednesday, August 14, 2019 12:09 AM
To: edk2-devel-groups-io <devel@edk2.groups.io>
Cc: edk2-rfc-groups-io <rfc@edk2.groups.io>; qemu devel list
<qemu-devel@nongnu.org>; Igor Mammedov <imammedo@redhat.com>;
Paolo Bonzini <pbonzini@redhat.com>; Yao, Jiewen
<jiewen.yao@intel.com>; Chen, Yingwen <yingwen.chen@intel.com>;
Nakajima, Jun <jun.nakajima@intel.com>; Boris Ostrovsky
<boris.ostrovsky@oracle.com>; Joao Marcal Lemos Martins
<joao.m.martins@oracle.com>; Phillip Goerl <phillip.goerl@oracle.com>
Subject: Re: CPU hotplug using SMM with QEMU+OVMF

On 08/13/19 16:16, Laszlo Ersek wrote:

Yingwen and Jiewen suggested the following process.

Legend:

- "New CPU": CPU being hot-added
- "Host CPU": existing CPU
- (Flash): code running from flash
- (SMM): code running from SMRAM

Steps:

(01) New CPU: (Flash) enter reset vector, Global SMI disabled by
default.

- What does "Global SMI disabled by default" mean? In particular, what
is "global" here?

[Jiewen] OK. Let's don’t use the term "global".

Do you mean that the CPU being hot-plugged should mask (by default)
broadcast SMIs? What about directed SMIs? (An attacker could try that
too.)

[Jiewen] I mean all SMIs are blocked for this specific hot-added CPU.

And what about other processors? (I'd assume step (01)) is not
relevant for other processors, but "global" is quite confusing here.)

[Jiewen] No impact to other processors.

- Does this part require a new branch somewhere in the OVMF SEC code?
How do we determine whether the CPU executing SEC is BSP or
hot-plugged AP?

[Jiewen] I think this is blocked from hardware perspective, since the first instruction.
There are some hardware specific registers can be used to determine if the CPU is new added.
I don’t think this must be same as the real hardware.
You are free to invent some registers in device model to be used in OVMF hot plug driver.

- How do we tell the hot-plugged AP where to start execution? (I.e. that
it should execute code at a particular pflash location.)

[Jiewen] Same real mode reset vector at FFFF:FFF0.

For example, in MpInitLib, we start a specific AP with INIT-SIPI-SIPI,
where "SIPI" stores the startup address in the "Interrupt Command
Register" (which is memory-mapped in xAPIC mode, and an MSR in x2APIC
mode, apparently). That doesn't apply here -- should QEMU auto-start
the new CPU?

[Jiewen] You can send INIT-SIPI-SIPI to new CPU only after it can access memory.
SIPI need give AP an below 1M memory address as waking vector.

- What memory is used as stack by the new CPU, when it runs code from
flash?

[Jiewen] Same as other CPU in normal boot. You can use special reserved memory.

QEMU does not emulate CAR (Cache As RAM). The new CPU doesn't have
access to SMRAM. And we cannot use AcpiNVS or Reserved memory,
because
a malicious OS could use other CPUs -- or PCI device DMA -- to attack
the stack (unless QEMU forcibly paused other CPUs upon hotplug; I'm
not sure).

[Jiewen] Excellent point!
I don’t think there is problem for real hardware, who always has CAR.
Can QEMU provide some CPU specific space, such as MMIO region?

- If an attempt is made to hotplug multiple CPUs in quick succession,
does something serialize those attempts?

[Jiewen] The BIOS need consider this as availability requirement.
I don’t have strong opinion.
You can design a system that required hotplug must be one-by-one, or fail the hot-add.
Or you can design a system that did not have such restriction.
Again, all we need to do is to maintain the integrity of SMM.
The availability should be considered as separate requirement.

Again, stack usage could be a concern, even with Cache-As-RAM --
HyperThreads (logical processors) on a single core don't have
dedicated cache.

[Jiewen] Agree with you on the virtual environment.
For real hardware, we do socket level hot-add only. So HT is not the concern.
But if you want to do that in virtual environment, a processor specific memory
should be considered.

Does CPU hotplug apply only at the socket level? If the CPU is
multi-core, what is responsible for hot-plugging all cores present in
the socket?

[Jiewen] Ditto.

(02) New CPU: (Flash) configure memory control to let it access global
host memory.

In QEMU/KVM guests, we don't have to enable memory explicitly, it just
exists and works.

In OVMF X64 SEC, we can't access RAM above 4GB, but that shouldn't be an
issue per se.

[Jiewen] Agree. I do not see the issue.

(03) New CPU: (Flash) send board message to tell host CPU (GPIO->SCI)
-- I am waiting for hot-add message.

Maybe we can simplify this in QEMU by broadcasting an SMI to existent
processors immediately upon plugging the new CPU.

(NOTE: Host CPU can only

send

instruction in SMM mode. -- The register is SMM only)

Sorry, I don't follow -- what register are we talking about here, and
why is the BSP needed to send anything at all? What "instruction" do you
have in mind?

[Jiewen] The new CPU does not enable SMI at reset.
At some point of time later, the CPU need enable SMI, right?
The "instruction" here means, the host CPUs need tell to CPU to enable SMI.

(04) Host CPU: (OS) get message from board that a new CPU is added.
(GPIO -> SCI)

(05) Host CPU: (OS) All CPUs enter SMM (SCI->SWSMI) (NOTE: New CPU
will not enter CPU because SMI is disabled)

I don't understand the OS involvement here. But, again, perhaps QEMU can
force all existent CPUs into SMM immediately upon adding the new CPU.

[Jiewen] OS here means the Host CPU running code in OS environment, not in SMM environment.

(06) Host CPU: (SMM) Save 38000, Update 38000 -- fill simple SMM
rebase code.

(07) Host CPU: (SMM) Send message to New CPU to Enable SMI.

Aha, so this is the SMM-only register you mention in step (03). Is the
register specified in the Intel SDM?

[Jiewen] Right. That is the register to let host CPU tell new CPU to enable SMI.
It is platform specific register. Not defined in SDM.
You may invent one in device model.

(08) New CPU: (Flash) Get message - Enable SMI.

(09) Host CPU: (SMM) Send SMI to the new CPU only.

(10) New CPU: (SMM) Response first SMI at 38000, and rebase SMBASE to
TSEG.

What code does the new CPU execute after it completes step (10)? Does it
halt?

[Jiewen] The new CPU exits SMM and return to original place - where it is
interrupted to enter SMM - running code on the flash.

(11) Host CPU: (SMM) Restore 38000.

These steps (i.e., (06) through (11)) don't appear RAS-specific. The
only platform-specific feature seems to be SMI masking register, which
could be extracted into a new SmmCpuFeaturesLib API.

Thus, would you please consider open sourcing firmware code for steps
(06) through (11)?

Alternatively -- and in particular because the stack for step (01)
concerns me --, we could approach this from a high-level, functional
perspective. The states that really matter are the relocated SMBASE for
the new CPU, and the state of the full system, right at the end of step
(11).

When the SMM setup quiesces during normal firmware boot, OVMF could
use
existent (finalized) SMBASE infomation to *pre-program* some virtual
QEMU hardware, with such state that would be expected, as "final" state,
of any new hotplugged CPU. Afterwards, if / when the hotplug actually
happens, QEMU could blanket-apply this state to the new CPU, and
broadcast a hardware SMI to all CPUs except the new one.

The hardware SMI should tell the firmware that the rest of the process
-- step (12) below, and onward -- is being requested.

If I understand right, this approach would produce an firmware & system
state that's identical to what's expected right after step (11):

- all SMBASEs relocated
- all preexistent CPUs in SMM
- new CPU halted / blocked from launch
- DRAM at 0x30000 / 0x38000 contains OS-owned data

Is my understanding correct that this is the expected state after step
(11)?

[Jiewen] I think you are correct.

Three more comments on the "SMBASE pre-config" approach:

- the virtual hardware providing this feature should become locked after
the configuration, until next platform reset

- the pre-config should occur via simple hardware accesses, so that it
can be replayed at S3 resume, i.e. as part of the S3 boot script

- from the pre-configured state, and the APIC ID, QEMU itself could
perhaps calculate the SMI stack location for the new processor.

(12) Host CPU: (SMM) Update located data structure to add the new CPU
information. (This step will involve CPU_SERVICE protocol)

I commented on EFI_SMM_CPU_SERVICE_PROTOCOL in upon bullet (4) of
<https://bugzilla.tianocore.org/show_bug.cgi?id=1512#c4>.

Calling EFI_SMM_ADD_PROCESSOR looks justified.

[Jiewen] I think you are correct.
Also REMOVE_PROCESSOR will be used for hot-remove action.

What are some of the other member functions used for? The scary one is
EFI_SMM_REGISTER_EXCEPTION_HANDLER.

[Jiewen] This is to register a new exception handler in SMM.
I don’t think this API is involved in hot-add.

===================== (now, the next SMI will bring all CPU into TSEG)

OK... but what component injects that SMI, and when?

[Jiewen] Any SMI event. It could be synchronized SMI or asynchronized SMI.
It could from software such as IO write, or hardware such as thermal event.

(13) New CPU: (Flash) run MRC code, to init its own memory.

Why is this needed esp. after step (10)? The new CPU has accessed DRAM
already. And why are we executing code from pflash, rather than from
SMRAM, given that we're past SMBASE relocation?

[Jiewen] On real hardware, it is needed because different CPU may have different capability to access different DIMM.
I do not think your virtual platform need it.

(14) New CPU: (Flash) Deadloop, and wait for INIT-SIPI-SIPI.

(15) Host CPU: (OS) Send INIT-SIPI-SIPI to pull new CPU in.

I'm confused by these steps. I thought that step (12) would complete the
hotplug, by updating the administrative data structures internally. And
the next SMI -- raised for the usual purposes, such as a software SMI
for variable access -- would be handled like it always is, except it
would also pull the new CPU into SMM too.

[Jiewen] The OS need use the new CPU at some point of time, right?
As such, the OS need pull the new CPU into its own environment by INIT-SIPI-SIPI.

Thanks!
Laszlo