On Tue, 27 Aug 2019 22:11:15 +0200
Laszlo Ersek <lersek@...> wrote:

On 08/27/19 18:23, Igor Mammedov wrote:

On Mon, 26 Aug 2019 17:30:43 +0200
Laszlo Ersek <lersek@...> wrote:

On 08/23/19 17:25, Kinney, Michael D wrote:

Hi Jiewen,

If a hot add CPU needs to run any code before the
first SMI, I would recommend is only executes code
from a write protected FLASH range without a stack
and then wait for the first SMI.

"without a stack" looks very risky to me. Even if we manage to implement
the guest code initially, we'll be trapped without a stack, should we
ever need to add more complex stuff there.

Do we need anything complex in relocation handler, though?
From what I'd imagine, minimum handler should
1: get address of TSEG, possibly read it from chipset

The TSEG base calculation is not trivial in this environment. The 32-bit
RAM size needs to be read from the CMOS (IO port accesses). Then the
extended TSEG size (if any) needs to be detected from PCI config space
(IO port accesses). Both CMOS and PCI config space requires IO port
writes too (not just reads). Even if there are enough registers for the
calculations, can we rely on these unprotected IO ports?

Also, can we switch to 32-bit mode without a stack? I assume it would be
necessary to switch to 32-bit mode for 32-bit arithmetic.

from SDM vol 3:
"
34.5.1 Initial SMM Execution Environment
After saving the current context of the processor, the processor initializes its core registers to the values shown in Table 34-4. Upon entering SMM, the PE and PG flags in control register CR0 are cleared, which places the processor in an environment similar to real-address mode. The differences between the SMM execution environment and the real-address mode execution environment are as follows:
• The addressable address space ranges from 0 to FFFFFFFFH (4 GBytes).
• The normal 64-KByte segment limit for real-address mode is increased to 4 GBytes.
• The default operand and address sizes are set to 16 bits, which restricts the addressable SMRAM address space to the 1-MByte real-address mode limit for native real-address-mode code. However, operand-size and address-size override prefixes can be used to access the address space beyond
^^^^^^^^
the 1-MByte.
"

Getting the initial APIC ID needs some CPUID instructions IIUC, which
clobber EAX through EDX, if I understand correctly. Given the register
pressure, CPUID might have to be one of the first instructions to call.

we could map at 30000 not 64K required for save area but 128K and use
2nd half as secure RAM for stack and intermediate data.

Firmware could put there pre-calculated pointer to TSEG after it's configured and locked down,
this way relocation handler won't have to figure out TSEG address on its own.

2: calculate its new SMBASE offset based on its APIC ID
3: save new SMBASE

For this OVMF use case, is any CPU init required
before the first SMI?

I expressed a preference for that too: "I wish we could simply wake the
new CPU [...] with an SMI".

http://mid.mail-archive.com/398b3327-0820-95af-a34d-1a4a1d50cf35@redhat.com

From Paolo's list of steps are steps (8a) and (8b)
really required?

07b - implies 08b

I agree about that implication, yes. *If* we send an INIT/SIPI/SIPI to
the new CPU, then the new CPU needs a HLT loop, I think.

It also could execute INIT reset, which leaves initialized SMM untouched
but otherwise CPU would be inactive.

8b could be trivial hlt loop and we most likely could skip 08a and signaling host CPU steps
but we need INIT/SIPI/SIPI sequence to wake up AP so it could handle pending SMI
before handling SIPI (so behavior would follow SDM).

See again my message linked above -- just after the quoted sentence, I
wrote, "IOW, if we could excise steps 07b, 08a, 08b".

But, I obviously defer to Paolo and Igor on that.

(I do believe we have a dilemma here. In QEMU, we probably prefer to
emulate physical hardware as faithfully as possible. However, we do not
have Cache-As-RAM (nor do we intend to, IIUC). Does that justify other
divergences from physical hardware too, such as waking just by virtue of
an SMI?)

So far we should be able to implement it per spec (at least SDM one),
but we would still need to invent chipset hardware
i.e. like adding to Q35 non exiting SMRAM and means to map/unmap it
to non-SMM address space.
(and I hope we could avoid adding "parked CPU" thingy)

I think we'll need a separate QEMU tree for this. I'm quite in the dark
-- I can't tell if I'll be able to do something in OVMF without actually
trying it. And for that, we'll need some proposed QEMU code that is
testable, but not upstream yet. (As I might realize that I'm unable to
make it work in OVMF.)

Let me prepare a QEMU branch with something usable for you.

To avoid inventing mgmt API for configuring SMRAM at 30000,
I'm suggesting to steal/alias top or bottom 128K of TSEG window to 30000.
This way OVMF would be able to set SMI relocation handler modifying
TSEG and pass TSEG base/other data to it as well.
Would it work for you or should we try more elaborate approach?

Can the SMI monarch use the Local
APIC to send a directed SMI to the hot added CPU?
The SMI monarch needs to know the APIC ID of the
hot added CPU. Do we also need to handle the case
where multiple CPUs are added at once? I think we
would need to serialize the use of 3000:8000 for the
SMM rebase operation on each hot added CPU.

I agree this would be a huge help.

We can serialize it (for normal hotplug flow) from ACPI handler
in the guest (i.e. non enforced serialization).
The only reason for serialization I see is not to allow
a bunch of new CPU trample over default SMBASE save area
at the same time.

If the default SMBASE area is corrupted due to concurrent access, could
that lead to invalid relocated SMBASE values? Possibly pointing into
normal RAM?

in case of broadcast SMI (btw does OVMF use broadcast SMIs?) several CPUs could end up
with the same SMBASE within SMRAM
1: default one: in case the 2nd CPU enters SMM after the 1st CPU saved new SMBASE but before it's called RSM
2: duplicated SMBASE: where the 2nd CPU saves its new SMBASE before the 1st calls RSM

while the 2nd could be counteracted with using locks, I don't see how 1st one could be avoided.
May be host CPU can send 2nd SMI so just relocated CPU could send an ACK from relocated SMBASE/with new SMI handler?

Thanks
Laszlo