Date   

[edk2-platforms]Vlv2TbltDevicePkg built failed for 'error: symbol `InternalX86DisablePaging64.0' undefined'.

gordontcp@...
 

Hi all,
I compiled Vlv2TbltDevicePkg in Windows 10 and operated according to the following URL:
https://github.com/tianocore/edk2-platforms/tree/master/Platform/Intel/Vlv2TbltDevicePkg
The compiled command is as follows:
build -a IA32 -a X64 -n 5 -t VS2019 -b DEBUG -p Vlv2TbltDevicePkg\PlatformPkgX64.dsc
The result is the following error message:
edk2p\Build\Vlv2TbltDevicePkgX64\DEBUG_VS2010x86\X64\MdePkg\Library\BaseLib\BaseLib\OUTPUT\X64\DisablePaging64.iii:35: error: symbol `InternalX86DisablePaging64.0' undefined
NMAKE : fatal error U1077: 'c:\nasm\nasm.EXE' : return code '0x1'

My version of Nasm is nasm-2.13.03, how can I solve the error about nasm?
Any suggestion is greatly appreciated!
Thanks!


Re: build command not found

Andrew Fish
 

On May 27, 2022, at 9:55 AM, Schultschik, Sven <sven.schultschik@...> wrote:

Hi all,

I currently work on integrate the edk2 build within a buildsystem.

I startet with the docu, made manual builds and wrote me a bash script to automate the steps. Worked all very well.

The first issue I had was a not correctly running "source edk2/edksetup.sh". I fixed by switching to /bin/bash from /bin/sh
The second issue was the failing tests which I fixed in setting manually the PYTHON_COMMAND to python3

Now the current issue which is still left.

build -p "Platform/StandaloneMm/PlatformStandaloneMmPkg/PlatformStandaloneMmRpmb.dsc" -b RELEASE -a AARCH64 -t GCC5 -n `nproc`
/bin/bash: line 1: build: command not found

For some reason the build command is not available. Doing the same steps manually it works, but not in the build environment.
All steps bevor are running without any error.

I'm currently looking for the place, where the build command get's "created and published".
When you source edksetup.sh It should put the path to the build command in your $PATH. Here is an example from my Mac:
/Volumes/Case/edk2/BaseTools/BinWrappers/PosixLike

We can see that is how it works from the command line.
/Volumes/Case/edk2(master)>which build
/Volumes/Case/edk2/BaseTools/BinWrappers/PosixLike/build

Also the -n takes a number of threads so it looks like ’nproc’ did not expand properly to a number.

I wonder why the edk2 build is so picky in the build environment I try to run it.
Yikes looks like edksetup.sh is missing the shebang. I’m guessing we ran some script to update headers at some point and removed it?

Could you file a Bugzilla for the missing shebang in edksetup.sh?

Thanks,

Andrew Fish

Thank you for any hints.

Regards

Sven





build command not found

Schultschik, Sven
 

Hi all,

I currently work on integrate the edk2 build within a buildsystem.

I startet with the docu, made manual builds and wrote me a bash script to automate the steps. Worked all very well.

The first issue I had was a not correctly running "source edk2/edksetup.sh". I fixed by switching to /bin/bash from /bin/sh
The second issue was the failing tests which I fixed in setting manually the PYTHON_COMMAND to python3

Now the current issue which is still left.

build -p "Platform/StandaloneMm/PlatformStandaloneMmPkg/PlatformStandaloneMmRpmb.dsc" -b RELEASE -a AARCH64 -t GCC5 -n `nproc`
/bin/bash: line 1: build: command not found

For some reason the build command is not available. Doing the same steps manually it works, but not in the build environment.
All steps bevor are running without any error.

I'm currently looking for the place, where the build command get's "created and published".

I wonder why the edk2 build is so picky in the build environment I try to run it.

Thank you for any hints.

Regards

Sven


Re: Failling tests on make BaseTools

Schultschik, Sven
 

Hi Rebecca,

thank you that was a very good hint. I now set the PYTHON_COMMAND manually to python3 and the tests working.

But the build behaves really odd in that environment, but that's another topic.

Regards Sven

-----Ursprüngliche Nachricht-----
Von: Rebecca Cran <quic_rcran@...>
Gesendet: Donnerstag, 26. Mai 2022 00:04
An: discuss@edk2.groups.io; Schultschik, Sven (DI PA DCP R&D 2) <sven.schultschik@...>
Betreff: Re: [edk2-discuss] Failling tests on make BaseTools

My initial guess would be that for some reason it's trying to use Python 2.x instead of 3.x.

--
Rebecca Cran

On 5/25/22 15:04, Schultschik, Sven wrote:
Hi all,

I currently work on integrate the edk2 build within a buildsystem.

I startet with the docu, made manual builds and wrote me a bash script to automate the steps. Worked very well till now.

Then I tried to add it to the build system and some of tests of the "make -C edks/BaseTools" fail now.

The checkout source is the same, the python version, the gcc version is the same.

I wonder why the tests are failing on one machine and are ok on the other ...

Thank you for any hints

======================================================================
FAIL: test_FMMT_FMMT (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/FMMT.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/FMMT.py", line 48
def __init__(self) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_BinaryFactoryProduct (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/BinaryFactoryProduct.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/BinaryFactoryProduct.py", line 34
type:list = []
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_BiosTree (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/BiosTree.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/BiosTree.py", line 31
def __init__(self, NodeName: str) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_BiosTreeNode (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/BiosTreeNode.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/BiosTreeNode.py", line 34
def __init__(self, name: str) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_FMMTOperation (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/FMMTOperation.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/FMMTOperation.py", line 16
def ViewFile(inputfile: str, ROOT_TYPE: str, layoutfile: str=None, outputfile: str=None) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_FMMTParser (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/FMMTParser.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/FMMTParser.py", line 15
def __init__(self, name: str, TYPE: str) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_FvHandler (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/FvHandler.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/FvHandler.py", line 16
def ChangeSize(TargetTree, size_delta: int=0) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_GuidTools (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/GuidTools.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/GuidTools.py", line 18
def ExecuteCommand(cmd: list) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_utils_FvLayoutPrint (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/utils/FvLayoutPrint.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/utils/FvLayoutPrint.py", line 9
def GetFormatter(layout_format: str):
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FirmwareStorageFormat_Common (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/Common.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/Common.py", line 42
def from_list(self, listformat: list) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FirmwareStorageFormat_FfsFileHeader
(CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/FfsFileHeader.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/FfsFileHeader.py", line 41
def FFS_FILE_SIZE(self) -> int:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FirmwareStorageFormat_FvHeader (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/FvHeader.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/FvHeader.py", line 77
def Refine_FV_EXT_ENTRY_OEM_TYPE_Header(nums: int) -> EFI_FIRMWARE_VOLUME_EXT_ENTRY_OEM_TYPE:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FirmwareStorageFormat_SectionHeader
(CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/SectionHeader.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/SectionHeader.py", line 22
def SECTION_SIZE(self) -> int:
^
SyntaxError: invalid syntax





How to generate UEFI capsule image and test UEFI capsule update?

gordontcp@...
 

Hi all,
We are planning to develop UEFI capsule F/W update, and we have seen \FmpDevicePkg\FmpDxe and MdeModulePkg\Application\CapsuleApp in https://github.com/tianocore/edk2. The examples are both related to capsule update issue. The problems we have are:

1. How to generate capsule image? Is there any sample code or operation flow?
2. Is it enough only to modify with FmpDxe and CapsuleApp(in edk2) for developing capsule update? Do other examples need to be considered? Such as \MdeModulePkg\Universal\EsrtFmpDxe in edk2 example.
3. How to test capsule F/W update process in UEFI shell? Is there a sample code for the app?

Any suggesting is highly appreciated!
Thanks!


Re: Failling tests on make BaseTools

Rebecca Cran
 

My initial guess would be that for some reason it's trying to use Python 2.x instead of 3.x.

--
Rebecca Cran

On 5/25/22 15:04, Schultschik, Sven wrote:
Hi all,

I currently work on integrate the edk2 build within a buildsystem.

I startet with the docu, made manual builds and wrote me a bash script to automate the steps. Worked very well till now.

Then I tried to add it to the build system and some of tests of the "make -C edks/BaseTools" fail now.

The checkout source is the same, the python version, the gcc version is the same.

I wonder why the tests are failing on one machine and are ok on the other ...

Thank you for any hints

======================================================================
FAIL: test_FMMT_FMMT (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/FMMT.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/FMMT.py", line 48
def __init__(self) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_BinaryFactoryProduct (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/BinaryFactoryProduct.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/BinaryFactoryProduct.py", line 34
type:list = []
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_BiosTree (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/BiosTree.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/BiosTree.py", line 31
def __init__(self, NodeName: str) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_BiosTreeNode (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/BiosTreeNode.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/BiosTreeNode.py", line 34
def __init__(self, name: str) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_FMMTOperation (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/FMMTOperation.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/FMMTOperation.py", line 16
def ViewFile(inputfile: str, ROOT_TYPE: str, layoutfile: str=None, outputfile: str=None) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_FMMTParser (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/FMMTParser.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/FMMTParser.py", line 15
def __init__(self, name: str, TYPE: str) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_FvHandler (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/FvHandler.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/FvHandler.py", line 16
def ChangeSize(TargetTree, size_delta: int=0) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_GuidTools (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/GuidTools.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/GuidTools.py", line 18
def ExecuteCommand(cmd: list) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_utils_FvLayoutPrint (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/utils/FvLayoutPrint.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/utils/FvLayoutPrint.py", line 9
def GetFormatter(layout_format: str):
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FirmwareStorageFormat_Common (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/Common.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/Common.py", line 42
def from_list(self, listformat: list) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FirmwareStorageFormat_FfsFileHeader (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/FfsFileHeader.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/FfsFileHeader.py", line 41
def FFS_FILE_SIZE(self) -> int:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FirmwareStorageFormat_FvHeader (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/FvHeader.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/FvHeader.py", line 77
def Refine_FV_EXT_ENTRY_OEM_TYPE_Header(nums: int) -> EFI_FIRMWARE_VOLUME_EXT_ENTRY_OEM_TYPE:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FirmwareStorageFormat_SectionHeader (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/SectionHeader.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/SectionHeader.py", line 22
def SECTION_SIZE(self) -> int:
^
SyntaxError: invalid syntax




Not able to Load Protocol gEfiWiFi2Protocol (Wifi 2)

Pranay Kadam
 

Hi Team,
I'm new to Edk2 platform. I am trying to write simple program to display available Wifi network in Uefi Shell.

I'm trying to load gEfiWiFi2ProtocolGuid Protocol using *LocateProtocol* function as below.
But I can see the return status as "NOT_FOUND". I don't know what am I missing here ?


DEBUG((DEBUG_INFO, "Loafing Protocol....."));
EFI_WIRELESS_MAC_CONNECTION_II_PROTOCOL *WifiNetwork;

Status = gBS->LocateProtocol(&gEfiWiFi2ProtocolGuid, NULL, (VOID**)(&WifiNetwork));

if(EFI_ERROR(Status))
{
DEBUG((DEBUG_ERROR, "Error %r",Status));
return Status;
}


Failling tests on make BaseTools

Schultschik, Sven
 

Hi all,

I currently work on integrate the edk2 build within a buildsystem.

I startet with the docu, made manual builds and wrote me a bash script to automate the steps. Worked very well till now.

Then I tried to add it to the build system and some of tests of the "make -C edks/BaseTools" fail now.

The checkout source is the same, the python version, the gcc version is the same.

I wonder why the tests are failing on one machine and are ok on the other ...

Thank you for any hints

======================================================================
FAIL: test_FMMT_FMMT (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/FMMT.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/FMMT.py", line 48
def __init__(self) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_BinaryFactoryProduct (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/BinaryFactoryProduct.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/BinaryFactoryProduct.py", line 34
type:list = []
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_BiosTree (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/BiosTree.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/BiosTree.py", line 31
def __init__(self, NodeName: str) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_BiosTreeNode (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/BiosTreeNode.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/BiosTreeNode.py", line 34
def __init__(self, name: str) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_FMMTOperation (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/FMMTOperation.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/FMMTOperation.py", line 16
def ViewFile(inputfile: str, ROOT_TYPE: str, layoutfile: str=None, outputfile: str=None) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_FMMTParser (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/FMMTParser.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/FMMTParser.py", line 15
def __init__(self, name: str, TYPE: str) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_FvHandler (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/FvHandler.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/FvHandler.py", line 16
def ChangeSize(TargetTree, size_delta: int=0) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_core_GuidTools (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/GuidTools.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/core/GuidTools.py", line 18
def ExecuteCommand(cmd: list) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FMMT_utils_FvLayoutPrint (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/utils/FvLayoutPrint.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FMMT/utils/FvLayoutPrint.py", line 9
def GetFormatter(layout_format: str):
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FirmwareStorageFormat_Common (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/Common.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/Common.py", line 42
def from_list(self, listformat: list) -> None:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FirmwareStorageFormat_FfsFileHeader (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/FfsFileHeader.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/FfsFileHeader.py", line 41
def FFS_FILE_SIZE(self) -> int:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FirmwareStorageFormat_FvHeader (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/FvHeader.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/FvHeader.py", line 77
def Refine_FV_EXT_ENTRY_OEM_TYPE_Header(nums: int) -> EFI_FIRMWARE_VOLUME_EXT_ENTRY_OEM_TYPE:
^
SyntaxError: invalid syntax


======================================================================
FAIL: test_FirmwareStorageFormat_SectionHeader (CheckPythonSyntax.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 49, in <lambda>
newmethod = lambda self: self.SingleFileTest(filename)
File "/home/builder/edk2/git/edk2/BaseTools/Tests/CheckPythonSyntax.py", line 27, in SingleFileTest
self.fail('syntax error: %s, Error is %s' % (filename, str(e)))
AssertionError: syntax error: /home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/SectionHeader.py, Error is File "/home/builder/edk2/git/edk2/BaseTools/Source/Python/FirmwareStorageFormat/SectionHeader.py", line 22
def SECTION_SIZE(self) -> int:
^
SyntaxError: invalid syntax


[OVMF] Unable to connect a driver to Raid controller

Dinakar Medavaram
 

Hi All,
We are working on a hypervisor for x86_64 and we brought up OVMF on our hypervisor. Our hypervisor pass through whole hardware i.e. whole PCI devices starting from root ports to all devices.
If we connect a USB startup disk, OVMF detects its and boots linux from USB stick.
But OVMF is not able to detect the disk connected to PCI bus. I could see PCI device recognised as Raid controllers ( 00 00 1F 02 ==> Mass Storage Controller - RAID controller) but no drivers gets attached to the controller.
Driver list below. Please suggest the right driver


Shell> pci
Seg Bus Dev Func
--- --- --- ----
00 00 00 00 ==> Bridge Device - Host/PCI bridge
Vendor 8086 Device 0150 Prog Interface 0
00 00 02 00 ==> Display Controller - VGA/8514 controller
Vendor 8086 Device 0162 Prog Interface 0
00 00 14 00 ==> Serial Bus Controllers - USB
Vendor 8086 Device 1E31 Prog Interface 30
00 00 16 00 ==> Simple Communications Controllers - Other communicati
on device
Vendor 8086 Device 1E3A Prog Interface 0
00 00 16 03 ==> Simple Communications Controllers - Serial controller

Vendor 8086 Device 1E3D Prog Interface 2
00 00 19 00 ==> Network Controller - Ethernet controller
Vendor 8086 Device 1502 Prog Interface 0
00 00 1A 00 ==> Serial Bus Controllers - USB
Vendor 8086 Device 1E2D Prog Interface 20
00 00 1B 00 ==> Multimedia Device - Mixed mode device
Vendor 8086 Device 1E20 Prog Interface 0
00 00 1D 00 ==> Serial Bus Controllers - USB
Vendor 8086 Device 1E26 Prog Interface 20
00 00 1E 00 ==> Bridge Device - PCI/PCI bridge
Vendor 8086 Device 244E Prog Interface 1
00 00 1F 00 ==> Bridge Device - PCI/ISA bridge
Vendor 8086 Device 1E47 Prog Interface 0
00 00 1F 02 ==> Mass Storage Controller - RAID controller
Vendor 8086 Device 2822 Prog Interface 0
00 00 1F 03 ==> Serial Bus Controllers - System Management Bus
Vendor 8086 Device 1E22 Prog Interface 0
Shell> drivers
T D
D Y C I
R P F A
V VERSION E G G #D #C DRIVER NAME IMAGE NAME
== ======== = = = == == =================================== ==========
3E 0000000A B - - 1 13 PCI Bus Driver PciBusDxe
40 00000010 ? - - - - Virtio PCI Driver VirtioPciDeviceDxe
41 00000010 ? - - - - Virtio Block Driver VirtioBlkDxe
42 00000010 ? - - - - <null string> GvtGopDxe
43 0000000A D - - 2 - Platform Console Management Driver ConPlatformDxe
44 0000000A D - - 1 - Platform Console Management Driver ConPlatformDxe
45 0000000A B - - 2 2 Console Splitter Driver ConSplitterDxe
46 0000000A ? - - - - Console Splitter Driver ConSplitterDxe
47 0000000A ? - - - - Console Splitter Driver ConSplitterDxe
48 0000000A B - - 1 1 Console Splitter Driver ConSplitterDxe
49 0000000A B - - 1 1 Console Splitter Driver ConSplitterDxe
4D 0000000A ? - - - - Graphics Console Driver GraphicsConsoleDxe
4E 0000000A B - - 1 1 Serial Terminal Driver TerminalDxe
4F 0000000A ? - - - - Generic Disk I/O Driver DiskIoDxe
50 0000000B ? - - - - Partition Driver(MBR/GPT/El Torito) PartitionDxe
53 0000000A ? - - - - SCSI Bus Driver ScsiBus
54 0000000A ? - - - - Scsi Disk Driver ScsiDisk
55 0000000A ? - - - - Sata Controller Init Driver SataController
56 00000010 ? - - - - AtaAtapiPassThru Driver AtaAtapiPassThruDxe
57 00000010 ? - - - - ATA Bus Driver AtaBusDxe
58 00000010 ? - - - - NVM Express Driver NvmExpressDxe
59 0000000A D - - 1 - PC-AT ISA Device Enumeration Driver IsaAcpi
5A 0000000A B - - 1 3 ISA Bus Driver IsaBusDxe
5B 0000000A ? - X - - PCI IDE/ATAPI Bus Driver IdeBusDxe
5C 0000000A ? - - - - Sata Controller Init Driver SataController
5D 0000000A B - - 1 1 ISA Serial Driver IsaSerialDxe
5E 0000000A D - - 1 - PS/2 Keyboard Driver Ps2KeyboardDxe
5F 0000000A ? - - - - FAT File System Driver Fat
60 00000010 ? - - - - UDF File System Driver UdfDxe
61 00000020 ? - - - - Usb Uhci Driver UhciDxe
62 00000030 D - - 2 - Usb Ehci Driver EhciDxe
63 00000030 D - - 1 - Usb Xhci Driver XhciDxe
64 0000000A D - - 3 - Usb Bus Driver UsbBusDxe
65 0000000A ? - - - - Usb Keyboard Driver UsbKbDxe
66 00000011 ? - - - - Usb Mass Storage Driver UsbMassStorageDxe
Shell>


Re: [edk2-devel] GSoC Contributor Proposals

Ethin Probst <harlydavidsen@...>
 

Hi Nate,
I've submitted my proposal. I ran into some accessibility problems with the "project technologies" part, so there might be random junk there that was added that I couldn't figure out how to get rid of.

On 4/13/22 22:33, Nate DeSimone wrote:
Hi Everyone,
I hope you are all working on your proposals. If you would like some feedback on your proposals before having to submit the final version of them on Tuesday next week, please do one of the following:
1. Upload your current draft proposal to the GSoC website.
2. Email Mike Kinney and I a link to your draft proposal.
To allow sufficient time for review and implementation of provided feedback, please get your drafts to us by 5pm pacific Friday (2022-04-16 0:00 UTC).
Best Regards,
Nate


Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Pedro Falcato <pedro.falcato@...>
 

GSoC tip: Try to spread your proposals out. If we don't have enough project slots for you, all your 3 proposals get rejected and your GSoC participation goes out the window.
Take a look at our project ideas and ask questions, and take a look at other orgs if you want. Also, don't forget that these project ideas are just suggestions, and you're free (and encouraged) to
build on top of them, or write your own proposal from scratch.

Best regards,
Pedro


On Wed, Apr 13, 2022 at 6:53 PM Ada Christine <adachristine18@...> wrote:
i can submit up to three proposals. I'll give all of this some thought and a little research and start putting at least two proposals together starting Friday. :)

On Wed, Apr 13, 2022, 15:15 Marvin Häuser <mhaeuser@...> wrote:

On 13. Apr 2022, at 16:38, Ada Christine <adachristine18@...> wrote:

i was replying via the groups.io web interface, I'm guessing that messed up
the thread? i haven't used mailing lists before and don't know how they
work. I'll use my mail client from here on.

I'm on board with not treating EFI as an operating system. the more i think
about it the more it looks like scope creep.

Agreed.

I'm not quite as enthusiastic
about it as i was at first glance.

I'm still keen on doing my gsoc proposal for edk, though, and even if this
task and the acpica application are decided to be out of scope unit
testing,

How about fuzz-testing? This is also something edk2 needs quite badly. At Acidanthera, we compile edk2 code in userspace outside the edk2 build system and fuzz with dummy applications.

clang integration

Pedro and Vitaly are looking for someone to finish ASan: https://edk2.groups.io/g/devel/topic/90010978#87991
There are working UBSan concepts, but they also need to be mainlined.

and source-level debugging are all relevant to
my interests.

how about your ideas for security stuff?

I want the entirety of MM to leverage SmmMemLib and to support SMAP. SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib would only work on MMRAM. Also evaluation of how to best avoid pointers in MM communication buffers would be nice.

There also is a bunch of other stuff, like working out moving a part of CpuDxe into DxeCore to have memory protection live immediately, memory protection in PEI, a replacement for the TE format (it’s buggy and most platforms mostly abandoned it over various issues), and alternatives to guarding critical code with SMM (like allowing NVRAM commits only as part of a reboot).

I personally find all of those projects very important, but I cannot promise many people agree. Especially those that impose global changes (most notably the TE replacement) may be very tedious to submit. Gladly, I believe you can submit multiple proposals (?)

Best regards,
Marvin

I'm not very knowledgeable about
trusted platform or secure boot but I'm willing to learn whatever is
necessary to get something spun up for my proposal.

On Wed, Apr 13, 2022, 12:05 Marvin Häuser <mhaeuser@...> wrote:

Do you use the “reply all” option in your mail client? Looks like my CCs
have been dropped again. Comments inline.

On 13. Apr 2022, at 12:54, Ada Christine <adachristine18@...>
wrote:
Hi, Marvin

Its similarity to my own latest experiment is the key to what grabbed my
attention. I have no particular use case in mind for it, but I see its
potential for anybody developing larger applications in that when a library
is changed there's no need to distribute a new version of the whole binary,
just the relevant library module.

I really do not like the trend of treating UEFI as a full-fledged OS - it
is not. The most used UEFI applications, OS loaders, are really not that
huge and are distributed as part of the OS image anyway. Even for less used
applications, you will always get a full snapshot anyhow. Gladly we don’t
have auto-update and package management yet. :)


I slept on it and it occurred to me that the whole thing could operate
similarly to the shell protocol in that the linker/loader is itself an
application that does a LoadImage() on the application needing dynamic
linking facilities.

That would mean the linker itself is shipped with every application that
requires it? Otherwise it doesn’t make much sense for it to be an app and
below’s problems apply.

If however the whole plan is making the linker as a DXE and including it
with the firmware, that I'm not quite as sure about. That would necessarily
tie any applications using dynamic linking to TianoCore or any firmware
distribution that derives from it.

I think that was the idea referred to as “edk2 core” by Steven, but I’d
like to hear his proposal to be sure. Virtually everyone uses edk2, so that
itself is not the problem, but versioning is. Vendors are slow to update
their snapshots or have just given up doing that entirely. Distributing it
for external applications like OS loaders would mean this can be leveraged
probably no earlier than 10 years from now. And for in-firmware things, I
have a hard time thinking about a use-case that outweighs the drawbacks.


To shift the topic slightly back to GSoC, however, I'm willing to work
on other items on the task list. Unit testing and an ACPICA application are
the alternative projects I had thought about. I need to choose fairly soon
as the proposal deadline is next Tuesday. I know a tiny bit about porting
ACPICA as I also have plans to incorporate it into my own project.

I have a few more ideas for security stuff, but Nate did not confirm them
as appropriate yet and I’m not here to drive you away from this specific
task (or the others). However, I’m still curious and concerned. :)

Best regards,
Marvin








--
Pedro Falcato


Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Pedro Falcato <pedro.falcato@...>
 

Hi Marvin, Ada,

Some comments:

I don't think the purpose of the dynamic linker is to treat EFI as a complete operating system, but to try to eliminate the static linking that may be needlessly duplicating
code that could instead be put in a single dynamic library. For instance, MdePkg and MdeModulePkg are linked into a *lot* of .efi, instead of being just a library. It'd be nice to see some
numbers on this (something like Google's bloaty could be run on every .efi file, in order to understand how much file space we would actually save).

Other comments inline.

On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser <mhaeuser@...> wrote:

On 13. Apr 2022, at 16:38, Ada Christine <adachristine18@...> wrote:

i was replying via the groups.io web interface, I'm guessing that messed up
the thread? i haven't used mailing lists before and don't know how they
work. I'll use my mail client from here on.

I'm on board with not treating EFI as an operating system. the more i think
about it the more it looks like scope creep.

Agreed.

I'm not quite as enthusiastic
about it as i was at first glance.

I'm still keen on doing my gsoc proposal for edk, though, and even if this
task and the acpica application are decided to be out of scope unit
testing,

How about fuzz-testing? This is also something edk2 needs quite badly. At Acidanthera, we compile edk2 code in userspace outside the edk2 build system and fuzz with dummy applications.

Note: fuzzing is also part of the LLVM instrumentation suite (see https://llvm.org/docs/LibFuzzer.html) and is something I could happily mentor.
clang integration

Pedro and Vitaly are looking for someone to finish ASan: https://edk2.groups.io/g/devel/topic/90010978#87991
There are working UBSan concepts, but they also need to be mainlined.

Is Vitaly going to be a mentor? I was assuming it was going to be me and some other, more senior, mentor (possibly Steven Shi, which I included in the task).
Anyway, re: ASAN, if the project includes ASAN, UBSAN and possibly some other sanitizer it's quite possible that it could be considered a large project (which means more hours but a larger stipend too). Fuzzing + coverage could
be very nice additions to this project idea.
Also, is stress-testing a decent idea?

and source-level debugging are all relevant to
my interests.

how about your ideas for security stuff?

I want the entirety of MM to leverage SmmMemLib and to support SMAP. SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib would only work on MMRAM. Also evaluation of how to best avoid pointers in MM communication buffers would be nice.

There also is a bunch of other stuff, like working out moving a part of CpuDxe into DxeCore to have memory protection live immediately, memory protection in PEI, a replacement for the TE format (it’s buggy and most platforms mostly abandoned it over various issues), and alternatives to guarding critical code with SMM (like allowing NVRAM commits only as part of a reboot).

I personally find all of those projects very important, but I cannot promise many people agree. Especially those that impose global changes (most notably the TE replacement) may be very tedious to submit. Gladly, I believe you can submit multiple proposals (?)

Best regards,
Marvin

I'm not very knowledgeable about
trusted platform or secure boot but I'm willing to learn whatever is
necessary to get something spun up for my proposal.

On Wed, Apr 13, 2022, 12:05 Marvin Häuser <mhaeuser@...> wrote:

Do you use the “reply all” option in your mail client? Looks like my CCs
have been dropped again. Comments inline.

On 13. Apr 2022, at 12:54, Ada Christine <adachristine18@...>
wrote:
Hi, Marvin

Its similarity to my own latest experiment is the key to what grabbed my
attention. I have no particular use case in mind for it, but I see its
potential for anybody developing larger applications in that when a library
is changed there's no need to distribute a new version of the whole binary,
just the relevant library module.

I really do not like the trend of treating UEFI as a full-fledged OS - it
is not. The most used UEFI applications, OS loaders, are really not that
huge and are distributed as part of the OS image anyway. Even for less used
applications, you will always get a full snapshot anyhow. Gladly we don’t
have auto-update and package management yet. :)


I slept on it and it occurred to me that the whole thing could operate
similarly to the shell protocol in that the linker/loader is itself an
application that does a LoadImage() on the application needing dynamic
linking facilities.

That would mean the linker itself is shipped with every application that
requires it? Otherwise it doesn’t make much sense for it to be an app and
below’s problems apply.

If however the whole plan is making the linker as a DXE and including it
with the firmware, that I'm not quite as sure about. That would necessarily
tie any applications using dynamic linking to TianoCore or any firmware
distribution that derives from it.

I think that was the idea referred to as “edk2 core” by Steven, but I’d
like to hear his proposal to be sure. Virtually everyone uses edk2, so that
itself is not the problem, but versioning is. Vendors are slow to update
their snapshots or have just given up doing that entirely. Distributing it
for external applications like OS loaders would mean this can be leveraged
probably no earlier than 10 years from now. And for in-firmware things, I
have a hard time thinking about a use-case that outweighs the drawbacks.


To shift the topic slightly back to GSoC, however, I'm willing to work
on other items on the task list. Unit testing and an ACPICA application are
the alternative projects I had thought about. I need to choose fairly soon
as the proposal deadline is next Tuesday. I know a tiny bit about porting
ACPICA as I also have plans to incorporate it into my own project.

I have a few more ideas for security stuff, but Nate did not confirm them
as appropriate yet and I’m not here to drive you away from this specific
task (or the others). However, I’m still curious and concerned. :)

Best regards,
Marvin








--
Pedro Falcato


Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Rebecca Cran
 

Oh I see what was meant - ignore me!
I do think it would be good to have more discussions on edk2-discuss though :)

--
Rebecca Cran

On 4/13/22 08:44, Rebecca Cran wrote:
On 4/13/22 00:54, Marvin Häuser wrote:
CC edk2-devel (you picked the logically correct list, but it’s pretty dead and barely anyone reads it)
Just a minor correction - it's edk2-discuss (discuss@edk2.groups.io) that was cc'd but that's pretty dead.
edk2-devel (devel@edk2.groups.io) is much more active!


Re: GSoC Proposal

Rebecca Cran
 

On 4/13/22 00:54, Marvin Häuser wrote:
CC edk2-devel (you picked the logically correct list, but it’s pretty dead and barely anyone reads it)
Just a minor correction - it's edk2-discuss (discuss@edk2.groups.io) that was cc'd but that's pretty dead.
edk2-devel (devel@edk2.groups.io) is much more active!

--
Rebecca Cran


Re: GenFw: Bad definition for symbol

Oliver Steffen
 

On 08.04.2022 14:08, Oliver Steffen wrote:
Hi everyone,

I get this error when building StandaloneMmPkg for AARCH64 using gcc 11
on Ubuntu 22.04:

Bad definition for symbol '_GLOBAL_OFFSET_TABLE_'@0x5a20 or
unsupported symbol type.
One solution is to add `-fno-stack-protector` to
`GCC5_AARCH64_CC_FLAGS`.

-- Oliver


Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Brian J. Johnson
 

On 4/15/22 23:23, Desimone, Nathaniel L wrote:
Hi Marvin,

-----Original Message-----
From: Marvin Häuser <mhaeuser@...>
Sent: Friday, April 15, 2022 9:44 AM
To: discuss@edk2.groups.io; Johnson, Brian <brian.johnson@...>;
Desimone, Nathaniel L <nathaniel.l.desimone@...>; Andrew Fish
<afish@...>; devel@edk2.groups.io
Cc: Pedro Falcato <pedro.falcato@...>; adachristine18@...;
Shi, Steven <steven.shi@...>
Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Hey Brian,

On 15.04.22 18:22, Brian J. Johnson wrote:
Nate, Andrew, Marvin, Pedro, Ada, et al,

This is a great discussion.  I've been debating where to weigh in...
Personally, I don't think this topic can get enough attention. Thanks! :)


I agree that some sort of library sharing to reduce image size would
be very helpful.  Although some parts of the BIOS are built
separately, large parts of it are compiled at the same time, so there
should be ample opportunity for tool-directed sharing.  Some form of
pre-linking modules together may be the easiest way to do that.  But
however it gets implemented, we should try to make it automatic:
requiring library writers to manually add a lot of metadata or write
thunks or shims would be a barrier to adoption.
My suggestion to use a private function array would indeed require shims,
however this would be the "better than nothing I guess" solution where only
the most used and most painful to duplicate functions are shared, e.g.,
memory and maybe common string printing functions. This would quickly and
efficiently achieve the goal of "reduce size". And some form of sophisticated
prelinking would be my proposal for something more systematic, which
might come a lot cheaper with the security features to be added.


I also agree that PEI has acquired far too much functionality over
time, and we badly need to find some way to reverse that.  PEI is
pulling in more and more of the h/w initialization, leaving DXE mainly
to provide the boot interface (ACPI, UEFI services, setup screens,
etc.)  That puts more and more pressure on XIP storage and
cache-as-RAM.  How can we encourage a change?
I'm not in the loop enough to comment on the XIP and CAR issues.
However, HW init moving to PEI (or more generally, into a single stage) is not
necessarily a bad thing, is it? As I've said before, I'm not sure what else to
think of PEI than "DXE light". Both "initialise some of the hardware" with no
clear scope boundary but with similar interfaces. If they were sort of fused to
provide a minimal "pre-memory PEI" and a more DXE-like "post-memory PEI"
without having a clear boundary between HW init steps, would that be too
bad? Now I'm *really* just blindly guessing because I barely looked into this
project, but isn't that somewhat the idea of SlimBootloader?

Your description of DXE sort of is it collapsing into an advanced BDS.
Nate earlier said he'd like a clear distinction between DXE and BDS, which I
didn't really get the point of.
I guess what I'm trying to express is where does DXE end and where
does BDS begin? A lot of drivers that are only used in BDS get
dispatched early in DXE. Conversely, it is entirely possible for some
DXE drivers to only run after we have technically entered the BDS
phase. The line where DXE ends and BDS begins is blurry and not
really a line but more like a rectangle.
To Isaac's point, there is a good argument to be made for beefing up
PEI's capabilities to the point that DXE and BDS more or less become
one thing. But what does that mean for PEI? Do we change from the
1-D PPI database to the 2-D Handle database in order to express
device topologies better for example? That would be useful for stuff
like the PEI UFS drivers as they run into difficulties expressing
LUN IDs that the DXE version does not for example. Not sure if I
would actually formally propose that in the PI WG but interesting
stuff to think about regardless.
Andrew pointed out that PEI was originally intended for memory init, and DXE for the rest. One nice aspect of that is that there's a simple, architected, consolidated handoff of state between them: the HOB list. That makes it easier to do "unusual" things in SEC+PEI (special security, address map changes, etc.) but have it all hidden from DXE, where the majority of "standard" code lives. The trend of moving more and more into PEI is eroding that advantage.

Maybe a similar handoff of state is needed between pre-memory and post-memory PEI? Or between DXE and BDS?

One could argue that we already have that for BDS: the UEFI boot services.

But if PEI consumed the HW init portions of DXE,
a more advanced BDS (which also powers e.g. the HID stuff on the way)
would be the logical remainder stage to separately support UEFI booting as a
payload (for SlimBootloader, coreboot, ...).

All of that was logically speaking. Even if the development was driven to its
extreme, PEI and DXE cores would still share a bunch of code and it might
make sense to share a foundation technically.

Best regards,
Marvin


Thanks,
Brian J. Johnson


On 4/15/22 03:15, Nate DeSimone wrote:
Hi Andrew,

On 4/14/22, 7:43 PM, "Andrew Fish" <afish@...> wrote:


On Apr 14, 2022, at 6:06 PM, Nate DeSimone
<nathaniel.l.desimone@...> wrote:

Hi Marvin,

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
Marvin Häuser
Sent: Thursday, April 14, 2022 12:56 AM
To: discuss@edk2.groups.io; Desimone, Nathaniel L
<nathaniel.l.desimone@...>
Cc: Pedro Falcato <pedro.falcato@...>; edk2-devel-groups-io
<devel@edk2.groups.io>; adachristine18@...; Shi, Steven
<steven.shi@...>
Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal

I feel like there are so many much easier solutions to this
problem that are at most limited by the clear specification. The
UEFI specification with regards to booting and all of that
obviously is of utmost importance.
If you have a better idea that retains compatibility with the
existing UEFI PI then I would be happy to hear it. Ultimately
anything we do needs to be a pure extension that retains
compatibility with old code. Given that restriction having the
ability to coalesce all the LibraryClasses into a single module and
have cross-module linking seems like the best way to handle it to me.

The UEFI PI specification parts that deal about internal
structure, as far as I know, are only in place to make it easy to
integrate Intel IP.
Its not just for Intel. The biggest reason for it to increase the
standardization of the boot flow across the PC ecosystem. We have
learned from experience that firmware is super critical to get a
product out the door but it is also difficult to write. So we try
to make it as reusable as humanly possible.

In fact, I don’t *know*, but I’m pretty sure the very strict
separation between PEI and DXE was preserved mostly because MRC
was 32-bit-only for a long time. Glad that seems to have been
resolved, AMD does memory init by PSP nowadays.
Having less complex early stages chain load more complex later
stages is a common design pattern in firmware, not just UEFI. For
example, your typical ARM system loads kinda like this:

PBL (SoC ROM) --> SBL (RAM Init) --> ARM Trust Zone --> Hypervisor
--> EDK II or U-Boot or LittleKernel (which runs android fastboot)

Comparing relative complexity I believe the Intel UEFI PI design is
actually pretty simple when you consider how much it gets done:

Ucode ROM --> SEC + PEI --> DXE + SMM + BDS

My biggest criticism of the design is that the strict separation
between PEI and DXE doesn't exist between DXE, SMM, and BDS 😊

There are a few reasons why PEI was 32-bit for quite some time. The
biggest one is the code size increase, 64-bit x86 code is 20-30%
larger than 32-bit x86 code. Since the only RAM Pre-Memory code has
access to is the cache onboard the processor and for security
reasons all that code has to fit within that RAM we generally do
everything we can to make that image as small as possible. Second,
64-bit requires a page table and since we desired to keep PEI
simple we tried to avoid that. Finally, the PI spec didn't allow a
64-bit PEI until recently. MRC is 32-bit code just because that is
what PEI happens to be. Porting it to 64-bit is not terribly
difficult.

Ultimately the mix of 32/64-bit does cause some difficulties. Any
data structures that get shared between PEI and DXE (HOBs, PCDs,
etc.) need to resolve to the same size and packing. LibraryClasses
need to be written to compile properly in both modes. In the case
of FSP API mode you need to resort to thunking between 32 and
64-bit modes during DXE. More or less we decided that the costs are
starting to outweigh the advantages.
I’d also point out that x86 VMs use X64 (x86-64) PEI today and it
works so the 32-bit/64-bit mix has nothing to do with UEFI/PI/edk2.

In the PC ecosystem a single chipset family can power thousands of
unique designs. So the  DRAM memory needs to be external, support
lots of different chipset packages(signal integrity...), support the
lowest cost through the highest cost DRAM and thousands of different
board layouts. So programing DRAM takes a masters degree in antenna
design. I’ve seen MRC (Memory Reference Code) with over a MiB of
DEBUG prints in it, and it literally is printing histograms of what
it is tuning. So all this code has to run before the system has any
DRAM, thus it is running using the cache as RAM. I’ve not looked at
the x86 architecture specs form the vendors in a while, but back in
the day they did not support page tables in ROM or pinned cached.
Now it might work, but if it breaks your CPU vendor blames you so
you don’t code PEI in X64….

We contributed the 1st edk2 ARM platform, Beagle Board, and It was a
long time ago but I seem to remember the mask ROM used a table in
NOR FLASH to init memory and then copied an image from NOR FLASH
into DRAM and jumped to it. So PEI is kind of not really needed and
we implemented a PrePEI and jumped directly to DXE.

Given I was around back in the day when all this stuff was designed
I can say SEC was always a place holder for security code, as
security code always has to run 1st. PEI (Pre EFI) was designed to
get DRAM programmed and then jump to DXE. It kind of also fell in
naturally to ACPI S3 flow since that was turning memory back on.
When we designed PEI we kind of though of it more like a boot loader
stage for the firmware that turned on memory and all the work would
happen in DXE. Then reality strikes and the existing BIOS assembly
programmers start learning C (lots of  cranky people) and they start
having to learn all about PEI to turn on memory. They had to write a
big chunk of code for the memory init in PEI. These folks had never
written any EFI code, so to them it was easier to move a lot of the
chipset init code into PEI as that is the world they had to figure
out to get memory turned on. I mean why learn EFI if you don’t have
to? So that is how we get so much code in IA32 (i386) on some
platforms. This start kind of biased future choices and how to
enable non edk2 code bases….
One of the big reasons a lot of code that should have been written in
DXE ended up in PEI is unfortunately due to the FSP and its inability
to support DXE code.


Thanks,

Andrew Fish



For many good reasons, Linux does not provide a stable kernel API.
This
allows to easily deploy breaking changes across the entire codebase.
Obviously, this is infeasible at a large scale when you need to
integrate IP blobs, but it would already help to break the
expectation that UEFI PI is a perfectly forwards- and
backwards-compatible system. DXE has SetMem and CopyMem as
part of
gBS. Maybe I don’t like it that much as part of the spec itself,
but it’s a good idea technically. I’d probably opt to have a
quickly accessible implementation detail of important function
pointers appended to PEI and DXE services, used by in-tree
modules. This may break both
forwards- and backwards-compatibility, but as it only applies to
in-tree modules, that is fine so long as we let go of the
compatibility notions. PPIs and protocols are an option too of
course, but they have a lookup performance penalty. Compared to
dynamic linking, that should hopefully be negligible however.

Absolutely optional to read, I don’t intend to waste anyone’s time
much, some philosophical stuff about my rationale:

If you started UEFI from scratch right now, would it have strictly
separated PEI and DXE?
For sure a clean slate design started today would look a lot
different than PEI/DXE... which was a clean slate design circa 1998
😊. In my opinion, if we were to suddenly go back to the drawing
board we would build something that is much closer to a full OS now
than we did back then. There have been cases where not being able
to use interrupt handlers and not having thread scheduling has
prevented implementation of desired features. The ARM guys built
LittleKernel (https://github.com/littlekernel/lk) for a lot of
these reasons. In the data center world some have decided to go to
the extreme of putting an entire copy of Linux in SPI so they can
do a network boot that downloads the OS image using BitTorrent!

The duplication between PEI and DXE core, and by extension MM
core, would be my most obvious place to start reducing size. I
would probably opt for a PEI/DXE hybrid where it starts in
„minimal mode“ (maybe think of PEI more like a microkernel here)
and after memory is initialised, the rest of DXE is loaded. And MM
core would get no loading at all, this requirement has gladly been
dropped ages ago. Just one prelinked snapshot of the address space
with a relocation table and a safe self-relocator on entry (this
is needed at the very least for ARM).

Ironically, with my idea for MM, dynamic loading would be free as
everything is prelinked anyway. The same is true for PEI XIP, it
is prelinked by nature.
Actually Post-Memory PEI can have non-prelinked PEIMs. And that
does get used for the PEI GOP driver.

What I do not like is the additional dynamic linking code at
load-time for non- XIP modules. Though, the more I think about it,
the more I wonder whether not the entirety of UEFI could be
composed of prelinked, relocatable memory snapshots without
traditional image loading entirely (for in-FW stuff). macOS has a
similar concept with its “Kernel Collections”.
Well, way
too much off-topic now. :)
If you make the assumption that 100% of the code is compiled all at
once then yes that works. UEFI was designed so that assumption does
not need to be true. There are good use cases for it: OpROMs,
generic OS loaders, network boot, etc.

Thanks,
Nate


Why am I explaining all this despite the fact everyone knows this
will never happen? Because I don’t like the notion of fixing
issues of an already overcomplicated system by adding even more
complicated stuff.
Especially
when the existing overcomplicated stuff is already uncomfortably
broken.

Best regards,
Marvin

For XIP PEI code… it will really help and would be very timely
since the
transition of PEI from 32-bit to 64-bit is going to increase the
size of PEI by ~20%.

Thanks,
Nate

From: Pedro Falcato <pedro.falcato@...>
Sent: Wednesday, April 13, 2022 11:43 AM
To: edk2-devel-groups-io <devel@edk2.groups.io>; Marvin Häuser
<mhaeuser@...>
Cc: discuss@edk2.groups.io; adachristine18@...; Desimone,
Nathaniel L <nathaniel.l.desimone@...>; Shi, Steven
<steven.shi@...>
Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Hi Marvin, Ada,

Some comments:

I don't think the purpose of the dynamic linker is to treat EFI
as a complete operating system, but to try to eliminate the
static linking that may be needlessly duplicating code that could
instead be put in a
single dynamic library. For instance, MdePkg and MdeModulePkg are
linked into a *lot* of .efi, instead of being just a library. It'd
be nice to see some numbers on this (something like Google's
bloaty could be run on every .efi file, in order to understand how
much file space we would actually save).

Other comments inline.

On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser
<mhaeuser@...<mailto:mhaeuser@...>> wrote:

On 13. Apr 2022, at 16:38, Ada Christine
<adachristine18@...<mailto:adachristine18@...>>
wrote:
i was replying via the groups.io<http://groups.io > web
interface, I'm guessing that messed up the thread? i haven't used
mailing lists before and don't know how they work. I'll use my
mail client from here on.

I'm on board with not treating EFI as an operating system. the
more i think about it the more it looks like scope creep.

Agreed.


I'm not quite as enthusiastic
about it as i was at first glance.

I'm still keen on doing my gsoc proposal for edk, though, and
even if this task and the acpica application are decided to be
out of scope unit testing,

How about fuzz-testing? This is also something edk2 needs quite
badly. At
Acidanthera, we compile edk2 code in userspace outside the edk2
build system and fuzz with dummy applications.

Note: fuzzing is also part of the LLVM instrumentation suite (see
https://llvm.org/docs/LibFuzzer.html ) and is something I could
happily mentor.
clang integration

Pedro and Vitaly are looking for someone to finish ASan:
https://edk2.groups.io/g/devel/topic/90010978#87991
There are working UBSan concepts, but they also need to be
mainlined.

Is Vitaly going to be a mentor? I was assuming it was going to be
me and
some other, more senior, mentor (possibly Steven Shi, which I
included in the task).
Anyway, re: ASAN, if the project includes ASAN, UBSAN and
possibly some other sanitizer it's quite possible that it could
be considered a large
project (which means more hours but a larger stipend too). Fuzzing
+ coverage could be very nice additions to this project idea.
Also, is stress-testing a decent idea?


and source-level debugging are all relevant to my interests.

how about your ideas for security stuff?

I want the entirety of MM to leverage SmmMemLib and to support
SMAP.
SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib
would
only work on MMRAM. Also evaluation of how to best avoid pointers
in MM communication buffers would be nice.

There also is a bunch of other stuff, like working out moving a
part of
CpuDxe into DxeCore to have memory protection live immediately,
memory protection in PEI, a replacement for the TE format (it’s
buggy and most platforms mostly abandoned it over various issues),
and alternatives to guarding critical code with SMM (like allowing
NVRAM commits only as part of a reboot).

I personally find all of those projects very important, but I
cannot promise many people agree. Especially those that impose
global changes (most notably the TE replacement) may be very
tedious to submit.
Gladly, I believe you can submit multiple proposals (?)

Best regards,
Marvin


I'm not very knowledgeable about
trusted platform or secure boot but I'm willing to learn whatever
is necessary to get something spun up for my proposal.

On Wed, Apr 13, 2022, 12:05 Marvin Häuser
<mhaeuser@...<mailto:mhaeuser@...>> wrote:


Do you use the “reply all” option in your mail client? Looks like
my CCs have been dropped again. Comments inline.

On 13. Apr 2022, at 12:54, Ada Christine
<adachristine18@...<mailto:adachristine18@...>>
wrote:
Hi, Marvin

Its similarity to my own latest experiment is the key to what
grabbed my attention. I have no particular use case in mind for
it, but I see its potential for anybody developing larger
applications in that when a library is changed there's no need to
distribute a new version of the whole binary, just the relevant
library module.

I really do not like the trend of treating UEFI as a full-fledged
OS - it is not. The most used UEFI applications, OS loaders, are
really not that huge and are distributed as part of the OS image
anyway.
Even for
less used applications, you will always get a full snapshot anyhow.
Gladly we don’t have auto-update and package management yet. :)


I slept on it and it occurred to me that the whole thing could
operate
similarly to the shell protocol in that the linker/loader is
itself an
application that does a LoadImage() on the application needing
dynamic
linking facilities.

That would mean the linker itself is shipped with every application
that requires it? Otherwise it doesn’t make much sense for it to
be an
app and below’s problems apply.

If however the whole plan is making the linker as a DXE and
including
it with the firmware, that I'm not quite as sure about. That would
necessarily tie any applications using dynamic linking to
TianoCore or
any firmware distribution that derives from it.

I think that was the idea referred to as “edk2 core” by Steven, but
I’d like to hear his proposal to be sure. Virtually everyone uses
edk2, so that itself is not the problem, but versioning is. Vendors
are slow to update their snapshots or have just given up doing that
entirely. Distributing it for external applications like OS loaders
would mean this can be leveraged probably no earlier than 10 years
from now. And for in-firmware things, I have a hard time thinking
about a
use-case that outweighs the drawbacks.


To shift the topic slightly back to GSoC, however, I'm willing to
work
on other items on the task list. Unit testing and an ACPICA
application are the alternative projects I had thought about. I need
to choose fairly soon as the proposal deadline is next Tuesday. I
know
a tiny bit about porting ACPICA as I also have plans to
incorporate it into my
own project.

I have a few more ideas for security stuff, but Nate did not confirm
them as appropriate yet and I’m not here to drive you away from
this
specific task (or the others). However, I’m still curious and
concerned. :)

Best regards,
Marvin








--
Pedro Falcato



















Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Ada Christine <adachristine18@...>
 

I've submitted an updated version of my proposal document. There's a little
more detail and I've shifted from using the term "dynamic linking" to "code
sharing" to reflect the fact that the question of which solution will fit
best is not yet answered. If there are any more important matters not
addressed in the proposal please do let me know and I will fit them in. For
those who can view it, the proposal is here:
https://summerofcode.withgoogle.com/proposals/details/whGX9tXL

Bless!

- Ada Christine

On Sat, Apr 16, 2022 at 4:24 AM Nate DeSimone <
nathaniel.l.desimone@...> wrote:

Hi Marvin,

-----Original Message-----
From: Marvin Häuser <mhaeuser@...>
Sent: Friday, April 15, 2022 9:44 AM
To: discuss@edk2.groups.io; Johnson, Brian <brian.johnson@...>;
Desimone, Nathaniel L <nathaniel.l.desimone@...>; Andrew Fish
<afish@...>; devel@edk2.groups.io
Cc: Pedro Falcato <pedro.falcato@...>; adachristine18@...;
Shi, Steven <steven.shi@...>
Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Hey Brian,

On 15.04.22 18:22, Brian J. Johnson wrote:
Nate, Andrew, Marvin, Pedro, Ada, et al,

This is a great discussion. I've been debating where to weigh in...
Personally, I don't think this topic can get enough attention. Thanks! :)


I agree that some sort of library sharing to reduce image size would
be very helpful. Although some parts of the BIOS are built
separately, large parts of it are compiled at the same time, so there
should be ample opportunity for tool-directed sharing. Some form of
pre-linking modules together may be the easiest way to do that. But
however it gets implemented, we should try to make it automatic:
requiring library writers to manually add a lot of metadata or write
thunks or shims would be a barrier to adoption.
My suggestion to use a private function array would indeed require shims,
however this would be the "better than nothing I guess" solution where
only
the most used and most painful to duplicate functions are shared, e.g.,
memory and maybe common string printing functions. This would quickly and
efficiently achieve the goal of "reduce size". And some form of
sophisticated
prelinking would be my proposal for something more systematic, which
might come a lot cheaper with the security features to be added.


I also agree that PEI has acquired far too much functionality over
time, and we badly need to find some way to reverse that. PEI is
pulling in more and more of the h/w initialization, leaving DXE mainly
to provide the boot interface (ACPI, UEFI services, setup screens,
etc.) That puts more and more pressure on XIP storage and
cache-as-RAM. How can we encourage a change?
I'm not in the loop enough to comment on the XIP and CAR issues.
However, HW init moving to PEI (or more generally, into a single stage)
is not
necessarily a bad thing, is it? As I've said before, I'm not sure what
else to
think of PEI than "DXE light". Both "initialise some of the hardware"
with no
clear scope boundary but with similar interfaces. If they were sort of
fused to
provide a minimal "pre-memory PEI" and a more DXE-like "post-memory PEI"
without having a clear boundary between HW init steps, would that be too
bad? Now I'm *really* just blindly guessing because I barely looked into
this
project, but isn't that somewhat the idea of SlimBootloader?

Your description of DXE sort of is it collapsing into an advanced BDS.
Nate earlier said he'd like a clear distinction between DXE and BDS,
which I
didn't really get the point of.
I guess what I'm trying to express is where does DXE end and where does
BDS begin? A lot of drivers that are only used in BDS get dispatched early
in DXE. Conversely, it is entirely possible for some DXE drivers to only
run after we have technically entered the BDS phase. The line where DXE
ends and BDS begins is blurry and not really a line but more like a
rectangle.

To Isaac's point, there is a good argument to be made for beefing up PEI's
capabilities to the point that DXE and BDS more or less become one thing.
But what does that mean for PEI? Do we change from the 1-D PPI database to
the 2-D Handle database in order to express device topologies better for
example? That would be useful for stuff like the PEI UFS drivers as they
run into difficulties expressing LUN IDs that the DXE version does not for
example. Not sure if I would actually formally propose that in the PI WG
but interesting stuff to think about regardless.

But if PEI consumed the HW init portions of DXE,
a more advanced BDS (which also powers e.g. the HID stuff on the way)
would be the logical remainder stage to separately support UEFI booting
as a
payload (for SlimBootloader, coreboot, ...).

All of that was logically speaking. Even if the development was driven
to its
extreme, PEI and DXE cores would still share a bunch of code and it might
make sense to share a foundation technically.

Best regards,
Marvin


Thanks,
Brian J. Johnson


On 4/15/22 03:15, Nate DeSimone wrote:
Hi Andrew,

On 4/14/22, 7:43 PM, "Andrew Fish" <afish@...> wrote:


On Apr 14, 2022, at 6:06 PM, Nate DeSimone
<nathaniel.l.desimone@...> wrote:

Hi Marvin,

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
Marvin Häuser
Sent: Thursday, April 14, 2022 12:56 AM
To: discuss@edk2.groups.io; Desimone, Nathaniel L
<nathaniel.l.desimone@...>
Cc: Pedro Falcato <pedro.falcato@...>; edk2-devel-groups-io
<devel@edk2.groups.io>; adachristine18@...; Shi, Steven
<steven.shi@...>
Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal

I feel like there are so many much easier solutions to this
problem that are at most limited by the clear specification. The
UEFI specification with regards to booting and all of that
obviously is of utmost importance.
If you have a better idea that retains compatibility with the
existing UEFI PI then I would be happy to hear it. Ultimately
anything we do needs to be a pure extension that retains
compatibility with old code. Given that restriction having the
ability to coalesce all the LibraryClasses into a single module and
have cross-module linking seems like the best way to handle it to
me.

The UEFI PI specification parts that deal about internal
structure, as far as I know, are only in place to make it easy to
integrate Intel IP.
Its not just for Intel. The biggest reason for it to increase the
standardization of the boot flow across the PC ecosystem. We have
learned from experience that firmware is super critical to get a
product out the door but it is also difficult to write. So we try
to make it as reusable as humanly possible.

In fact, I don’t *know*, but I’m pretty sure the very strict
separation between PEI and DXE was preserved mostly because MRC
was 32-bit-only for a long time. Glad that seems to have been
resolved, AMD does memory init by PSP nowadays.
Having less complex early stages chain load more complex later
stages is a common design pattern in firmware, not just UEFI. For
example, your typical ARM system loads kinda like this:

PBL (SoC ROM) --> SBL (RAM Init) --> ARM Trust Zone --> Hypervisor
--> EDK II or U-Boot or LittleKernel (which runs android fastboot)

Comparing relative complexity I believe the Intel UEFI PI design is
actually pretty simple when you consider how much it gets done:

Ucode ROM --> SEC + PEI --> DXE + SMM + BDS

My biggest criticism of the design is that the strict separation
between PEI and DXE doesn't exist between DXE, SMM, and BDS 😊

There are a few reasons why PEI was 32-bit for quite some time. The
biggest one is the code size increase, 64-bit x86 code is 20-30%
larger than 32-bit x86 code. Since the only RAM Pre-Memory code has
access to is the cache onboard the processor and for security
reasons all that code has to fit within that RAM we generally do
everything we can to make that image as small as possible. Second,
64-bit requires a page table and since we desired to keep PEI
simple we tried to avoid that. Finally, the PI spec didn't allow a
64-bit PEI until recently. MRC is 32-bit code just because that is
what PEI happens to be. Porting it to 64-bit is not terribly
difficult.

Ultimately the mix of 32/64-bit does cause some difficulties. Any
data structures that get shared between PEI and DXE (HOBs, PCDs,
etc.) need to resolve to the same size and packing. LibraryClasses
need to be written to compile properly in both modes. In the case
of FSP API mode you need to resort to thunking between 32 and
64-bit modes during DXE. More or less we decided that the costs are
starting to outweigh the advantages.
I’d also point out that x86 VMs use X64 (x86-64) PEI today and it
works so the 32-bit/64-bit mix has nothing to do with UEFI/PI/edk2.

In the PC ecosystem a single chipset family can power thousands of
unique designs. So the DRAM memory needs to be external, support
lots of different chipset packages(signal integrity...), support the
lowest cost through the highest cost DRAM and thousands of different
board layouts. So programing DRAM takes a masters degree in antenna
design. I’ve seen MRC (Memory Reference Code) with over a MiB of
DEBUG prints in it, and it literally is printing histograms of what
it is tuning. So all this code has to run before the system has any
DRAM, thus it is running using the cache as RAM. I’ve not looked at
the x86 architecture specs form the vendors in a while, but back in
the day they did not support page tables in ROM or pinned cached.
Now it might work, but if it breaks your CPU vendor blames you so
you don’t code PEI in X64….

We contributed the 1st edk2 ARM platform, Beagle Board, and It was a
long time ago but I seem to remember the mask ROM used a table in
NOR FLASH to init memory and then copied an image from NOR FLASH
into DRAM and jumped to it. So PEI is kind of not really needed and
we implemented a PrePEI and jumped directly to DXE.

Given I was around back in the day when all this stuff was designed
I can say SEC was always a place holder for security code, as
security code always has to run 1st. PEI (Pre EFI) was designed to
get DRAM programmed and then jump to DXE. It kind of also fell in
naturally to ACPI S3 flow since that was turning memory back on.
When we designed PEI we kind of though of it more like a boot loader
stage for the firmware that turned on memory and all the work would
happen in DXE. Then reality strikes and the existing BIOS assembly
programmers start learning C (lots of cranky people) and they start
having to learn all about PEI to turn on memory. They had to write a
big chunk of code for the memory init in PEI. These folks had never
written any EFI code, so to them it was easier to move a lot of the
chipset init code into PEI as that is the world they had to figure
out to get memory turned on. I mean why learn EFI if you don’t have
to? So that is how we get so much code in IA32 (i386) on some
platforms. This start kind of biased future choices and how to
enable non edk2 code bases….
One of the big reasons a lot of code that should have been written in
DXE ended up in PEI is unfortunately due to the FSP and its inability
to support DXE code.


Thanks,

Andrew Fish



For many good reasons, Linux does not provide a stable kernel API.
This
allows to easily deploy breaking changes across the entire
codebase.
Obviously, this is infeasible at a large scale when you need to
integrate IP blobs, but it would already help to break the
expectation that UEFI PI is a perfectly forwards- and
backwards-compatible system. DXE has SetMem and CopyMem as
part of
gBS. Maybe I don’t like it that much as part of the spec itself,
but it’s a good idea technically. I’d probably opt to have a
quickly accessible implementation detail of important function
pointers appended to PEI and DXE services, used by in-tree
modules. This may break both
forwards- and backwards-compatibility, but as it only applies to
in-tree modules, that is fine so long as we let go of the
compatibility notions. PPIs and protocols are an option too of
course, but they have a lookup performance penalty. Compared to
dynamic linking, that should hopefully be negligible however.

Absolutely optional to read, I don’t intend to waste anyone’s time
much, some philosophical stuff about my rationale:

If you started UEFI from scratch right now, would it have strictly
separated PEI and DXE?
For sure a clean slate design started today would look a lot
different than PEI/DXE... which was a clean slate design circa 1998
😊. In my opinion, if we were to suddenly go back to the drawing
board we would build something that is much closer to a full OS now
than we did back then. There have been cases where not being able
to use interrupt handlers and not having thread scheduling has
prevented implementation of desired features. The ARM guys built
LittleKernel (https://github.com/littlekernel/lk) for a lot of
these reasons. In the data center world some have decided to go to
the extreme of putting an entire copy of Linux in SPI so they can
do a network boot that downloads the OS image using BitTorrent!

The duplication between PEI and DXE core, and by extension MM
core, would be my most obvious place to start reducing size. I
would probably opt for a PEI/DXE hybrid where it starts in
„minimal mode“ (maybe think of PEI more like a microkernel here)
and after memory is initialised, the rest of DXE is loaded. And MM
core would get no loading at all, this requirement has gladly been
dropped ages ago. Just one prelinked snapshot of the address space
with a relocation table and a safe self-relocator on entry (this
is needed at the very least for ARM).

Ironically, with my idea for MM, dynamic loading would be free as
everything is prelinked anyway. The same is true for PEI XIP, it
is prelinked by nature.
Actually Post-Memory PEI can have non-prelinked PEIMs. And that
does get used for the PEI GOP driver.

What I do not like is the additional dynamic linking code at
load-time for non- XIP modules. Though, the more I think about it,
the more I wonder whether not the entirety of UEFI could be
composed of prelinked, relocatable memory snapshots without
traditional image loading entirely (for in-FW stuff). macOS has a
similar concept with its “Kernel Collections”.
Well, way
too much off-topic now. :)
If you make the assumption that 100% of the code is compiled all at
once then yes that works. UEFI was designed so that assumption does
not need to be true. There are good use cases for it: OpROMs,
generic OS loaders, network boot, etc.

Thanks,
Nate


Why am I explaining all this despite the fact everyone knows this
will never happen? Because I don’t like the notion of fixing
issues of an already overcomplicated system by adding even more
complicated stuff.
Especially
when the existing overcomplicated stuff is already uncomfortably
broken.

Best regards,
Marvin

For XIP PEI code… it will really help and would be very timely
since the
transition of PEI from 32-bit to 64-bit is going to increase the
size of PEI by ~20%.

Thanks,
Nate

From: Pedro Falcato <pedro.falcato@...>
Sent: Wednesday, April 13, 2022 11:43 AM
To: edk2-devel-groups-io <devel@edk2.groups.io>; Marvin Häuser
<mhaeuser@...>
Cc: discuss@edk2.groups.io; adachristine18@...; Desimone,
Nathaniel L <nathaniel.l.desimone@...>; Shi, Steven
<steven.shi@...>
Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Hi Marvin, Ada,

Some comments:

I don't think the purpose of the dynamic linker is to treat EFI
as a complete operating system, but to try to eliminate the
static linking that may be needlessly duplicating code that could
instead be put in a
single dynamic library. For instance, MdePkg and MdeModulePkg are
linked into a *lot* of .efi, instead of being just a library. It'd
be nice to see some numbers on this (something like Google's
bloaty could be run on every .efi file, in order to understand how
much file space we would actually save).

Other comments inline.

On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser
<mhaeuser@...<mailto:mhaeuser@...>> wrote:

On 13. Apr 2022, at 16:38, Ada Christine
<adachristine18@...<mailto:adachristine18@...>>
wrote:
i was replying via the groups.io<http://groups.io > web
interface, I'm guessing that messed up the thread? i haven't used
mailing lists before and don't know how they work. I'll use my
mail client from here on.

I'm on board with not treating EFI as an operating system. the
more i think about it the more it looks like scope creep.

Agreed.


I'm not quite as enthusiastic
about it as i was at first glance.

I'm still keen on doing my gsoc proposal for edk, though, and
even if this task and the acpica application are decided to be
out of scope unit testing,

How about fuzz-testing? This is also something edk2 needs quite
badly. At
Acidanthera, we compile edk2 code in userspace outside the edk2
build system and fuzz with dummy applications.

Note: fuzzing is also part of the LLVM instrumentation suite (see
https://llvm.org/docs/LibFuzzer.html ) and is something I could
happily mentor.
clang integration

Pedro and Vitaly are looking for someone to finish ASan:
https://edk2.groups.io/g/devel/topic/90010978#87991
There are working UBSan concepts, but they also need to be
mainlined.

Is Vitaly going to be a mentor? I was assuming it was going to be
me and
some other, more senior, mentor (possibly Steven Shi, which I
included in the task).
Anyway, re: ASAN, if the project includes ASAN, UBSAN and
possibly some other sanitizer it's quite possible that it could
be considered a large
project (which means more hours but a larger stipend too). Fuzzing
+ coverage could be very nice additions to this project idea.
Also, is stress-testing a decent idea?


and source-level debugging are all relevant to my interests.

how about your ideas for security stuff?

I want the entirety of MM to leverage SmmMemLib and to support
SMAP.
SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib
would
only work on MMRAM. Also evaluation of how to best avoid pointers
in MM communication buffers would be nice.

There also is a bunch of other stuff, like working out moving a
part of
CpuDxe into DxeCore to have memory protection live immediately,
memory protection in PEI, a replacement for the TE format (it’s
buggy and most platforms mostly abandoned it over various issues),
and alternatives to guarding critical code with SMM (like allowing
NVRAM commits only as part of a reboot).

I personally find all of those projects very important, but I
cannot promise many people agree. Especially those that impose
global changes (most notably the TE replacement) may be very
tedious to submit.
Gladly, I believe you can submit multiple proposals (?)

Best regards,
Marvin


I'm not very knowledgeable about
trusted platform or secure boot but I'm willing to learn whatever
is necessary to get something spun up for my proposal.

On Wed, Apr 13, 2022, 12:05 Marvin Häuser
<mhaeuser@...<mailto:mhaeuser@...>> wrote:


Do you use the “reply all” option in your mail client? Looks like
my CCs have been dropped again. Comments inline.

On 13. Apr 2022, at 12:54, Ada Christine
<adachristine18@...<mailto:adachristine18@...>>
wrote:
Hi, Marvin

Its similarity to my own latest experiment is the key to what
grabbed my attention. I have no particular use case in mind for
it, but I see its potential for anybody developing larger
applications in that when a library is changed there's no need to
distribute a new version of the whole binary, just the relevant
library module.

I really do not like the trend of treating UEFI as a full-fledged
OS - it is not. The most used UEFI applications, OS loaders, are
really not that huge and are distributed as part of the OS image
anyway.
Even for
less used applications, you will always get a full snapshot
anyhow.
Gladly we don’t have auto-update and package management yet. :)


I slept on it and it occurred to me that the whole thing could
operate
similarly to the shell protocol in that the linker/loader is
itself an
application that does a LoadImage() on the application needing
dynamic
linking facilities.

That would mean the linker itself is shipped with every
application
that requires it? Otherwise it doesn’t make much sense for it to
be an
app and below’s problems apply.

If however the whole plan is making the linker as a DXE and
including
it with the firmware, that I'm not quite as sure about. That would
necessarily tie any applications using dynamic linking to
TianoCore or
any firmware distribution that derives from it.

I think that was the idea referred to as “edk2 core” by Steven,
but
I’d like to hear his proposal to be sure. Virtually everyone uses
edk2, so that itself is not the problem, but versioning is.
Vendors
are slow to update their snapshots or have just given up doing
that
entirely. Distributing it for external applications like OS
loaders
would mean this can be leveraged probably no earlier than 10 years
from now. And for in-firmware things, I have a hard time thinking
about a
use-case that outweighs the drawbacks.


To shift the topic slightly back to GSoC, however, I'm willing to
work
on other items on the task list. Unit testing and an ACPICA
application are the alternative projects I had thought about. I
need
to choose fairly soon as the proposal deadline is next Tuesday. I
know
a tiny bit about porting ACPICA as I also have plans to
incorporate it into my
own project.

I have a few more ideas for security stuff, but Nate did not
confirm
them as appropriate yet and I’m not here to drive you away from
this
specific task (or the others). However, I’m still curious and
concerned. :)

Best regards,
Marvin








--
Pedro Falcato

























Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Nate DeSimone
 

Hi Marvin,

-----Original Message-----
From: Marvin Häuser <mhaeuser@...>
Sent: Friday, April 15, 2022 9:44 AM
To: discuss@edk2.groups.io; Johnson, Brian <brian.johnson@...>;
Desimone, Nathaniel L <nathaniel.l.desimone@...>; Andrew Fish
<afish@...>; devel@edk2.groups.io
Cc: Pedro Falcato <pedro.falcato@...>; adachristine18@...;
Shi, Steven <steven.shi@...>
Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Hey Brian,

On 15.04.22 18:22, Brian J. Johnson wrote:
Nate, Andrew, Marvin, Pedro, Ada, et al,

This is a great discussion.  I've been debating where to weigh in...
Personally, I don't think this topic can get enough attention. Thanks! :)


I agree that some sort of library sharing to reduce image size would
be very helpful.  Although some parts of the BIOS are built
separately, large parts of it are compiled at the same time, so there
should be ample opportunity for tool-directed sharing.  Some form of
pre-linking modules together may be the easiest way to do that.  But
however it gets implemented, we should try to make it automatic:
requiring library writers to manually add a lot of metadata or write
thunks or shims would be a barrier to adoption.
My suggestion to use a private function array would indeed require shims,
however this would be the "better than nothing I guess" solution where only
the most used and most painful to duplicate functions are shared, e.g.,
memory and maybe common string printing functions. This would quickly and
efficiently achieve the goal of "reduce size". And some form of sophisticated
prelinking would be my proposal for something more systematic, which
might come a lot cheaper with the security features to be added.


I also agree that PEI has acquired far too much functionality over
time, and we badly need to find some way to reverse that.  PEI is
pulling in more and more of the h/w initialization, leaving DXE mainly
to provide the boot interface (ACPI, UEFI services, setup screens,
etc.)  That puts more and more pressure on XIP storage and
cache-as-RAM.  How can we encourage a change?
I'm not in the loop enough to comment on the XIP and CAR issues.
However, HW init moving to PEI (or more generally, into a single stage) is not
necessarily a bad thing, is it? As I've said before, I'm not sure what else to
think of PEI than "DXE light". Both "initialise some of the hardware" with no
clear scope boundary but with similar interfaces. If they were sort of fused to
provide a minimal "pre-memory PEI" and a more DXE-like "post-memory PEI"
without having a clear boundary between HW init steps, would that be too
bad? Now I'm *really* just blindly guessing because I barely looked into this
project, but isn't that somewhat the idea of SlimBootloader?

Your description of DXE sort of is it collapsing into an advanced BDS.
Nate earlier said he'd like a clear distinction between DXE and BDS, which I
didn't really get the point of.
I guess what I'm trying to express is where does DXE end and where does BDS begin? A lot of drivers that are only used in BDS get dispatched early in DXE. Conversely, it is entirely possible for some DXE drivers to only run after we have technically entered the BDS phase. The line where DXE ends and BDS begins is blurry and not really a line but more like a rectangle.

To Isaac's point, there is a good argument to be made for beefing up PEI's capabilities to the point that DXE and BDS more or less become one thing. But what does that mean for PEI? Do we change from the 1-D PPI database to the 2-D Handle database in order to express device topologies better for example? That would be useful for stuff like the PEI UFS drivers as they run into difficulties expressing LUN IDs that the DXE version does not for example. Not sure if I would actually formally propose that in the PI WG but interesting stuff to think about regardless.

But if PEI consumed the HW init portions of DXE,
a more advanced BDS (which also powers e.g. the HID stuff on the way)
would be the logical remainder stage to separately support UEFI booting as a
payload (for SlimBootloader, coreboot, ...).

All of that was logically speaking. Even if the development was driven to its
extreme, PEI and DXE cores would still share a bunch of code and it might
make sense to share a foundation technically.

Best regards,
Marvin


Thanks,
Brian J. Johnson


On 4/15/22 03:15, Nate DeSimone wrote:
Hi Andrew,

On 4/14/22, 7:43 PM, "Andrew Fish" <afish@...> wrote:


On Apr 14, 2022, at 6:06 PM, Nate DeSimone
<nathaniel.l.desimone@...> wrote:

Hi Marvin,

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
Marvin Häuser
Sent: Thursday, April 14, 2022 12:56 AM
To: discuss@edk2.groups.io; Desimone, Nathaniel L
<nathaniel.l.desimone@...>
Cc: Pedro Falcato <pedro.falcato@...>; edk2-devel-groups-io
<devel@edk2.groups.io>; adachristine18@...; Shi, Steven
<steven.shi@...>
Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal

I feel like there are so many much easier solutions to this
problem that are at most limited by the clear specification. The
UEFI specification with regards to booting and all of that
obviously is of utmost importance.
If you have a better idea that retains compatibility with the
existing UEFI PI then I would be happy to hear it. Ultimately
anything we do needs to be a pure extension that retains
compatibility with old code. Given that restriction having the
ability to coalesce all the LibraryClasses into a single module and
have cross-module linking seems like the best way to handle it to me.

The UEFI PI specification parts that deal about internal
structure, as far as I know, are only in place to make it easy to
integrate Intel IP.
Its not just for Intel. The biggest reason for it to increase the
standardization of the boot flow across the PC ecosystem. We have
learned from experience that firmware is super critical to get a
product out the door but it is also difficult to write. So we try
to make it as reusable as humanly possible.

In fact, I don’t *know*, but I’m pretty sure the very strict
separation between PEI and DXE was preserved mostly because MRC
was 32-bit-only for a long time. Glad that seems to have been
resolved, AMD does memory init by PSP nowadays.
Having less complex early stages chain load more complex later
stages is a common design pattern in firmware, not just UEFI. For
example, your typical ARM system loads kinda like this:

PBL (SoC ROM) --> SBL (RAM Init) --> ARM Trust Zone --> Hypervisor
--> EDK II or U-Boot or LittleKernel (which runs android fastboot)

Comparing relative complexity I believe the Intel UEFI PI design is
actually pretty simple when you consider how much it gets done:

Ucode ROM --> SEC + PEI --> DXE + SMM + BDS

My biggest criticism of the design is that the strict separation
between PEI and DXE doesn't exist between DXE, SMM, and BDS 😊

There are a few reasons why PEI was 32-bit for quite some time. The
biggest one is the code size increase, 64-bit x86 code is 20-30%
larger than 32-bit x86 code. Since the only RAM Pre-Memory code has
access to is the cache onboard the processor and for security
reasons all that code has to fit within that RAM we generally do
everything we can to make that image as small as possible. Second,
64-bit requires a page table and since we desired to keep PEI
simple we tried to avoid that. Finally, the PI spec didn't allow a
64-bit PEI until recently. MRC is 32-bit code just because that is
what PEI happens to be. Porting it to 64-bit is not terribly
difficult.

Ultimately the mix of 32/64-bit does cause some difficulties. Any
data structures that get shared between PEI and DXE (HOBs, PCDs,
etc.) need to resolve to the same size and packing. LibraryClasses
need to be written to compile properly in both modes. In the case
of FSP API mode you need to resort to thunking between 32 and
64-bit modes during DXE. More or less we decided that the costs are
starting to outweigh the advantages.
I’d also point out that x86 VMs use X64 (x86-64) PEI today and it
works so the 32-bit/64-bit mix has nothing to do with UEFI/PI/edk2.

In the PC ecosystem a single chipset family can power thousands of
unique designs. So the  DRAM memory needs to be external, support
lots of different chipset packages(signal integrity...), support the
lowest cost through the highest cost DRAM and thousands of different
board layouts. So programing DRAM takes a masters degree in antenna
design. I’ve seen MRC (Memory Reference Code) with over a MiB of
DEBUG prints in it, and it literally is printing histograms of what
it is tuning. So all this code has to run before the system has any
DRAM, thus it is running using the cache as RAM. I’ve not looked at
the x86 architecture specs form the vendors in a while, but back in
the day they did not support page tables in ROM or pinned cached.
Now it might work, but if it breaks your CPU vendor blames you so
you don’t code PEI in X64….

We contributed the 1st edk2 ARM platform, Beagle Board, and It was a
long time ago but I seem to remember the mask ROM used a table in
NOR FLASH to init memory and then copied an image from NOR FLASH
into DRAM and jumped to it. So PEI is kind of not really needed and
we implemented a PrePEI and jumped directly to DXE.

Given I was around back in the day when all this stuff was designed
I can say SEC was always a place holder for security code, as
security code always has to run 1st. PEI (Pre EFI) was designed to
get DRAM programmed and then jump to DXE. It kind of also fell in
naturally to ACPI S3 flow since that was turning memory back on.
When we designed PEI we kind of though of it more like a boot loader
stage for the firmware that turned on memory and all the work would
happen in DXE. Then reality strikes and the existing BIOS assembly
programmers start learning C (lots of  cranky people) and they start
having to learn all about PEI to turn on memory. They had to write a
big chunk of code for the memory init in PEI. These folks had never
written any EFI code, so to them it was easier to move a lot of the
chipset init code into PEI as that is the world they had to figure
out to get memory turned on. I mean why learn EFI if you don’t have
to? So that is how we get so much code in IA32 (i386) on some
platforms. This start kind of biased future choices and how to
enable non edk2 code bases….
One of the big reasons a lot of code that should have been written in
DXE ended up in PEI is unfortunately due to the FSP and its inability
to support DXE code.


Thanks,

Andrew Fish



For many good reasons, Linux does not provide a stable kernel API.
This
allows to easily deploy breaking changes across the entire codebase.
Obviously, this is infeasible at a large scale when you need to
integrate IP blobs, but it would already help to break the
expectation that UEFI PI is a perfectly forwards- and
backwards-compatible system. DXE has SetMem and CopyMem as
part of
gBS. Maybe I don’t like it that much as part of the spec itself,
but it’s a good idea technically. I’d probably opt to have a
quickly accessible implementation detail of important function
pointers appended to PEI and DXE services, used by in-tree
modules. This may break both
forwards- and backwards-compatibility, but as it only applies to
in-tree modules, that is fine so long as we let go of the
compatibility notions. PPIs and protocols are an option too of
course, but they have a lookup performance penalty. Compared to
dynamic linking, that should hopefully be negligible however.

Absolutely optional to read, I don’t intend to waste anyone’s time
much, some philosophical stuff about my rationale:

If you started UEFI from scratch right now, would it have strictly
separated PEI and DXE?
For sure a clean slate design started today would look a lot
different than PEI/DXE... which was a clean slate design circa 1998
😊. In my opinion, if we were to suddenly go back to the drawing
board we would build something that is much closer to a full OS now
than we did back then. There have been cases where not being able
to use interrupt handlers and not having thread scheduling has
prevented implementation of desired features. The ARM guys built
LittleKernel (https://github.com/littlekernel/lk) for a lot of
these reasons. In the data center world some have decided to go to
the extreme of putting an entire copy of Linux in SPI so they can
do a network boot that downloads the OS image using BitTorrent!

The duplication between PEI and DXE core, and by extension MM
core, would be my most obvious place to start reducing size. I
would probably opt for a PEI/DXE hybrid where it starts in
„minimal mode“ (maybe think of PEI more like a microkernel here)
and after memory is initialised, the rest of DXE is loaded. And MM
core would get no loading at all, this requirement has gladly been
dropped ages ago. Just one prelinked snapshot of the address space
with a relocation table and a safe self-relocator on entry (this
is needed at the very least for ARM).

Ironically, with my idea for MM, dynamic loading would be free as
everything is prelinked anyway. The same is true for PEI XIP, it
is prelinked by nature.
Actually Post-Memory PEI can have non-prelinked PEIMs. And that
does get used for the PEI GOP driver.

What I do not like is the additional dynamic linking code at
load-time for non- XIP modules. Though, the more I think about it,
the more I wonder whether not the entirety of UEFI could be
composed of prelinked, relocatable memory snapshots without
traditional image loading entirely (for in-FW stuff). macOS has a
similar concept with its “Kernel Collections”.
Well, way
too much off-topic now. :)
If you make the assumption that 100% of the code is compiled all at
once then yes that works. UEFI was designed so that assumption does
not need to be true. There are good use cases for it: OpROMs,
generic OS loaders, network boot, etc.

Thanks,
Nate


Why am I explaining all this despite the fact everyone knows this
will never happen? Because I don’t like the notion of fixing
issues of an already overcomplicated system by adding even more
complicated stuff.
Especially
when the existing overcomplicated stuff is already uncomfortably
broken.

Best regards,
Marvin

For XIP PEI code… it will really help and would be very timely
since the
transition of PEI from 32-bit to 64-bit is going to increase the
size of PEI by ~20%.

Thanks,
Nate

From: Pedro Falcato <pedro.falcato@...>
Sent: Wednesday, April 13, 2022 11:43 AM
To: edk2-devel-groups-io <devel@edk2.groups.io>; Marvin Häuser
<mhaeuser@...>
Cc: discuss@edk2.groups.io; adachristine18@...; Desimone,
Nathaniel L <nathaniel.l.desimone@...>; Shi, Steven
<steven.shi@...>
Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Hi Marvin, Ada,

Some comments:

I don't think the purpose of the dynamic linker is to treat EFI
as a complete operating system, but to try to eliminate the
static linking that may be needlessly duplicating code that could
instead be put in a
single dynamic library. For instance, MdePkg and MdeModulePkg are
linked into a *lot* of .efi, instead of being just a library. It'd
be nice to see some numbers on this (something like Google's
bloaty could be run on every .efi file, in order to understand how
much file space we would actually save).

Other comments inline.

On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser
<mhaeuser@...<mailto:mhaeuser@...>> wrote:

On 13. Apr 2022, at 16:38, Ada Christine
<adachristine18@...<mailto:adachristine18@...>>
wrote:
i was replying via the groups.io<http://groups.io > web
interface, I'm guessing that messed up the thread? i haven't used
mailing lists before and don't know how they work. I'll use my
mail client from here on.

I'm on board with not treating EFI as an operating system. the
more i think about it the more it looks like scope creep.

Agreed.


I'm not quite as enthusiastic
about it as i was at first glance.

I'm still keen on doing my gsoc proposal for edk, though, and
even if this task and the acpica application are decided to be
out of scope unit testing,

How about fuzz-testing? This is also something edk2 needs quite
badly. At
Acidanthera, we compile edk2 code in userspace outside the edk2
build system and fuzz with dummy applications.

Note: fuzzing is also part of the LLVM instrumentation suite (see
https://llvm.org/docs/LibFuzzer.html ) and is something I could
happily mentor.
clang integration

Pedro and Vitaly are looking for someone to finish ASan:
https://edk2.groups.io/g/devel/topic/90010978#87991
There are working UBSan concepts, but they also need to be
mainlined.

Is Vitaly going to be a mentor? I was assuming it was going to be
me and
some other, more senior, mentor (possibly Steven Shi, which I
included in the task).
Anyway, re: ASAN, if the project includes ASAN, UBSAN and
possibly some other sanitizer it's quite possible that it could
be considered a large
project (which means more hours but a larger stipend too). Fuzzing
+ coverage could be very nice additions to this project idea.
Also, is stress-testing a decent idea?


and source-level debugging are all relevant to my interests.

how about your ideas for security stuff?

I want the entirety of MM to leverage SmmMemLib and to support
SMAP.
SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib
would
only work on MMRAM. Also evaluation of how to best avoid pointers
in MM communication buffers would be nice.

There also is a bunch of other stuff, like working out moving a
part of
CpuDxe into DxeCore to have memory protection live immediately,
memory protection in PEI, a replacement for the TE format (it’s
buggy and most platforms mostly abandoned it over various issues),
and alternatives to guarding critical code with SMM (like allowing
NVRAM commits only as part of a reboot).

I personally find all of those projects very important, but I
cannot promise many people agree. Especially those that impose
global changes (most notably the TE replacement) may be very
tedious to submit.
Gladly, I believe you can submit multiple proposals (?)

Best regards,
Marvin


I'm not very knowledgeable about
trusted platform or secure boot but I'm willing to learn whatever
is necessary to get something spun up for my proposal.

On Wed, Apr 13, 2022, 12:05 Marvin Häuser
<mhaeuser@...<mailto:mhaeuser@...>> wrote:


Do you use the “reply all” option in your mail client? Looks like
my CCs have been dropped again. Comments inline.

On 13. Apr 2022, at 12:54, Ada Christine
<adachristine18@...<mailto:adachristine18@...>>
wrote:
Hi, Marvin

Its similarity to my own latest experiment is the key to what
grabbed my attention. I have no particular use case in mind for
it, but I see its potential for anybody developing larger
applications in that when a library is changed there's no need to
distribute a new version of the whole binary, just the relevant
library module.

I really do not like the trend of treating UEFI as a full-fledged
OS - it is not. The most used UEFI applications, OS loaders, are
really not that huge and are distributed as part of the OS image
anyway.
Even for
less used applications, you will always get a full snapshot anyhow.
Gladly we don’t have auto-update and package management yet. :)


I slept on it and it occurred to me that the whole thing could
operate
similarly to the shell protocol in that the linker/loader is
itself an
application that does a LoadImage() on the application needing
dynamic
linking facilities.

That would mean the linker itself is shipped with every application
that requires it? Otherwise it doesn’t make much sense for it to
be an
app and below’s problems apply.

If however the whole plan is making the linker as a DXE and
including
it with the firmware, that I'm not quite as sure about. That would
necessarily tie any applications using dynamic linking to
TianoCore or
any firmware distribution that derives from it.

I think that was the idea referred to as “edk2 core” by Steven, but
I’d like to hear his proposal to be sure. Virtually everyone uses
edk2, so that itself is not the problem, but versioning is. Vendors
are slow to update their snapshots or have just given up doing that
entirely. Distributing it for external applications like OS loaders
would mean this can be leveraged probably no earlier than 10 years
from now. And for in-firmware things, I have a hard time thinking
about a
use-case that outweighs the drawbacks.


To shift the topic slightly back to GSoC, however, I'm willing to
work
on other items on the task list. Unit testing and an ACPICA
application are the alternative projects I had thought about. I need
to choose fairly soon as the proposal deadline is next Tuesday. I
know
a tiny bit about porting ACPICA as I also have plans to
incorporate it into my
own project.

I have a few more ideas for security stuff, but Nate did not confirm
them as appropriate yet and I’m not here to drive you away from
this
specific task (or the others). However, I’m still curious and
concerned. :)

Best regards,
Marvin








--
Pedro Falcato




















Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Marvin Häuser <mhaeuser@...>
 

Hey Isaac,

Thanks for your input! :)

On 15.04.22 20:47, Oram, Isaac W wrote:
In practice, it feels like we have evolved far beyond original PEI scope for natural and good reasons. Initially, it was very resource constrained. 8KB CAR for data only. That is no longer a real constraint for most HW. And thus we get things like MP services early, writable variables early, complex PCD databases, and such.
Looking at the amount of things that are duplicated from DXE to PEI (MP, variable services, PCD, GOP, ...), this sounds like all the more reason to me to have some unified technological foundation for PEI and DXE core. With mixed-architecture execution decaying (at least for the time being), what'd be the point of duplicating this code in both stages? Of course this hypothetical shared foundation should be flexible enough to be able to build distinct PEI and DXE cores to allow for UEFI PI and UEFI to be implemented separately from each other.


I don't think it is bad. The PEI environment is much friendlier for developing and debugging ubiquitous silicon initialization than DXE. Ubiquitous meaning you pretty much need to do it on any boot path to any boot device. Pre Memory PEI and Post Memory PEI seem to be the right design for many things. It generally simplifies handshaking between early and later silicon code.

Thus I like:
SEC: Basics
PEI Pre-Mem: Get memory
PEI Post-Mem: Complete all "always required" silicon init. Can also do silicon init for more optional platform features. Should not do silicon init for dynamically optional features.
Standalone MM: Kicked off from post memory PEI.
DXE: Setup driver and UEFI environments
BDS: Initialize boot target specific required HW, provide any required user interface, boot OS
Yes! :)


I think that this kind of design division makes a solution very flexible for supporting a wide range of designs. It allows more generic/reusable DXE and BDS. It makes SMM a more optional solution. We have launched Linux as an embedded OS or target OS from DXE IPL, DXE Core, BDS, etc. It supports that kind of flexibility. It enables LinuxBoot @ BDS and one day in place of DXE (Universal Payload).
In that hypothetical model, yeah, I think DXE and BDS are *mostly* the same thing. BDS of course is a distinctly defined sub-process of determining the boot option, but the whole DXE+BDS stage combination would just be the generic "user experience" portion of UEFI. And I totally see things being interchangeable both ways, be that e.g. Slimbootloader + UEFI DXE/BDS or UEFI PEI + LinuxBoot.

And fundamentally, it can reduce complexity because we can reduce the handshaking across phases. That tends to introduce complexity that the above helps reduce.

We tried to incorporate a lot of this in the Minimum Platform Architecture spec and it has been working fairly well. It is hard to migrate the large mass of existing code, so we don't have a perfect example of this in action. But the vision is that you can design such that the phases are well bounded and very testable. And this will help silicon development, board porting, and adding features. If we succeed, we probably come back to ask for changes to make the pre-memory to post-memory handoff better. I find few people really know that well and we have repeated performance problems. But generally, there is plenty of flexibility in the existing architecture and phases to tune this for many answers.

Specific to the discussion of DLL like behavior. I agree it should be handled by tools and not a unique and complex answer. It is actually a trivial change to take existing PEIM (for example) and just add the entry point as a library class constructor and then you can link a bunch of PEIM into a single image. It would be pretty minor to make this easy and supported I would think. You lose dynamic dispatch, obviously. But it is a short hack to eliminate early library duplication.
This would actually be a great thing to have upstream (as in, something more straight-forward that'd allow modules to be merged without modifications, especially taking Depex into account somehow)! Amaranth will purge any sort of linking (at the very least downstream) and will mostly be just one monolith. I do not see that as an alternative to the previously discussed implementations of and alternatives to dynamic linking however, but as something complementary. Has there been any prior work on this?

Best regards,
Marvin


Regards,
Isaac

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Marvin Häuser
Sent: Friday, April 15, 2022 9:44 AM
To: discuss@edk2.groups.io; Johnson, Brian <brian.johnson@...>; Desimone, Nathaniel L <nathaniel.l.desimone@...>; Andrew Fish <afish@...>; devel@edk2.groups.io
Cc: Pedro Falcato <pedro.falcato@...>; adachristine18@...; Shi, Steven <steven.shi@...>
Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Hey Brian,

On 15.04.22 18:22, Brian J. Johnson wrote:
Nate, Andrew, Marvin, Pedro, Ada, et al,

This is a great discussion.  I've been debating where to weigh in...
Personally, I don't think this topic can get enough attention. Thanks! :)

I agree that some sort of library sharing to reduce image size would
be very helpful.  Although some parts of the BIOS are built
separately, large parts of it are compiled at the same time, so there
should be ample opportunity for tool-directed sharing.  Some form of
pre-linking modules together may be the easiest way to do that.  But
however it gets implemented, we should try to make it automatic:
requiring library writers to manually add a lot of metadata or write
thunks or shims would be a barrier to adoption.
My suggestion to use a private function array would indeed require shims, however this would be the "better than nothing I guess" solution where only the most used and most painful to duplicate functions are shared, e.g., memory and maybe common string printing functions. This would quickly and efficiently achieve the goal of "reduce size". And some form of sophisticated prelinking would be my proposal for something more systematic, which might come a lot cheaper with the security features to be added.

I also agree that PEI has acquired far too much functionality over
time, and we badly need to find some way to reverse that.  PEI is
pulling in more and more of the h/w initialization, leaving DXE mainly
to provide the boot interface (ACPI, UEFI services, setup screens,
etc.)  That puts more and more pressure on XIP storage and
cache-as-RAM.  How can we encourage a change?
I'm not in the loop enough to comment on the XIP and CAR issues.
However, HW init moving to PEI (or more generally, into a single stage) is not necessarily a bad thing, is it? As I've said before, I'm not sure what else to think of PEI than "DXE light". Both "initialise some of the hardware" with no clear scope boundary but with similar interfaces. If they were sort of fused to provide a minimal "pre-memory PEI" and a more DXE-like "post-memory PEI" without having a clear boundary between HW init steps, would that be too bad? Now I'm *really* just blindly guessing because I barely looked into this project, but isn't that somewhat the idea of SlimBootloader?

Your description of DXE sort of is it collapsing into an advanced BDS.
Nate earlier said he'd like a clear distinction between DXE and BDS, which I didn't really get the point of. But if PEI consumed the HW init portions of DXE, a more advanced BDS (which also powers e.g. the HID stuff on the way) would be the logical remainder stage to separately support UEFI booting as a payload (for SlimBootloader, coreboot, ...).

All of that was logically speaking. Even if the development was driven to its extreme, PEI and DXE cores would still share a bunch of code and it might make sense to share a foundation technically.

Best regards,
Marvin

Thanks,
Brian J. Johnson


On 4/15/22 03:15, Nate DeSimone wrote:
Hi Andrew,

On 4/14/22, 7:43 PM, "Andrew Fish" <afish@...> wrote:

On Apr 14, 2022, at 6:06 PM, Nate DeSimone
<nathaniel.l.desimone@...> wrote:

Hi Marvin,

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
Marvin Häuser
Sent: Thursday, April 14, 2022 12:56 AM
To: discuss@edk2.groups.io; Desimone, Nathaniel L
<nathaniel.l.desimone@...>
Cc: Pedro Falcato <pedro.falcato@...>; edk2-devel-groups-io
<devel@edk2.groups.io>; adachristine18@...; Shi, Steven
<steven.shi@...>
Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal

I feel like there are so many much easier solutions to this
problem that are at most limited by the clear specification. The
UEFI specification with regards to booting and all of that
obviously is of utmost importance.
If you have a better idea that retains compatibility with the
existing UEFI PI then I would be happy to hear it. Ultimately
anything we do needs to be a pure extension that retains
compatibility with old code. Given that restriction having the
ability to coalesce all the LibraryClasses into a single module and
have cross-module linking seems like the best way to handle it to me.

The UEFI PI specification parts that deal about internal
structure, as far as I know, are only in place to make it easy to
integrate Intel IP.
Its not just for Intel. The biggest reason for it to increase the
standardization of the boot flow across the PC ecosystem. We have
learned from experience that firmware is super critical to get a
product out the door but it is also difficult to write. So we try
to make it as reusable as humanly possible.

In fact, I don’t *know*, but I’m pretty sure the very strict
separation between PEI and DXE was preserved mostly because MRC
was 32-bit-only for a long time. Glad that seems to have been
resolved, AMD does memory init by PSP nowadays.
Having less complex early stages chain load more complex later
stages is a common design pattern in firmware, not just UEFI. For
example, your typical ARM system loads kinda like this:

PBL (SoC ROM) --> SBL (RAM Init) --> ARM Trust Zone --> Hypervisor
--> EDK II or U-Boot or LittleKernel (which runs android fastboot)

Comparing relative complexity I believe the Intel UEFI PI design is
actually pretty simple when you consider how much it gets done:

Ucode ROM --> SEC + PEI --> DXE + SMM + BDS

My biggest criticism of the design is that the strict separation
between PEI and DXE doesn't exist between DXE, SMM, and BDS 😊

There are a few reasons why PEI was 32-bit for quite some time. The
biggest one is the code size increase, 64-bit x86 code is 20-30%
larger than 32-bit x86 code. Since the only RAM Pre-Memory code has
access to is the cache onboard the processor and for security
reasons all that code has to fit within that RAM we generally do
everything we can to make that image as small as possible. Second,
64-bit requires a page table and since we desired to keep PEI
simple we tried to avoid that. Finally, the PI spec didn't allow a
64-bit PEI until recently. MRC is 32-bit code just because that is
what PEI happens to be. Porting it to 64-bit is not terribly
difficult.

Ultimately the mix of 32/64-bit does cause some difficulties. Any
data structures that get shared between PEI and DXE (HOBs, PCDs,
etc.) need to resolve to the same size and packing. LibraryClasses
need to be written to compile properly in both modes. In the case
of FSP API mode you need to resort to thunking between 32 and
64-bit modes during DXE. More or less we decided that the costs are
starting to outweigh the advantages.
I’d also point out that x86 VMs use X64 (x86-64) PEI today and it
works so the 32-bit/64-bit mix has nothing to do with UEFI/PI/edk2.

In the PC ecosystem a single chipset family can power thousands of
unique designs. So the  DRAM memory needs to be external, support
lots of different chipset packages(signal integrity...), support the
lowest cost through the highest cost DRAM and thousands of different
board layouts. So programing DRAM takes a masters degree in antenna
design. I’ve seen MRC (Memory Reference Code) with over a MiB of
DEBUG prints in it, and it literally is printing histograms of what
it is tuning. So all this code has to run before the system has any
DRAM, thus it is running using the cache as RAM. I’ve not looked at
the x86 architecture specs form the vendors in a while, but back in
the day they did not support page tables in ROM or pinned cached.
Now it might work, but if it breaks your CPU vendor blames you so
you don’t code PEI in X64….

We contributed the 1st edk2 ARM platform, Beagle Board, and It was a
long time ago but I seem to remember the mask ROM used a table in
NOR FLASH to init memory and then copied an image from NOR FLASH
into DRAM and jumped to it. So PEI is kind of not really needed and
we implemented a PrePEI and jumped directly to DXE.

Given I was around back in the day when all this stuff was designed
I can say SEC was always a place holder for security code, as
security code always has to run 1st. PEI (Pre EFI) was designed to
get DRAM programmed and then jump to DXE. It kind of also fell in
naturally to ACPI S3 flow since that was turning memory back on.
When we designed PEI we kind of though of it more like a boot loader
stage for the firmware that turned on memory and all the work would
happen in DXE. Then reality strikes and the existing BIOS assembly
programmers start learning C (lots of  cranky people) and they start
having to learn all about PEI to turn on memory. They had to write a
big chunk of code for the memory init in PEI. These folks had never
written any EFI code, so to them it was easier to move a lot of the
chipset init code into PEI as that is the world they had to figure
out to get memory turned on. I mean why learn EFI if you don’t have
to? So that is how we get so much code in IA32 (i386) on some
platforms. This start kind of biased future choices and how to
enable non edk2 code bases….
One of the big reasons a lot of code that should have been written in
DXE ended up in PEI is unfortunately due to the FSP and its inability
to support DXE code.

Thanks,

Andrew Fish


For many good reasons, Linux does not provide a stable kernel API.
This
allows to easily deploy breaking changes across the entire codebase.
Obviously, this is infeasible at a large scale when you need to
integrate IP blobs, but it would already help to break the
expectation that UEFI PI is a perfectly forwards- and
backwards-compatible system. DXE has SetMem and CopyMem as part of
gBS. Maybe I don’t like it that much as part of the spec itself,
but it’s a good idea technically. I’d probably opt to have a
quickly accessible implementation detail of important function
pointers appended to PEI and DXE services, used by in-tree
modules. This may break both
forwards- and backwards-compatibility, but as it only applies to
in-tree modules, that is fine so long as we let go of the
compatibility notions. PPIs and protocols are an option too of
course, but they have a lookup performance penalty. Compared to
dynamic linking, that should hopefully be negligible however.

Absolutely optional to read, I don’t intend to waste anyone’s time
much, some philosophical stuff about my rationale:

If you started UEFI from scratch right now, would it have strictly
separated PEI and DXE?
For sure a clean slate design started today would look a lot
different than PEI/DXE... which was a clean slate design circa 1998
😊. In my opinion, if we were to suddenly go back to the drawing
board we would build something that is much closer to a full OS now
than we did back then. There have been cases where not being able
to use interrupt handlers and not having thread scheduling has
prevented implementation of desired features. The ARM guys built
LittleKernel (https://github.com/littlekernel/lk) for a lot of
these reasons. In the data center world some have decided to go to
the extreme of putting an entire copy of Linux in SPI so they can
do a network boot that downloads the OS image using BitTorrent!

The duplication between PEI and DXE core, and by extension MM
core, would be my most obvious place to start reducing size. I
would probably opt for a PEI/DXE hybrid where it starts in
„minimal mode“ (maybe think of PEI more like a microkernel here)
and after memory is initialised, the rest of DXE is loaded. And MM
core would get no loading at all, this requirement has gladly been
dropped ages ago. Just one prelinked snapshot of the address space
with a relocation table and a safe self-relocator on entry (this
is needed at the very least for ARM).

Ironically, with my idea for MM, dynamic loading would be free as
everything is prelinked anyway. The same is true for PEI XIP, it
is prelinked by nature.
Actually Post-Memory PEI can have non-prelinked PEIMs. And that
does get used for the PEI GOP driver.

What I do not like is the additional dynamic linking code at
load-time for non- XIP modules. Though, the more I think about it,
the more I wonder whether not the entirety of UEFI could be
composed of prelinked, relocatable memory snapshots without
traditional image loading entirely (for in-FW stuff). macOS has a
similar concept with its “Kernel Collections”.
Well, way
too much off-topic now. :)
If you make the assumption that 100% of the code is compiled all at
once then yes that works. UEFI was designed so that assumption does
not need to be true. There are good use cases for it: OpROMs,
generic OS loaders, network boot, etc.

Thanks,
Nate

Why am I explaining all this despite the fact everyone knows this
will never happen? Because I don’t like the notion of fixing
issues of an already overcomplicated system by adding even more
complicated stuff.
Especially
when the existing overcomplicated stuff is already uncomfortably
broken.

Best regards,
Marvin

For XIP PEI code… it will really help and would be very timely
since the
transition of PEI from 32-bit to 64-bit is going to increase the
size of PEI by ~20%.
Thanks,
Nate

From: Pedro Falcato <pedro.falcato@...>
Sent: Wednesday, April 13, 2022 11:43 AM
To: edk2-devel-groups-io <devel@edk2.groups.io>; Marvin Häuser
<mhaeuser@...>
Cc: discuss@edk2.groups.io; adachristine18@...; Desimone,
Nathaniel L <nathaniel.l.desimone@...>; Shi, Steven
<steven.shi@...>
Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Hi Marvin, Ada,

Some comments:

I don't think the purpose of the dynamic linker is to treat EFI
as a complete operating system, but to try to eliminate the
static linking that may be needlessly duplicating code that could
instead be put in a
single dynamic library. For instance, MdePkg and MdeModulePkg are
linked into a *lot* of .efi, instead of being just a library. It'd
be nice to see some numbers on this (something like Google's
bloaty could be run on every .efi file, in order to understand how
much file space we would actually save).
Other comments inline.

On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser
<mhaeuser@...<mailto:mhaeuser@...>> wrote:
On 13. Apr 2022, at 16:38, Ada Christine
<adachristine18@...<mailto:adachristine18@...>> wrote:
i was replying via the groups.io<http://groups.io > web
interface, I'm guessing that messed up the thread? i haven't used
mailing lists before and don't know how they work. I'll use my
mail client from here on.

I'm on board with not treating EFI as an operating system. the
more i think about it the more it looks like scope creep.

Agreed.


I'm not quite as enthusiastic
about it as i was at first glance.

I'm still keen on doing my gsoc proposal for edk, though, and
even if this task and the acpica application are decided to be
out of scope unit testing,

How about fuzz-testing? This is also something edk2 needs quite
badly. At
Acidanthera, we compile edk2 code in userspace outside the edk2
build system and fuzz with dummy applications.
Note: fuzzing is also part of the LLVM instrumentation suite (see
https://llvm.org/docs/LibFuzzer.html ) and is something I could
happily mentor.
clang integration

Pedro and Vitaly are looking for someone to finish ASan:
https://edk2.groups.io/g/devel/topic/90010978#87991
There are working UBSan concepts, but they also need to be
mainlined.

Is Vitaly going to be a mentor? I was assuming it was going to be
me and
some other, more senior, mentor (possibly Steven Shi, which I
included in the task).
Anyway, re: ASAN, if the project includes ASAN, UBSAN and
possibly some other sanitizer it's quite possible that it could
be considered a large
project (which means more hours but a larger stipend too). Fuzzing
+ coverage could be very nice additions to this project idea.
Also, is stress-testing a decent idea?


and source-level debugging are all relevant to my interests.

how about your ideas for security stuff?

I want the entirety of MM to leverage SmmMemLib and to support SMAP.
SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib would
only work on MMRAM. Also evaluation of how to best avoid pointers
in MM communication buffers would be nice.
There also is a bunch of other stuff, like working out moving a
part of
CpuDxe into DxeCore to have memory protection live immediately,
memory protection in PEI, a replacement for the TE format (it’s
buggy and most platforms mostly abandoned it over various issues),
and alternatives to guarding critical code with SMM (like allowing
NVRAM commits only as part of a reboot).
I personally find all of those projects very important, but I
cannot promise many people agree. Especially those that impose
global changes (most notably the TE replacement) may be very
tedious to submit.
Gladly, I believe you can submit multiple proposals (?)

Best regards,
Marvin


I'm not very knowledgeable about
trusted platform or secure boot but I'm willing to learn whatever
is necessary to get something spun up for my proposal.

On Wed, Apr 13, 2022, 12:05 Marvin Häuser
<mhaeuser@...<mailto:mhaeuser@...>> wrote:

Do you use the “reply all” option in your mail client? Looks like
my CCs have been dropped again. Comments inline.

On 13. Apr 2022, at 12:54, Ada Christine
<adachristine18@...<mailto:adachristine18@...>>
wrote:
Hi, Marvin

Its similarity to my own latest experiment is the key to what
grabbed my attention. I have no particular use case in mind for
it, but I see its potential for anybody developing larger
applications in that when a library is changed there's no need to
distribute a new version of the whole binary, just the relevant
library module.

I really do not like the trend of treating UEFI as a full-fledged
OS - it is not. The most used UEFI applications, OS loaders, are
really not that huge and are distributed as part of the OS image
anyway.
Even for
less used applications, you will always get a full snapshot anyhow.
Gladly we don’t have auto-update and package management yet. :)


I slept on it and it occurred to me that the whole thing could
operate
similarly to the shell protocol in that the linker/loader is
itself an
application that does a LoadImage() on the application needing
dynamic
linking facilities.

That would mean the linker itself is shipped with every application
that requires it? Otherwise it doesn’t make much sense for it to
be an
app and below’s problems apply.

If however the whole plan is making the linker as a DXE and
including
it with the firmware, that I'm not quite as sure about. That would
necessarily tie any applications using dynamic linking to
TianoCore or
any firmware distribution that derives from it.

I think that was the idea referred to as “edk2 core” by Steven, but
I’d like to hear his proposal to be sure. Virtually everyone uses
edk2, so that itself is not the problem, but versioning is. Vendors
are slow to update their snapshots or have just given up doing that
entirely. Distributing it for external applications like OS loaders
would mean this can be leveraged probably no earlier than 10 years
from now. And for in-firmware things, I have a hard time thinking
about a
use-case that outweighs the drawbacks.

To shift the topic slightly back to GSoC, however, I'm willing to
work
on other items on the task list. Unit testing and an ACPICA
application are the alternative projects I had thought about. I need
to choose fairly soon as the proposal deadline is next Tuesday. I
know
a tiny bit about porting ACPICA as I also have plans to
incorporate it into my
own project.
I have a few more ideas for security stuff, but Nate did not confirm
them as appropriate yet and I’m not here to drive you away from this
specific task (or the others). However, I’m still curious and
concerned. :)

Best regards,
Marvin








--
Pedro Falcato























Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Isaac Oram
 

In practice, it feels like we have evolved far beyond original PEI scope for natural and good reasons. Initially, it was very resource constrained. 8KB CAR for data only. That is no longer a real constraint for most HW. And thus we get things like MP services early, writable variables early, complex PCD databases, and such.

I don't think it is bad. The PEI environment is much friendlier for developing and debugging ubiquitous silicon initialization than DXE. Ubiquitous meaning you pretty much need to do it on any boot path to any boot device. Pre Memory PEI and Post Memory PEI seem to be the right design for many things. It generally simplifies handshaking between early and later silicon code.

Thus I like:
SEC: Basics
PEI Pre-Mem: Get memory
PEI Post-Mem: Complete all "always required" silicon init. Can also do silicon init for more optional platform features. Should not do silicon init for dynamically optional features.
Standalone MM: Kicked off from post memory PEI.
DXE: Setup driver and UEFI environments
BDS: Initialize boot target specific required HW, provide any required user interface, boot OS

I think that this kind of design division makes a solution very flexible for supporting a wide range of designs. It allows more generic/reusable DXE and BDS. It makes SMM a more optional solution. We have launched Linux as an embedded OS or target OS from DXE IPL, DXE Core, BDS, etc. It supports that kind of flexibility. It enables LinuxBoot @ BDS and one day in place of DXE (Universal Payload).
And fundamentally, it can reduce complexity because we can reduce the handshaking across phases. That tends to introduce complexity that the above helps reduce.

We tried to incorporate a lot of this in the Minimum Platform Architecture spec and it has been working fairly well. It is hard to migrate the large mass of existing code, so we don't have a perfect example of this in action. But the vision is that you can design such that the phases are well bounded and very testable. And this will help silicon development, board porting, and adding features. If we succeed, we probably come back to ask for changes to make the pre-memory to post-memory handoff better. I find few people really know that well and we have repeated performance problems. But generally, there is plenty of flexibility in the existing architecture and phases to tune this for many answers.

Specific to the discussion of DLL like behavior. I agree it should be handled by tools and not a unique and complex answer. It is actually a trivial change to take existing PEIM (for example) and just add the entry point as a library class constructor and then you can link a bunch of PEIM into a single image. It would be pretty minor to make this easy and supported I would think. You lose dynamic dispatch, obviously. But it is a short hack to eliminate early library duplication.

Regards,
Isaac

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Marvin Häuser
Sent: Friday, April 15, 2022 9:44 AM
To: discuss@edk2.groups.io; Johnson, Brian <brian.johnson@...>; Desimone, Nathaniel L <nathaniel.l.desimone@...>; Andrew Fish <afish@...>; devel@edk2.groups.io
Cc: Pedro Falcato <pedro.falcato@...>; adachristine18@...; Shi, Steven <steven.shi@...>
Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Hey Brian,

On 15.04.22 18:22, Brian J. Johnson wrote:
Nate, Andrew, Marvin, Pedro, Ada, et al,

This is a great discussion.  I've been debating where to weigh in...
Personally, I don't think this topic can get enough attention. Thanks! :)


I agree that some sort of library sharing to reduce image size would
be very helpful.  Although some parts of the BIOS are built
separately, large parts of it are compiled at the same time, so there
should be ample opportunity for tool-directed sharing.  Some form of
pre-linking modules together may be the easiest way to do that.  But
however it gets implemented, we should try to make it automatic:
requiring library writers to manually add a lot of metadata or write
thunks or shims would be a barrier to adoption.
My suggestion to use a private function array would indeed require shims, however this would be the "better than nothing I guess" solution where only the most used and most painful to duplicate functions are shared, e.g., memory and maybe common string printing functions. This would quickly and efficiently achieve the goal of "reduce size". And some form of sophisticated prelinking would be my proposal for something more systematic, which might come a lot cheaper with the security features to be added.


I also agree that PEI has acquired far too much functionality over
time, and we badly need to find some way to reverse that.  PEI is
pulling in more and more of the h/w initialization, leaving DXE mainly
to provide the boot interface (ACPI, UEFI services, setup screens,
etc.)  That puts more and more pressure on XIP storage and
cache-as-RAM.  How can we encourage a change?
I'm not in the loop enough to comment on the XIP and CAR issues.
However, HW init moving to PEI (or more generally, into a single stage) is not necessarily a bad thing, is it? As I've said before, I'm not sure what else to think of PEI than "DXE light". Both "initialise some of the hardware" with no clear scope boundary but with similar interfaces. If they were sort of fused to provide a minimal "pre-memory PEI" and a more DXE-like "post-memory PEI" without having a clear boundary between HW init steps, would that be too bad? Now I'm *really* just blindly guessing because I barely looked into this project, but isn't that somewhat the idea of SlimBootloader?

Your description of DXE sort of is it collapsing into an advanced BDS.
Nate earlier said he'd like a clear distinction between DXE and BDS, which I didn't really get the point of. But if PEI consumed the HW init portions of DXE, a more advanced BDS (which also powers e.g. the HID stuff on the way) would be the logical remainder stage to separately support UEFI booting as a payload (for SlimBootloader, coreboot, ...).

All of that was logically speaking. Even if the development was driven to its extreme, PEI and DXE cores would still share a bunch of code and it might make sense to share a foundation technically.

Best regards,
Marvin


Thanks,
Brian J. Johnson


On 4/15/22 03:15, Nate DeSimone wrote:
Hi Andrew,

On 4/14/22, 7:43 PM, "Andrew Fish" <afish@...> wrote:


On Apr 14, 2022, at 6:06 PM, Nate DeSimone
<nathaniel.l.desimone@...> wrote:

Hi Marvin,

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
Marvin Häuser
Sent: Thursday, April 14, 2022 12:56 AM
To: discuss@edk2.groups.io; Desimone, Nathaniel L
<nathaniel.l.desimone@...>
Cc: Pedro Falcato <pedro.falcato@...>; edk2-devel-groups-io
<devel@edk2.groups.io>; adachristine18@...; Shi, Steven
<steven.shi@...>
Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal

I feel like there are so many much easier solutions to this
problem that are at most limited by the clear specification. The
UEFI specification with regards to booting and all of that
obviously is of utmost importance.
If you have a better idea that retains compatibility with the
existing UEFI PI then I would be happy to hear it. Ultimately
anything we do needs to be a pure extension that retains
compatibility with old code. Given that restriction having the
ability to coalesce all the LibraryClasses into a single module and
have cross-module linking seems like the best way to handle it to me.

The UEFI PI specification parts that deal about internal
structure, as far as I know, are only in place to make it easy to
integrate Intel IP.
Its not just for Intel. The biggest reason for it to increase the
standardization of the boot flow across the PC ecosystem. We have
learned from experience that firmware is super critical to get a
product out the door but it is also difficult to write. So we try
to make it as reusable as humanly possible.

In fact, I don’t *know*, but I’m pretty sure the very strict
separation between PEI and DXE was preserved mostly because MRC
was 32-bit-only for a long time. Glad that seems to have been
resolved, AMD does memory init by PSP nowadays.
Having less complex early stages chain load more complex later
stages is a common design pattern in firmware, not just UEFI. For
example, your typical ARM system loads kinda like this:

PBL (SoC ROM) --> SBL (RAM Init) --> ARM Trust Zone --> Hypervisor
--> EDK II or U-Boot or LittleKernel (which runs android fastboot)

Comparing relative complexity I believe the Intel UEFI PI design is
actually pretty simple when you consider how much it gets done:

Ucode ROM --> SEC + PEI --> DXE + SMM + BDS

My biggest criticism of the design is that the strict separation
between PEI and DXE doesn't exist between DXE, SMM, and BDS 😊

There are a few reasons why PEI was 32-bit for quite some time. The
biggest one is the code size increase, 64-bit x86 code is 20-30%
larger than 32-bit x86 code. Since the only RAM Pre-Memory code has
access to is the cache onboard the processor and for security
reasons all that code has to fit within that RAM we generally do
everything we can to make that image as small as possible. Second,
64-bit requires a page table and since we desired to keep PEI
simple we tried to avoid that. Finally, the PI spec didn't allow a
64-bit PEI until recently. MRC is 32-bit code just because that is
what PEI happens to be. Porting it to 64-bit is not terribly
difficult.

Ultimately the mix of 32/64-bit does cause some difficulties. Any
data structures that get shared between PEI and DXE (HOBs, PCDs,
etc.) need to resolve to the same size and packing. LibraryClasses
need to be written to compile properly in both modes. In the case
of FSP API mode you need to resort to thunking between 32 and
64-bit modes during DXE. More or less we decided that the costs are
starting to outweigh the advantages.
I’d also point out that x86 VMs use X64 (x86-64) PEI today and it
works so the 32-bit/64-bit mix has nothing to do with UEFI/PI/edk2.

In the PC ecosystem a single chipset family can power thousands of
unique designs. So the  DRAM memory needs to be external, support
lots of different chipset packages(signal integrity...), support the
lowest cost through the highest cost DRAM and thousands of different
board layouts. So programing DRAM takes a masters degree in antenna
design. I’ve seen MRC (Memory Reference Code) with over a MiB of
DEBUG prints in it, and it literally is printing histograms of what
it is tuning. So all this code has to run before the system has any
DRAM, thus it is running using the cache as RAM. I’ve not looked at
the x86 architecture specs form the vendors in a while, but back in
the day they did not support page tables in ROM or pinned cached.
Now it might work, but if it breaks your CPU vendor blames you so
you don’t code PEI in X64….

We contributed the 1st edk2 ARM platform, Beagle Board, and It was a
long time ago but I seem to remember the mask ROM used a table in
NOR FLASH to init memory and then copied an image from NOR FLASH
into DRAM and jumped to it. So PEI is kind of not really needed and
we implemented a PrePEI and jumped directly to DXE.

Given I was around back in the day when all this stuff was designed
I can say SEC was always a place holder for security code, as
security code always has to run 1st. PEI (Pre EFI) was designed to
get DRAM programmed and then jump to DXE. It kind of also fell in
naturally to ACPI S3 flow since that was turning memory back on.
When we designed PEI we kind of though of it more like a boot loader
stage for the firmware that turned on memory and all the work would
happen in DXE. Then reality strikes and the existing BIOS assembly
programmers start learning C (lots of  cranky people) and they start
having to learn all about PEI to turn on memory. They had to write a
big chunk of code for the memory init in PEI. These folks had never
written any EFI code, so to them it was easier to move a lot of the
chipset init code into PEI as that is the world they had to figure
out to get memory turned on. I mean why learn EFI if you don’t have
to? So that is how we get so much code in IA32 (i386) on some
platforms. This start kind of biased future choices and how to
enable non edk2 code bases….
One of the big reasons a lot of code that should have been written in
DXE ended up in PEI is unfortunately due to the FSP and its inability
to support DXE code.


Thanks,

Andrew Fish



For many good reasons, Linux does not provide a stable kernel API.
This
allows to easily deploy breaking changes across the entire codebase.
Obviously, this is infeasible at a large scale when you need to
integrate IP blobs, but it would already help to break the
expectation that UEFI PI is a perfectly forwards- and
backwards-compatible system. DXE has SetMem and CopyMem as part of
gBS. Maybe I don’t like it that much as part of the spec itself,
but it’s a good idea technically. I’d probably opt to have a
quickly accessible implementation detail of important function
pointers appended to PEI and DXE services, used by in-tree
modules. This may break both
forwards- and backwards-compatibility, but as it only applies to
in-tree modules, that is fine so long as we let go of the
compatibility notions. PPIs and protocols are an option too of
course, but they have a lookup performance penalty. Compared to
dynamic linking, that should hopefully be negligible however.

Absolutely optional to read, I don’t intend to waste anyone’s time
much, some philosophical stuff about my rationale:

If you started UEFI from scratch right now, would it have strictly
separated PEI and DXE?
For sure a clean slate design started today would look a lot
different than PEI/DXE... which was a clean slate design circa 1998
😊. In my opinion, if we were to suddenly go back to the drawing
board we would build something that is much closer to a full OS now
than we did back then. There have been cases where not being able
to use interrupt handlers and not having thread scheduling has
prevented implementation of desired features. The ARM guys built
LittleKernel (https://github.com/littlekernel/lk) for a lot of
these reasons. In the data center world some have decided to go to
the extreme of putting an entire copy of Linux in SPI so they can
do a network boot that downloads the OS image using BitTorrent!

The duplication between PEI and DXE core, and by extension MM
core, would be my most obvious place to start reducing size. I
would probably opt for a PEI/DXE hybrid where it starts in
„minimal mode“ (maybe think of PEI more like a microkernel here)
and after memory is initialised, the rest of DXE is loaded. And MM
core would get no loading at all, this requirement has gladly been
dropped ages ago. Just one prelinked snapshot of the address space
with a relocation table and a safe self-relocator on entry (this
is needed at the very least for ARM).

Ironically, with my idea for MM, dynamic loading would be free as
everything is prelinked anyway. The same is true for PEI XIP, it
is prelinked by nature.
Actually Post-Memory PEI can have non-prelinked PEIMs. And that
does get used for the PEI GOP driver.

What I do not like is the additional dynamic linking code at
load-time for non- XIP modules. Though, the more I think about it,
the more I wonder whether not the entirety of UEFI could be
composed of prelinked, relocatable memory snapshots without
traditional image loading entirely (for in-FW stuff). macOS has a
similar concept with its “Kernel Collections”.
Well, way
too much off-topic now. :)
If you make the assumption that 100% of the code is compiled all at
once then yes that works. UEFI was designed so that assumption does
not need to be true. There are good use cases for it: OpROMs,
generic OS loaders, network boot, etc.

Thanks,
Nate


Why am I explaining all this despite the fact everyone knows this
will never happen? Because I don’t like the notion of fixing
issues of an already overcomplicated system by adding even more
complicated stuff.
Especially
when the existing overcomplicated stuff is already uncomfortably
broken.

Best regards,
Marvin

For XIP PEI code… it will really help and would be very timely
since the
transition of PEI from 32-bit to 64-bit is going to increase the
size of PEI by ~20%.

Thanks,
Nate

From: Pedro Falcato <pedro.falcato@...>
Sent: Wednesday, April 13, 2022 11:43 AM
To: edk2-devel-groups-io <devel@edk2.groups.io>; Marvin Häuser
<mhaeuser@...>
Cc: discuss@edk2.groups.io; adachristine18@...; Desimone,
Nathaniel L <nathaniel.l.desimone@...>; Shi, Steven
<steven.shi@...>
Subject: Re: [edk2-devel] [edk2-discuss] GSoC Proposal

Hi Marvin, Ada,

Some comments:

I don't think the purpose of the dynamic linker is to treat EFI
as a complete operating system, but to try to eliminate the
static linking that may be needlessly duplicating code that could
instead be put in a
single dynamic library. For instance, MdePkg and MdeModulePkg are
linked into a *lot* of .efi, instead of being just a library. It'd
be nice to see some numbers on this (something like Google's
bloaty could be run on every .efi file, in order to understand how
much file space we would actually save).

Other comments inline.

On Wed, Apr 13, 2022 at 4:15 PM Marvin Häuser
<mhaeuser@...<mailto:mhaeuser@...>> wrote:

On 13. Apr 2022, at 16:38, Ada Christine
<adachristine18@...<mailto:adachristine18@...>> wrote:
i was replying via the groups.io<http://groups.io > web
interface, I'm guessing that messed up the thread? i haven't used
mailing lists before and don't know how they work. I'll use my
mail client from here on.

I'm on board with not treating EFI as an operating system. the
more i think about it the more it looks like scope creep.

Agreed.


I'm not quite as enthusiastic
about it as i was at first glance.

I'm still keen on doing my gsoc proposal for edk, though, and
even if this task and the acpica application are decided to be
out of scope unit testing,

How about fuzz-testing? This is also something edk2 needs quite
badly. At
Acidanthera, we compile edk2 code in userspace outside the edk2
build system and fuzz with dummy applications.

Note: fuzzing is also part of the LLVM instrumentation suite (see
https://llvm.org/docs/LibFuzzer.html ) and is something I could
happily mentor.
clang integration

Pedro and Vitaly are looking for someone to finish ASan:
https://edk2.groups.io/g/devel/topic/90010978#87991
There are working UBSan concepts, but they also need to be
mainlined.

Is Vitaly going to be a mentor? I was assuming it was going to be
me and
some other, more senior, mentor (possibly Steven Shi, which I
included in the task).
Anyway, re: ASAN, if the project includes ASAN, UBSAN and
possibly some other sanitizer it's quite possible that it could
be considered a large
project (which means more hours but a larger stipend too). Fuzzing
+ coverage could be very nice additions to this project idea.
Also, is stress-testing a decent idea?


and source-level debugging are all relevant to my interests.

how about your ideas for security stuff?

I want the entirety of MM to leverage SmmMemLib and to support SMAP.
SmmMemLib would then handle UEFI->MMRAM and BaseMemoryLib would
only work on MMRAM. Also evaluation of how to best avoid pointers
in MM communication buffers would be nice.

There also is a bunch of other stuff, like working out moving a
part of
CpuDxe into DxeCore to have memory protection live immediately,
memory protection in PEI, a replacement for the TE format (it’s
buggy and most platforms mostly abandoned it over various issues),
and alternatives to guarding critical code with SMM (like allowing
NVRAM commits only as part of a reboot).

I personally find all of those projects very important, but I
cannot promise many people agree. Especially those that impose
global changes (most notably the TE replacement) may be very
tedious to submit.
Gladly, I believe you can submit multiple proposals (?)

Best regards,
Marvin


I'm not very knowledgeable about
trusted platform or secure boot but I'm willing to learn whatever
is necessary to get something spun up for my proposal.

On Wed, Apr 13, 2022, 12:05 Marvin Häuser
<mhaeuser@...<mailto:mhaeuser@...>> wrote:


Do you use the “reply all” option in your mail client? Looks like
my CCs have been dropped again. Comments inline.

On 13. Apr 2022, at 12:54, Ada Christine
<adachristine18@...<mailto:adachristine18@...>>
wrote:
Hi, Marvin

Its similarity to my own latest experiment is the key to what
grabbed my attention. I have no particular use case in mind for
it, but I see its potential for anybody developing larger
applications in that when a library is changed there's no need to
distribute a new version of the whole binary, just the relevant
library module.

I really do not like the trend of treating UEFI as a full-fledged
OS - it is not. The most used UEFI applications, OS loaders, are
really not that huge and are distributed as part of the OS image
anyway.
Even for
less used applications, you will always get a full snapshot anyhow.
Gladly we don’t have auto-update and package management yet. :)


I slept on it and it occurred to me that the whole thing could
operate
similarly to the shell protocol in that the linker/loader is
itself an
application that does a LoadImage() on the application needing
dynamic
linking facilities.

That would mean the linker itself is shipped with every application
that requires it? Otherwise it doesn’t make much sense for it to
be an
app and below’s problems apply.

If however the whole plan is making the linker as a DXE and
including
it with the firmware, that I'm not quite as sure about. That would
necessarily tie any applications using dynamic linking to
TianoCore or
any firmware distribution that derives from it.

I think that was the idea referred to as “edk2 core” by Steven, but
I’d like to hear his proposal to be sure. Virtually everyone uses
edk2, so that itself is not the problem, but versioning is. Vendors
are slow to update their snapshots or have just given up doing that
entirely. Distributing it for external applications like OS loaders
would mean this can be leveraged probably no earlier than 10 years
from now. And for in-firmware things, I have a hard time thinking
about a
use-case that outweighs the drawbacks.


To shift the topic slightly back to GSoC, however, I'm willing to
work
on other items on the task list. Unit testing and an ACPICA
application are the alternative projects I had thought about. I need
to choose fairly soon as the proposal deadline is next Tuesday. I
know
a tiny bit about porting ACPICA as I also have plans to
incorporate it into my
own project.

I have a few more ideas for security stuff, but Nate did not confirm
them as appropriate yet and I’m not here to drive you away from this
specific task (or the others). However, I’m still curious and
concerned. :)

Best regards,
Marvin








--
Pedro Falcato



















181 - 200 of 1159