(+Ard)

On 03/24/21 02:49, Hernandez Miramontes, Jose Miguel wrote:

Hi

Is anyone familiar with this file? (Edk2\BaseTools\Scripts\GccBase.lds)
I would like to understand more what it does and why it is needed.

When looking at the .efi output of genfw, it seems the .build-id section generated by gcc is discarded.

git-blame is your friend. It leads to commit 7fd5d619806d ("BaseTools
GCC: drop GNU notes section from EFI image", 2016-08-02).

Thanks
Laszlo

What would it take to preserve it?

When reading this, https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html,
I understood that this section is needed so GDB can find the symbols when they are extracted from the binary.


objdump -fh PeiCore.dll

PeiCore.dll: file format elf32-i386
architecture: i386, flags 0x00000013:
HAS_RELOC, EXEC_P, HAS_SYMS
start address 0x00006089

Sections:
Idx Name Size VMA LMA File off Algn
0 .text 000060bc 00000220 00000220 00000080 2**5
CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
1 .data 00000348 000062e0 000062e0 00006140 2**5
CONTENTS, ALLOC, LOAD, RELOC, DATA
2 .build-id 00000024 00006628 00006628 00006488 2**2
CONTENTS, READONLY

$objdump -fh PeiCore.efi

PeiCore.efi: file format pei-i386
architecture: i386, flags 0x0000010b:
HAS_RELOC, EXEC_P, HAS_DEBUG, D_PAGED
start address 0x00006089

Sections:
Idx Name Size VMA LMA File off Algn
0 .text 000060c0 00000220 00000220 00000220 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
1 .data 00000400 000062e0 000062e0 000062e0 2**2
CONTENTS, ALLOC, LOAD, DATA
2 .reloc 000001a0 000066e0 000066e0 000066e0 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
Jose Miguel Hernandez Miramontes
BIOS Engineer
jose.miguel.hernandez.miramontes@...
+1 (512) 362-1230
Intel Corporation

On Wed, 24 Mar 2021 at 16:18, Laszlo Ersek <lersek@...> wrote:

(+Ard)

On 03/24/21 02:49, Hernandez Miramontes, Jose Miguel wrote:

Hi

Is anyone familiar with this file? (Edk2\BaseTools\Scripts\GccBase.lds)
I would like to understand more what it does and why it is needed.

When looking at the .efi output of genfw, it seems the .build-id section generated by gcc is discarded.

git-blame is your friend. It leads to commit 7fd5d619806d ("BaseTools
GCC: drop GNU notes section from EFI image", 2016-08-02).

The build-id is used by Linux distros when they ship debug symbols
with production shared libraries. The build-id permits the loader to
look up the correct file, and confirm that the versions match.

In EDK2, the ELF binary is only an intermediate artifact, and so I
fail to see why we would need build IDs here. If the ELF binary was
built with debug symbols, they will be in the binary itself, not in a
side loaded symbol file.

On Wed, 24 Mar 2021 at 16:56, Hernandez Miramontes, Jose Miguel
<jose.miguel.hernandez.miramontes@...> wrote:

yes, but if the section is not removed debugger would be able to find the debug symbols. Like how it happens with PDB files.
We could leave the symbols in the binaries, but it will consume more space. Space in the SPI is usually a constrain

I am not sure how your build environment works exactly, but when using
GCC to build for ARM, we already have this feature: the CodeView NB10
entry in the debug table in the PE/COFF metadata is populated with the
path to the .dll on the host from which the EFI executable was
generated. The debugger extracts this data to locate the file on the
build machine, and loads the symbols.

I am just wondering if it is possible to fix this so that the section is there in the final efi.
If I were to try it, where would I start?

I wouldn't be in favor of hacking this up: the ELF to PE/COFF
conversion in EDK2 is already highly dubious, and it would be much
better to stick with the PE/COFF semantics for carrying such metadata,
like the CodeView NB10 entry does.

Jose,

Caveat Emptor as Xcode clang is my daily driver, so I don’t really use GCC.

As Ard mentioned the ELF image is converted to a PE/COFF executable for EFI, and the symbols are stripped so the build ID is not going to help you. In PE/COFF there is something called a Debug Directory Entry[1]. It can point to a .debug section, or info in any section. The Debug Directory Entry starts with a 4 byte structure that defines its layout [1].

The 2nd issue you will have is the PE/COFF images are relocatable executables so they are not loaded at their linked address. Thus to load symbols you need to know the slide (delta of load address to link address) which you can derive from the load address of the image, and a path to the external symbols file. PE/COFF has the property that all the PE/COFF headers are loaded into memory when a PE/COFF image is loaded, this will be important later.

When you get past early PEI[2] and code is no longer XIP (eXecute In Place) you need to know the load address of the code to load symbols. So you end up with these options:
1) Load symbols for PC (This works for XIP too).
2) Load symbols for frame. Really just 1) for each frame PC.
3) Load symbols based on some table that has all the load info. This is the example from the EFI Spec [3].

To implement 1) you start with the PC and walk backwards in memory looking for the PE/COFF (or DOS) header. That header is the load address of the module. From the headers you can find the debug directory entry. The Debug Directory entry can then be decoded and that gives you (maybe) a unique identifier and the file name of the symbols the debugger needs to load. At that point you can tell the debugger this debug info matches the module loaded at this address. If your debugger knows PE/COFF you just have to point at the DOS or PE/COFF header.

So given I use lldb and the macOS flavor of lldb does not parse PE/COFF we had to write lldb Python to do all of the above and some more. So above is the how to do everything yourself, or theory of operation.

There is also the primitive look at the serial logs and load symbols by hand using debugger commands.

In terms of `git grep` in the edk2 look for EFI_IMAGE_DEBUG_DIRECTORY_ENTRY. There is a lib function called PeCoffLoaderGetPdbPointer() that returns a pointer to the PE/COFF Debug Directory Entry, it has the name pdb as that is the name of the windows symbols file that the Debug Directory Entry generally points to for VC++. For Xcode/clang the Debug Directory Entry entry has the UUID of mach-O executable (that got converted into a PE/COFF) and lldb can look up the symbols that way, other toolchains need the path to the debug file to load. So for example the OVMF exception handler [4] will dump out info about the faulting PC.

Sorry for the very generic answer, but I though it might be helpful for other folks too. You might want to ask a question about your target architecture, toolchain, and debugger on the mailing to find out what of the above is available for your toolchain with the edk2 for free. If you are interested in adding more features above is an outline of how it works and you can always ask more detailed questions on the mailing list.

[1] https://github.com/tianocore/edk2/blob/master/MdePkg/Include/IndustryStandard/PeImage.h#L609

[2] The build kicks out an <FvName>.Fv.map file that contains the info you need to load symbols.
For example: Build/OvmfX64/DEBUG_XCODE5/FV/PEIFV.Fv.map
EFI_FV_TOTAL_SIZE = 0xe0000
EFI_FV_TAKEN_SIZE = 0x43728
EFI_FV_SPACE_SIZE = 0x9c8d8

PeiCore (Fixed Flash Address, BaseAddress=0x0000820120, EntryPoint=0x000082b012)
(GUID=52C05B14-0B98-496C-BC3B-04B50211D680 .textbaseaddress=0x0000820360 .databaseaddress=0x000082e420)



PcdPeim (Fixed Flash Address, BaseAddress=0x000082ed00, EntryPoint=0x0000831a62)
(GUID=9B3ADA4F-AE56-4C24-8DEA-F03B7558AE50 .textbaseaddress=0x000082ef40 .databaseaddress=0x0000834ac0)
...

[3] https://github.com/tianocore/edk2/blob/master/MdePkg/Include/Guid/DebugImageInfoTable.h

[4] UefiCpuPkg/Library/CpuExceptionHandlerLib/CpuExceptionCommon.c:130: PdbPointer = PeCoffLoaderGetPdbPointer ((VOID *) Pe32Data);

Thanks,

Andrew Fish

yes, but if the section is not removed debugger would be able to find the debug symbols. Like how it happens with PDB files.
We could leave the symbols in the binaries, but it will consume more space. Space in the SPI is usually a constrain

I am just wondering if it is possible to fix this so that the section is there in the final efi.
If I were to try it, where would I start?

Jose Miguel Hernandez Miramontes
BIOS Engineer
jose.miguel.hernandez.miramontes@...
+1 (512) 362-1230
Intel Corporation

On Mar 24, 2021, at 8:56 AM, Hernandez Miramontes, Jose Miguel <jose.miguel.hernandez.miramontes@...> wrote:

yes, but if the section is not removed debugger would be able to find the debug symbols. Like how it happens with PDB files.

But gdb has no idea that the PE/COFF image exists in some random memory location. So you are going to have to do an ‘add-symbol-file <filename> -o offset` [1] to even let gdb know about your image. If you are not planning on using add-symbol-file what are you planning to use? If the gdb was PE/COFF aware it could load symbols its self if you point it at the PE/COFF?

We could leave the symbols in the binaries, but it will consume more space. Space in the SPI is usually a constrain

In PE/COFF the debug directory entry points to the external symbol file and with works for Windows/Linux/macOS. Conceptually if gdb knows about PE/COFF you should be able to just point gdb start of the image and it could load symbols. But I don’t know of a gdb command that can do that.

I am just wondering if it is possible to fix this so that the section is there in the final efi.
If I were to try it, where would I start?

GenFw is converting ELF to PE/COFF and it is quite brittle and intertwined. I’d insert any new section at the end as you don’t want to break relative offsets between text and data etc. If GenFw sees the image is an ELF file it will automatically attempt to convert it to PE/COFF [1]. That calls ConvertElf() [3] and then that calls the Elf32 or Elf64 functions. As I mentioned if you look closely [4] there are a lot of intertwined assumptions, so if you try to add a new section best to do it to the end of the PE/COFF.

If you have a better strategy than add-symbol-file please share with the community. I ask as it kind of seems if gdb knows enough about PE/COFF to find your magic section, it would know enough to parse the standard PE/COFF Debug Directory Entry. Regardless just having to point at the PE/COFF load address would be somewhat simpler than also having to decode the Debug Directory Entry in a script.

I don’t know much about gdb and your magic section, but I’ve got clang working with mach-O via lldb so I can help answer generic edk2 or PE/COFF questions about how to write debugger scripts, so feel free to ask questions on the list.

[1] https://2018.osfc.io/uploads/talk/paper/15/Debugging_UEFI_Firmware_Linux_Workshop_OSFC_19.pdf
https://retrage.github.io/2019/12/05/debugging-ovmf-en.html

[2] https://github.com/tianocore/edk2/blob/master/BaseTools/Source/C/GenFw/GenFw.c#L2058

[3] https://github.com/tianocore/edk2/blob/master/BaseTools/Source/C/GenFw/ElfConvert.c#L167

[4] https://github.com/tianocore/edk2/blob/master/BaseTools/Source/C/GenFw/Elf64Convert.c#L762

Thanks,

Andrew Fish

Jose Miguel Hernandez Miramontes
BIOS Engineer
jose.miguel.hernandez.miramontes@... <mailto:jose.miguel.hernandez.miramontes@...>
+1 (512) 362-1230
Intel Corporation

-----Original Message-----
From: Ard Biesheuvel <ardb@... <mailto:ardb@...>>
Sent: Wednesday, March 24, 2021 10:26 AM
To: Laszlo Ersek <lersek@... <mailto:lersek@...>>
Cc: discuss@edk2.groups.io <mailto:discuss@edk2.groups.io>; Hernandez Miramontes, Jose Miguel <jose.miguel.hernandez.miramontes@... <mailto:jose.miguel.hernandez.miramontes@...>>; Feng, Bob C <bob.c.feng@... <mailto:bob.c.feng@...>>; Saucedo Tejada, Genaro <genaro.saucedo.tejada@... <mailto:genaro.saucedo.tejada@...>>; Ortiz Garcia, Jorge E <jorge.e.ortiz.garcia@... <mailto:jorge.e.ortiz.garcia@...>>; Ard Biesheuvel (TianoCore) <ardb+tianocore@... <mailto:ardb+tianocore@...>>
Subject: Re: [edk2-discuss] GccBase.lds

On Wed, 24 Mar 2021 at 16:18, Laszlo Ersek <lersek@...> wrote:

(+Ard)

On 03/24/21 02:49, Hernandez Miramontes, Jose Miguel wrote:

Hi

Is anyone familiar with this file?
(Edk2\BaseTools\Scripts\GccBase.lds)
I would like to understand more what it does and why it is needed.

When looking at the .efi output of genfw, it seems the .build-id section generated by gcc is discarded.

git-blame is your friend. It leads to commit 7fd5d619806d ("BaseTools
GCC: drop GNU notes section from EFI image", 2016-08-02).

The build-id is used by Linux distros when they ship debug symbols with production shared libraries. The build-id permits the loader to look up the correct file, and confirm that the versions match.

In EDK2, the ELF binary is only an intermediate artifact, and so I fail to see why we would need build IDs here. If the ELF binary was built with debug symbols, they will be in the binary itself, not in a side loaded symbol file.

On Mar 26, 2021, at 8:43 AM, Hernandez Miramontes, Jose Miguel <jose.miguel.hernandez.miramontes@...> wrote:

Thanks,
I didn’t notice that the path was stored in the debug directory entry.
All the detailed information was very helpful.
I was able to confirm that it is there.

We have script that would parse for the PE32 header in memory, find the PDB path and then load the symbols.
But it doesn’t work when the binary is compiled with GCC.
Since the information is there, we just need to fix it to deal with the deltas.

It only needs the base address and the path to the binary with full symbols.
I confirmed I could manually load it so the error must be in how it finds the path.

Whoot! Glad you got it working and thanks for sharing maybe that will inspire others to give this a try. I dumped out a lot of info as I thought other folks may find it useful too.

Thanks,

Andrew Fish

Thanks again

Jose Miguel Hernandez Miramontes
BIOS Engineer
jose.miguel.hernandez.miramontes@... <mailto:jose.miguel.hernandez.miramontes@...>
Intel Corporation

From: Andrew Fish <afish@...>
Sent: Thursday, March 25, 2021 3:40 PM
To: discuss@edk2.groups.io; Hernandez Miramontes, Jose Miguel <jose.miguel.hernandez.miramontes@...>
Cc: Ard Biesheuvel <ardb@...>; Laszlo Ersek <lersek@...>; Feng, Bob C <bob.c.feng@...>; Saucedo Tejada, Genaro <genaro.saucedo.tejada@...>; Ortiz Garcia, Jorge E <jorge.e.ortiz.garcia@...>; Ard Biesheuvel (TianoCore) <ardb+tianocore@...>
Subject: Re: [edk2-discuss] GccBase.lds




On Mar 24, 2021, at 8:56 AM, Hernandez Miramontes, Jose Miguel <jose.miguel.hernandez.miramontes@... <mailto:jose.miguel.hernandez.miramontes@...>> wrote:

yes, but if the section is not removed debugger would be able to find the debug symbols. Like how it happens with PDB files.

But gdb has no idea that the PE/COFF image exists in some random memory location. So you are going to have to do an ‘add-symbol-file <filename> -o offset` [1] to even let gdb know about your image. If you are not planning on using add-symbol-file what are you planning to use? If the gdb was PE/COFF aware it could load symbols its self if you point it at the PE/COFF?


We could leave the symbols in the binaries, but it will consume more space. Space in the SPI is usually a constrain


In PE/COFF the debug directory entry points to the external symbol file and with works for Windows/Linux/macOS. Conceptually if gdb knows about PE/COFF you should be able to just point gdb start of the image and it could load symbols. But I don’t know of a gdb command that can do that.


I am just wondering if it is possible to fix this so that the section is there in the final efi.
If I were to try it, where would I start?


GenFw is converting ELF to PE/COFF and it is quite brittle and intertwined. I’d insert any new section at the end as you don’t want to break relative offsets between text and data etc. If GenFw sees the image is an ELF file it will automatically attempt to convert it to PE/COFF [1]. That calls ConvertElf() [3] and then that calls the Elf32 or Elf64 functions. As I mentioned if you look closely [4] there are a lot of intertwined assumptions, so if you try to add a new section best to do it to the end of the PE/COFF.

If you have a better strategy than add-symbol-file please share with the community. I ask as it kind of seems if gdb knows enough about PE/COFF to find your magic section, it would know enough to parse the standard PE/COFF Debug Directory Entry. Regardless just having to point at the PE/COFF load address would be somewhat simpler than also having to decode the Debug Directory Entry in a script.

I don’t know much about gdb and your magic section, but I’ve got clang working with mach-O via lldb so I can help answer generic edk2 or PE/COFF questions about how to write debugger scripts, so feel free to ask questions on the list.

[1] https://2018.osfc.io/uploads/talk/paper/15/Debugging_UEFI_Firmware_Linux_Workshop_OSFC_19.pdf
https://retrage.github.io/2019/12/05/debugging-ovmf-en.html

[2] https://github.com/tianocore/edk2/blob/master/BaseTools/Source/C/GenFw/GenFw.c#L2058

[3] https://github.com/tianocore/edk2/blob/master/BaseTools/Source/C/GenFw/ElfConvert.c#L167

[4] https://github.com/tianocore/edk2/blob/master/BaseTools/Source/C/GenFw/Elf64Convert.c#L762

Thanks,

Andrew Fish


Jose Miguel Hernandez Miramontes
BIOS Engineer
jose.miguel.hernandez.miramontes@... <mailto:jose.miguel.hernandez.miramontes@...>
+1 (512) 362-1230
Intel Corporation

-----Original Message-----
From: Ard Biesheuvel <ardb@... <mailto:ardb@...>>
Sent: Wednesday, March 24, 2021 10:26 AM
To: Laszlo Ersek <lersek@... <mailto:lersek@...>>
Cc: discuss@edk2.groups.io <mailto:discuss@edk2.groups.io>; Hernandez Miramontes, Jose Miguel <jose.miguel.hernandez.miramontes@... <mailto:jose.miguel.hernandez.miramontes@...>>; Feng, Bob C <bob.c.feng@... <mailto:bob.c.feng@...>>; Saucedo Tejada, Genaro <genaro.saucedo.tejada@... <mailto:genaro.saucedo.tejada@...>>; Ortiz Garcia, Jorge E <jorge.e.ortiz.garcia@... <mailto:jorge.e.ortiz.garcia@...>>; Ard Biesheuvel (TianoCore) <ardb+tianocore@... <mailto:ardb+tianocore@...>>
Subject: Re: [edk2-discuss] GccBase.lds

On Wed, 24 Mar 2021 at 16:18, Laszlo Ersek <lersek@... <mailto:lersek@...>> wrote:


(+Ard)

On 03/24/21 02:49, Hernandez Miramontes, Jose Miguel wrote:

Hi

Is anyone familiar with this file?
(Edk2\BaseTools\Scripts\GccBase.lds)
I would like to understand more what it does and why it is needed.

When looking at the .efi output of genfw, it seems the .build-id section generated by gcc is discarded.

git-blame is your friend. It leads to commit 7fd5d619806d ("BaseTools
GCC: drop GNU notes section from EFI image", 2016-08-02).


The build-id is used by Linux distros when they ship debug symbols with production shared libraries. The build-id permits the loader to look up the correct file, and confirm that the versions match.

In EDK2, the ELF binary is only an intermediate artifact, and so I fail to see why we would need build IDs here. If the ELF binary was built with debug symbols, they will be in the binary itself, not in a side loaded symbol file.