"StdLibPkg" branch on edk2-staging


Maciej Rabeda
 

Hi,

I have submitted a new edk2-staging branch named "StdLibPkg".
https://github.com/tianocore/edk2-staging/tree/StdLibPkg

Branch contains initial implementation of C standard library and intrinsic library for UEFI to smoothen open-source submodule integration (instead of implementing those functions in each new package introducing an external submodule dependent on C stdlib).

More details on the package, its contents and feature scope are placed in that branch, in StdLibPkg/Readme.md file.
https://github.com/tianocore/edk2-staging/blob/StdLibPkg/StdLibPkg/Readme.md

Feel free to play around. Any and all feedback is welcome.

Thanks,
Maciej


Rebecca Cran
 

Are you aware of the edk2-libc project at https://github.com/tianocore/edk2-libc ?


--
Rebecca Cran

On 7/27/21 9:48 AM, Maciej Rabeda wrote:
Hi,

I have submitted a new edk2-staging branch named "StdLibPkg".
https://github.com/tianocore/edk2-staging/tree/StdLibPkg

Branch contains initial implementation of C standard library and intrinsic library for UEFI to smoothen open-source submodule integration (instead of implementing those functions in each new package introducing an external submodule dependent on C stdlib).

More details on the package, its contents and feature scope are placed in that branch, in StdLibPkg/Readme.md file.
https://github.com/tianocore/edk2-staging/blob/StdLibPkg/StdLibPkg/Readme.md

Feel free to play around. Any and all feedback is welcome.

Thanks,
Maciej




Maciej Rabeda
 

Naturally. Since it is there, why is it not used for brotli, openssl in CryptoPkg or jansson in RedfishPkg?

On 28-Jul-21 17:34, Rebecca Cran wrote:
Are you aware of the edk2-libc project at https://github.com/tianocore/edk2-libc ?


Rebecca Cran
 

I think it's mostly been abandoned, and people are expected to use native UEFI functions instead. Personally I'd like to see it continue to be maintained.


--
Rebecca Cran

On 7/28/21 9:45 AM, Maciej Rabeda wrote:
Naturally. Since it is there, why is it not used for brotli, openssl in CryptoPkg or jansson in RedfishPkg?

On 28-Jul-21 17:34, Rebecca Cran wrote:
Are you aware of the edk2-libc project at https://github.com/tianocore/edk2-libc ?





Tim Lewis
 

I would point out that there was significant work on libc in the past (see https://github.com/andreiw/UefiToolsPkg) but never any help to upstream these fixes, including making sure that many Linux tools can easily be ported. I myself have used it to port several BSD utilities over, but each time there will little nits with the existing port and even small patches were returned with "no current maintainer"

In terms of making other projects use libc, I think the other objection was that it was never optimized for non-shell usage. Other projects (lie https://github.com/tianocore/edk2-staging/tree/CdePkg ) have tried to remedy this, but never with source checked in. But it allows support under PEI, DXE, etc.

Tim

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Rebecca Cran
Sent: Wednesday, July 28, 2021 8:34 AM
To: devel@edk2.groups.io; maciej.rabeda@linux.intel.com
Subject: Re: [edk2-devel] "StdLibPkg" branch on edk2-staging

Are you aware of the edk2-libc project at https://github.com/tianocore/edk2-libc ?


--
Rebecca Cran


On 7/27/21 9:48 AM, Maciej Rabeda wrote:
Hi,

I have submitted a new edk2-staging branch named "StdLibPkg".
https://github.com/tianocore/edk2-staging/tree/StdLibPkg

Branch contains initial implementation of C standard library and
intrinsic library for UEFI to smoothen open-source submodule
integration (instead of implementing those functions in each new
package introducing an external submodule dependent on C stdlib).

More details on the package, its contents and feature scope are placed
in that branch, in StdLibPkg/Readme.md file.
https://github.com/tianocore/edk2-staging/blob/StdLibPkg/StdLibPkg/Readme.md


Feel free to play around. Any and all feedback is welcome.

Thanks,
Maciej





Rebecca Cran
 

CC'ing the Tianocore stewards.


I've had problems getting a recent patch committed just to make edk2-libc work against current edk2 master.

Tianocore stewards: do we need new or additional maintainers for edk2-libc, or is there a plan to deprecate it?


The current maintainers are listed as:

StdLib, StdLibPrivateInternalFiles
W: https://github.com/tianocore/tianocore.github.io/wiki/StdLib
M: Daryl McDaniel <edk2-lists@mc2research.org>
M: Jaben Carsey <jaben.carsey@intel.com>


--
Rebecca Cran

On 7/28/21 4:25 PM, tim.lewis@insyde.com wrote:
I would point out that there was significant work on libc in the past (see https://github.com/andreiw/UefiToolsPkg) but never any help to upstream these fixes, including making sure that many Linux tools can easily be ported. I myself have used it to port several BSD utilities over, but each time there will little nits with the existing port and even small patches were returned with "no current maintainer"

In terms of making other projects use libc, I think the other objection was that it was never optimized for non-shell usage. Other projects (lie https://github.com/tianocore/edk2-staging/tree/CdePkg ) have tried to remedy this, but never with source checked in. But it allows support under PEI, DXE, etc.

Tim

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Rebecca Cran
Sent: Wednesday, July 28, 2021 8:34 AM
To: devel@edk2.groups.io; maciej.rabeda@linux.intel.com
Subject: Re: [edk2-devel] "StdLibPkg" branch on edk2-staging

Are you aware of the edk2-libc project at https://github.com/tianocore/edk2-libc ?


Leif Lindholm
 

Hi Rebecca,

I think the truth is we're a bit resigned about edk2-libc in general.
My view is it should be either maintained or deprecated (or replaced).

And maintainership *should* mean at least keeping it up to date with
security fixes.

I personally have no enthusiasm for the topic, but if there existed:
- An up-to-date codebase.
- A plan for keeping the coodbase up-to-date (as opposed to a plan to
keep the codebase up-to-date).
- At least a couple of maintainers.
I would have no objection to that existing under the TianoCore
umbrella.

Best Regards,

Leif

On Wed, Jul 28, 2021 at 16:44:02 -0600, Rebecca Cran wrote:
CC'ing the Tianocore stewards.


I've had problems getting a recent patch committed just to make edk2-libc
work against current edk2 master.

Tianocore stewards: do we need new or additional maintainers for edk2-libc,
or is there a plan to deprecate it?


The current maintainers are listed as:

StdLib, StdLibPrivateInternalFiles
W: https://github.com/tianocore/tianocore.github.io/wiki/StdLib
M: Daryl McDaniel <edk2-lists@mc2research.org>
M: Jaben Carsey <jaben.carsey@intel.com>


--
Rebecca Cran


On 7/28/21 4:25 PM, tim.lewis@insyde.com wrote:
I would point out that there was significant work on libc in the
past (see https://github.com/andreiw/UefiToolsPkg) but never any
help to upstream these fixes, including making sure that many Linux
tools can easily be ported. I myself have used it to port several
BSD utilities over, but each time there will little nits with the
existing port and even small patches were returned with "no current
maintainer"

In terms of making other projects use libc, I think the other
objection was that it was never optimized for non-shell
usage. Other projects (lie
https://github.com/tianocore/edk2-staging/tree/CdePkg ) have tried
to remedy this, but never with source checked in. But it allows
support under PEI, DXE, etc.

Tim

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Rebecca Cran
Sent: Wednesday, July 28, 2021 8:34 AM
To: devel@edk2.groups.io; maciej.rabeda@linux.intel.com
Subject: Re: [edk2-devel] "StdLibPkg" branch on edk2-staging

Are you aware of the edk2-libc project at https://github.com/tianocore/edk2-libc ?


Andrew Fish
 

I’d guess I’d also ask the why not NewLib? Especially since Red Hat helps out with edk2….

Thanks,

Andrew Fish

On Aug 4, 2021, at 4:03 AM, Leif Lindholm <leif@nuviainc.com> wrote:

Hi Rebecca,

I think the truth is we're a bit resigned about edk2-libc in general.
My view is it should be either maintained or deprecated (or replaced).

And maintainership *should* mean at least keeping it up to date with
security fixes.

I personally have no enthusiasm for the topic, but if there existed:
- An up-to-date codebase.
- A plan for keeping the coodbase up-to-date (as opposed to a plan to
keep the codebase up-to-date).
- At least a couple of maintainers.
I would have no objection to that existing under the TianoCore
umbrella.

Best Regards,

Leif

On Wed, Jul 28, 2021 at 16:44:02 -0600, Rebecca Cran wrote:
CC'ing the Tianocore stewards.


I've had problems getting a recent patch committed just to make edk2-libc
work against current edk2 master.

Tianocore stewards: do we need new or additional maintainers for edk2-libc,
or is there a plan to deprecate it?


The current maintainers are listed as:

StdLib, StdLibPrivateInternalFiles
W: https://github.com/tianocore/tianocore.github.io/wiki/StdLib
M: Daryl McDaniel <edk2-lists@mc2research.org>
M: Jaben Carsey <jaben.carsey@intel.com>


--
Rebecca Cran


On 7/28/21 4:25 PM, tim.lewis@insyde.com wrote:
I would point out that there was significant work on libc in the
past (see https://github.com/andreiw/UefiToolsPkg) but never any
help to upstream these fixes, including making sure that many Linux
tools can easily be ported. I myself have used it to port several
BSD utilities over, but each time there will little nits with the
existing port and even small patches were returned with "no current
maintainer"

In terms of making other projects use libc, I think the other
objection was that it was never optimized for non-shell
usage. Other projects (lie
https://github.com/tianocore/edk2-staging/tree/CdePkg ) have tried
to remedy this, but never with source checked in. But it allows
support under PEI, DXE, etc.

Tim

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Rebecca Cran
Sent: Wednesday, July 28, 2021 8:34 AM
To: devel@edk2.groups.io; maciej.rabeda@linux.intel.com
Subject: Re: [edk2-devel] "StdLibPkg" branch on edk2-staging

Are you aware of the edk2-libc project at https://github.com/tianocore/edk2-libc ?





Kilian Kegel
 

Hi all,

 

I plan to come back with the https://github.com/tianocore/edk2-staging/tree/CdePkg end of this year

and to unroll the source code of fundamental parts of my C Library and discuss that along

the 30. anniversary of “The C Book” https://publications.gbdirect.co.uk/c_book/chapter9

 

Additionally I will discuss the details you need to know:

  1. about the differences C90/C95/C99 in language and library
  2. how to get HOSTED ENVIRONMENT for UEFI POST drivers
  3. how to get Intel/AMD TimeStampCounter based precise time calibration - platform,

processor  and chipset independent for x86 UEFI platforms

  1. how to arrange space optimization for UEFI FW as an embedded platform
  2. how to test and verify the compatibility of a Standard C Library implementation,

since “CI“ won’t help at all in this regards

  1. how to implement a printf-core that performs narrow and wide mode at once for space optimization
  2. how to implement a scanf-core that performs narrow and wide mode at once for space optimization
  3. how to implement realloc() sub-allocator on UEFI memory management functions for PEI and DXE

 

Note: If UEFI is going to provide Standard-C-Functions for all drivers, you will have soon e.g. “sprintf()”

in all drivers. The amount of code for a proper sprintf() + scanf() + wscanf() + strtok() + wcstok() + malloc() … implementation

that is not tailored for embedded FW, will overrun the BIOS FLASH space instantly.

I consider commercial BIOS implementations that runs really many drivers (> 100) to start a platform.

 

Currently I implement more feature to CdePkg/Torito C Library:

  1. Improve UEFI driver cross development for VisualStudio and EDK2
  2. substitution of “DEBUG” traces by CdePkg-traces

that allows usage of predefined DEBUG-messages to run in a CdePkg-based driver.

 

Sole purpose is to have DEBUG/RELEASE mode _NOT_ globally, but locally to get

trace messages selectively for specific drivers only.

(to speed up POST with traces, that consumes a significant amount of time during

working hours of a BIOS developer)

 

  1. add SMM support.

Shell, DXE, PEI (pre and post memory) is already available for 2 years:

https://edk2.groups.io/g/devel/message/51562?p=,,,20,0,0,0::Created,,CdePkg,20,2,0,65191785

 

  1. get ACPICA tools https://acpica.org/ to UEFI Shell

 

I need to implement some non-standard / Microsoft specific C functions

to get that running for Andrew Fish.

 

That time links from https://github.com/KilianKegel are under construction and are

not necessarily  up-to-date.

Same for https://github.com/tianocore/edk2-staging/tree/CdePkg that is not used for long …

 

But https://github.com/KilianKegel/CdePkg.git driver set and sample set runs instantly

on the latest BIOS of a commercial vendor on a Intel IOTG TGL-H (Tiger Lake H) board using the Microsoft

build environment.

 

@Tim: I’d really appreciate if CdePkg could hold additionally Insyde BIOS support. Regrettably

I do not have access to Insyde source code.

 

@Maciej

I can not see any “wide” functions here. https://github.com/DevSolar/pdclib

from WCHAR.H or WCTYPE.H. That is a serious limitation in the UEFI environment.

 

To create UEFI-Shell programs for Personal Computers this link can be used.

https://github.com/KilianKegel/Visual-ANSI-C-for-UEFI-Shell#visual-ansi-c-for-uefi-shell

 

Best regards,

Kilian

 

From: Andrew Fish via groups.io
Sent: Thursday, August 5, 2021 05:35 AM
To: edk2-devel-groups-io; Leif Lindholm
Cc: Rebecca Cran; tim.lewis@...; maciej.rabeda@...; Mike Kinney
Subject: Re: [edk2-devel] "StdLibPkg" branch on edk2-staging

 

I’d guess I’d also ask the why not NewLib? Especially since Red Hat helps out with edk2….

Thanks,

Andrew Fish

> On Aug 4, 2021, at 4:03 AM, Leif Lindholm <leif@...> wrote:
>
> Hi Rebecca,
>
> I think the truth is we're a bit resigned about edk2-libc in general.
> My view is it should be either maintained or deprecated (or replaced).
>
> And maintainership *should* mean at least keeping it up to date with
> security fixes.
>
> I personally have no enthusiasm for the topic, but if there existed:
> - An up-to-date codebase.
> - A plan for keeping the coodbase up-to-date (as opposed to a plan to
>  keep the codebase up-to-date).
> - At least a couple of maintainers.
> I would have no objection to that existing under the TianoCore
> umbrella.
>
> Best Regards,
>
> Leif
>
> On Wed, Jul 28, 2021 at 16:44:02 -0600, Rebecca Cran wrote:
>> CC'ing the Tianocore stewards.
>>
>>
>> I've had problems getting a recent patch committed just to make edk2-libc
>> work against current edk2 master.
>>
>> Tianocore stewards: do we need new or additional maintainers for edk2-libc,
>> or is there a plan to deprecate it?
>>
>>
>> The current maintainers are listed as:
>>
>> StdLib, StdLibPrivateInternalFiles
>> W: https://github.com/tianocore/tianocore.github.io/wiki/StdLib
>> M: Daryl McDaniel <edk2-lists@...>
>> M: Jaben Carsey <jaben.carsey@...>
>>
>>
>> --
>> Rebecca Cran
>>
>>
>> On 7/28/21 4:25 PM, tim.lewis@... wrote:
>>> I would point out that there was significant work on libc in the
>> past (see https://github.com/andreiw/UefiToolsPkg) but never any
>> help to upstream these fixes, including making sure that many Linux
>> tools can easily be ported. I myself have used it to port several
>> BSD utilities over, but each time there will little nits with the
>> existing port and even small patches were returned with "no current
>> maintainer"
>>>
>>> In terms of making other projects use libc, I think the other
>>> objection was that it was never optimized for non-shell
>>> usage. Other projects (lie
>>> https://github.com/tianocore/edk2-staging/tree/CdePkg ) have tried
>>> to remedy this, but never with source checked in. But it allows
>>> support under PEI, DXE, etc.
>>>
>>> Tim
>>>
>>> -----Original Message-----
>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Rebecca Cran
>>> Sent: Wednesday, July 28, 2021 8:34 AM
>>> To: devel@edk2.groups.io; maciej.rabeda@...
>>> Subject: Re: [edk2-devel] "StdLibPkg" branch on edk2-staging
>>>
>>> Are you aware of the edk2-libc project at https://github.com/tianocore/edk2-libc ?
>>>
>>>
>
>
>
>
>





 


Kilian Kegel
 

Hi Maciej,

 

I have updated and fixed build errors in branch CdePkg (https://github.com/tianocore/edk2-staging.git).

 

Also a POST trace is recorded on my MinnowBoard and stored here:

https://raw.githubusercontent.com/tianocore/edk2-staging/CdePkg/UEFIBinaries/RELEASE.log

All implemented Standard-C-functions from CdePkg/Torito C Library demonstrates

its basic behavior.

There are also a lot of WCHAR.H- and WCTYPE.H-functions.

For the wide-functions I just perform the characters 0..0x1FF instead of all 65536

different wchar_t characters due to runtime regards…

 

Regrettably the MinnowBoard is deprecated https://www.silicom-usa.com/pr/eol/minnowboard-turbot

but the CdePkg also runs in the EMULATOR: https://github.com/tianocore/edk2-staging/tree/CdePkg#howto

 

Best Regards,

Kilian

From: Kilian Kegel
Sent: Thursday, August 5, 2021 06:14 AM
To: devel@edk2.groups.io; afish@...; Leif Lindholm
Cc: Rebecca Cran; tim.lewis@...; maciej.rabeda@...; Mike Kinney
Subject: Re: [edk2-devel] "StdLibPkg" branch on edk2-staging

 

Hi all,

 

I plan to come back with the https://github.com/tianocore/edk2-staging/tree/CdePkg end of this year

and to unroll the source code of fundamental parts of my C Library and discuss that along

the 30. anniversary of “The C Book” https://publications.gbdirect.co.uk/c_book/chapter9

 

Additionally I will discuss the details you need to know:

  1. about the differences C90/C95/C99 in language and library
  2. how to get HOSTED ENVIRONMENT for UEFI POST drivers
  3. how to get Intel/AMD TimeStampCounter based precise time calibration - platform,

processor  and chipset independent for x86 UEFI platforms

  1. how to arrange space optimization for UEFI FW as an embedded platform
  2. how to test and verify the compatibility of a Standard C Library implementation,

since “CI“ won’t help at all in this regards

  1. how to implement a printf-core that performs narrow and wide mode at once for space optimization
  2. how to implement a scanf-core that performs narrow and wide mode at once for space optimization
  3. how to implement realloc() sub-allocator on UEFI memory management functions for PEI and DXE

 

Note: If UEFI is going to provide Standard-C-Functions for all drivers, you will have soon e.g. “sprintf()”

in all drivers. The amount of code for a proper sprintf() + scanf() + wscanf() + strtok() + wcstok() + malloc() … implementation

that is not tailored for embedded FW, will overrun the BIOS FLASH space instantly.

I consider commercial BIOS implementations that runs really many drivers (> 100) to start a platform.

 

Currently I implement more feature to CdePkg/Torito C Library:

  1. Improve UEFI driver cross development for VisualStudio and EDK2
  2. substitution of “DEBUG” traces by CdePkg-traces

that allows usage of predefined DEBUG-messages to run in a CdePkg-based driver.

 

Sole purpose is to have DEBUG/RELEASE mode _NOT_ globally, but locally to get

trace messages selectively for specific drivers only.

(to speed up POST with traces, that consumes a significant amount of time during

working hours of a BIOS developer)

 

  1. add SMM support.

Shell, DXE, PEI (pre and post memory) is already available for 2 years:

https://edk2.groups.io/g/devel/message/51562?p=,,,20,0,0,0::Created,,CdePkg,20,2,0,65191785

 

  1. get ACPICA tools https://acpica.org/ to UEFI Shell

 

I need to implement some non-standard / Microsoft specific C functions

to get that running for Andrew Fish.

 

That time links from https://github.com/KilianKegel are under construction and are

not necessarily  up-to-date.

Same for https://github.com/tianocore/edk2-staging/tree/CdePkg that is not used for long …

 

But https://github.com/KilianKegel/CdePkg.git driver set and sample set runs instantly

on the latest BIOS of a commercial vendor on a Intel IOTG TGL-H (Tiger Lake H) board using the Microsoft

build environment.

 

@Tim: I’d really appreciate if CdePkg could hold additionally Insyde BIOS support. Regrettably

I do not have access to Insyde source code.

 

@Maciej

I can not see any “wide” functions here. https://github.com/DevSolar/pdclib

from WCHAR.H or WCTYPE.H. That is a serious limitation in the UEFI environment.

 

To create UEFI-Shell programs for Personal Computers this link can be used.

https://github.com/KilianKegel/Visual-ANSI-C-for-UEFI-Shell#visual-ansi-c-for-uefi-shell

 

Best regards,

Kilian

 

From: Andrew Fish via groups.io
Sent: Thursday, August 5, 2021 05:35 AM
To: edk2-devel-groups-io; Leif Lindholm
Cc: Rebecca Cran; tim.lewis@...; maciej.rabeda@...; Mike Kinney
Subject: Re: [edk2-devel] "StdLibPkg" branch on edk2-staging

 

I’d guess I’d also ask the why not NewLib? Especially since Red Hat helps out with edk2….

Thanks,

Andrew Fish

> On Aug 4, 2021, at 4:03 AM, Leif Lindholm <leif@...> wrote:
>
> Hi Rebecca,
>
> I think the truth is we're a bit resigned about edk2-libc in general.
> My view is it should be either maintained or deprecated (or replaced).
>
> And maintainership *should* mean at least keeping it up to date with
> security fixes.
>
> I personally have no enthusiasm for the topic, but if there existed:
> - An up-to-date codebase.
> - A plan for keeping the coodbase up-to-date (as opposed to a plan to
>  keep the codebase up-to-date).
> - At least a couple of maintainers.
> I would have no objection to that existing under the TianoCore
> umbrella.
>
> Best Regards,
>
> Leif
>
> On Wed, Jul 28, 2021 at 16:44:02 -0600, Rebecca Cran wrote:
>> CC'ing the Tianocore stewards.
>>
>>
>> I've had problems getting a recent patch committed just to make edk2-libc
>> work against current edk2 master.
>>
>> Tianocore stewards: do we need new or additional maintainers for edk2-libc,
>> or is there a plan to deprecate it?
>>
>>
>> The current maintainers are listed as:
>>
>> StdLib, StdLibPrivateInternalFiles
>> W: https://github.com/tianocore/tianocore.github.io/wiki/StdLib
>> M: Daryl McDaniel <edk2-lists@...>
>> M: Jaben Carsey <jaben.carsey@...>
>>
>>
>> --
>> Rebecca Cran
>>
>>
>> On 7/28/21 4:25 PM, tim.lewis@... wrote:
>>> I would point out that there was significant work on libc in the
>> past (see https://github.com/andreiw/UefiToolsPkg) but never any
>> help to upstream these fixes, including making sure that many Linux
>> tools can easily be ported. I myself have used it to port several
>> BSD utilities over, but each time there will little nits with the
>> existing port and even small patches were returned with "no current
>> maintainer"
>>>
>>> In terms of making other projects use libc, I think the other
>>> objection was that it was never optimized for non-shell
>>> usage. Other projects (lie
>>> https://github.com/tianocore/edk2-staging/tree/CdePkg ) have tried
>>> to remedy this, but never with source checked in. But it allows
>>> support under PEI, DXE, etc.
>>>
>>> Tim
>>>
>>> -----Original Message-----
>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Rebecca Cran
>>> Sent: Wednesday, July 28, 2021 8:34 AM
>>> To: devel@edk2.groups.io; maciej.rabeda@...
>>> Subject: Re: [edk2-devel] "StdLibPkg" branch on edk2-staging
>>>
>>> Are you aware of the edk2-libc project at https://github.com/tianocore/edk2-libc ?
>>>
>>>
>
>
>
>
>