GSOC 2021 EXT4 driver Project


Pedro Falcato
 

Hi everyone,

Me and my project have been selected for GSoC this year, under Michael Kinney and bret. Thank you for the opportunity to collaborate with you and improve Tianocore!
If anyone has any questions, please fire away :)
How do I get started? I'd like to find some easier tasks as to start trying out patch submission and generally programming in a firmware environment.
Also, I've been talking with my mentors and a relevant question to ask the mailing list is: Where should we put the EXT4 driver?
Michael said there are other filesystems in MdeModulePkg, but it might be getting too big and proposed the following options:

1) EXT4 in new package in edk2 repo as a peer to FatPkg.
2) EXT4 in edk2 repo in MdeModulePkg
3) EXT4 in edk2-platforms advanced feature package.
4) EXT4 in edk2 advanced feature package

As someone that's still learning how to navigate the project's tree(s), this is a bit over my head and so I'd like your opinion on the matter.
Also, I would love if someone could point me to some good reading material and/or examples of the package/build system, as I couldn't find documentation on those
and my previous experiment with Tianocore involved looking at FatPkg and mindlessly copying what it was doing.

Thanks,

Pedro 


Michael D Kinney
 

Hi Pedro,

Here is a link to background materials to get started with EDK II.

https://github.com/tianocore-training/Tianocore_Training_Contents/wiki

The detailed specifications for the EDK II build systems and coding style are:

https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Draft-Specification

Best regards,

Mike

===================

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Pedro Falcato
Sent: Monday, May 24, 2021 12:27 PM
To: devel@edk2.groups.io
Subject: [edk2-devel] GSOC 2021 EXT4 driver Project

Hi everyone,

Me and my project have been selected for GSoC this year, under Michael Kinney and bret. Thank you for the opportunity to collaborate with you and improve Tianocore!
If anyone has any questions, please fire away :)
How do I get started? I'd like to find some easier tasks as to start trying out patch submission and generally programming in a firmware environment.
Also, I've been talking with my mentors and a relevant question to ask the mailing list is: Where should we put the EXT4 driver?
Michael said there are other filesystems in MdeModulePkg, but it might be getting too big and proposed the following options:

1) EXT4 in new package in edk2 repo as a peer to FatPkg.
2) EXT4 in edk2 repo in MdeModulePkg
3) EXT4 in edk2-platforms advanced feature package.
4) EXT4 in edk2 advanced feature package

As someone that's still learning how to navigate the project's tree(s), this is a bit over my head and so I'd like your opinion on the matter.
Also, I would love if someone could point me to some good reading material and/or examples of the package/build system, as I couldn't find documentation on those
and my previous experiment with Tianocore involved looking at FatPkg and mindlessly copying what it was doing.

Thanks,

Pedro


Michael Brown
 

On 24/05/2021 20:26, Pedro Falcato wrote:
Me and my project have been selected for GSoC this year, under Michael Kinney and bret. Thank you for the opportunity to collaborate with you and improve Tianocore!
If anyone has any questions, please fire away :)
How do I get started? I'd like to find some easier tasks as to start trying out patch submission and generally programming in a firmware environment.
Also, I've been talking with my mentors and a relevant question to ask the mailing list is: Where should we put the EXT4 driver?
Michael said there are other filesystems in MdeModulePkg, but it might be getting too big and proposed the following options:
1) EXT4 in new package in edk2 repo as a peer to FatPkg.
2) EXT4 in edk2 repo in MdeModulePkg
3) EXT4 in edk2-platforms advanced feature package.
4) EXT4 in edk2 advanced feature package
FWIW, the most natural positioning seems to me to be as a separate Ext4Pkg similar to FatPkg.

For the sake of your own sanity, you probably want to start by maintaining Ext4Pkg as a completely standalone git repo that you happen to place within your local edk2 checkout directory. This will let you develop in isolation without worrying about where the code will eventually live, at least until the code has reached a state of maturity that makes it worth considering its eventual home. Keeping it in a separate Ext4Pkg repo will also enforce some level of discipline on your coding: you'll naturally avoid being tempted into letting your development work leak outside of Ext4Pkg into other packages.

In terms of questions, there are two that immediately spring to mind:

- Are you planning on making this read-only, or read-write?

- If you're planning on making it read-write: what simplifications will you be able to make to the write path in the UEFI environment? (UEFI is basically a single-user environment in which all filesystem writes should be flushed immediately: you may want to consider a simplified write path that never uses journalling, for example).

HTH,

Michael


Pedro Falcato
 

Hi Michael,

Sorry for the delay, I've been quite busy with work these past few days.

Yes, it sounds good, I'll keep it on a separate repo until it's actually more or less mature.

As for your questions,

1) I'm planning to make this read-write, but most importantly, I want it to be solid. If I can't reach write support, it's okay, since it's just a stretch goal.
My idea was to get solid read-only support, as it's naturally the most important part of a firmware filesystem driver (and as far as I know, there are not many
EFI applications/bootloaders that write to the filesystem). If I get the desired solidity and unit tests going, I want to make it write-capable too.

2) Although I'd love to avoid journaling, which is a matter I'm not too familiar with, I'm not entirely sure what simplifications may be done because for one,
what happens if the power cuts during a write? It's unclear to me how a FW filesystem driver might work on a damaged filesystem like that, since it's not at all similar to an OS,
which usually can do some sort of 'fsck' invocation to repair a filesystem before it's mounted. Is the firmware essentially unable to boot to those partitions until
someone gets a recovery drive of some sort that has a 'fsck' on it? I hope the FAT32 code has some answers for me, but I haven't had the time to go look at it that closely just yet.
It might be that the chance of this happening is minimal, but that doesn't sit right with me.

Also, one question: Does firmware code need the usual synchronization primitives (only spinlocks in this case, I would assume) or is it just assumed that it's a single threaded
environment? I know UEFI doesn't have threads but there are places in code that use things like EFI_MP_SERVICES, can the APs never touch certain code (like filesystem code, for example)?

Best Regards,

Pedro


Michael Brown
 

On 28/05/2021 18:16, Pedro Falcato wrote:
2) Although I'd love to avoid journaling, which is a matter I'm not too familiar with, I'm not entirely sure what simplifications may be done because for one,
what happens if the power cuts during a write? It's unclear to me how a FW filesystem driver might work on a damaged filesystem like that, since it's not at all similar to an OS,
which usually can do some sort of 'fsck' invocation to repair a filesystem before it's mounted. Is the firmware essentially unable to boot to those partitions until
someone gets a recovery drive of some sort that has a 'fsck' on it? I hope the FAT32 code has some answers for me, but I haven't had the time to go look at it that closely just yet.
It might be that the chance of this happening is minimal, but that doesn't sit right with me.
I don't know the internals of Ext4 journalling well enough to comment in detail, but my guess is that you are likely to find that some aspects are required for correctness but some aspects are required only for fast write performance with multiple concurrent processes (which is completely irrelevant for boot firmware).

Also, one question: Does firmware code need the usual synchronization primitives (only spinlocks in this case, I would assume) or is it just assumed that it's a single threaded
environment? I know UEFI doesn't have threads but there are places in code that use things like EFI_MP_SERVICES, can the APs never touch certain code (like filesystem code, for example)?
UEFI is fundamentally single-process, single-threaded, and based on polling rather than interrupts. It is _almost_ a clean design in which code never needs to worry about locking or other forms of synchronization.

Unfortunately this design is compromised by the existence of UEFI timers. There is no way to hook in a useful hardware interrupt (e.g. for a NIC received packet), but there are timer interrupts that will fire at unpredictable times and can result in arbitrary callbacks being invoked.

This introduces a requirement for some kind of synchronization, which UEFI handles via RaiseTPL()/RestoreTPL(). You can use RaiseTPL() to effectively disable timer interrupts and thereby guarantee that your code will not be reentered.

There is essentially no formal specification of what code should be allowed to run at each TPL, so your only viable option is to dig through existing EDK2 implementations to infer the de facto requirements.

Michael


Michael D Kinney
 

Hi Pedro,

 

After thinking about this, I think I would prefer Option (4).

 

The proposed landing zone would be a new Ext4Pkg in the edk2-platforms repository in the Features directory.

 

edk2-platforms/Features/Intel/Ext4Pkg

 

All the features in that directory and not specific to Intel.  There are other email discussions about moving some of that content up a level, so an alternative path would be:

 

              edk2-platforms/Features/Ext4Pkg

 

Best regards,

 

Mike

 

 

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Pedro Falcato
Sent: Monday, May 24, 2021 12:27 PM
To: devel@edk2.groups.io
Subject: [edk2-devel] GSOC 2021 EXT4 driver Project

 

Hi everyone,

Me and my project have been selected for GSoC this year, under Michael Kinney and bret. Thank you for the opportunity to collaborate with you and improve Tianocore!
If anyone has any questions, please fire away :)
How do I get started? I'd like to find some easier tasks as to start trying out patch submission and generally programming in a firmware environment.
Also, I've been talking with my mentors and a relevant question to ask the mailing list is: Where should we put the EXT4 driver?
Michael said there are other filesystems in MdeModulePkg, but it might be getting too big and proposed the following options:

1) EXT4 in new package in edk2 repo as a peer to FatPkg.
2) EXT4 in edk2 repo in MdeModulePkg
3) EXT4 in edk2-platforms advanced feature package.
4) EXT4 in edk2 advanced feature package

As someone that's still learning how to navigate the project's tree(s), this is a bit over my head and so I'd like your opinion on the matter.
Also, I would love if someone could point me to some good reading material and/or examples of the package/build system, as I couldn't find documentation on those
and my previous experiment with Tianocore involved looking at FatPkg and mindlessly copying what it was doing.

Thanks,

Pedro 


Leif Lindholm
 

edk2-platforms/Features/Ext4Pkg sounds good to me too.

/
Leif

On Thu, Jun 10, 2021 at 17:38:17 +0000, Michael D Kinney wrote:
Hi Pedro,

After thinking about this, I think I would prefer Option (4).

The proposed landing zone would be a new Ext4Pkg in the edk2-platforms repository in the Features directory.

edk2-platforms/Features/Intel/Ext4Pkg

All the features in that directory and not specific to Intel. There are other email discussions about moving some of that content up a level, so an alternative path would be:

edk2-platforms/Features/Ext4Pkg

Best regards,

Mike


From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Pedro Falcato
Sent: Monday, May 24, 2021 12:27 PM
To: devel@edk2.groups.io
Subject: [edk2-devel] GSOC 2021 EXT4 driver Project

Hi everyone,

Me and my project have been selected for GSoC this year, under Michael Kinney and bret. Thank you for the opportunity to collaborate with you and improve Tianocore!
If anyone has any questions, please fire away :)
How do I get started? I'd like to find some easier tasks as to start trying out patch submission and generally programming in a firmware environment.
Also, I've been talking with my mentors and a relevant question to ask the mailing list is: Where should we put the EXT4 driver?
Michael said there are other filesystems in MdeModulePkg, but it might be getting too big and proposed the following options:

1) EXT4 in new package in edk2 repo as a peer to FatPkg.
2) EXT4 in edk2 repo in MdeModulePkg
3) EXT4 in edk2-platforms advanced feature package.
4) EXT4 in edk2 advanced feature package

As someone that's still learning how to navigate the project's tree(s), this is a bit over my head and so I'd like your opinion on the matter.
Also, I would love if someone could point me to some good reading material and/or examples of the package/build system, as I couldn't find documentation on those
and my previous experiment with Tianocore involved looking at FatPkg and mindlessly copying what it was doing.

Thanks,

Pedro






Michael Kubacki
 

I had done some of the early work in moving content to edk2-platforms/Features. Just wanted to add my support for maintaining this as a vendor neutral feature there.

#4 in the options list is not really what AdvancedFeaturePkg intended for, there's more details here - https://github.com/tianocore/edk2-platforms/tree/master/Features/Intel#AdvancedFeaturePkg.

- Michael

On 6/10/2021 1:54 PM, Leif Lindholm wrote:
edk2-platforms/Features/Ext4Pkg sounds good to me too.
/
Leif
On Thu, Jun 10, 2021 at 17:38:17 +0000, Michael D Kinney wrote:
Hi Pedro,

After thinking about this, I think I would prefer Option (4).

The proposed landing zone would be a new Ext4Pkg in the edk2-platforms repository in the Features directory.

edk2-platforms/Features/Intel/Ext4Pkg

All the features in that directory and not specific to Intel. There are other email discussions about moving some of that content up a level, so an alternative path would be:

edk2-platforms/Features/Ext4Pkg

Best regards,

Mike


From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Pedro Falcato
Sent: Monday, May 24, 2021 12:27 PM
To: devel@edk2.groups.io
Subject: [edk2-devel] GSOC 2021 EXT4 driver Project

Hi everyone,

Me and my project have been selected for GSoC this year, under Michael Kinney and bret. Thank you for the opportunity to collaborate with you and improve Tianocore!
If anyone has any questions, please fire away :)
How do I get started? I'd like to find some easier tasks as to start trying out patch submission and generally programming in a firmware environment.
Also, I've been talking with my mentors and a relevant question to ask the mailing list is: Where should we put the EXT4 driver?
Michael said there are other filesystems in MdeModulePkg, but it might be getting too big and proposed the following options:

1) EXT4 in new package in edk2 repo as a peer to FatPkg.
2) EXT4 in edk2 repo in MdeModulePkg
3) EXT4 in edk2-platforms advanced feature package.
4) EXT4 in edk2 advanced feature package

As someone that's still learning how to navigate the project's tree(s), this is a bit over my head and so I'd like your opinion on the matter.
Also, I would love if someone could point me to some good reading material and/or examples of the package/build system, as I couldn't find documentation on those
and my previous experiment with Tianocore involved looking at FatPkg and mindlessly copying what it was doing.

Thanks,

Pedro






Michael D Kinney
 

Hi Michael,

Thanks for the feedback.

#4 should be clarified as adding a new feature package called Ext4Pkg. Not adding EXT4 to the existing AdvancedFeaturePkg.

Mike

-----Original Message-----
From: Michael Kubacki <mikuback@linux.microsoft.com>
Sent: Friday, June 11, 2021 11:23 AM
To: devel@edk2.groups.io; leif@nuviainc.com; Kinney, Michael D <michael.d.kinney@intel.com>
Cc: pedro.falcato@gmail.com
Subject: Re: [edk2-devel] GSOC 2021 EXT4 driver Project

I had done some of the early work in moving content to
edk2-platforms/Features. Just wanted to add my support for maintaining
this as a vendor neutral feature there.

#4 in the options list is not really what AdvancedFeaturePkg intended
for, there's more details here -
https://github.com/tianocore/edk2-platforms/tree/master/Features/Intel#AdvancedFeaturePkg.

- Michael

On 6/10/2021 1:54 PM, Leif Lindholm wrote:
edk2-platforms/Features/Ext4Pkg sounds good to me too.

/
Leif

On Thu, Jun 10, 2021 at 17:38:17 +0000, Michael D Kinney wrote:
Hi Pedro,

After thinking about this, I think I would prefer Option (4).

The proposed landing zone would be a new Ext4Pkg in the edk2-platforms repository in the Features directory.

edk2-platforms/Features/Intel/Ext4Pkg

All the features in that directory and not specific to Intel. There are other email discussions about moving some of
that content up a level, so an alternative path would be:

edk2-platforms/Features/Ext4Pkg

Best regards,

Mike


From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Pedro Falcato
Sent: Monday, May 24, 2021 12:27 PM
To: devel@edk2.groups.io
Subject: [edk2-devel] GSOC 2021 EXT4 driver Project

Hi everyone,

Me and my project have been selected for GSoC this year, under Michael Kinney and bret. Thank you for the opportunity
to collaborate with you and improve Tianocore!
If anyone has any questions, please fire away :)
How do I get started? I'd like to find some easier tasks as to start trying out patch submission and generally
programming in a firmware environment.
Also, I've been talking with my mentors and a relevant question to ask the mailing list is: Where should we put the
EXT4 driver?
Michael said there are other filesystems in MdeModulePkg, but it might be getting too big and proposed the following
options:

1) EXT4 in new package in edk2 repo as a peer to FatPkg.
2) EXT4 in edk2 repo in MdeModulePkg
3) EXT4 in edk2-platforms advanced feature package.
4) EXT4 in edk2 advanced feature package

As someone that's still learning how to navigate the project's tree(s), this is a bit over my head and so I'd like your
opinion on the matter.
Also, I would love if someone could point me to some good reading material and/or examples of the package/build system,
as I couldn't find documentation on those
and my previous experiment with Tianocore involved looking at FatPkg and mindlessly copying what it was doing.

Thanks,

Pedro