Applying for GSoC 2022: Add Rust Support to EDK II


 

Hello everyone, I am a 2nd-year University Student from India. I am
interested in applying for adding Rust support to EDK2. I have already
introduced myself to the mailing list earlier
(https://edk2.groups.io/g/devel/message/87637) and have even submitted
some patches for the edkii-rust branch in edk2-staging (which were not
merged since that branch seems to be abandoned now).
- https://edk2.groups.io/g/devel/message/87753
- https://edk2.groups.io/g/devel/message/87754
- https://edk2.groups.io/g/devel/message/87755
- https://edk2.groups.io/g/devel/message/87756

Anyway, since no mentor has been listed for this project, I was
wondering who should I discuss the proposal with? Normally, I think
one is supposed to discuss the proposal details with a mentor in form
of a google doc or something before submitting an application. So
should I directly start by submitting a proposal through the GSoC
application portal? Or is there someone I should contact first?

Ayush Singh


Pedro Falcato
 

Hi Ayush!

I had a decent look at your proposal, and I have some questions (note that you can always re-submit proposals until April 19th):

1) How are you planning to reconcile the cargo build system with EDK2's build system?

2) If you go ahead with cargo, will the new Rust code require a lot of tiny crates? I'm personally not very enthusiastic about having lots of dependencies all over the place,
especially somewhere Tianocore doesn't control. I've seen Rust crates that were integrated into an operating system (https://fuchsia.googlesource.com/fuchsia/+/refs/heads/main/third_party/rust_crates/),
where they essentially maintain their own versions of the crate, with the "proper" build system that the project uses.

3) How much of std are you going to implement? Note that traditionally, EDK2, like a lot of this firmware/kernel/bootloader code, doesn't actually implement much of the standard library (and that's the reasoning behind edk2-libc).

4) Do you have any idea of what DXE driver you're going to implement? Do you have any experience writing drivers?

5) Any plans for generation of C <-> Rust bindings? Does https://github.com/rust-lang/rust-bindgen work out of the box?

Best regards,
Pedro 

On Mon, Apr 4, 2022 at 6:18 PM Ayush Singh <ayushdevel1325@...> wrote:
Hello everyone, I am a 2nd-year University Student from India. I am
interested in applying for adding Rust support to EDK2. I have already
introduced myself to the mailing list earlier
(https://edk2.groups.io/g/devel/message/87637) and have even submitted
some patches for the edkii-rust branch in edk2-staging (which were not
merged since that branch seems to be abandoned now).
- https://edk2.groups.io/g/devel/message/87753
- https://edk2.groups.io/g/devel/message/87754
- https://edk2.groups.io/g/devel/message/87755
- https://edk2.groups.io/g/devel/message/87756

Anyway, since no mentor has been listed for this project, I was
wondering who should I discuss the proposal with? Normally, I think
one is supposed to discuss the proposal details with a mentor in form
of a google doc or something before submitting an application. So
should I directly start by submitting a proposal through the GSoC
application portal? Or is there someone I should contact first?

Ayush Singh







--
Pedro Falcato


 

Hi Pedro,

The answers to the specific questions are given below. I will also
update my proposal later.

1) How are you planning to reconcile the cargo build system with EDK2's build system?
Well, a lot of work for this was already done in edkii-rust branch of
edk2-staging. So most of the .dsc and other files needed for Rust are
already present, they will just need to be updated/verified.

The main change I have in mind right now is using the build-std cargo
feature (https://doc.rust-lang.org/cargo/reference/unstable.html#build-std)
instead of cargo-xbuild. It basically can do what cargo-xbuild did but
is baked into cargo instead of an external tool.

I am planning on supporting the following combinations of building:
1. C source + Rust source mixed in INF (Library or Module)
2. Pure Rust Module only
3. Pure Rust Module + Pure Rust Library with Cargo Dependency
4. Pure Rust Module + C Library with EDKII Dependency
5. C Module + Pure Rust Library with EDKII Dependency
6. Pure Rust Module + Pure Rust Library with EDKII Dependency

2) If you go ahead with cargo, will the new Rust code require a lot of tiny crates? I'm personally not very enthusiastic about having lots of dependencies all over the place,
Shouldn't the decision of how many crates to use be left up to the
driver creators? I personally also like minimizing external
dependencies, especially in a project such as this, even if I have to
reimplement some things myself. But using cargo has a lot of benefits,
especially in the 2, 3, 4, and 6 building combinations.

Also, building with EDKII tools will also be a first-class citizen, so
cargo won't be the only option.

3) How much of std are you going to implement? Note that traditionally, EDK2, like a lot of this firmware/kernel/bootloader code, doesn't actually implement much of the standard library (and that's the reasoning behind edk2-libc).
First and foremost, I will implement a global allocator using the
`AllocatePool()` boot service. This will allow me to compile `alloc`
crate which is the crate that actually contains all the types such as
`std::collections`, `Vector`, `Box`, `Rc`, `String`, etc. All of these
types should now magically work theoretically, however since UEFI is a
tier 3 target in Rust, there is a moderate chance that manual
intervention will be needed here. Thus a lot of testing will be needed
here.

Once that is done, I will work to add `std::ffi::OsString` support.
This is important since the default strings in Rust use UTF-8 while
the default string type in UEFI is 16bit Unicode.

Finally, I will work on writing a basic test framework since the
default Rust testing framework cannot be used in uefi environment.
This will be done using `custom-test-frameworks feature
(https://doc.rust-lang.org/unstable-book/language-features/custom-test-frameworks.html).

I am open to suggestions if you believe that any other parts of std
need to be implemented. However, the above portion is what I had in
mind.

4) Do you have any idea of what DXE driver you're going to implement? Do you have any experience writing drivers?
No, I do not yet have any experience with writing UEFI drivers. I have
found some resources online about writing drivers with edk2 but this
will be my first time actually doing it. That is why I have mentioned
in the proposal that I will be trying to write a small driver in C
first during the community bonding period.

As for the specific driver, while I don't have any specific driver in
mind, and thus will need help from mentors to decide, I do have an
idea about what the driver should contain. They are as follows:
1. It should be complex enough to contain the common parts of edk2
that are actually used in production drivers. So drivers like
HelloWorld are out.
2. It should be able to serve as an example for writing DXE drivers in Rust.
3. It should also serve as a test of sorts, but I think that's a given
by the virtue of previous points.

5) Any plans for generation of C <-> Rust bindings? Does https://github.com/rust-lang/rust-bindgen work out of the box?
I'm not quite sure about this yet. Bingen is great but there is no
need to use it if there are only a few C methods/structures that I
need to use. On the other hand, it is great when there is a lot of C
code that needs to be used from Rust. It might be helpful in 1 and 5
build combinations, but I think this needs additional discussion.

It can also be useful for `ShellPkg`, `NetworkPkg`, etc. However, I
think I will get a better understanding by actually writing a driver.


Feel free to give suggestions and ask more questions. I would be happy
to answer them.

Ayush Singh.


Nate DeSimone
 

Hi Ayush,

Great to meet you and welcome to the TianoCore project! Great to hear you are interested! Apologize for the tardiness in my response. Implementing Rust support sounds like a wonderful project and one that would really help advance the state of the art for UEFI firmware development! I am looking for someone with Rust experience that can help mentor this project. My usage of Rust at time of writing has not advanced very far beyond "Hello World." While I can give a great deal of knowledge and background on UEFI and EDK II, my ability to recommend how that be applied to a Rust binding is limited. However, I do know enough to suspect the vast majority of the work will be figuring out how to integrate the vast array of libraries that EDK II provides into a coherent and clean Rust binding. The one aspect of this project that I think will be interesting is figuring out is what to do about std:: in Rust. From what I have seen of the functionality there more or less assumes the existence of a libc implementation for the platform, which is not necessarily true for DXE and is absolutely not true for PEI. I would be interested in hearing your thought on how to handle that elegantly.

I'm sorry that your patches haven't gotten much attention thus far. Once I find mentor(s) for the Rust project I'll make sure they pick those up and take a look at the work you have done thus far.

Hope this helps and welcome to the project!

With Best Regards,
Nate

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Ayush Singh
Sent: Monday, April 4, 2022 10:18 AM
To: devel@edk2.groups.io
Cc: Bret.Barkelew@...; Desimone, Nathaniel L <nathaniel.l.desimone@...>; mhaeuser@...
Subject: [edk2-devel] Applying for GSoC 2022: Add Rust Support to EDK II

Hello everyone, I am a 2nd-year University Student from India. I am interested in applying for adding Rust support to EDK2. I have already introduced myself to the mailing list earlier
(https://edk2.groups.io/g/devel/message/87637) and have even submitted some patches for the edkii-rust branch in edk2-staging (which were not merged since that branch seems to be abandoned now).
- https://edk2.groups.io/g/devel/message/87753
- https://edk2.groups.io/g/devel/message/87754
- https://edk2.groups.io/g/devel/message/87755
- https://edk2.groups.io/g/devel/message/87756

Anyway, since no mentor has been listed for this project, I was wondering who should I discuss the proposal with? Normally, I think one is supposed to discuss the proposal details with a mentor in form of a google doc or something before submitting an application. So should I directly start by submitting a proposal through the GSoC application portal? Or is there someone I should contact first?

Ayush Singh


 

Hi Nate

Thanks for the response.

For the std implementation, I do have some idea how to go about implementing it now. The most important thing I realized is that most of the std isn't actually std. For example, std::collection, Vector, Box, Rc, etc are all actually part of alloc and not std. The things that really are part of std include threads, i/o, etc.

I have taken a look at some other people's projects who have tried implementing libstd for other targets and it seems it is possible to write an implementation without libc. It's just very difficult since in most OS besides Linux, the syscall ABI is not stable enough and using libc is just easier and recommended.

As for my earlier patches, Jiewen told me that edkii-rust branch is no longer maintained and that they are now using a different uefi rust implementation for their work.

I did also find that it will be possible to make the std with stable Rust even though if internals use nightly, so that's cool. Some useful projects about writing libstd for new platform that I found are below:

Ayush Singh

On Fri, 8 Apr, 2022, 2:33 am Desimone, Nathaniel L, <nathaniel.l.desimone@...> wrote:
Hi Ayush,

Great to meet you and welcome to the TianoCore project! Great to hear you are interested! Apologize for the tardiness in my response. Implementing Rust support sounds like a wonderful project and one that would really help advance the state of the art for UEFI firmware development! I am looking for someone with Rust experience that can help mentor this project. My usage of Rust at time of writing has not advanced very far beyond "Hello World." While I can give a great deal of knowledge and background on UEFI and EDK II, my ability to recommend how that be applied to a Rust binding is limited. However, I do know enough to suspect the vast majority of the work will be figuring out how to integrate the vast array of libraries that EDK II provides into a coherent and clean Rust binding. The one aspect of this project that I think will be interesting is figuring out is what to do about std:: in Rust. From what I have seen of the functionality there more or less assumes the existence of a libc implementation for the platform, which is not necessarily true for DXE and is absolutely not true for PEI. I would be interested in hearing your thought on how to handle that elegantly.

I'm sorry that your patches haven't gotten much attention thus far. Once I find mentor(s) for the Rust project I'll make sure they pick those up and take a look at the work you have done thus far.

Hope this helps and welcome to the project!

With Best Regards,
Nate

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Ayush Singh
Sent: Monday, April 4, 2022 10:18 AM
To: devel@edk2.groups.io
Cc: Bret.Barkelew@...; Desimone, Nathaniel L <nathaniel.l.desimone@...>; mhaeuser@...
Subject: [edk2-devel] Applying for GSoC 2022: Add Rust Support to EDK II

Hello everyone, I am a 2nd-year University Student from India. I am interested in applying for adding Rust support to EDK2. I have already introduced myself to the mailing list earlier
(https://edk2.groups.io/g/devel/message/87637) and have even submitted some patches for the edkii-rust branch in edk2-staging (which were not merged since that branch seems to be abandoned now).
- https://edk2.groups.io/g/devel/message/87753
- https://edk2.groups.io/g/devel/message/87754
- https://edk2.groups.io/g/devel/message/87755
- https://edk2.groups.io/g/devel/message/87756

Anyway, since no mentor has been listed for this project, I was wondering who should I discuss the proposal with? Normally, I think one is supposed to discuss the proposal details with a mentor in form of a google doc or something before submitting an application. So should I directly start by submitting a proposal through the GSoC application portal? Or is there someone I should contact first?

Ayush Singh






Nate DeSimone
 

Hi Ayush,

 

Very interesting and intellectually stimulating, thank you! It makes sense that the Rust authors would opt to depend on libc for syscalls, at least on most UNIX designs there is an assumption that the library that implements the syscall interface *is* libc and any library that provides a syscall wrapper is just a different implementation of libc.

 

For UEFI, the closest equivalent we have to syscalls is the PEI services and UEFI boot services tables, which are basically just a bunch of C function pointers. Based on your research it sounds like in should be possible to build on top of some of this work that has already been done and create a version of std that is pure Rust with perhaps the exception of some C function pointer calls out to the UEFI services for memory allocation and whatnot. Memory allocation will be interesting because DXE provides a proper heap but PEI only allows pages (which are 4KB chunks of RAM) to be freed. As such it would probably make sense to build a Rust implementation of a heap that allocates and frees pages as necessary so that it will be possible to use std on both PEI and DXE.

 

With regard to Jiewen’s rust-firmware project, my personal opinion is that his approach is more long term and aspirational. Given that EDK II is now deployed on ~4 billion devices around the world, I don’t think a wholesale conversion from 0% Rust code to 100% pure Rust code across the entire industry is realistic. A much more pragmatic option in my opinion would be to allow some mix of C and Rust code to co-exist and if firmware implementations evolve towards a greater mix of Rust code over time than something like Jiewen’s proposal could become feasible. But in the short term my opinion is slowly introducing Rust over time is the only feasible option. I understand Jiewen’s reasons for preferring a 100% Rust; from a security standpoint that is the only way to get the full benefits of Rust’s type safety checks. It is also my opinion that type safety is not a silver bullet; especially in the firmware world where we have to do raw writes to physical memory for MMIO there will always be a ton of unsafe code even if it is all pure Rust.

 

Thanks for looking into this and for the well-researched answer!

 

Best Regards,

Nate

 

From: Ayush Singh <ayushdevel1325@...>
Sent: Friday, April 8, 2022 5:41 AM
To: Desimone, Nathaniel L <nathaniel.l.desimone@...>
Cc: edk2-devel-groups-io <devel@edk2.groups.io>; Bret Barkelew <Bret.Barkelew@...>; Marvin Häuser <mhaeuser@...>; Yao, Jiewen <jiewen.yao@...>
Subject: Re: [edk2-devel] Applying for GSoC 2022: Add Rust Support to EDK II

 

Hi Nate

 

Thanks for the response.

 

For the std implementation, I do have some idea how to go about implementing it now. The most important thing I realized is that most of the std isn't actually std. For example, std::collection, Vector, Box, Rc, etc are all actually part of alloc and not std. The things that really are part of std include threads, i/o, etc.

 

I have taken a look at some other people's projects who have tried implementing libstd for other targets and it seems it is possible to write an implementation without libc. It's just very difficult since in most OS besides Linux, the syscall ABI is not stable enough and using libc is just easier and recommended.

 

As for my earlier patches, Jiewen told me that edkii-rust branch is no longer maintained and that they are now using a different uefi rust implementation for their work.

 

I did also find that it will be possible to make the std with stable Rust even though if internals use nightly, so that's cool. Some useful projects about writing libstd for new platform that I found are below:

 

Ayush Singh

 

On Fri, 8 Apr, 2022, 2:33 am Desimone, Nathaniel L, <nathaniel.l.desimone@...> wrote:

Hi Ayush,

Great to meet you and welcome to the TianoCore project! Great to hear you are interested! Apologize for the tardiness in my response. Implementing Rust support sounds like a wonderful project and one that would really help advance the state of the art for UEFI firmware development! I am looking for someone with Rust experience that can help mentor this project. My usage of Rust at time of writing has not advanced very far beyond "Hello World." While I can give a great deal of knowledge and background on UEFI and EDK II, my ability to recommend how that be applied to a Rust binding is limited. However, I do know enough to suspect the vast majority of the work will be figuring out how to integrate the vast array of libraries that EDK II provides into a coherent and clean Rust binding. The one aspect of this project that I think will be interesting is figuring out is what to do about std:: in Rust. From what I have seen of the functionality there more or less assumes the existence of a libc implementation for the platform, which is not necessarily true for DXE and is absolutely not true for PEI. I would be interested in hearing your thought on how to handle that elegantly.

I'm sorry that your patches haven't gotten much attention thus far. Once I find mentor(s) for the Rust project I'll make sure they pick those up and take a look at the work you have done thus far.

Hope this helps and welcome to the project!

With Best Regards,
Nate

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Ayush Singh
Sent: Monday, April 4, 2022 10:18 AM
To: devel@edk2.groups.io
Cc: Bret.Barkelew@...; Desimone, Nathaniel L <nathaniel.l.desimone@...>; mhaeuser@...
Subject: [edk2-devel] Applying for GSoC 2022: Add Rust Support to EDK II

Hello everyone, I am a 2nd-year University Student from India. I am interested in applying for adding Rust support to EDK2. I have already introduced myself to the mailing list earlier
(https://edk2.groups.io/g/devel/message/87637) and have even submitted some patches for the edkii-rust branch in edk2-staging (which were not merged since that branch seems to be abandoned now).
- https://edk2.groups.io/g/devel/message/87753
- https://edk2.groups.io/g/devel/message/87754
- https://edk2.groups.io/g/devel/message/87755
- https://edk2.groups.io/g/devel/message/87756

Anyway, since no mentor has been listed for this project, I was wondering who should I discuss the proposal with? Normally, I think one is supposed to discuss the proposal details with a mentor in form of a google doc or something before submitting an application. So should I directly start by submitting a proposal through the GSoC application portal? Or is there someone I should contact first?

Ayush Singh





 

Thanks,

I have updated my Proposal based on the discussion on the mailing
list, so feel free to give feedback on that.

As far as my views go, I agree that it's not realistic to convert all
edk2 to Rust. I think of this project to be about making Rust an
option to do firmware development in, and stand alongside C. Maybe
Rust will replace all the C someday, but I would be more interested in
writing new parts in Rust rather than rewriting something that already
works perfectly fine.

And yes, it is pretty difficult to remove unsafe from firmware
development, but I think it might actually be possible to write UEFI
applications without unsafe, although that might just be me being
optimistic. Also, unsafe Rust is at worst as unsafe as normal C, so it
still should be a slight improvement. The best thing would be when we
can use collections and stuff in firmware development. That would
definitely be pretty cool.

Ayush


On Thu, Apr 14, 2022 at 6:30 AM Desimone, Nathaniel L
<nathaniel.l.desimone@...> wrote:

Hi Ayush,



Very interesting and intellectually stimulating, thank you! It makes sense that the Rust authors would opt to depend on libc for syscalls, at least on most UNIX designs there is an assumption that the library that implements the syscall interface *is* libc and any library that provides a syscall wrapper is just a different implementation of libc.



For UEFI, the closest equivalent we have to syscalls is the PEI services and UEFI boot services tables, which are basically just a bunch of C function pointers. Based on your research it sounds like in should be possible to build on top of some of this work that has already been done and create a version of std that is pure Rust with perhaps the exception of some C function pointer calls out to the UEFI services for memory allocation and whatnot. Memory allocation will be interesting because DXE provides a proper heap but PEI only allows pages (which are 4KB chunks of RAM) to be freed. As such it would probably make sense to build a Rust implementation of a heap that allocates and frees pages as necessary so that it will be possible to use std on both PEI and DXE.



With regard to Jiewen’s rust-firmware project, my personal opinion is that his approach is more long term and aspirational. Given that EDK II is now deployed on ~4 billion devices around the world, I don’t think a wholesale conversion from 0% Rust code to 100% pure Rust code across the entire industry is realistic. A much more pragmatic option in my opinion would be to allow some mix of C and Rust code to co-exist and if firmware implementations evolve towards a greater mix of Rust code over time than something like Jiewen’s proposal could become feasible. But in the short term my opinion is slowly introducing Rust over time is the only feasible option. I understand Jiewen’s reasons for preferring a 100% Rust; from a security standpoint that is the only way to get the full benefits of Rust’s type safety checks. It is also my opinion that type safety is not a silver bullet; especially in the firmware world where we have to do raw writes to physical memory for MMIO there will always be a ton of unsafe code even if it is all pure Rust.



Thanks for looking into this and for the well-researched answer!



Best Regards,

Nate



From: Ayush Singh <ayushdevel1325@...>
Sent: Friday, April 8, 2022 5:41 AM
To: Desimone, Nathaniel L <nathaniel.l.desimone@...>
Cc: edk2-devel-groups-io <devel@edk2.groups.io>; Bret Barkelew <Bret.Barkelew@...>; Marvin Häuser <mhaeuser@...>; Yao, Jiewen <jiewen.yao@...>
Subject: Re: [edk2-devel] Applying for GSoC 2022: Add Rust Support to EDK II



Hi Nate



Thanks for the response.



For the std implementation, I do have some idea how to go about implementing it now. The most important thing I realized is that most of the std isn't actually std. For example, std::collection, Vector, Box, Rc, etc are all actually part of alloc and not std. The things that really are part of std include threads, i/o, etc.



I have taken a look at some other people's projects who have tried implementing libstd for other targets and it seems it is possible to write an implementation without libc. It's just very difficult since in most OS besides Linux, the syscall ABI is not stable enough and using libc is just easier and recommended.



As for my earlier patches, Jiewen told me that edkii-rust branch is no longer maintained and that they are now using a different uefi rust implementation for their work.



I did also find that it will be possible to make the std with stable Rust even though if internals use nightly, so that's cool. Some useful projects about writing libstd for new platform that I found are below:

- https://github.com/betrusted-io/rust/tree/1.54.0.5

- https://github.com/japaric/steed



Ayush Singh



On Fri, 8 Apr, 2022, 2:33 am Desimone, Nathaniel L, <nathaniel.l.desimone@...> wrote:

Hi Ayush,

Great to meet you and welcome to the TianoCore project! Great to hear you are interested! Apologize for the tardiness in my response. Implementing Rust support sounds like a wonderful project and one that would really help advance the state of the art for UEFI firmware development! I am looking for someone with Rust experience that can help mentor this project. My usage of Rust at time of writing has not advanced very far beyond "Hello World." While I can give a great deal of knowledge and background on UEFI and EDK II, my ability to recommend how that be applied to a Rust binding is limited. However, I do know enough to suspect the vast majority of the work will be figuring out how to integrate the vast array of libraries that EDK II provides into a coherent and clean Rust binding. The one aspect of this project that I think will be interesting is figuring out is what to do about std:: in Rust. From what I have seen of the functionality there more or less assumes the existence of a libc implementation for the platform, which is not necessarily true for DXE and is absolutely not true for PEI. I would be interested in hearing your thought on how to handle that elegantly.

I'm sorry that your patches haven't gotten much attention thus far. Once I find mentor(s) for the Rust project I'll make sure they pick those up and take a look at the work you have done thus far.

Hope this helps and welcome to the project!

With Best Regards,
Nate

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Ayush Singh
Sent: Monday, April 4, 2022 10:18 AM
To: devel@edk2.groups.io
Cc: Bret.Barkelew@...; Desimone, Nathaniel L <nathaniel.l.desimone@...>; mhaeuser@...
Subject: [edk2-devel] Applying for GSoC 2022: Add Rust Support to EDK II

Hello everyone, I am a 2nd-year University Student from India. I am interested in applying for adding Rust support to EDK2. I have already introduced myself to the mailing list earlier
(https://edk2.groups.io/g/devel/message/87637) and have even submitted some patches for the edkii-rust branch in edk2-staging (which were not merged since that branch seems to be abandoned now).
- https://edk2.groups.io/g/devel/message/87753
- https://edk2.groups.io/g/devel/message/87754
- https://edk2.groups.io/g/devel/message/87755
- https://edk2.groups.io/g/devel/message/87756

Anyway, since no mentor has been listed for this project, I was wondering who should I discuss the proposal with? Normally, I think one is supposed to discuss the proposal details with a mentor in form of a google doc or something before submitting an application. So should I directly start by submitting a proposal through the GSoC application portal? Or is there someone I should contact first?

Ayush Singh