UEFI accessibility mandate


Ethin Probst <harlydavidsen@...>
 

Hello all,

I'm new here, and was recommended to the TianoCore project by someone over at the UEFI forum. I've run across TianoCore before, and like the project.
Before anyone gets worried by the subject line, no, this is not any kind of legal thing. Its just something I believe needs to happen. :)
Back in 2016-2017 I contacted the UEFI forum about two problems, one of which was the format of the spec, which I figured out on my own. The other problem was not so easily dealt with. Te other problem relates to accessibility of UEFI-compliant systems and platform firmware to persons with disabilities. As it currently stands, such a thing is nonexistent. To be fair, I completely understand the difficulty that such a thing would require, and I would fully agree if we still used the PC-AT BIOS systems -- yes, indeed, I would never suggest this kind of thing on such a system given that there was no actual standard of any kind for BIOSes. However, now that UEFI is here, we have such a possibility.
As it currently stands, people with low vision or blind people have access to their computers in the general sense (I am blind myself). We can do pretty much anything anyone else could do. We can code, play games, all that. There are few things that we cannot do. One of those things is managing our systems firmware in the preboot environment.
As it stands now, I can only boot other OSes or disks via memorization. While that worked on BIOS machines (I have, or had, an old Toshiba laptop that was BIOS-based), it no longer works because UEFI is mercurial. When I access the boot menu now, I play a game of chance. If the cards are in my favor, the OS I want to boot boots, and I can go on my way. But if the cards aren't in my favor, I end up making something happen that was unintended, and, worst of all, I have no idea what I did.
However, the boot menu is only one portion of a platform firmware UI. What about the setup utility or other utilities offered by computer manufacturers? What about diagnostic utilities, bootloaders, etc? What do I do with those? Well, I only have one option -- sited assistance. If I go into my computers setup utility, I cannot trust myself and say to myself, "OK, I know what I'm doing. All I need to do is change x and save and quit." No, I can't do that, because memorizing such a complex interface is extremely difficult, and its something I wouldn't expect anyone to do.
My proposal is simple, and I'm posting it here because I'd like comments and feedback before it actually gets implemented (it will take a lot of time, I'm sure): mandate, in the UEFI specification, that accessibility features for persons with disabilities must be implemented and documented, and, if such features are not implemented, then that vendor is not compliant with the specification. Place strict minimum requirements for what the accessibility features should do and how they should work.
Now, I'm sure someone out there will ask me how this can be done. Well, that's why I've joined the group -- though as I familiarize myself with EDK2 development and all that I may actually be able to participate as more than just an accessibility expert, of sorts.
As a side note, I have been blind all my life. I was born with retinopathy of prematurity (ROP), which resulted because I was born at 26 weeks. My retina was detached, and, though the doctors attempted to fix it, it would not remain attached, and there is no chance of it getting fixed now. I would neither want nor care for such a cure, however. I have lived my entire life blind, and while the thought of gaining site back is appealing, I am unwilling to go through the years and years of rewiring and reconditioning of my brain that would be required for me to survive with site. To me, it is simply not worth the cost.
But back to the discussion at hand: I would be happy to discuss how the accessibility features would work and what would be required. Even standardizing, through the specification, a key combination to toggle the accessibility features would be nice, as that would alleviate the major problem of a blind person buying a new computer and not knowing how to enable the accessibility features. The overarching goal would be to make the preboot environment (including applications run within it) accessible and usable by blind and visually impaired people as a boot service only. It would be superfluous to make this a runtime service, as all major OSes already have accessibility features. Plus, managing such a thing would be impossible to do.
This email has gotten quite long, so I will suspend the discussion of functionality and how I would like it to work for a future email once everyone has gotten on board.

Thank you for your time and consideration.


Leif Lindholm
 

Hi Ethin,

Apologies for the delay in responding - I personally don't tend to
read the RFC list (and I know that applies to some other developers
who have now subscribed).

On Sun, Aug 11, 2019 at 06:15:18PM -0700, Ethin Probst wrote:
Hello all,

I'm new here, and was recommended to the TianoCore project by
someone over at the UEFI forum. I've run across TianoCore before,
and like the project.
Before anyone gets worried by the subject line, no, this is not any
kind of legal thing. Its just something I believe needs to
happen. :)
Back in 2016-2017 I contacted the UEFI forum about two problems, one
of which was the format of the spec, which I figured out on my
own. The other problem was not so easily dealt with. Te other
problem relates to accessibility of UEFI-compliant systems and
platform firmware to persons with disabilities. As it currently
stands, such a thing is nonexistent. To be fair, I completely
understand the difficulty that such a thing would require, and I
would fully agree if we still used the PC-AT BIOS systems -- yes,
indeed, I would never suggest this kind of thing on such a system
given that there was no actual standard of any kind for
BIOSes. However, now that UEFI is here, we have such a possibility.
As it currently stands, people with low vision or blind people have
access to their computers in the general sense (I am blind
myself). We can do pretty much anything anyone else could do. We can
code, play games, all that. There are few things that we cannot
do. One of those things is managing our systems firmware in the
preboot environment.
As it stands now, I can only boot other OSes or disks via
memorization. While that worked on BIOS machines (I have, or had, an
old Toshiba laptop that was BIOS-based), it no longer works because
UEFI is mercurial. When I access the boot menu now, I play a game of
chance. If the cards are in my favor, the OS I want to boot boots,
and I can go on my way. But if the cards aren't in my favor, I end
up making something happen that was unintended, and, worst of all, I
have no idea what I did.
However, the boot menu is only one portion of a platform firmware
UI. What about the setup utility or other utilities offered by
computer manufacturers? What about diagnostic utilities,
bootloaders, etc? What do I do with those? Well, I only have one
option -- sited assistance. If I go into my computers setup utility,
I cannot trust myself and say to myself, "OK, I know what I'm
doing. All I need to do is change x and save and quit." No, I can't
do that, because memorizing such a complex interface is extremely
difficult, and its something I wouldn't expect anyone to do.
My proposal is simple, and I'm posting it here because I'd like
comments and feedback before it actually gets implemented (it will
take a lot of time, I'm sure): mandate, in the UEFI specification,
that accessibility features for persons with disabilities must be
implemented and documented, and, if such features are not
implemented, then that vendor is not compliant with the
specification. Place strict minimum requirements for what the
accessibility features should do and how they should work.
So, first of all, I need to point out that what you suggest would be
very difficult for the UEFI Forum to mandate. *But* I don't think this
needs to be a problem.

UEFI primarily exists as a means through which interoperability can be
guaranteed. This means it does not tend to so much mandate what is
actually implemented in a platform, as it defines how it behaves if it
is. This is not 100% true, but suffice to say that mandating the
functionality you are asking for *without* having a reference
implementation available would likely simply be ignored.

If you want to turn thumbscrews, get Microsoft to add the requirement
to their windows logo requirements :)

However, as much as it is difficult to force people to develop this
support, if a functional reference implementation was published under
a permissive open source license, it would take a pretty spectacularly
incompetent product owner not to incorporate it in their products.

Now, I'm sure someone out there will ask me how this can be
done. Well, that's why I've joined the group -- though as I
familiarize myself with EDK2 development and all that I may actually
be able to participate as more than just an accessibility expert, of
sorts.
I have added Rafael Machado on cc. He did a PhD on firmware
accessibility, including a prototype implementing an audio driver for
EDK2. The resulting thesis can be found at
https://arxiv.org/abs/1712.03186 .

As a side note, I have been blind all my life. I was born with
retinopathy of prematurity (ROP), which resulted because I was born
at 26 weeks. My retina was detached, and, though the doctors
attempted to fix it, it would not remain attached, and there is no
chance of it getting fixed now. I would neither want nor care for
such a cure, however. I have lived my entire life blind, and while
the thought of gaining site back is appealing, I am unwilling to go
through the years and years of rewiring and reconditioning of my
brain that would be required for me to survive with site. To me, it
is simply not worth the cost.
But back to the discussion at hand: I would be happy to discuss how
the accessibility features would work and what would be
required. Even standardizing, through the specification, a key
combination to toggle the accessibility features would be nice, as
that would alleviate the major problem of a blind person buying a
new computer and not knowing how to enable the accessibility
features. The overarching goal would be to make the preboot
environment (including applications run within it) accessible and
usable by blind and visually impaired people as a boot service
only.
Yes, I agree, operating systems are already doing their things.

Again, mandating such a key combination is not necessarily something
that the UEFI specification can do. Some devices won't necessarily
have a keyboard.

Some things the UEFI specification could do is:
- Add protocols for Audio output (and possibly input) and talking to
audio codecs.
- Add audio hooks for the Human Interaction Interface subsystem (HII).

EDK2 could then start adding device drivers producing these protocols,
and enable the support for some reference platforms.

Would we need any specific support for things like refreshable braille
displays?

It would be superfluous to make this a runtime service, as all
major OSes already have accessibility features. Plus, managing such
a thing would be impossible to do.
Agreed.

This email has gotten quite long, so I will suspend the discussion
of functionality and how I would like it to work for a future email
once everyone has gotten on board.
Best Regards,

Leif


Thank you for your time and consideration.



Ethin Probst
 

While conquering refreshable braille displays would be nice, I think
we should take it in stages. (Some blind people would want this in
along with the accessibility subsystems, however I do not believe that
would be the wisest approach.) Instead, I think we should do what you
suggest: implement protocols for communicating with audio devices
(both input and output) and add audio hooks for the HII. (We could
even make those audio hooks just general hooks/events instead of
partitioning them for a particular purpose.)
You mentioned that not all devices 'have a keyboard'. This is most
definitely true, but not all devices have audio outputs and/or inputs
either. Therefore, if we can't mandate it for *all* devices, perhaps
we could only mandate it (or at least somehow infer it as a
requirement) for devices that both have an interaction mechanism (any
form of interacting with the system directly via the HII and not, say,
over the wire, because that would be superfluous) and have
mechanisms/devices/drivers/... for audio output at minimum. (Audio
input is not necessarily something that is required unless the device
is controlled primarily by voice, but I think having voice controls is
a bit beyond the UEFI specification and is not something UEFI should
worry about. Voice control is not something you should have in the
preboot environment anyway, correct?)
So, I think the first stage we should work on should include:
- Implementing protocols for communicating with audio outputs (don't
worry about inputs right now)
- Implementing protocols/hooks/events that can be intercepted in the
HII by a foreground/background UEFI driver/application
- Implementing some kind of method to allow the accessibility
protocols to read what is drawn to the text when navigating around
The third item -- when implemented properly -- will prevent the
accessibility subsystem from reading everything on the screen every
time the "cursor" moves. Perhaps another good idea would be to
implement some kind of priority system for text-to-speech, like speech
dispatcher does
(https://docs.microsoft.com/en-us/cortana/skills/speech-synthesis-markup-language).

On 9/10/19, Leif Lindholm <leif.lindholm@linaro.org> wrote:
Hi Ethin,

Apologies for the delay in responding - I personally don't tend to
read the RFC list (and I know that applies to some other developers
who have now subscribed).

On Sun, Aug 11, 2019 at 06:15:18PM -0700, Ethin Probst wrote:
Hello all,

I'm new here, and was recommended to the TianoCore project by
someone over at the UEFI forum. I've run across TianoCore before,
and like the project.
Before anyone gets worried by the subject line, no, this is not any
kind of legal thing. Its just something I believe needs to
happen. :)
Back in 2016-2017 I contacted the UEFI forum about two problems, one
of which was the format of the spec, which I figured out on my
own. The other problem was not so easily dealt with. Te other
problem relates to accessibility of UEFI-compliant systems and
platform firmware to persons with disabilities. As it currently
stands, such a thing is nonexistent. To be fair, I completely
understand the difficulty that such a thing would require, and I
would fully agree if we still used the PC-AT BIOS systems -- yes,
indeed, I would never suggest this kind of thing on such a system
given that there was no actual standard of any kind for
BIOSes. However, now that UEFI is here, we have such a possibility.
As it currently stands, people with low vision or blind people have
access to their computers in the general sense (I am blind
myself). We can do pretty much anything anyone else could do. We can
code, play games, all that. There are few things that we cannot
do. One of those things is managing our systems firmware in the
preboot environment.
As it stands now, I can only boot other OSes or disks via
memorization. While that worked on BIOS machines (I have, or had, an
old Toshiba laptop that was BIOS-based), it no longer works because
UEFI is mercurial. When I access the boot menu now, I play a game of
chance. If the cards are in my favor, the OS I want to boot boots,
and I can go on my way. But if the cards aren't in my favor, I end
up making something happen that was unintended, and, worst of all, I
have no idea what I did.
However, the boot menu is only one portion of a platform firmware
UI. What about the setup utility or other utilities offered by
computer manufacturers? What about diagnostic utilities,
bootloaders, etc? What do I do with those? Well, I only have one
option -- sited assistance. If I go into my computers setup utility,
I cannot trust myself and say to myself, "OK, I know what I'm
doing. All I need to do is change x and save and quit." No, I can't
do that, because memorizing such a complex interface is extremely
difficult, and its something I wouldn't expect anyone to do.
My proposal is simple, and I'm posting it here because I'd like
comments and feedback before it actually gets implemented (it will
take a lot of time, I'm sure): mandate, in the UEFI specification,
that accessibility features for persons with disabilities must be
implemented and documented, and, if such features are not
implemented, then that vendor is not compliant with the
specification. Place strict minimum requirements for what the
accessibility features should do and how they should work.
So, first of all, I need to point out that what you suggest would be
very difficult for the UEFI Forum to mandate. *But* I don't think this
needs to be a problem.

UEFI primarily exists as a means through which interoperability can be
guaranteed. This means it does not tend to so much mandate what is
actually implemented in a platform, as it defines how it behaves if it
is. This is not 100% true, but suffice to say that mandating the
functionality you are asking for *without* having a reference
implementation available would likely simply be ignored.

If you want to turn thumbscrews, get Microsoft to add the requirement
to their windows logo requirements :)

However, as much as it is difficult to force people to develop this
support, if a functional reference implementation was published under
a permissive open source license, it would take a pretty spectacularly
incompetent product owner not to incorporate it in their products.

Now, I'm sure someone out there will ask me how this can be
done. Well, that's why I've joined the group -- though as I
familiarize myself with EDK2 development and all that I may actually
be able to participate as more than just an accessibility expert, of
sorts.
I have added Rafael Machado on cc. He did a PhD on firmware
accessibility, including a prototype implementing an audio driver for
EDK2. The resulting thesis can be found at
https://arxiv.org/abs/1712.03186 .

As a side note, I have been blind all my life. I was born with
retinopathy of prematurity (ROP), which resulted because I was born
at 26 weeks. My retina was detached, and, though the doctors
attempted to fix it, it would not remain attached, and there is no
chance of it getting fixed now. I would neither want nor care for
such a cure, however. I have lived my entire life blind, and while
the thought of gaining site back is appealing, I am unwilling to go
through the years and years of rewiring and reconditioning of my
brain that would be required for me to survive with site. To me, it
is simply not worth the cost.
But back to the discussion at hand: I would be happy to discuss how
the accessibility features would work and what would be
required. Even standardizing, through the specification, a key
combination to toggle the accessibility features would be nice, as
that would alleviate the major problem of a blind person buying a
new computer and not knowing how to enable the accessibility
features. The overarching goal would be to make the preboot
environment (including applications run within it) accessible and
usable by blind and visually impaired people as a boot service
only.
Yes, I agree, operating systems are already doing their things.

Again, mandating such a key combination is not necessarily something
that the UEFI specification can do. Some devices won't necessarily
have a keyboard.

Some things the UEFI specification could do is:
- Add protocols for Audio output (and possibly input) and talking to
audio codecs.
- Add audio hooks for the Human Interaction Interface subsystem (HII).

EDK2 could then start adding device drivers producing these protocols,
and enable the support for some reference platforms.

Would we need any specific support for things like refreshable braille
displays?

It would be superfluous to make this a runtime service, as all
major OSes already have accessibility features. Plus, managing such
a thing would be impossible to do.
Agreed.

This email has gotten quite long, so I will suspend the discussion
of functionality and how I would like it to work for a future email
once everyone has gotten on board.
Best Regards,

Leif


Thank you for your time and consideration.


--
Signed,
Ethin D. Probst


Ethin Probst
 

Woops, I just realized while reading my message after I'd sent it that
I had linked to the wrong document. The priority system is outlined
over here (https://freebsoft.org/doc/speechd/ssip.html#Top). This
document discusses SsIP -- the Speech Synthesis Interface Protocol.

On 9/10/19, Ethin Probst <harlydavidsen@gmail.com> wrote:
While conquering refreshable braille displays would be nice, I think
we should take it in stages. (Some blind people would want this in
along with the accessibility subsystems, however I do not believe that
would be the wisest approach.) Instead, I think we should do what you
suggest: implement protocols for communicating with audio devices
(both input and output) and add audio hooks for the HII. (We could
even make those audio hooks just general hooks/events instead of
partitioning them for a particular purpose.)
You mentioned that not all devices 'have a keyboard'. This is most
definitely true, but not all devices have audio outputs and/or inputs
either. Therefore, if we can't mandate it for *all* devices, perhaps
we could only mandate it (or at least somehow infer it as a
requirement) for devices that both have an interaction mechanism (any
form of interacting with the system directly via the HII and not, say,
over the wire, because that would be superfluous) and have
mechanisms/devices/drivers/... for audio output at minimum. (Audio
input is not necessarily something that is required unless the device
is controlled primarily by voice, but I think having voice controls is
a bit beyond the UEFI specification and is not something UEFI should
worry about. Voice control is not something you should have in the
preboot environment anyway, correct?)
So, I think the first stage we should work on should include:
- Implementing protocols for communicating with audio outputs (don't
worry about inputs right now)
- Implementing protocols/hooks/events that can be intercepted in the
HII by a foreground/background UEFI driver/application
- Implementing some kind of method to allow the accessibility
protocols to read what is drawn to the text when navigating around
The third item -- when implemented properly -- will prevent the
accessibility subsystem from reading everything on the screen every
time the "cursor" moves. Perhaps another good idea would be to
implement some kind of priority system for text-to-speech, like speech
dispatcher does
(https://docs.microsoft.com/en-us/cortana/skills/speech-synthesis-markup-language).

On 9/10/19, Leif Lindholm <leif.lindholm@linaro.org> wrote:
Hi Ethin,

Apologies for the delay in responding - I personally don't tend to
read the RFC list (and I know that applies to some other developers
who have now subscribed).

On Sun, Aug 11, 2019 at 06:15:18PM -0700, Ethin Probst wrote:
Hello all,

I'm new here, and was recommended to the TianoCore project by
someone over at the UEFI forum. I've run across TianoCore before,
and like the project.
Before anyone gets worried by the subject line, no, this is not any
kind of legal thing. Its just something I believe needs to
happen. :)
Back in 2016-2017 I contacted the UEFI forum about two problems, one
of which was the format of the spec, which I figured out on my
own. The other problem was not so easily dealt with. Te other
problem relates to accessibility of UEFI-compliant systems and
platform firmware to persons with disabilities. As it currently
stands, such a thing is nonexistent. To be fair, I completely
understand the difficulty that such a thing would require, and I
would fully agree if we still used the PC-AT BIOS systems -- yes,
indeed, I would never suggest this kind of thing on such a system
given that there was no actual standard of any kind for
BIOSes. However, now that UEFI is here, we have such a possibility.
As it currently stands, people with low vision or blind people have
access to their computers in the general sense (I am blind
myself). We can do pretty much anything anyone else could do. We can
code, play games, all that. There are few things that we cannot
do. One of those things is managing our systems firmware in the
preboot environment.
As it stands now, I can only boot other OSes or disks via
memorization. While that worked on BIOS machines (I have, or had, an
old Toshiba laptop that was BIOS-based), it no longer works because
UEFI is mercurial. When I access the boot menu now, I play a game of
chance. If the cards are in my favor, the OS I want to boot boots,
and I can go on my way. But if the cards aren't in my favor, I end
up making something happen that was unintended, and, worst of all, I
have no idea what I did.
However, the boot menu is only one portion of a platform firmware
UI. What about the setup utility or other utilities offered by
computer manufacturers? What about diagnostic utilities,
bootloaders, etc? What do I do with those? Well, I only have one
option -- sited assistance. If I go into my computers setup utility,
I cannot trust myself and say to myself, "OK, I know what I'm
doing. All I need to do is change x and save and quit." No, I can't
do that, because memorizing such a complex interface is extremely
difficult, and its something I wouldn't expect anyone to do.
My proposal is simple, and I'm posting it here because I'd like
comments and feedback before it actually gets implemented (it will
take a lot of time, I'm sure): mandate, in the UEFI specification,
that accessibility features for persons with disabilities must be
implemented and documented, and, if such features are not
implemented, then that vendor is not compliant with the
specification. Place strict minimum requirements for what the
accessibility features should do and how they should work.
So, first of all, I need to point out that what you suggest would be
very difficult for the UEFI Forum to mandate. *But* I don't think this
needs to be a problem.

UEFI primarily exists as a means through which interoperability can be
guaranteed. This means it does not tend to so much mandate what is
actually implemented in a platform, as it defines how it behaves if it
is. This is not 100% true, but suffice to say that mandating the
functionality you are asking for *without* having a reference
implementation available would likely simply be ignored.

If you want to turn thumbscrews, get Microsoft to add the requirement
to their windows logo requirements :)

However, as much as it is difficult to force people to develop this
support, if a functional reference implementation was published under
a permissive open source license, it would take a pretty spectacularly
incompetent product owner not to incorporate it in their products.

Now, I'm sure someone out there will ask me how this can be
done. Well, that's why I've joined the group -- though as I
familiarize myself with EDK2 development and all that I may actually
be able to participate as more than just an accessibility expert, of
sorts.
I have added Rafael Machado on cc. He did a PhD on firmware
accessibility, including a prototype implementing an audio driver for
EDK2. The resulting thesis can be found at
https://arxiv.org/abs/1712.03186 .

As a side note, I have been blind all my life. I was born with
retinopathy of prematurity (ROP), which resulted because I was born
at 26 weeks. My retina was detached, and, though the doctors
attempted to fix it, it would not remain attached, and there is no
chance of it getting fixed now. I would neither want nor care for
such a cure, however. I have lived my entire life blind, and while
the thought of gaining site back is appealing, I am unwilling to go
through the years and years of rewiring and reconditioning of my
brain that would be required for me to survive with site. To me, it
is simply not worth the cost.
But back to the discussion at hand: I would be happy to discuss how
the accessibility features would work and what would be
required. Even standardizing, through the specification, a key
combination to toggle the accessibility features would be nice, as
that would alleviate the major problem of a blind person buying a
new computer and not knowing how to enable the accessibility
features. The overarching goal would be to make the preboot
environment (including applications run within it) accessible and
usable by blind and visually impaired people as a boot service
only.
Yes, I agree, operating systems are already doing their things.

Again, mandating such a key combination is not necessarily something
that the UEFI specification can do. Some devices won't necessarily
have a keyboard.

Some things the UEFI specification could do is:
- Add protocols for Audio output (and possibly input) and talking to
audio codecs.
- Add audio hooks for the Human Interaction Interface subsystem (HII).

EDK2 could then start adding device drivers producing these protocols,
and enable the support for some reference platforms.

Would we need any specific support for things like refreshable braille
displays?

It would be superfluous to make this a runtime service, as all
major OSes already have accessibility features. Plus, managing such
a thing would be impossible to do.
Agreed.

This email has gotten quite long, so I will suspend the discussion
of functionality and how I would like it to work for a future email
once everyone has gotten on board.
Best Regards,

Leif


Thank you for your time and consideration.



--
Signed,
Ethin D. Probst
--
Signed,
Ethin D. Probst


Rafael Machado <rafaelrodrigues.machado@...>
 

Hi Ethin and all

As Leif mentioned, my mastering thesys was the implementation of a
reference code so a driver for audio at pre-OS would be possible,
eliminating the first barrier for the creation of a screen reader at UEFI,
that is the lack of audio drivers.

My project was based on Intel High Definition Audio, so it is not a generic
code that would fit at all scenarios, but it a first step.
The idea was to proceed this work using the Google Summer of Code
initiative,and I will try my best to make this happen in future editions of
this initiative.
Unfortunately my thesys was written in portuguese, because there was almost
nothing related to BIOS in Portuguese, so the decision was to write in
Portuguese to help the community to attract brasilian developers. As soon
as I find some slot, I'll translate to english.
But you are not alone. Just in Brazil I discovered around 30 blind
developers that area waiting for accessibility at BIOS. Also had some
contacts from US and Ireland.
So it is a market to be explored. Everyone will win.
The companies, because they will attract more people and their families,
and the ones who need accessibility.
By the way, most of the population will need accessible solutions if they
live enough. This includes the uefi community members, that today do not
have special needs, but that may require it in future.
Accessibility is not a wast of money.
This ia what I would like to show to the UEFI community.
Hope to have the opportunity to present this in some future event.

Since this discussion was started, who knows what are the next steps?
My code is available at github if someone gets interested (ps.: The code
has failures. It was just a Prof of Concept)

https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Lets keep the discussion, because it is important.

Thanks and Regards
Rafael R. Machado


Em ter, 10 de set de 2019 17:00, Ethin Probst <harlydavidsen@gmail.com>
escreveu:

Woops, I just realized while reading my message after I'd sent it that
I had linked to the wrong document. The priority system is outlined
over here (https://freebsoft.org/doc/speechd/ssip.html#Top). This
document discusses SsIP -- the Speech Synthesis Interface Protocol.

On 9/10/19, Ethin Probst <harlydavidsen@gmail.com> wrote:
While conquering refreshable braille displays would be nice, I think
we should take it in stages. (Some blind people would want this in
along with the accessibility subsystems, however I do not believe that
would be the wisest approach.) Instead, I think we should do what you
suggest: implement protocols for communicating with audio devices
(both input and output) and add audio hooks for the HII. (We could
even make those audio hooks just general hooks/events instead of
partitioning them for a particular purpose.)
You mentioned that not all devices 'have a keyboard'. This is most
definitely true, but not all devices have audio outputs and/or inputs
either. Therefore, if we can't mandate it for *all* devices, perhaps
we could only mandate it (or at least somehow infer it as a
requirement) for devices that both have an interaction mechanism (any
form of interacting with the system directly via the HII and not, say,
over the wire, because that would be superfluous) and have
mechanisms/devices/drivers/... for audio output at minimum. (Audio
input is not necessarily something that is required unless the device
is controlled primarily by voice, but I think having voice controls is
a bit beyond the UEFI specification and is not something UEFI should
worry about. Voice control is not something you should have in the
preboot environment anyway, correct?)
So, I think the first stage we should work on should include:
- Implementing protocols for communicating with audio outputs (don't
worry about inputs right now)
- Implementing protocols/hooks/events that can be intercepted in the
HII by a foreground/background UEFI driver/application
- Implementing some kind of method to allow the accessibility
protocols to read what is drawn to the text when navigating around
The third item -- when implemented properly -- will prevent the
accessibility subsystem from reading everything on the screen every
time the "cursor" moves. Perhaps another good idea would be to
implement some kind of priority system for text-to-speech, like speech
dispatcher does
(
https://docs.microsoft.com/en-us/cortana/skills/speech-synthesis-markup-language
).

On 9/10/19, Leif Lindholm <leif.lindholm@linaro.org> wrote:
Hi Ethin,

Apologies for the delay in responding - I personally don't tend to
read the RFC list (and I know that applies to some other developers
who have now subscribed).

On Sun, Aug 11, 2019 at 06:15:18PM -0700, Ethin Probst wrote:
Hello all,

I'm new here, and was recommended to the TianoCore project by
someone over at the UEFI forum. I've run across TianoCore before,
and like the project.
Before anyone gets worried by the subject line, no, this is not any
kind of legal thing. Its just something I believe needs to
happen. :)
Back in 2016-2017 I contacted the UEFI forum about two problems, one
of which was the format of the spec, which I figured out on my
own. The other problem was not so easily dealt with. Te other
problem relates to accessibility of UEFI-compliant systems and
platform firmware to persons with disabilities. As it currently
stands, such a thing is nonexistent. To be fair, I completely
understand the difficulty that such a thing would require, and I
would fully agree if we still used the PC-AT BIOS systems -- yes,
indeed, I would never suggest this kind of thing on such a system
given that there was no actual standard of any kind for
BIOSes. However, now that UEFI is here, we have such a possibility.
As it currently stands, people with low vision or blind people have
access to their computers in the general sense (I am blind
myself). We can do pretty much anything anyone else could do. We can
code, play games, all that. There are few things that we cannot
do. One of those things is managing our systems firmware in the
preboot environment.
As it stands now, I can only boot other OSes or disks via
memorization. While that worked on BIOS machines (I have, or had, an
old Toshiba laptop that was BIOS-based), it no longer works because
UEFI is mercurial. When I access the boot menu now, I play a game of
chance. If the cards are in my favor, the OS I want to boot boots,
and I can go on my way. But if the cards aren't in my favor, I end
up making something happen that was unintended, and, worst of all, I
have no idea what I did.
However, the boot menu is only one portion of a platform firmware
UI. What about the setup utility or other utilities offered by
computer manufacturers? What about diagnostic utilities,
bootloaders, etc? What do I do with those? Well, I only have one
option -- sited assistance. If I go into my computers setup utility,
I cannot trust myself and say to myself, "OK, I know what I'm
doing. All I need to do is change x and save and quit." No, I can't
do that, because memorizing such a complex interface is extremely
difficult, and its something I wouldn't expect anyone to do.
My proposal is simple, and I'm posting it here because I'd like
comments and feedback before it actually gets implemented (it will
take a lot of time, I'm sure): mandate, in the UEFI specification,
that accessibility features for persons with disabilities must be
implemented and documented, and, if such features are not
implemented, then that vendor is not compliant with the
specification. Place strict minimum requirements for what the
accessibility features should do and how they should work.
So, first of all, I need to point out that what you suggest would be
very difficult for the UEFI Forum to mandate. *But* I don't think this
needs to be a problem.

UEFI primarily exists as a means through which interoperability can be
guaranteed. This means it does not tend to so much mandate what is
actually implemented in a platform, as it defines how it behaves if it
is. This is not 100% true, but suffice to say that mandating the
functionality you are asking for *without* having a reference
implementation available would likely simply be ignored.

If you want to turn thumbscrews, get Microsoft to add the requirement
to their windows logo requirements :)

However, as much as it is difficult to force people to develop this
support, if a functional reference implementation was published under
a permissive open source license, it would take a pretty spectacularly
incompetent product owner not to incorporate it in their products.

Now, I'm sure someone out there will ask me how this can be
done. Well, that's why I've joined the group -- though as I
familiarize myself with EDK2 development and all that I may actually
be able to participate as more than just an accessibility expert, of
sorts.
I have added Rafael Machado on cc. He did a PhD on firmware
accessibility, including a prototype implementing an audio driver for
EDK2. The resulting thesis can be found at
https://arxiv.org/abs/1712.03186 .

As a side note, I have been blind all my life. I was born with
retinopathy of prematurity (ROP), which resulted because I was born
at 26 weeks. My retina was detached, and, though the doctors
attempted to fix it, it would not remain attached, and there is no
chance of it getting fixed now. I would neither want nor care for
such a cure, however. I have lived my entire life blind, and while
the thought of gaining site back is appealing, I am unwilling to go
through the years and years of rewiring and reconditioning of my
brain that would be required for me to survive with site. To me, it
is simply not worth the cost.
But back to the discussion at hand: I would be happy to discuss how
the accessibility features would work and what would be
required. Even standardizing, through the specification, a key
combination to toggle the accessibility features would be nice, as
that would alleviate the major problem of a blind person buying a
new computer and not knowing how to enable the accessibility
features. The overarching goal would be to make the preboot
environment (including applications run within it) accessible and
usable by blind and visually impaired people as a boot service
only.
Yes, I agree, operating systems are already doing their things.

Again, mandating such a key combination is not necessarily something
that the UEFI specification can do. Some devices won't necessarily
have a keyboard.

Some things the UEFI specification could do is:
- Add protocols for Audio output (and possibly input) and talking to
audio codecs.
- Add audio hooks for the Human Interaction Interface subsystem (HII).

EDK2 could then start adding device drivers producing these protocols,
and enable the support for some reference platforms.

Would we need any specific support for things like refreshable braille
displays?

It would be superfluous to make this a runtime service, as all
major OSes already have accessibility features. Plus, managing such
a thing would be impossible to do.
Agreed.

This email has gotten quite long, so I will suspend the discussion
of functionality and how I would like it to work for a future email
once everyone has gotten on board.
Best Regards,

Leif


Thank you for your time and consideration.



--
Signed,
Ethin D. Probst

--
Signed,
Ethin D. Probst


Ethin Probst
 

I've posted this to the largest forum of blind visually impaired
people I know of to date and have gotten some excellent feedback from
them. I also provided in that post a link to this discussion. However,
I can't possibly forward all their questions onto this group (and I
doubt I can answer them all). If you wish to join the discussion over
there, its over at https://forum.audiogames.net/post/461041/#p461041.
(If we can find a way to merge the two discussions, that would make
everything easier. If we can't, I'm fine with that.)

On 9/10/19, Rafael Machado <rafaelrodrigues.machado@gmail.com> wrote:
Hi Ethin and all

As Leif mentioned, my mastering thesys was the implementation of a
reference code so a driver for audio at pre-OS would be possible,
eliminating the first barrier for the creation of a screen reader at UEFI,
that is the lack of audio drivers.

My project was based on Intel High Definition Audio, so it is not a generic
code that would fit at all scenarios, but it a first step.
The idea was to proceed this work using the Google Summer of Code
initiative,and I will try my best to make this happen in future editions of
this initiative.
Unfortunately my thesys was written in portuguese, because there was almost
nothing related to BIOS in Portuguese, so the decision was to write in
Portuguese to help the community to attract brasilian developers. As soon
as I find some slot, I'll translate to english.
But you are not alone. Just in Brazil I discovered around 30 blind
developers that area waiting for accessibility at BIOS. Also had some
contacts from US and Ireland.
So it is a market to be explored. Everyone will win.
The companies, because they will attract more people and their families,
and the ones who need accessibility.
By the way, most of the population will need accessible solutions if they
live enough. This includes the uefi community members, that today do not
have special needs, but that may require it in future.
Accessibility is not a wast of money.
This ia what I would like to show to the UEFI community.
Hope to have the opportunity to present this in some future event.

Since this discussion was started, who knows what are the next steps?
My code is available at github if someone gets interested (ps.: The code
has failures. It was just a Prof of Concept)

https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Lets keep the discussion, because it is important.

Thanks and Regards
Rafael R. Machado


Em ter, 10 de set de 2019 17:00, Ethin Probst <harlydavidsen@gmail.com>
escreveu:

Woops, I just realized while reading my message after I'd sent it that
I had linked to the wrong document. The priority system is outlined
over here (https://freebsoft.org/doc/speechd/ssip.html#Top). This
document discusses SsIP -- the Speech Synthesis Interface Protocol.

On 9/10/19, Ethin Probst <harlydavidsen@gmail.com> wrote:
While conquering refreshable braille displays would be nice, I think
we should take it in stages. (Some blind people would want this in
along with the accessibility subsystems, however I do not believe that
would be the wisest approach.) Instead, I think we should do what you
suggest: implement protocols for communicating with audio devices
(both input and output) and add audio hooks for the HII. (We could
even make those audio hooks just general hooks/events instead of
partitioning them for a particular purpose.)
You mentioned that not all devices 'have a keyboard'. This is most
definitely true, but not all devices have audio outputs and/or inputs
either. Therefore, if we can't mandate it for *all* devices, perhaps
we could only mandate it (or at least somehow infer it as a
requirement) for devices that both have an interaction mechanism (any
form of interacting with the system directly via the HII and not, say,
over the wire, because that would be superfluous) and have
mechanisms/devices/drivers/... for audio output at minimum. (Audio
input is not necessarily something that is required unless the device
is controlled primarily by voice, but I think having voice controls is
a bit beyond the UEFI specification and is not something UEFI should
worry about. Voice control is not something you should have in the
preboot environment anyway, correct?)
So, I think the first stage we should work on should include:
- Implementing protocols for communicating with audio outputs (don't
worry about inputs right now)
- Implementing protocols/hooks/events that can be intercepted in the
HII by a foreground/background UEFI driver/application
- Implementing some kind of method to allow the accessibility
protocols to read what is drawn to the text when navigating around
The third item -- when implemented properly -- will prevent the
accessibility subsystem from reading everything on the screen every
time the "cursor" moves. Perhaps another good idea would be to
implement some kind of priority system for text-to-speech, like speech
dispatcher does
(
https://docs.microsoft.com/en-us/cortana/skills/speech-synthesis-markup-language
).

On 9/10/19, Leif Lindholm <leif.lindholm@linaro.org> wrote:
Hi Ethin,

Apologies for the delay in responding - I personally don't tend to
read the RFC list (and I know that applies to some other developers
who have now subscribed).

On Sun, Aug 11, 2019 at 06:15:18PM -0700, Ethin Probst wrote:
Hello all,

I'm new here, and was recommended to the TianoCore project by
someone over at the UEFI forum. I've run across TianoCore before,
and like the project.
Before anyone gets worried by the subject line, no, this is not any
kind of legal thing. Its just something I believe needs to
happen. :)
Back in 2016-2017 I contacted the UEFI forum about two problems, one
of which was the format of the spec, which I figured out on my
own. The other problem was not so easily dealt with. Te other
problem relates to accessibility of UEFI-compliant systems and
platform firmware to persons with disabilities. As it currently
stands, such a thing is nonexistent. To be fair, I completely
understand the difficulty that such a thing would require, and I
would fully agree if we still used the PC-AT BIOS systems -- yes,
indeed, I would never suggest this kind of thing on such a system
given that there was no actual standard of any kind for
BIOSes. However, now that UEFI is here, we have such a possibility.
As it currently stands, people with low vision or blind people have
access to their computers in the general sense (I am blind
myself). We can do pretty much anything anyone else could do. We can
code, play games, all that. There are few things that we cannot
do. One of those things is managing our systems firmware in the
preboot environment.
As it stands now, I can only boot other OSes or disks via
memorization. While that worked on BIOS machines (I have, or had, an
old Toshiba laptop that was BIOS-based), it no longer works because
UEFI is mercurial. When I access the boot menu now, I play a game of
chance. If the cards are in my favor, the OS I want to boot boots,
and I can go on my way. But if the cards aren't in my favor, I end
up making something happen that was unintended, and, worst of all, I
have no idea what I did.
However, the boot menu is only one portion of a platform firmware
UI. What about the setup utility or other utilities offered by
computer manufacturers? What about diagnostic utilities,
bootloaders, etc? What do I do with those? Well, I only have one
option -- sited assistance. If I go into my computers setup utility,
I cannot trust myself and say to myself, "OK, I know what I'm
doing. All I need to do is change x and save and quit." No, I can't
do that, because memorizing such a complex interface is extremely
difficult, and its something I wouldn't expect anyone to do.
My proposal is simple, and I'm posting it here because I'd like
comments and feedback before it actually gets implemented (it will
take a lot of time, I'm sure): mandate, in the UEFI specification,
that accessibility features for persons with disabilities must be
implemented and documented, and, if such features are not
implemented, then that vendor is not compliant with the
specification. Place strict minimum requirements for what the
accessibility features should do and how they should work.
So, first of all, I need to point out that what you suggest would be
very difficult for the UEFI Forum to mandate. *But* I don't think this
needs to be a problem.

UEFI primarily exists as a means through which interoperability can be
guaranteed. This means it does not tend to so much mandate what is
actually implemented in a platform, as it defines how it behaves if it
is. This is not 100% true, but suffice to say that mandating the
functionality you are asking for *without* having a reference
implementation available would likely simply be ignored.

If you want to turn thumbscrews, get Microsoft to add the requirement
to their windows logo requirements :)

However, as much as it is difficult to force people to develop this
support, if a functional reference implementation was published under
a permissive open source license, it would take a pretty spectacularly
incompetent product owner not to incorporate it in their products.

Now, I'm sure someone out there will ask me how this can be
done. Well, that's why I've joined the group -- though as I
familiarize myself with EDK2 development and all that I may actually
be able to participate as more than just an accessibility expert, of
sorts.
I have added Rafael Machado on cc. He did a PhD on firmware
accessibility, including a prototype implementing an audio driver for
EDK2. The resulting thesis can be found at
https://arxiv.org/abs/1712.03186 .

As a side note, I have been blind all my life. I was born with
retinopathy of prematurity (ROP), which resulted because I was born
at 26 weeks. My retina was detached, and, though the doctors
attempted to fix it, it would not remain attached, and there is no
chance of it getting fixed now. I would neither want nor care for
such a cure, however. I have lived my entire life blind, and while
the thought of gaining site back is appealing, I am unwilling to go
through the years and years of rewiring and reconditioning of my
brain that would be required for me to survive with site. To me, it
is simply not worth the cost.
But back to the discussion at hand: I would be happy to discuss how
the accessibility features would work and what would be
required. Even standardizing, through the specification, a key
combination to toggle the accessibility features would be nice, as
that would alleviate the major problem of a blind person buying a
new computer and not knowing how to enable the accessibility
features. The overarching goal would be to make the preboot
environment (including applications run within it) accessible and
usable by blind and visually impaired people as a boot service
only.
Yes, I agree, operating systems are already doing their things.

Again, mandating such a key combination is not necessarily something
that the UEFI specification can do. Some devices won't necessarily
have a keyboard.

Some things the UEFI specification could do is:
- Add protocols for Audio output (and possibly input) and talking to
audio codecs.
- Add audio hooks for the Human Interaction Interface subsystem (HII).

EDK2 could then start adding device drivers producing these protocols,
and enable the support for some reference platforms.

Would we need any specific support for things like refreshable braille
displays?

It would be superfluous to make this a runtime service, as all
major OSes already have accessibility features. Plus, managing such
a thing would be impossible to do.
Agreed.

This email has gotten quite long, so I will suspend the discussion
of functionality and how I would like it to work for a future email
once everyone has gotten on board.
Best Regards,

Leif


Thank you for your time and consideration.



--
Signed,
Ethin D. Probst

--
Signed,
Ethin D. Probst


--
Signed,
Ethin D. Probst


Rafael Machado <rafaelrodrigues.machado@...>
 

Hi Ethin
I will contact the community you mentioned to get feedback and align
expectations.

Hope to hear comments and feedback of other member of discussion (here at
EDK2, that is where the work will happen).

Thanks and Regards
Rafael

Em ter, 10 de set de 2019 às 20:50, Ethin Probst <harlydavidsen@gmail.com>
escreveu:

I've posted this to the largest forum of blind visually impaired
people I know of to date and have gotten some excellent feedback from
them. I also provided in that post a link to this discussion. However,
I can't possibly forward all their questions onto this group (and I
doubt I can answer them all). If you wish to join the discussion over
there, its over at https://forum.audiogames.net/post/461041/#p461041.
(If we can find a way to merge the two discussions, that would make
everything easier. If we can't, I'm fine with that.)

On 9/10/19, Rafael Machado <rafaelrodrigues.machado@gmail.com> wrote:
Hi Ethin and all

As Leif mentioned, my mastering thesys was the implementation of a
reference code so a driver for audio at pre-OS would be possible,
eliminating the first barrier for the creation of a screen reader at
UEFI,
that is the lack of audio drivers.

My project was based on Intel High Definition Audio, so it is not a
generic
code that would fit at all scenarios, but it a first step.
The idea was to proceed this work using the Google Summer of Code
initiative,and I will try my best to make this happen in future editions
of
this initiative.
Unfortunately my thesys was written in portuguese, because there was
almost
nothing related to BIOS in Portuguese, so the decision was to write in
Portuguese to help the community to attract brasilian developers. As soon
as I find some slot, I'll translate to english.
But you are not alone. Just in Brazil I discovered around 30 blind
developers that area waiting for accessibility at BIOS. Also had some
contacts from US and Ireland.
So it is a market to be explored. Everyone will win.
The companies, because they will attract more people and their families,
and the ones who need accessibility.
By the way, most of the population will need accessible solutions if they
live enough. This includes the uefi community members, that today do not
have special needs, but that may require it in future.
Accessibility is not a wast of money.
This ia what I would like to show to the UEFI community.
Hope to have the opportunity to present this in some future event.

Since this discussion was started, who knows what are the next steps?
My code is available at github if someone gets interested (ps.: The code
has failures. It was just a Prof of Concept)

https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Lets keep the discussion, because it is important.

Thanks and Regards
Rafael R. Machado


Em ter, 10 de set de 2019 17:00, Ethin Probst <harlydavidsen@gmail.com>
escreveu:

Woops, I just realized while reading my message after I'd sent it that
I had linked to the wrong document. The priority system is outlined
over here (https://freebsoft.org/doc/speechd/ssip.html#Top). This
document discusses SsIP -- the Speech Synthesis Interface Protocol.

On 9/10/19, Ethin Probst <harlydavidsen@gmail.com> wrote:
While conquering refreshable braille displays would be nice, I think
we should take it in stages. (Some blind people would want this in
along with the accessibility subsystems, however I do not believe that
would be the wisest approach.) Instead, I think we should do what you
suggest: implement protocols for communicating with audio devices
(both input and output) and add audio hooks for the HII. (We could
even make those audio hooks just general hooks/events instead of
partitioning them for a particular purpose.)
You mentioned that not all devices 'have a keyboard'. This is most
definitely true, but not all devices have audio outputs and/or inputs
either. Therefore, if we can't mandate it for *all* devices, perhaps
we could only mandate it (or at least somehow infer it as a
requirement) for devices that both have an interaction mechanism (any
form of interacting with the system directly via the HII and not, say,
over the wire, because that would be superfluous) and have
mechanisms/devices/drivers/... for audio output at minimum. (Audio
input is not necessarily something that is required unless the device
is controlled primarily by voice, but I think having voice controls is
a bit beyond the UEFI specification and is not something UEFI should
worry about. Voice control is not something you should have in the
preboot environment anyway, correct?)
So, I think the first stage we should work on should include:
- Implementing protocols for communicating with audio outputs (don't
worry about inputs right now)
- Implementing protocols/hooks/events that can be intercepted in the
HII by a foreground/background UEFI driver/application
- Implementing some kind of method to allow the accessibility
protocols to read what is drawn to the text when navigating around
The third item -- when implemented properly -- will prevent the
accessibility subsystem from reading everything on the screen every
time the "cursor" moves. Perhaps another good idea would be to
implement some kind of priority system for text-to-speech, like speech
dispatcher does
(
https://docs.microsoft.com/en-us/cortana/skills/speech-synthesis-markup-language
).

On 9/10/19, Leif Lindholm <leif.lindholm@linaro.org> wrote:
Hi Ethin,

Apologies for the delay in responding - I personally don't tend to
read the RFC list (and I know that applies to some other developers
who have now subscribed).

On Sun, Aug 11, 2019 at 06:15:18PM -0700, Ethin Probst wrote:
Hello all,

I'm new here, and was recommended to the TianoCore project by
someone over at the UEFI forum. I've run across TianoCore before,
and like the project.
Before anyone gets worried by the subject line, no, this is not any
kind of legal thing. Its just something I believe needs to
happen. :)
Back in 2016-2017 I contacted the UEFI forum about two problems, one
of which was the format of the spec, which I figured out on my
own. The other problem was not so easily dealt with. Te other
problem relates to accessibility of UEFI-compliant systems and
platform firmware to persons with disabilities. As it currently
stands, such a thing is nonexistent. To be fair, I completely
understand the difficulty that such a thing would require, and I
would fully agree if we still used the PC-AT BIOS systems -- yes,
indeed, I would never suggest this kind of thing on such a system
given that there was no actual standard of any kind for
BIOSes. However, now that UEFI is here, we have such a possibility.
As it currently stands, people with low vision or blind people have
access to their computers in the general sense (I am blind
myself). We can do pretty much anything anyone else could do. We can
code, play games, all that. There are few things that we cannot
do. One of those things is managing our systems firmware in the
preboot environment.
As it stands now, I can only boot other OSes or disks via
memorization. While that worked on BIOS machines (I have, or had, an
old Toshiba laptop that was BIOS-based), it no longer works because
UEFI is mercurial. When I access the boot menu now, I play a game of
chance. If the cards are in my favor, the OS I want to boot boots,
and I can go on my way. But if the cards aren't in my favor, I end
up making something happen that was unintended, and, worst of all, I
have no idea what I did.
However, the boot menu is only one portion of a platform firmware
UI. What about the setup utility or other utilities offered by
computer manufacturers? What about diagnostic utilities,
bootloaders, etc? What do I do with those? Well, I only have one
option -- sited assistance. If I go into my computers setup utility,
I cannot trust myself and say to myself, "OK, I know what I'm
doing. All I need to do is change x and save and quit." No, I can't
do that, because memorizing such a complex interface is extremely
difficult, and its something I wouldn't expect anyone to do.
My proposal is simple, and I'm posting it here because I'd like
comments and feedback before it actually gets implemented (it will
take a lot of time, I'm sure): mandate, in the UEFI specification,
that accessibility features for persons with disabilities must be
implemented and documented, and, if such features are not
implemented, then that vendor is not compliant with the
specification. Place strict minimum requirements for what the
accessibility features should do and how they should work.
So, first of all, I need to point out that what you suggest would be
very difficult for the UEFI Forum to mandate. *But* I don't think
this
needs to be a problem.

UEFI primarily exists as a means through which interoperability can
be
guaranteed. This means it does not tend to so much mandate what is
actually implemented in a platform, as it defines how it behaves if
it
is. This is not 100% true, but suffice to say that mandating the
functionality you are asking for *without* having a reference
implementation available would likely simply be ignored.

If you want to turn thumbscrews, get Microsoft to add the requirement
to their windows logo requirements :)

However, as much as it is difficult to force people to develop this
support, if a functional reference implementation was published under
a permissive open source license, it would take a pretty
spectacularly
incompetent product owner not to incorporate it in their products.

Now, I'm sure someone out there will ask me how this can be
done. Well, that's why I've joined the group -- though as I
familiarize myself with EDK2 development and all that I may actually
be able to participate as more than just an accessibility expert, of
sorts.
I have added Rafael Machado on cc. He did a PhD on firmware
accessibility, including a prototype implementing an audio driver for
EDK2. The resulting thesis can be found at
https://arxiv.org/abs/1712.03186 .

As a side note, I have been blind all my life. I was born with
retinopathy of prematurity (ROP), which resulted because I was born
at 26 weeks. My retina was detached, and, though the doctors
attempted to fix it, it would not remain attached, and there is no
chance of it getting fixed now. I would neither want nor care for
such a cure, however. I have lived my entire life blind, and while
the thought of gaining site back is appealing, I am unwilling to go
through the years and years of rewiring and reconditioning of my
brain that would be required for me to survive with site. To me, it
is simply not worth the cost.
But back to the discussion at hand: I would be happy to discuss how
the accessibility features would work and what would be
required. Even standardizing, through the specification, a key
combination to toggle the accessibility features would be nice, as
that would alleviate the major problem of a blind person buying a
new computer and not knowing how to enable the accessibility
features. The overarching goal would be to make the preboot
environment (including applications run within it) accessible and
usable by blind and visually impaired people as a boot service
only.
Yes, I agree, operating systems are already doing their things.

Again, mandating such a key combination is not necessarily something
that the UEFI specification can do. Some devices won't necessarily
have a keyboard.

Some things the UEFI specification could do is:
- Add protocols for Audio output (and possibly input) and talking to
audio codecs.
- Add audio hooks for the Human Interaction Interface subsystem
(HII).

EDK2 could then start adding device drivers producing these
protocols,
and enable the support for some reference platforms.

Would we need any specific support for things like refreshable
braille
displays?

It would be superfluous to make this a runtime service, as all
major OSes already have accessibility features. Plus, managing such
a thing would be impossible to do.
Agreed.

This email has gotten quite long, so I will suspend the discussion
of functionality and how I would like it to work for a future email
once everyone has gotten on board.
Best Regards,

Leif


Thank you for your time and consideration.



--
Signed,
Ethin D. Probst

--
Signed,
Ethin D. Probst



--
Signed,
Ethin D. Probst


Rafael Machado <rafaelrodrigues.machado@...>
 

Hi Ethin

I just get registered at the forum, but have no permission to post there.
The message "Sorry! no permission to post a reply" is presented to my user.

Thanks
Rafael

Em sex, 13 de set de 2019 às 08:16, Rafael Machado <
rafaelrodrigues.machado@gmail.com> escreveu:

Hi Ethin
I will contact the community you mentioned to get feedback and align
expectations.

Hope to hear comments and feedback of other member of discussion (here at
EDK2, that is where the work will happen).

Thanks and Regards
Rafael

Em ter, 10 de set de 2019 às 20:50, Ethin Probst <harlydavidsen@gmail.com>
escreveu:

I've posted this to the largest forum of blind visually impaired
people I know of to date and have gotten some excellent feedback from
them. I also provided in that post a link to this discussion. However,
I can't possibly forward all their questions onto this group (and I
doubt I can answer them all). If you wish to join the discussion over
there, its over at https://forum.audiogames.net/post/461041/#p461041.
(If we can find a way to merge the two discussions, that would make
everything easier. If we can't, I'm fine with that.)

On 9/10/19, Rafael Machado <rafaelrodrigues.machado@gmail.com> wrote:
Hi Ethin and all

As Leif mentioned, my mastering thesys was the implementation of a
reference code so a driver for audio at pre-OS would be possible,
eliminating the first barrier for the creation of a screen reader at
UEFI,
that is the lack of audio drivers.

My project was based on Intel High Definition Audio, so it is not a
generic
code that would fit at all scenarios, but it a first step.
The idea was to proceed this work using the Google Summer of Code
initiative,and I will try my best to make this happen in future
editions of
this initiative.
Unfortunately my thesys was written in portuguese, because there was
almost
nothing related to BIOS in Portuguese, so the decision was to write in
Portuguese to help the community to attract brasilian developers. As
soon
as I find some slot, I'll translate to english.
But you are not alone. Just in Brazil I discovered around 30 blind
developers that area waiting for accessibility at BIOS. Also had some
contacts from US and Ireland.
So it is a market to be explored. Everyone will win.
The companies, because they will attract more people and their families,
and the ones who need accessibility.
By the way, most of the population will need accessible solutions if
they
live enough. This includes the uefi community members, that today do not
have special needs, but that may require it in future.
Accessibility is not a wast of money.
This ia what I would like to show to the UEFI community.
Hope to have the opportunity to present this in some future event.

Since this discussion was started, who knows what are the next steps?
My code is available at github if someone gets interested (ps.: The code
has failures. It was just a Prof of Concept)

https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Lets keep the discussion, because it is important.

Thanks and Regards
Rafael R. Machado


Em ter, 10 de set de 2019 17:00, Ethin Probst <harlydavidsen@gmail.com>
escreveu:

Woops, I just realized while reading my message after I'd sent it that
I had linked to the wrong document. The priority system is outlined
over here (https://freebsoft.org/doc/speechd/ssip.html#Top). This
document discusses SsIP -- the Speech Synthesis Interface Protocol.

On 9/10/19, Ethin Probst <harlydavidsen@gmail.com> wrote:
While conquering refreshable braille displays would be nice, I think
we should take it in stages. (Some blind people would want this in
along with the accessibility subsystems, however I do not believe
that
would be the wisest approach.) Instead, I think we should do what you
suggest: implement protocols for communicating with audio devices
(both input and output) and add audio hooks for the HII. (We could
even make those audio hooks just general hooks/events instead of
partitioning them for a particular purpose.)
You mentioned that not all devices 'have a keyboard'. This is most
definitely true, but not all devices have audio outputs and/or inputs
either. Therefore, if we can't mandate it for *all* devices, perhaps
we could only mandate it (or at least somehow infer it as a
requirement) for devices that both have an interaction mechanism (any
form of interacting with the system directly via the HII and not,
say,
over the wire, because that would be superfluous) and have
mechanisms/devices/drivers/... for audio output at minimum. (Audio
input is not necessarily something that is required unless the device
is controlled primarily by voice, but I think having voice controls
is
a bit beyond the UEFI specification and is not something UEFI should
worry about. Voice control is not something you should have in the
preboot environment anyway, correct?)
So, I think the first stage we should work on should include:
- Implementing protocols for communicating with audio outputs (don't
worry about inputs right now)
- Implementing protocols/hooks/events that can be intercepted in the
HII by a foreground/background UEFI driver/application
- Implementing some kind of method to allow the accessibility
protocols to read what is drawn to the text when navigating around
The third item -- when implemented properly -- will prevent the
accessibility subsystem from reading everything on the screen every
time the "cursor" moves. Perhaps another good idea would be to
implement some kind of priority system for text-to-speech, like
speech
dispatcher does
(
https://docs.microsoft.com/en-us/cortana/skills/speech-synthesis-markup-language
).

On 9/10/19, Leif Lindholm <leif.lindholm@linaro.org> wrote:
Hi Ethin,

Apologies for the delay in responding - I personally don't tend to
read the RFC list (and I know that applies to some other developers
who have now subscribed).

On Sun, Aug 11, 2019 at 06:15:18PM -0700, Ethin Probst wrote:
Hello all,

I'm new here, and was recommended to the TianoCore project by
someone over at the UEFI forum. I've run across TianoCore before,
and like the project.
Before anyone gets worried by the subject line, no, this is not any
kind of legal thing. Its just something I believe needs to
happen. :)
Back in 2016-2017 I contacted the UEFI forum about two problems,
one
of which was the format of the spec, which I figured out on my
own. The other problem was not so easily dealt with. Te other
problem relates to accessibility of UEFI-compliant systems and
platform firmware to persons with disabilities. As it currently
stands, such a thing is nonexistent. To be fair, I completely
understand the difficulty that such a thing would require, and I
would fully agree if we still used the PC-AT BIOS systems -- yes,
indeed, I would never suggest this kind of thing on such a system
given that there was no actual standard of any kind for
BIOSes. However, now that UEFI is here, we have such a possibility.
As it currently stands, people with low vision or blind people have
access to their computers in the general sense (I am blind
myself). We can do pretty much anything anyone else could do. We
can
code, play games, all that. There are few things that we cannot
do. One of those things is managing our systems firmware in the
preboot environment.
As it stands now, I can only boot other OSes or disks via
memorization. While that worked on BIOS machines (I have, or had,
an
old Toshiba laptop that was BIOS-based), it no longer works because
UEFI is mercurial. When I access the boot menu now, I play a game
of
chance. If the cards are in my favor, the OS I want to boot boots,
and I can go on my way. But if the cards aren't in my favor, I end
up making something happen that was unintended, and, worst of all,
I
have no idea what I did.
However, the boot menu is only one portion of a platform firmware
UI. What about the setup utility or other utilities offered by
computer manufacturers? What about diagnostic utilities,
bootloaders, etc? What do I do with those? Well, I only have one
option -- sited assistance. If I go into my computers setup
utility,
I cannot trust myself and say to myself, "OK, I know what I'm
doing. All I need to do is change x and save and quit." No, I can't
do that, because memorizing such a complex interface is extremely
difficult, and its something I wouldn't expect anyone to do.
My proposal is simple, and I'm posting it here because I'd like
comments and feedback before it actually gets implemented (it will
take a lot of time, I'm sure): mandate, in the UEFI specification,
that accessibility features for persons with disabilities must be
implemented and documented, and, if such features are not
implemented, then that vendor is not compliant with the
specification. Place strict minimum requirements for what the
accessibility features should do and how they should work.
So, first of all, I need to point out that what you suggest would be
very difficult for the UEFI Forum to mandate. *But* I don't think
this
needs to be a problem.

UEFI primarily exists as a means through which interoperability can
be
guaranteed. This means it does not tend to so much mandate what is
actually implemented in a platform, as it defines how it behaves if
it
is. This is not 100% true, but suffice to say that mandating the
functionality you are asking for *without* having a reference
implementation available would likely simply be ignored.

If you want to turn thumbscrews, get Microsoft to add the
requirement
to their windows logo requirements :)

However, as much as it is difficult to force people to develop this
support, if a functional reference implementation was published
under
a permissive open source license, it would take a pretty
spectacularly
incompetent product owner not to incorporate it in their products.

Now, I'm sure someone out there will ask me how this can be
done. Well, that's why I've joined the group -- though as I
familiarize myself with EDK2 development and all that I may
actually
be able to participate as more than just an accessibility expert,
of
sorts.
I have added Rafael Machado on cc. He did a PhD on firmware
accessibility, including a prototype implementing an audio driver
for
EDK2. The resulting thesis can be found at
https://arxiv.org/abs/1712.03186 .

As a side note, I have been blind all my life. I was born with
retinopathy of prematurity (ROP), which resulted because I was born
at 26 weeks. My retina was detached, and, though the doctors
attempted to fix it, it would not remain attached, and there is no
chance of it getting fixed now. I would neither want nor care for
such a cure, however. I have lived my entire life blind, and while
the thought of gaining site back is appealing, I am unwilling to go
through the years and years of rewiring and reconditioning of my
brain that would be required for me to survive with site. To me, it
is simply not worth the cost.
But back to the discussion at hand: I would be happy to discuss how
the accessibility features would work and what would be
required. Even standardizing, through the specification, a key
combination to toggle the accessibility features would be nice, as
that would alleviate the major problem of a blind person buying a
new computer and not knowing how to enable the accessibility
features. The overarching goal would be to make the preboot
environment (including applications run within it) accessible and
usable by blind and visually impaired people as a boot service
only.
Yes, I agree, operating systems are already doing their things.

Again, mandating such a key combination is not necessarily something
that the UEFI specification can do. Some devices won't necessarily
have a keyboard.

Some things the UEFI specification could do is:
- Add protocols for Audio output (and possibly input) and talking to
audio codecs.
- Add audio hooks for the Human Interaction Interface subsystem
(HII).

EDK2 could then start adding device drivers producing these
protocols,
and enable the support for some reference platforms.

Would we need any specific support for things like refreshable
braille
displays?

It would be superfluous to make this a runtime service, as all
major OSes already have accessibility features. Plus, managing such
a thing would be impossible to do.
Agreed.

This email has gotten quite long, so I will suspend the discussion
of functionality and how I would like it to work for a future email
once everyone has gotten on board.
Best Regards,

Leif


Thank you for your time and consideration.



--
Signed,
Ethin D. Probst

--
Signed,
Ethin D. Probst



--
Signed,
Ethin D. Probst


Ethin Probst
 

You need to post about yourself in this topic to actually get
unrestricted (its a way of stopping bots):
https://forum.audiogames.net/post/461466/#p461466
Sorry I didn't mention that, I forgot about that. :(

On 9/13/19, Rafael Machado <rafaelrodrigues.machado@gmail.com> wrote:
Hi Ethin

I just get registered at the forum, but have no permission to post there.
The message "Sorry! no permission to post a reply" is presented to my
user.

Thanks
Rafael

Em sex, 13 de set de 2019 às 08:16, Rafael Machado <
rafaelrodrigues.machado@gmail.com> escreveu:

Hi Ethin
I will contact the community you mentioned to get feedback and align
expectations.

Hope to hear comments and feedback of other member of discussion (here at
EDK2, that is where the work will happen).

Thanks and Regards
Rafael

Em ter, 10 de set de 2019 às 20:50, Ethin Probst
<harlydavidsen@gmail.com>
escreveu:

I've posted this to the largest forum of blind visually impaired
people I know of to date and have gotten some excellent feedback from
them. I also provided in that post a link to this discussion. However,
I can't possibly forward all their questions onto this group (and I
doubt I can answer them all). If you wish to join the discussion over
there, its over at https://forum.audiogames.net/post/461041/#p461041.
(If we can find a way to merge the two discussions, that would make
everything easier. If we can't, I'm fine with that.)

On 9/10/19, Rafael Machado <rafaelrodrigues.machado@gmail.com> wrote:
Hi Ethin and all

As Leif mentioned, my mastering thesys was the implementation of a
reference code so a driver for audio at pre-OS would be possible,
eliminating the first barrier for the creation of a screen reader at
UEFI,
that is the lack of audio drivers.

My project was based on Intel High Definition Audio, so it is not a
generic
code that would fit at all scenarios, but it a first step.
The idea was to proceed this work using the Google Summer of Code
initiative,and I will try my best to make this happen in future
editions of
this initiative.
Unfortunately my thesys was written in portuguese, because there was
almost
nothing related to BIOS in Portuguese, so the decision was to write in
Portuguese to help the community to attract brasilian developers. As
soon
as I find some slot, I'll translate to english.
But you are not alone. Just in Brazil I discovered around 30 blind
developers that area waiting for accessibility at BIOS. Also had some
contacts from US and Ireland.
So it is a market to be explored. Everyone will win.
The companies, because they will attract more people and their
families,
and the ones who need accessibility.
By the way, most of the population will need accessible solutions if
they
live enough. This includes the uefi community members, that today do
not
have special needs, but that may require it in future.
Accessibility is not a wast of money.
This ia what I would like to show to the UEFI community.
Hope to have the opportunity to present this in some future event.

Since this discussion was started, who knows what are the next steps?
My code is available at github if someone gets interested (ps.: The
code
has failures. It was just a Prof of Concept)

https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Lets keep the discussion, because it is important.

Thanks and Regards
Rafael R. Machado


Em ter, 10 de set de 2019 17:00, Ethin Probst
<harlydavidsen@gmail.com>
escreveu:

Woops, I just realized while reading my message after I'd sent it
that
I had linked to the wrong document. The priority system is outlined
over here (https://freebsoft.org/doc/speechd/ssip.html#Top). This
document discusses SsIP -- the Speech Synthesis Interface Protocol.

On 9/10/19, Ethin Probst <harlydavidsen@gmail.com> wrote:
While conquering refreshable braille displays would be nice, I
think
we should take it in stages. (Some blind people would want this in
along with the accessibility subsystems, however I do not believe
that
would be the wisest approach.) Instead, I think we should do what
you
suggest: implement protocols for communicating with audio devices
(both input and output) and add audio hooks for the HII. (We could
even make those audio hooks just general hooks/events instead of
partitioning them for a particular purpose.)
You mentioned that not all devices 'have a keyboard'. This is most
definitely true, but not all devices have audio outputs and/or
inputs
either. Therefore, if we can't mandate it for *all* devices,
perhaps
we could only mandate it (or at least somehow infer it as a
requirement) for devices that both have an interaction mechanism
(any
form of interacting with the system directly via the HII and not,
say,
over the wire, because that would be superfluous) and have
mechanisms/devices/drivers/... for audio output at minimum. (Audio
input is not necessarily something that is required unless the
device
is controlled primarily by voice, but I think having voice controls
is
a bit beyond the UEFI specification and is not something UEFI
should
worry about. Voice control is not something you should have in the
preboot environment anyway, correct?)
So, I think the first stage we should work on should include:
- Implementing protocols for communicating with audio outputs
(don't
worry about inputs right now)
- Implementing protocols/hooks/events that can be intercepted in
the
HII by a foreground/background UEFI driver/application
- Implementing some kind of method to allow the accessibility
protocols to read what is drawn to the text when navigating around
The third item -- when implemented properly -- will prevent the
accessibility subsystem from reading everything on the screen every
time the "cursor" moves. Perhaps another good idea would be to
implement some kind of priority system for text-to-speech, like
speech
dispatcher does
(
https://docs.microsoft.com/en-us/cortana/skills/speech-synthesis-markup-language
).

On 9/10/19, Leif Lindholm <leif.lindholm@linaro.org> wrote:
Hi Ethin,

Apologies for the delay in responding - I personally don't tend to
read the RFC list (and I know that applies to some other
developers
who have now subscribed).

On Sun, Aug 11, 2019 at 06:15:18PM -0700, Ethin Probst wrote:
Hello all,

I'm new here, and was recommended to the TianoCore project by
someone over at the UEFI forum. I've run across TianoCore before,
and like the project.
Before anyone gets worried by the subject line, no, this is not
any
kind of legal thing. Its just something I believe needs to
happen. :)
Back in 2016-2017 I contacted the UEFI forum about two problems,
one
of which was the format of the spec, which I figured out on my
own. The other problem was not so easily dealt with. Te other
problem relates to accessibility of UEFI-compliant systems and
platform firmware to persons with disabilities. As it currently
stands, such a thing is nonexistent. To be fair, I completely
understand the difficulty that such a thing would require, and I
would fully agree if we still used the PC-AT BIOS systems -- yes,
indeed, I would never suggest this kind of thing on such a system
given that there was no actual standard of any kind for
BIOSes. However, now that UEFI is here, we have such a
possibility.
As it currently stands, people with low vision or blind people
have
access to their computers in the general sense (I am blind
myself). We can do pretty much anything anyone else could do. We
can
code, play games, all that. There are few things that we cannot
do. One of those things is managing our systems firmware in the
preboot environment.
As it stands now, I can only boot other OSes or disks via
memorization. While that worked on BIOS machines (I have, or had,
an
old Toshiba laptop that was BIOS-based), it no longer works
because
UEFI is mercurial. When I access the boot menu now, I play a game
of
chance. If the cards are in my favor, the OS I want to boot
boots,
and I can go on my way. But if the cards aren't in my favor, I
end
up making something happen that was unintended, and, worst of
all,
I
have no idea what I did.
However, the boot menu is only one portion of a platform firmware
UI. What about the setup utility or other utilities offered by
computer manufacturers? What about diagnostic utilities,
bootloaders, etc? What do I do with those? Well, I only have one
option -- sited assistance. If I go into my computers setup
utility,
I cannot trust myself and say to myself, "OK, I know what I'm
doing. All I need to do is change x and save and quit." No, I
can't
do that, because memorizing such a complex interface is extremely
difficult, and its something I wouldn't expect anyone to do.
My proposal is simple, and I'm posting it here because I'd like
comments and feedback before it actually gets implemented (it
will
take a lot of time, I'm sure): mandate, in the UEFI
specification,
that accessibility features for persons with disabilities must be
implemented and documented, and, if such features are not
implemented, then that vendor is not compliant with the
specification. Place strict minimum requirements for what the
accessibility features should do and how they should work.
So, first of all, I need to point out that what you suggest would
be
very difficult for the UEFI Forum to mandate. *But* I don't think
this
needs to be a problem.

UEFI primarily exists as a means through which interoperability
can
be
guaranteed. This means it does not tend to so much mandate what is
actually implemented in a platform, as it defines how it behaves
if
it
is. This is not 100% true, but suffice to say that mandating the
functionality you are asking for *without* having a reference
implementation available would likely simply be ignored.

If you want to turn thumbscrews, get Microsoft to add the
requirement
to their windows logo requirements :)

However, as much as it is difficult to force people to develop
this
support, if a functional reference implementation was published
under
a permissive open source license, it would take a pretty
spectacularly
incompetent product owner not to incorporate it in their products.

Now, I'm sure someone out there will ask me how this can be
done. Well, that's why I've joined the group -- though as I
familiarize myself with EDK2 development and all that I may
actually
be able to participate as more than just an accessibility expert,
of
sorts.
I have added Rafael Machado on cc. He did a PhD on firmware
accessibility, including a prototype implementing an audio driver
for
EDK2. The resulting thesis can be found at
https://arxiv.org/abs/1712.03186 .

As a side note, I have been blind all my life. I was born with
retinopathy of prematurity (ROP), which resulted because I was
born
at 26 weeks. My retina was detached, and, though the doctors
attempted to fix it, it would not remain attached, and there is
no
chance of it getting fixed now. I would neither want nor care for
such a cure, however. I have lived my entire life blind, and
while
the thought of gaining site back is appealing, I am unwilling to
go
through the years and years of rewiring and reconditioning of my
brain that would be required for me to survive with site. To me,
it
is simply not worth the cost.
But back to the discussion at hand: I would be happy to discuss
how
the accessibility features would work and what would be
required. Even standardizing, through the specification, a key
combination to toggle the accessibility features would be nice,
as
that would alleviate the major problem of a blind person buying a
new computer and not knowing how to enable the accessibility
features. The overarching goal would be to make the preboot
environment (including applications run within it) accessible and
usable by blind and visually impaired people as a boot service
only.
Yes, I agree, operating systems are already doing their things.

Again, mandating such a key combination is not necessarily
something
that the UEFI specification can do. Some devices won't necessarily
have a keyboard.

Some things the UEFI specification could do is:
- Add protocols for Audio output (and possibly input) and talking
to
audio codecs.
- Add audio hooks for the Human Interaction Interface subsystem
(HII).

EDK2 could then start adding device drivers producing these
protocols,
and enable the support for some reference platforms.

Would we need any specific support for things like refreshable
braille
displays?

It would be superfluous to make this a runtime service, as all
major OSes already have accessibility features. Plus, managing
such
a thing would be impossible to do.
Agreed.

This email has gotten quite long, so I will suspend the
discussion
of functionality and how I would like it to work for a future
email
once everyone has gotten on board.
Best Regards,

Leif


Thank you for your time and consideration.



--
Signed,
Ethin D. Probst

--
Signed,
Ethin D. Probst



--
Signed,
Ethin D. Probst
--
Signed,
Ethin D. Probst


Andrew Fish <afish@...>
 

Rafael,

Please let us know what you find out. I probably don''t have the time to help implement this feature, but I happy to help work on the architecture and design for UEFI accessibility on the edk2 mailing lists, and I can also represent what ever we come up with at the UEFI Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given there is no guideline on how a User Interface (UI) works. If accessibility requires some from of hardware abstraction, like audio, then we could likely get that into the UEFI Spec. What might be possible is an EDK2 reference implementation of accessibility. Maybe we could use the reference implementation to write a UEFI white paper on design for accessibility? I there is an open source implementation, and an official design guide this would make it much easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS Loader has a UI for the Full Disk Encryption password. If you press the power button quickly 3 times at the disk encryption password prompt accessibility is enabled and Voice Over gets turned on. You then get localized voice prompts when you move between UI elements. Since this is the OS loader all the resources are stored on the disk. You quickly run into a range of challenges since, audio is hard, abstracting audio is hard (what codec does firmware have to support), Audio files are not small and firmware is size constrained, the need to localize the audio responses causes even more size issues, the boot options are usually written by an OS installer so how would firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We choose to go to the Moon!" speech sometimes we chose to do things "not because they are easy, but because they are hard; because that goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept". If we have a design that means we can break the problem up into smaller parts, and maybe we can find people that have expertise in that part to build a chunk at a time. If we could implement the prototype in OVMF that would show how it works, but run on everyones machines, so that would be really helpful for demos and design review.

Thanks,

Andrew Fish

On Sep 13, 2019, at 4:16 AM, Rafael Machado <rafaelrodrigues.machado@gmail.com> wrote:

Hi Ethin
I will contact the community you mentioned to get feedback and align expectations.

Hope to hear comments and feedback of other member of discussion (here at EDK2, that is where the work will happen).

Thanks and Regards
Rafael

Em ter, 10 de set de 2019 às 20:50, Ethin Probst <harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com>> escreveu:
I've posted this to the largest forum of blind visually impaired
people I know of to date and have gotten some excellent feedback from
them. I also provided in that post a link to this discussion. However,
I can't possibly forward all their questions onto this group (and I
doubt I can answer them all). If you wish to join the discussion over
there, its over at https://forum.audiogames.net/post/461041/#p461041.
(If we can find a way to merge the two discussions, that would make
everything easier. If we can't, I'm fine with that.)

On 9/10/19, Rafael Machado <rafaelrodrigues.machado@gmail.com <mailto:rafaelrodrigues.machado@gmail.com>> wrote:
Hi Ethin and all

As Leif mentioned, my mastering thesys was the implementation of a
reference code so a driver for audio at pre-OS would be possible,
eliminating the first barrier for the creation of a screen reader at UEFI,
that is the lack of audio drivers.

My project was based on Intel High Definition Audio, so it is not a generic
code that would fit at all scenarios, but it a first step.
The idea was to proceed this work using the Google Summer of Code
initiative,and I will try my best to make this happen in future editions of
this initiative.
Unfortunately my thesys was written in portuguese, because there was almost
nothing related to BIOS in Portuguese, so the decision was to write in
Portuguese to help the community to attract brasilian developers. As soon
as I find some slot, I'll translate to english.
But you are not alone. Just in Brazil I discovered around 30 blind
developers that area waiting for accessibility at BIOS. Also had some
contacts from US and Ireland.
So it is a market to be explored. Everyone will win.
The companies, because they will attract more people and their families,
and the ones who need accessibility.
By the way, most of the population will need accessible solutions if they
live enough. This includes the uefi community members, that today do not
have special needs, but that may require it in future.
Accessibility is not a wast of money.
This ia what I would like to show to the UEFI community.
Hope to have the opportunity to present this in some future event.

Since this discussion was started, who knows what are the next steps?
My code is available at github if someone gets interested (ps.: The code
has failures. It was just a Prof of Concept)

https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Lets keep the discussion, because it is important.

Thanks and Regards
Rafael R. Machado


Em ter, 10 de set de 2019 17:00, Ethin Probst <harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com>>
escreveu:

Woops, I just realized while reading my message after I'd sent it that
I had linked to the wrong document. The priority system is outlined
over here (https://freebsoft.org/doc/speechd/ssip.html#Top). This
document discusses SsIP -- the Speech Synthesis Interface Protocol.

On 9/10/19, Ethin Probst <harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com>> wrote:
While conquering refreshable braille displays would be nice, I think
we should take it in stages. (Some blind people would want this in
along with the accessibility subsystems, however I do not believe that
would be the wisest approach.) Instead, I think we should do what you
suggest: implement protocols for communicating with audio devices
(both input and output) and add audio hooks for the HII. (We could
even make those audio hooks just general hooks/events instead of
partitioning them for a particular purpose.)
You mentioned that not all devices 'have a keyboard'. This is most
definitely true, but not all devices have audio outputs and/or inputs
either. Therefore, if we can't mandate it for *all* devices, perhaps
we could only mandate it (or at least somehow infer it as a
requirement) for devices that both have an interaction mechanism (any
form of interacting with the system directly via the HII and not, say,
over the wire, because that would be superfluous) and have
mechanisms/devices/drivers/... for audio output at minimum. (Audio
input is not necessarily something that is required unless the device
is controlled primarily by voice, but I think having voice controls is
a bit beyond the UEFI specification and is not something UEFI should
worry about. Voice control is not something you should have in the
preboot environment anyway, correct?)
So, I think the first stage we should work on should include:
- Implementing protocols for communicating with audio outputs (don't
worry about inputs right now)
- Implementing protocols/hooks/events that can be intercepted in the
HII by a foreground/background UEFI driver/application
- Implementing some kind of method to allow the accessibility
protocols to read what is drawn to the text when navigating around
The third item -- when implemented properly -- will prevent the
accessibility subsystem from reading everything on the screen every
time the "cursor" moves. Perhaps another good idea would be to
implement some kind of priority system for text-to-speech, like speech
dispatcher does
(
https://docs.microsoft.com/en-us/cortana/skills/speech-synthesis-markup-language
).

On 9/10/19, Leif Lindholm <leif.lindholm@linaro.org <mailto:leif.lindholm@linaro.org>> wrote:
Hi Ethin,

Apologies for the delay in responding - I personally don't tend to
read the RFC list (and I know that applies to some other developers
who have now subscribed).

On Sun, Aug 11, 2019 at 06:15:18PM -0700, Ethin Probst wrote:
Hello all,

I'm new here, and was recommended to the TianoCore project by
someone over at the UEFI forum. I've run across TianoCore before,
and like the project.
Before anyone gets worried by the subject line, no, this is not any
kind of legal thing. Its just something I believe needs to
happen. :)
Back in 2016-2017 I contacted the UEFI forum about two problems, one
of which was the format of the spec, which I figured out on my
own. The other problem was not so easily dealt with. Te other
problem relates to accessibility of UEFI-compliant systems and
platform firmware to persons with disabilities. As it currently
stands, such a thing is nonexistent. To be fair, I completely
understand the difficulty that such a thing would require, and I
would fully agree if we still used the PC-AT BIOS systems -- yes,
indeed, I would never suggest this kind of thing on such a system
given that there was no actual standard of any kind for
BIOSes. However, now that UEFI is here, we have such a possibility.
As it currently stands, people with low vision or blind people have
access to their computers in the general sense (I am blind
myself). We can do pretty much anything anyone else could do. We can
code, play games, all that. There are few things that we cannot
do. One of those things is managing our systems firmware in the
preboot environment.
As it stands now, I can only boot other OSes or disks via
memorization. While that worked on BIOS machines (I have, or had, an
old Toshiba laptop that was BIOS-based), it no longer works because
UEFI is mercurial. When I access the boot menu now, I play a game of
chance. If the cards are in my favor, the OS I want to boot boots,
and I can go on my way. But if the cards aren't in my favor, I end
up making something happen that was unintended, and, worst of all, I
have no idea what I did.
However, the boot menu is only one portion of a platform firmware
UI. What about the setup utility or other utilities offered by
computer manufacturers? What about diagnostic utilities,
bootloaders, etc? What do I do with those? Well, I only have one
option -- sited assistance. If I go into my computers setup utility,
I cannot trust myself and say to myself, "OK, I know what I'm
doing. All I need to do is change x and save and quit." No, I can't
do that, because memorizing such a complex interface is extremely
difficult, and its something I wouldn't expect anyone to do.
My proposal is simple, and I'm posting it here because I'd like
comments and feedback before it actually gets implemented (it will
take a lot of time, I'm sure): mandate, in the UEFI specification,
that accessibility features for persons with disabilities must be
implemented and documented, and, if such features are not
implemented, then that vendor is not compliant with the
specification. Place strict minimum requirements for what the
accessibility features should do and how they should work.
So, first of all, I need to point out that what you suggest would be
very difficult for the UEFI Forum to mandate. *But* I don't think this
needs to be a problem.

UEFI primarily exists as a means through which interoperability can be
guaranteed. This means it does not tend to so much mandate what is
actually implemented in a platform, as it defines how it behaves if it
is. This is not 100% true, but suffice to say that mandating the
functionality you are asking for *without* having a reference
implementation available would likely simply be ignored.

If you want to turn thumbscrews, get Microsoft to add the requirement
to their windows logo requirements :)

However, as much as it is difficult to force people to develop this
support, if a functional reference implementation was published under
a permissive open source license, it would take a pretty spectacularly
incompetent product owner not to incorporate it in their products.

Now, I'm sure someone out there will ask me how this can be
done. Well, that's why I've joined the group -- though as I
familiarize myself with EDK2 development and all that I may actually
be able to participate as more than just an accessibility expert, of
sorts.
I have added Rafael Machado on cc. He did a PhD on firmware
accessibility, including a prototype implementing an audio driver for
EDK2. The resulting thesis can be found at
https://arxiv.org/abs/1712.03186 .

As a side note, I have been blind all my life. I was born with
retinopathy of prematurity (ROP), which resulted because I was born
at 26 weeks. My retina was detached, and, though the doctors
attempted to fix it, it would not remain attached, and there is no
chance of it getting fixed now. I would neither want nor care for
such a cure, however. I have lived my entire life blind, and while
the thought of gaining site back is appealing, I am unwilling to go
through the years and years of rewiring and reconditioning of my
brain that would be required for me to survive with site. To me, it
is simply not worth the cost.
But back to the discussion at hand: I would be happy to discuss how
the accessibility features would work and what would be
required. Even standardizing, through the specification, a key
combination to toggle the accessibility features would be nice, as
that would alleviate the major problem of a blind person buying a
new computer and not knowing how to enable the accessibility
features. The overarching goal would be to make the preboot
environment (including applications run within it) accessible and
usable by blind and visually impaired people as a boot service
only.
Yes, I agree, operating systems are already doing their things.

Again, mandating such a key combination is not necessarily something
that the UEFI specification can do. Some devices won't necessarily
have a keyboard.

Some things the UEFI specification could do is:
- Add protocols for Audio output (and possibly input) and talking to
audio codecs.
- Add audio hooks for the Human Interaction Interface subsystem (HII).

EDK2 could then start adding device drivers producing these protocols,
and enable the support for some reference platforms.

Would we need any specific support for things like refreshable braille
displays?

It would be superfluous to make this a runtime service, as all
major OSes already have accessibility features. Plus, managing such
a thing would be impossible to do.
Agreed.

This email has gotten quite long, so I will suspend the discussion
of functionality and how I would like it to work for a future email
once everyone has gotten on board.
Best Regards,

Leif


Thank you for your time and consideration.



--
Signed,
Ethin D. Probst

--
Signed,
Ethin D. Probst



--
Signed,
Ethin D. Probst


Laszlo Ersek
 

On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo


Rafael Machado <rafaelrodrigues.machado@...>
 

Hi Everyone
Sorry for the delay on the response, to many things happening at the same
time.
I will try to answer e-mails to this thread every Saturday or Sunday
morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time to
help implement this feature, but I happy to help work on the architecture
and design for UEFI accessibility on the edk2 mailing lists, and I >>can
also represent what ever we come up with at the UEFI Spec Work Group.
During my MSc I had to study a lot the audio and BIOS architectures. The
idea was to eliminate the first barrier to the creation of a screen reader
for pre-OS environment, that was the lack of some open implementation of
audio control and actions at UEFI. To do that I studied the Intel High
Definition Audio Spec and a lot of UEFI specs to understand better how to
do that.
The initial target was to do all this development at OVMF, but as far as I
could get, the sound card is not available to OVMF. as Laszlo mentioned at
this e-mail there are some projects that may help on this, but at the time
I was working on my MSc I didn't find this, so I did everything on a real
system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and
working on a MSc degree at the same time having a 40hours/week job, being a
father and a husband is not an easy task, but it got to what I was
expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that
later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA
memory access. For the ones that have good years, it's possible to listen
some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was that
one of my write operations was causing some misunderstanding between the
audio controller and the audio codec. The controller was sending packets
with 16bit depth, but the codec was processing them as 8bit depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing at
the OS level.
The reference code, that is just a proof-of-concept, and that has several
things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to UEFI
drivers after some discussion. Everything is released as BDS so companies
can use without IP problems.
Just to give some more information about the need of this kind of solution.
There is a lot of blind people that work with hardware support, so
formatting disk, configuring RAID and booting dual-boot systems is always a
challenge to them. Even set the BIOS clock. How to do that without the
system's feedback?

It would be hard to have a UEFI mandate for accessibility, given there is
no guideline on how a User Interface (UI) works. If accessibility requires
some from of hardware abstraction, like audio, then we could >>likely get
that into the UEFI Spec. What might be possible is an EDK2 reference
implementation of accessibility. Maybe we could use the reference
implementation to write a UEFI white paper on design >>for accessibility? I
there is an open source implementation, and an official design guide this
would make it much easier for advocacy groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2 papers
is one of my targets since the beginning of my MSc almost 5 years ago.

I've got some experience with accessibility as the macOS EFI OS Loader
has a UI for the Full Disk Encryption password. If you press the power
button quickly 3 times at the disk encryption password prompt
accessibility is enabled and Voice Over gets turned on. You then get
localized voice prompts when you move between UI elements. Since this is
the OS loader all the resources are stored on the disk. You >>quickly run
into a range of challenges since, audio is hard, abstracting audio is hard
(what codec does firmware have to support), Audio files are not small and
firmware is size constrained, the need to localize >>the audio responses
causes even more size issues, the boot options are usually written by an OS
installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so no
audio files would need to be stored. There are some open-source
implementations that are not GPL.
This was described at my MSc as future works that can continue what I have
started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things "not
because they are easy, but because they are hard; because that >>goal will
serve to organize and measure the best of our energies and skills, because
that challenge is one that we are willing to accept". If we have a design
that means we can break the problem up into >>smaller parts, and maybe we
can find people that have expertise in that part to build a chunk at a
time. If we could implement the prototype in OVMF that would show how it
works, but run on everyone's >>machines, so that would be really helpful
for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible future
steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with
proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver
to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also
producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application
remains functional.
- 4) Separate controller and codec code by creating
an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate out
the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder audio
files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the need of
audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my thesis
to English.
I wrote everything in Portuguese because there was not material about UEFI
to the Brazilian audience, and another target I have is to show companies
that we have people that can work at this kind of projects in Brazil,
bringing this kind of development to south america. (Yes, I have
complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@redhat.com>
escreveu:

On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo


Rafael Machado <rafaelrodrigues.machado@...>
 

Hi everyone.
So, based on everything was mentioned here.

The idea is to propose the creation of two protocols:

- EFI_AUDIO_OUTPUT_PROTOCOL: This protocol should be the responsible for
initializing the audio controller, that in the case of my MSc work does
initialization of the RING buffers, that are used as containers to the
audio streams that need to be processed.

- EFI_AUDIO_CODEC_PROTOCOL: This protocol should be responsible for
initializing the codec (each codec may need different init commands), and
also is used to control things like mute, volume level and this kind of
things. (can see the volume control actions at the last video I mentioned
on the previous e-mail)

Does this approach works at non-x86 platforms? (I don't have knowledge in
ARM platforms, so feedback from the community will be well received.)

Hope to hear some voices :)

Thanks and Regards
Rafael

Em sáb, 21 de set de 2019 às 09:36, Rafael Machado <
rafaelrodrigues.machado@gmail.com> escreveu:

Hi Everyone
Sorry for the delay on the response, to many things happening at the same
time.
I will try to answer e-mails to this thread every Saturday or Sunday
morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time to
help implement this feature, but I happy to help work on the architecture
and design for UEFI accessibility on the edk2 mailing lists, and I >>can
also represent what ever we come up with at the UEFI Spec Work Group.
During my MSc I had to study a lot the audio and BIOS architectures. The
idea was to eliminate the first barrier to the creation of a screen reader
for pre-OS environment, that was the lack of some open implementation of
audio control and actions at UEFI. To do that I studied the Intel High
Definition Audio Spec and a lot of UEFI specs to understand better how to
do that.
The initial target was to do all this development at OVMF, but as far as I
could get, the sound card is not available to OVMF. as Laszlo mentioned at
this e-mail there are some projects that may help on this, but at the time
I was working on my MSc I didn't find this, so I did everything on a real
system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and
working on a MSc degree at the same time having a 40hours/week job, being a
father and a husband is not an easy task, but it got to what I was
expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that
later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA
memory access. For the ones that have good years, it's possible to listen
some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was that
one of my write operations was causing some misunderstanding between the
audio controller and the audio codec. The controller was sending packets
with 16bit depth, but the codec was processing them as 8bit depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing at
the OS level.
The reference code, that is just a proof-of-concept, and that has several
things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to UEFI
drivers after some discussion. Everything is released as BDS so companies
can use without IP problems.
Just to give some more information about the need of this kind of
solution. There is a lot of blind people that work with hardware support,
so formatting disk, configuring RAID and booting dual-boot systems is
always a challenge to them. Even set the BIOS clock. How to do that without
the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given there
is no guideline on how a User Interface (UI) works. If accessibility
requires some from of hardware abstraction, like audio, then we could
likely get that into the UEFI Spec. What might be possible is an EDK2
reference implementation of accessibility. Maybe we could use the reference
implementation to write a UEFI white paper on design >>for accessibility? I
there is an open source implementation, and an official design guide this
would make it much easier for advocacy groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2 papers
is one of my targets since the beginning of my MSc almost 5 years ago.

I've got some experience with accessibility as the macOS EFI OS Loader
has a UI for the Full Disk Encryption password. If you press the power
button quickly 3 times at the disk encryption password prompt
accessibility is enabled and Voice Over gets turned on. You then get
localized voice prompts when you move between UI elements. Since this is
the OS loader all the resources are stored on the disk. You >>quickly run
into a range of challenges since, audio is hard, abstracting audio is hard
(what codec does firmware have to support), Audio files are not small and
firmware is size constrained, the need to localize >>the audio responses
causes even more size issues, the boot options are usually written by an OS
installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so no
audio files would need to be stored. There are some open-source
implementations that are not GPL.
This was described at my MSc as future works that can continue what I have
started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things "not
because they are easy, but because they are hard; because that >>goal will
serve to organize and measure the best of our energies and skills, because
that challenge is one that we are willing to accept". If we have a design
that means we can break the problem up into >>smaller parts, and maybe we
can find people that have expertise in that part to build a chunk at a
time. If we could implement the prototype in OVMF that would show how it
works, but run on everyone's >>machines, so that would be really helpful
for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible future
steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with
proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver
to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also
producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application
remains functional.
- 4) Separate controller and codec code by creating
an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate out
the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder audio
files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the need
of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my thesis
to English.
I wrote everything in Portuguese because there was not material about UEFI
to the Brazilian audience, and another target I have is to show companies
that we have people that can work at this kind of projects in Brazil,
bringing this kind of development to south america. (Yes, I have
complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@redhat.com>
escreveu:

On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo


Ethin Probst
 

If other platforms are PCI-based (i.e. allow us to scan the PCI bus
and figure out where (in MMIO space) the HDA controller is mapped to),
then it (theoretically) sould work. I don't know for sure though; I'm
not very knowledgeable in other CPU architectures.

On 9/23/19, Rafael Machado <rafaelrodrigues.machado@gmail.com> wrote:
Hi everyone.
So, based on everything was mentioned here.

The idea is to propose the creation of two protocols:

- EFI_AUDIO_OUTPUT_PROTOCOL: This protocol should be the responsible for
initializing the audio controller, that in the case of my MSc work does
initialization of the RING buffers, that are used as containers to the
audio streams that need to be processed.

- EFI_AUDIO_CODEC_PROTOCOL: This protocol should be responsible for
initializing the codec (each codec may need different init commands), and
also is used to control things like mute, volume level and this kind of
things. (can see the volume control actions at the last video I mentioned
on the previous e-mail)

Does this approach works at non-x86 platforms? (I don't have knowledge in
ARM platforms, so feedback from the community will be well received.)

Hope to hear some voices :)

Thanks and Regards
Rafael

Em sáb, 21 de set de 2019 às 09:36, Rafael Machado <
rafaelrodrigues.machado@gmail.com> escreveu:

Hi Everyone
Sorry for the delay on the response, to many things happening at the same
time.
I will try to answer e-mails to this thread every Saturday or Sunday
morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time
to
help implement this feature, but I happy to help work on the architecture
and design for UEFI accessibility on the edk2 mailing lists, and I >>can
also represent what ever we come up with at the UEFI Spec Work Group.
During my MSc I had to study a lot the audio and BIOS architectures. The
idea was to eliminate the first barrier to the creation of a screen
reader
for pre-OS environment, that was the lack of some open implementation of
audio control and actions at UEFI. To do that I studied the Intel High
Definition Audio Spec and a lot of UEFI specs to understand better how to
do that.
The initial target was to do all this development at OVMF, but as far as
I
could get, the sound card is not available to OVMF. as Laszlo mentioned
at
this e-mail there are some projects that may help on this, but at the
time
I was working on my MSc I didn't find this, so I did everything on a real
system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and
working on a MSc degree at the same time having a 40hours/week job, being
a
father and a husband is not an easy task, but it got to what I was
expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that
later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA
memory access. For the ones that have good years, it's possible to listen
some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was
that
one of my write operations was causing some misunderstanding between the
audio controller and the audio codec. The controller was sending packets
with 16bit depth, but the codec was processing them as 8bit depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing at
the OS level.
The reference code, that is just a proof-of-concept, and that has several
things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to UEFI
drivers after some discussion. Everything is released as BDS so companies
can use without IP problems.
Just to give some more information about the need of this kind of
solution. There is a lot of blind people that work with hardware support,
so formatting disk, configuring RAID and booting dual-boot systems is
always a challenge to them. Even set the BIOS clock. How to do that
without
the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given there
is no guideline on how a User Interface (UI) works. If accessibility
requires some from of hardware abstraction, like audio, then we could
likely get that into the UEFI Spec. What might be possible is an EDK2
reference implementation of accessibility. Maybe we could use the
reference
implementation to write a UEFI white paper on design >>for accessibility?
I
there is an open source implementation, and an official design guide this
would make it much easier for advocacy groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2 papers
is one of my targets since the beginning of my MSc almost 5 years ago.

I've got some experience with accessibility as the macOS EFI OS Loader
has a UI for the Full Disk Encryption password. If you press the power
button quickly 3 times at the disk encryption password prompt
accessibility is enabled and Voice Over gets turned on. You then get
localized voice prompts when you move between UI elements. Since this is
the OS loader all the resources are stored on the disk. You >>quickly run
into a range of challenges since, audio is hard, abstracting audio is
hard
(what codec does firmware have to support), Audio files are not small and
firmware is size constrained, the need to localize >>the audio responses
causes even more size issues, the boot options are usually written by an
OS
installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so
no
audio files would need to be stored. There are some open-source
implementations that are not GPL.
This was described at my MSc as future works that can continue what I
have
started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things "not
because they are easy, but because they are hard; because that >>goal
will
serve to organize and measure the best of our energies and skills,
because
that challenge is one that we are willing to accept". If we have a design
that means we can break the problem up into >>smaller parts, and maybe we
can find people that have expertise in that part to build a chunk at a
time. If we could implement the prototype in OVMF that would show how it
works, but run on everyone's >>machines, so that would be really helpful
for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible future
steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with
proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver
to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also
producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application
remains functional.
- 4) Separate controller and codec code by creating
an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate
out
the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder
audio
files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the need
of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my
thesis
to English.
I wrote everything in Portuguese because there was not material about
UEFI
to the Brazilian audience, and another target I have is to show companies
that we have people that can work at this kind of projects in Brazil,
bringing this kind of development to south america. (Yes, I have
complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@redhat.com>
escreveu:

On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo
--
Signed,
Ethin D. Probst


Andrew Fish <afish@...>
 

Rafael,

Did you do much research into CODECs? Like which one(s) should be supported? I assume th CODEC implies the audio file formats that can be decoded? Also how large are the audio files?

Is the CODEC protocol more of a plug-in for the Intel HDA? By that I mean it only works on that hardware, or does it work generically on top of any EFI_AUDIO_OUTPUT_PROTOCOL?

I was starting to think about how to store to audio and deal with dynamic configuration. I guess one crazy idea could be to have the OS create the audio files using text to speech, and maybe store them on the EFI System Partition. For example when an OS installs it is going to write the EFI Boot Variable that contains a Description of the boot option. Maybe we could convert that to audio and add a nvram variable that points to the given audio file?

Thanks,

Andrew Fish

On Sep 23, 2019, at 6:20 AM, Rafael Machado <rafaelrodrigues.machado@gmail.com> wrote:

Hi everyone.
So, based on everything was mentioned here.

The idea is to propose the creation of two protocols:

- EFI_AUDIO_OUTPUT_PROTOCOL: This protocol should be the responsible for initializing the audio controller, that in the case of my MSc work does initialization of the RING buffers, that are used as containers to the audio streams that need to be processed.

- EFI_AUDIO_CODEC_PROTOCOL: This protocol should be responsible for initializing the codec (each codec may need different init commands), and also is used to control things like mute, volume level and this kind of things. (can see the volume control actions at the last video I mentioned on the previous e-mail)

Does this approach works at non-x86 platforms? (I don't have knowledge in ARM platforms, so feedback from the community will be well received.)

Hope to hear some voices :)

Thanks and Regards
Rafael

Em sáb, 21 de set de 2019 às 09:36, Rafael Machado <rafaelrodrigues.machado@gmail.com <mailto:rafaelrodrigues.machado@gmail.com>> escreveu:
Hi Everyone
Sorry for the delay on the response, to many things happening at the same time.
I will try to answer e-mails to this thread every Saturday or Sunday morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time to help implement this feature, but I happy to help work on the architecture and design for UEFI accessibility on the edk2 mailing lists, and I >>can also represent what ever we come up with at the UEFI Spec Work Group.
During my MSc I had to study a lot the audio and BIOS architectures. The idea was to eliminate the first barrier to the creation of a screen reader for pre-OS environment, that was the lack of some open implementation of audio control and actions at UEFI. To do that I studied the Intel High Definition Audio Spec and a lot of UEFI specs to understand better how to do that.
The initial target was to do all this development at OVMF, but as far as I could get, the sound card is not available to OVMF. as Laszlo mentioned at this e-mail there are some projects that may help on this, but at the time I was working on my MSc I didn't find this, so I did everything on a real system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and working on a MSc degree at the same time having a 40hours/week job, being a father and a husband is not an easy task, but it got to what I was expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA memory access. For the ones that have good years, it's possible to listen some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was that one of my write operations was causing some misunderstanding between the audio controller and the audio codec. The controller was sending packets with 16bit depth, but the codec was processing them as 8bit depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing at the OS level.
The reference code, that is just a proof-of-concept, and that has several things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to UEFI drivers after some discussion. Everything is released as BDS so companies can use without IP problems.
Just to give some more information about the need of this kind of solution. There is a lot of blind people that work with hardware support, so formatting disk, configuring RAID and booting dual-boot systems is always a challenge to them. Even set the BIOS clock. How to do that without the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given there is no guideline on how a User Interface (UI) works. If accessibility requires some from of hardware abstraction, like audio, then we could >>likely get that into the UEFI Spec. What might be possible is an EDK2 reference implementation of accessibility. Maybe we could use the reference implementation to write a UEFI white paper on design >>for accessibility? I there is an open source implementation, and an official design guide this would make it much easier for advocacy groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2 papers is one of my targets since the beginning of my MSc almost 5 years ago.

I've got some experience with accessibility as the macOS EFI OS Loader has a UI for the Full Disk Encryption password. If you press the power button quickly 3 times at the disk encryption password prompt >>accessibility is enabled and Voice Over gets turned on. You then get localized voice prompts when you move between UI elements. Since this is the OS loader all the resources are stored on the disk. You >>quickly run into a range of challenges since, audio is hard, abstracting audio is hard (what codec does firmware have to support), Audio files are not small and firmware is size constrained, the need to localize >>the audio responses causes even more size issues, the boot options are usually written by an OS installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so no audio files would need to be stored. There are some open-source implementations that are not GPL.
This was described at my MSc as future works that can continue what I have started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We choose to go to the Moon!" speech sometimes we chose to do things "not because they are easy, but because they are hard; because that >>goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept". If we have a design that means we can break the problem up into >>smaller parts, and maybe we can find people that have expertise in that part to build a chunk at a time. If we could implement the prototype in OVMF that would show how it works, but run on everyone's >>machines, so that would be really helpful for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible future steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application remains functional.
- 4) Separate controller and codec code by creating an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate out the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder audio files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the need of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my thesis to English.
I wrote everything in Portuguese because there was not material about UEFI to the Brazilian audience, and another target I have is to show companies that we have people that can work at this kind of projects in Brazil, bringing this kind of development to south america. (Yes, I have complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@redhat.com <mailto:lersek@redhat.com>> escreveu:
On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo


Rafael Machado <rafaelrodrigues.machado@...>
 

Hi Everyone

Answering Ethin:
If other platforms are PCI-based (i.e. allow us to scan the PCI bus
and figure out where (in MMIO space) the HDA controller is mapped to),
then it (theoretically) sould work. I don't know for sure though; I'm
not very knowledgeable in other CPU architectures
Answer: Neither do I. Lets wait some ARM expert to give some opinion :)

Answering Andrew
Did you do much research into CODECs? Like which one(s) should be
supported? I assume the CODEC implies the audio file formats that can be
decoded? Also how large are the audio files?
Answer: During my research I have studied some codec specs, and the way the
codecs works is really similar. Normally they need to receive some packets
to initialize the nodes (node is the name of each internal component of the
codec), by receiving some commands (named verbs) that are defined at the
HDASpec. For example the verb "power state" (0xF05) is used to set the
power state of each node on the codec, to really enable the chip or put the
chip to sleep.
The verbs can be found at this document
https://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/high-definition-audio-specification.pdf,
at page 216.
Not all verbs are standardized, for example the verb "Amplifier Gain Mute"
(0xB--), each vendor can decide the command. In the case of the ASUS system
I have used, the codec is a Conexant CX20752 (
https://www.datasheets360.com/pdf/7550682360275196829), and at the
datasheet we can see that the "Amplifier Gain Mute" verb is 0xB00 and 0xB20
(right and left channel in the case of a stereo stream) at page 33. These
verbs that are vendor defined creates the problems to think on a generic
solution.
About the second part of the question, the Codecs are independent, so they
are only responsible to process signals that are passed to them in a stream
using a DMA buffer, or a command to change their behavior (increase /
decrease volume for example). So for the codec to process correctly the
buffers managed by the audio controller, both need to be configured with
the same stream format, something like 2 channel 8 bit depth for example.
At the second video I send previously (
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be) the problem
was that the controller was configured to manage a stream with 2 channel
and 8 bits depth, but the codec was processing the data as 2 channel 16 bit
depth.
About the audio files size, at my work I didn't use any compressed format,
because I was running out of time to finish my project, so I was processing
raw audio data, that is much bigger than other formats like MP3. In case we
decide to use audio files at this first stage, maybe we will need to port
some audio format lib to UEFI also.

Is the CODEC protocol more of a plug-in for the Intel HDA? By that I mean
it only works on that hardware, or does it work generically on top of
any EFI_AUDIO_OUTPUT_PROTOCOL?
Answer: My understanding is that the CODEC protocol will need the audio
controller to be already configured, because the communication with the
codec is done using the addresses that are related to the AUDIO controller.
So the access to the AUDIO Codec is managed by the Audio controller. We
need to write the commands at the address that was previously allocated and
set as the CORB/RIRB (Command Output Ring Buffer / Response Input Ring
Buffer), and set at the HDA controller registers (page 36 of the HDA Spec)

I was starting to think about how to store to audio and deal with dynamic
configuration. I guess one crazy idea could be to have the OS create the
audio files using text to speech, and maybe store them on the EFI System
Partition. For example when an OS installs it is going to write the EFI
Boot Variable that contains a Description of the boot option. Maybe we
could convert that to audio and add a nvram variable that points to the
given audio file?
Answer: This is one option. One question. How would the OS knows the bios
options and menus so it could create these files?

Thanks and Regards
Rafael


Em seg, 23 de set de 2019 às 12:11, Andrew Fish <afish@apple.com> escreveu:

Rafael,

Did you do much research into CODECs? Like which one(s) should be
supported? I assume th CODEC implies the audio file formats that can be
decoded? Also how large are the audio files?

Is the CODEC protocol more of a plug-in for the Intel HDA? By that I mean
it only works on that hardware, or does it work generically on top of
any EFI_AUDIO_OUTPUT_PROTOCOL?

I was starting to think about how to store to audio and deal with dynamic
configuration. I guess one crazy idea could be to have the OS create the
audio files using text to speech, and maybe store them on the EFI System
Partition. For example when an OS installs it is going to write the EFI
Boot Variable that contains a Description of the boot option. Maybe we
could convert that to audio and add a nvram variable that points to the
given audio file?

Thanks,

Andrew Fish

On Sep 23, 2019, at 6:20 AM, Rafael Machado <
rafaelrodrigues.machado@gmail.com> wrote:

Hi everyone.
So, based on everything was mentioned here.

The idea is to propose the creation of two protocols:

- EFI_AUDIO_OUTPUT_PROTOCOL: This protocol should be the responsible for
initializing the audio controller, that in the case of my MSc work does
initialization of the RING buffers, that are used as containers to the
audio streams that need to be processed.

- EFI_AUDIO_CODEC_PROTOCOL: This protocol should be responsible for
initializing the codec (each codec may need different init commands), and
also is used to control things like mute, volume level and this kind of
things. (can see the volume control actions at the last video I mentioned
on the previous e-mail)

Does this approach works at non-x86 platforms? (I don't have knowledge in
ARM platforms, so feedback from the community will be well received.)

Hope to hear some voices :)

Thanks and Regards
Rafael

Em sáb, 21 de set de 2019 às 09:36, Rafael Machado <
rafaelrodrigues.machado@gmail.com> escreveu:

Hi Everyone
Sorry for the delay on the response, to many things happening at the same
time.
I will try to answer e-mails to this thread every Saturday or Sunday
morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing lists,
and I >>can also represent what ever we come up with at the UEFI Spec Work
Group.
During my MSc I had to study a lot the audio and BIOS architectures. The
idea was to eliminate the first barrier to the creation of a screen reader
for pre-OS environment, that was the lack of some open implementation of
audio control and actions at UEFI. To do that I studied the Intel High
Definition Audio Spec and a lot of UEFI specs to understand better how to
do that.
The initial target was to do all this development at OVMF, but as far as
I could get, the sound card is not available to OVMF. as Laszlo mentioned
at this e-mail there are some projects that may help on this, but at the
time I was working on my MSc I didn't find this, so I did everything on a
real system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and
working on a MSc degree at the same time having a 40hours/week job, being a
father and a husband is not an easy task, but it got to what I was
expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that
later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA
memory access. For the ones that have good years, it's possible to listen
some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was
that one of my write operations was causing some misunderstanding between
the audio controller and the audio codec. The controller was sending
packets with 16bit depth, but the codec was processing them as 8bit depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing at
the OS level.
The reference code, that is just a proof-of-concept, and that has several
things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to UEFI
drivers after some discussion. Everything is released as BDS so companies
can use without IP problems.
Just to give some more information about the need of this kind of
solution. There is a lot of blind people that work with hardware support,
so formatting disk, configuring RAID and booting dual-boot systems is
always a challenge to them. Even set the BIOS clock. How to do that without
the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given there
is no guideline on how a User Interface (UI) works. If accessibility
requires some from of hardware abstraction, like audio, then we could
likely get that into the UEFI Spec. What might be possible is an EDK2
reference implementation of accessibility. Maybe we could use the reference
implementation to write a UEFI white paper on design >>for accessibility? I
there is an open source implementation, and an official design guide this
would make it much easier for advocacy groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2 papers
is one of my targets since the beginning of my MSc almost 5 years ago.

I've got some experience with accessibility as the macOS EFI OS Loader
has a UI for the Full Disk Encryption password. If you press the power
button quickly 3 times at the disk encryption password prompt
accessibility is enabled and Voice Over gets turned on. You then get
localized voice prompts when you move between UI elements. Since this is
the OS loader all the resources are stored on the disk. You >>quickly run
into a range of challenges since, audio is hard, abstracting audio is hard
(what codec does firmware have to support), Audio files are not small and
firmware is size constrained, the need to localize >>the audio responses
causes even more size issues, the boot options are usually written by an OS
installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so
no audio files would need to be stored. There are some open-source
implementations that are not GPL.
This was described at my MSc as future works that can continue what I
have started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things "not
because they are easy, but because they are hard; because that >>goal will
serve to organize and measure the best of our energies and skills, because
that challenge is one that we are willing to accept". If we have a design
that means we can break the problem up into >>smaller parts, and maybe we
can find people that have expertise in that part to build a chunk at a
time. If we could implement the prototype in OVMF that would show how it
works, but run on everyone's >>machines, so that would be really helpful
for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible future
steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with
proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver
to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also
producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application
remains functional.
- 4) Separate controller and codec code by creating
an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate out
the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder
audio files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the need
of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my
thesis to English.
I wrote everything in Portuguese because there was not material about
UEFI to the Brazilian audience, and another target I have is to show
companies that we have people that can work at this kind of projects in
Brazil, bringing this kind of development to south america. (Yes, I have
complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@redhat.com>
escreveu:

On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo


Andrew Fish <afish@...>
 

On Sep 24, 2019, at 5:06 AM, Rafael Machado <rafaelrodrigues.machado@gmail.com> wrote:

Hi Everyone

Answering Ethin:
If other platforms are PCI-based (i.e. allow us to scan the PCI bus
and figure out where (in MMIO space) the HDA controller is mapped to),
then it (theoretically) sould work. I don't know for sure though; I'm
not very knowledgeable in other CPU architectures
Answer: Neither do I. Lets wait some ARM expert to give some opinion :)
Rafael,

I'm not exactly the ARM expert, but what we have seen is as long as en EFI PCI driver follows the EFI rules for DMA it should work just fine on all CPU architectures. The historical problem is if you don't follow the rules on x86 DMA still works since the memory coherency is maintained by hardware on x86. Since the coherency requires software on ARM, you have to do it right.

Answering Andrew
Did you do much research into CODECs? Like which one(s) should be supported? I assume the CODEC implies the audio file formats that can be decoded? Also how large are the audio files?
Answer: During my research I have studied some codec specs, and the way the codecs works is really similar. Normally they need to receive some packets to initialize the nodes (node is the name of each internal component of the codec), by receiving some commands (named verbs) that are defined at the HDASpec. For example the verb "power state" (0xF05) is used to set the power state of each node on the codec, to really enable the chip or put the chip to sleep.
The verbs can be found at this document https://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/high-definition-audio-specification.pdf, at page 216.
Not all verbs are standardized, for example the verb "Amplifier Gain Mute" (0xB--), each vendor can decide the command. In the case of the ASUS system I have used, the codec is a Conexant CX20752 (https://www.datasheets360.com/pdf/7550682360275196829), and at the datasheet we can see that the "Amplifier Gain Mute" verb is 0xB00 and 0xB20 (right and left channel in the case of a stereo stream) at page 33. These verbs that are vendor defined creates the problems to think on a generic solution.
About the second part of the question, the Codecs are independent, so they are only responsible to process signals that are passed to them in a stream using a DMA buffer, or a command to change their behavior (increase / decrease volume for example). So for the codec to process correctly the buffers managed by the audio controller, both need to be configured with the same stream format, something like 2 channel 8 bit depth for example. At the second video I send previously (https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be) the problem was that the controller was configured to manage a stream with 2 channel and 8 bits depth, but the codec was processing the data as 2 channel 16 bit depth.
About the audio files size, at my work I didn't use any compressed format, because I was running out of time to finish my project, so I was processing raw audio data, that is much bigger than other formats like MP3. In case we decide to use audio files at this first stage, maybe we will need to port some audio format lib to UEFI also.
Thanks I tracked down a friend of mine who knows audio so I think I grok it better now. It seems HDA is a hardware standard. Seems like most of the work in an HDA driver is doing the setup to decode the data.

As far as a file format we might be able to use PCD, or as we know it a wav file. It might also be possible to use MP3, but there have been IP issues in the past encoding MP3 and I'm not sure how that will work out. The only issue with WAV file is size, but we may be able to compress the WAV files with a decompressor that already lives in the ROM. Saving file space is probably as simple as wrapping the file with a header that describes the file type and encryption scheme.

Is the CODEC protocol more of a plug-in for the Intel HDA? By that I mean it only works on that hardware, or does it work generically on top of any EFI_AUDIO_OUTPUT_PROTOCOL?
Answer: My understanding is that the CODEC protocol will need the audio controller to be already configured, because the communication with the codec is done using the addresses that are related to the AUDIO controller. So the access to the AUDIO Codec is managed by the Audio controller. We need to write the commands at the address that was previously allocated and set as the CORB/RIRB (Command Output Ring Buffer / Response Input Ring Buffer), and set at the HDA controller registers (page 36 of the HDA Spec)

I was starting to think about how to store to audio and deal with dynamic configuration. I guess one crazy idea could be to have the OS create the audio files using text to speech, and maybe store them on the EFI System Partition. For example when an OS installs it is going to write the EFI Boot Variable that contains a Description of the boot option. Maybe we could convert that to audio and add a nvram variable that points to the given audio file?
Answer: This is one option. One question. How would the OS knows the bios options and menus so it could create these files?
The EFI Boot options are NVRAM based, so we could use NVRAM for that. For the HII (Setup) it may be possible to pass the data up to the OS and then have it encoded. We might be able to make it visible to EFI via writing to the disk on the EFI System Partition with an NVARM variable for any extra information we needed. I see to remember we talked about passing data up to the OS at one point so we could have an OS based setup utility. I don't remember the details. Mike Rothman wrote the HII part of the spec so I can hit him up for ideas when we try to figure this out.

Thanks,

Andrew Fish

Thanks and Regards
Rafael


Em seg, 23 de set de 2019 às 12:11, Andrew Fish <afish@apple.com <mailto:afish@apple.com>> escreveu:
Rafael,

Did you do much research into CODECs? Like which one(s) should be supported? I assume th CODEC implies the audio file formats that can be decoded? Also how large are the audio files?

Is the CODEC protocol more of a plug-in for the Intel HDA? By that I mean it only works on that hardware, or does it work generically on top of any EFI_AUDIO_OUTPUT_PROTOCOL?

I was starting to think about how to store to audio and deal with dynamic configuration. I guess one crazy idea could be to have the OS create the audio files using text to speech, and maybe store them on the EFI System Partition. For example when an OS installs it is going to write the EFI Boot Variable that contains a Description of the boot option. Maybe we could convert that to audio and add a nvram variable that points to the given audio file?

Thanks,

Andrew Fish

On Sep 23, 2019, at 6:20 AM, Rafael Machado <rafaelrodrigues.machado@gmail.com <mailto:rafaelrodrigues.machado@gmail.com>> wrote:

Hi everyone.
So, based on everything was mentioned here.

The idea is to propose the creation of two protocols:

- EFI_AUDIO_OUTPUT_PROTOCOL: This protocol should be the responsible for initializing the audio controller, that in the case of my MSc work does initialization of the RING buffers, that are used as containers to the audio streams that need to be processed.

- EFI_AUDIO_CODEC_PROTOCOL: This protocol should be responsible for initializing the codec (each codec may need different init commands), and also is used to control things like mute, volume level and this kind of things. (can see the volume control actions at the last video I mentioned on the previous e-mail)

Does this approach works at non-x86 platforms? (I don't have knowledge in ARM platforms, so feedback from the community will be well received.)

Hope to hear some voices :)

Thanks and Regards
Rafael

Em sáb, 21 de set de 2019 às 09:36, Rafael Machado <rafaelrodrigues.machado@gmail.com <mailto:rafaelrodrigues.machado@gmail.com>> escreveu:
Hi Everyone
Sorry for the delay on the response, to many things happening at the same time.
I will try to answer e-mails to this thread every Saturday or Sunday morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time to help implement this feature, but I happy to help work on the architecture and design for UEFI accessibility on the edk2 mailing lists, and I >>can also represent what ever we come up with at the UEFI Spec Work Group.
During my MSc I had to study a lot the audio and BIOS architectures. The idea was to eliminate the first barrier to the creation of a screen reader for pre-OS environment, that was the lack of some open implementation of audio control and actions at UEFI. To do that I studied the Intel High Definition Audio Spec and a lot of UEFI specs to understand better how to do that.
The initial target was to do all this development at OVMF, but as far as I could get, the sound card is not available to OVMF. as Laszlo mentioned at this e-mail there are some projects that may help on this, but at the time I was working on my MSc I didn't find this, so I did everything on a real system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and working on a MSc degree at the same time having a 40hours/week job, being a father and a husband is not an easy task, but it got to what I was expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA memory access. For the ones that have good years, it's possible to listen some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was that one of my write operations was causing some misunderstanding between the audio controller and the audio codec. The controller was sending packets with 16bit depth, but the codec was processing them as 8bit depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing at the OS level.
The reference code, that is just a proof-of-concept, and that has several things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to UEFI drivers after some discussion. Everything is released as BDS so companies can use without IP problems.
Just to give some more information about the need of this kind of solution. There is a lot of blind people that work with hardware support, so formatting disk, configuring RAID and booting dual-boot systems is always a challenge to them. Even set the BIOS clock. How to do that without the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given there is no guideline on how a User Interface (UI) works. If accessibility requires some from of hardware abstraction, like audio, then we could >>likely get that into the UEFI Spec. What might be possible is an EDK2 reference implementation of accessibility. Maybe we could use the reference implementation to write a UEFI white paper on design >>for accessibility? I there is an open source implementation, and an official design guide this would make it much easier for advocacy groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2 papers is one of my targets since the beginning of my MSc almost 5 years ago.

I've got some experience with accessibility as the macOS EFI OS Loader has a UI for the Full Disk Encryption password. If you press the power button quickly 3 times at the disk encryption password prompt >>accessibility is enabled and Voice Over gets turned on. You then get localized voice prompts when you move between UI elements. Since this is the OS loader all the resources are stored on the disk. You >>quickly run into a range of challenges since, audio is hard, abstracting audio is hard (what codec does firmware have to support), Audio files are not small and firmware is size constrained, the need to localize >>the audio responses causes even more size issues, the boot options are usually written by an OS installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so no audio files would need to be stored. There are some open-source implementations that are not GPL.
This was described at my MSc as future works that can continue what I have started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We choose to go to the Moon!" speech sometimes we chose to do things "not because they are easy, but because they are hard; because that >>goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept". If we have a design that means we can break the problem up into >>smaller parts, and maybe we can find people that have expertise in that part to build a chunk at a time. If we could implement the prototype in OVMF that would show how it works, but run on everyone's >>machines, so that would be really helpful for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible future steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application remains functional.
- 4) Separate controller and codec code by creating an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate out the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder audio files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the need of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my thesis to English.
I wrote everything in Portuguese because there was not material about UEFI to the Brazilian audience, and another target I have is to show companies that we have people that can work at this kind of projects in Brazil, bringing this kind of development to south america. (Yes, I have complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@redhat.com <mailto:lersek@redhat.com>> escreveu:
On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo


Ethin Probst
 

The patent for MP3 expired a year or so ago, if memory serves. (I
don't know how long ago it expired but it definitely has expired, so
you don't need to worry about IP (I don't think).) An OS setup utility
would be nice if it weren't for the fact that it would prevent you
from modifying things like secure boot. Why can't we just either (1)
embed the audio files in NVRAM or (2) embed a speech synthesizer
directly into the UEFI system (the synthesizer would not need to be
standardized, just easily embeddable) and then send PCM samples
directly to the HDA device using that synthesizer? ESpeak is one such
synthesizer; I don't know how we'd go about modifying it but given
enough time we could probably make it fit in the UEFI environment.

On 9/24/19, Andrew Fish <afish@apple.com> wrote:


On Sep 24, 2019, at 5:06 AM, Rafael Machado
<rafaelrodrigues.machado@gmail.com> wrote:

Hi Everyone

Answering Ethin:
If other platforms are PCI-based (i.e. allow us to scan the PCI bus
and figure out where (in MMIO space) the HDA controller is mapped to),
then it (theoretically) sould work. I don't know for sure though; I'm
not very knowledgeable in other CPU architectures
Answer: Neither do I. Lets wait some ARM expert to give some opinion :)
Rafael,

I'm not exactly the ARM expert, but what we have seen is as long as en EFI
PCI driver follows the EFI rules for DMA it should work just fine on all CPU
architectures. The historical problem is if you don't follow the rules on
x86 DMA still works since the memory coherency is maintained by hardware on
x86. Since the coherency requires software on ARM, you have to do it right.


Answering Andrew
Did you do much research into CODECs? Like which one(s) should be
supported? I assume the CODEC implies the audio file formats that can
be decoded? Also how large are the audio files?
Answer: During my research I have studied some codec specs, and the way
the codecs works is really similar. Normally they need to receive some
packets to initialize the nodes (node is the name of each internal
component of the codec), by receiving some commands (named verbs) that are
defined at the HDASpec. For example the verb "power state" (0xF05) is
used to set the power state of each node on the codec, to really enable
the chip or put the chip to sleep.
The verbs can be found at this document
https://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/high-definition-audio-specification.pdf
<https://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/high-definition-audio-specification.pdf>,
at page 216.
Not all verbs are standardized, for example the verb "Amplifier Gain Mute"
(0xB--), each vendor can decide the command. In the case of the ASUS
system I have used, the codec is a Conexant CX20752
(https://www.datasheets360.com/pdf/7550682360275196829
<https://www.datasheets360.com/pdf/7550682360275196829>), and at the
datasheet we can see that the "Amplifier Gain Mute" verb is 0xB00 and
0xB20 (right and left channel in the case of a stereo stream) at page 33.
These verbs that are vendor defined creates the problems to think on a
generic solution.
About the second part of the question, the Codecs are independent, so they
are only responsible to process signals that are passed to them in a
stream using a DMA buffer, or a command to change their behavior (increase
/ decrease volume for example). So for the codec to process correctly the
buffers managed by the audio controller, both need to be configured with
the same stream format, something like 2 channel 8 bit depth for example.
At the second video I send previously
(https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
<https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be>) the
problem was that the controller was configured to manage a stream with 2
channel and 8 bits depth, but the codec was processing the data as 2
channel 16 bit depth.
About the audio files size, at my work I didn't use any compressed format,
because I was running out of time to finish my project, so I was
processing raw audio data, that is much bigger than other formats like
MP3. In case we decide to use audio files at this first stage, maybe we
will need to port some audio format lib to UEFI also.
Thanks I tracked down a friend of mine who knows audio so I think I grok it
better now. It seems HDA is a hardware standard. Seems like most of the work
in an HDA driver is doing the setup to decode the data.

As far as a file format we might be able to use PCD, or as we know it a wav
file. It might also be possible to use MP3, but there have been IP issues in
the past encoding MP3 and I'm not sure how that will work out. The only
issue with WAV file is size, but we may be able to compress the WAV files
with a decompressor that already lives in the ROM. Saving file space is
probably as simple as wrapping the file with a header that describes the
file type and encryption scheme.

Is the CODEC protocol more of a plug-in for the Intel HDA? By that I
mean it only works on that hardware, or does it work generically on top
of any EFI_AUDIO_OUTPUT_PROTOCOL?
Answer: My understanding is that the CODEC protocol will need the audio
controller to be already configured, because the communication with the
codec is done using the addresses that are related to the AUDIO
controller. So the access to the AUDIO Codec is managed by the Audio
controller. We need to write the commands at the address that was
previously allocated and set as the CORB/RIRB (Command Output Ring Buffer
/ Response Input Ring Buffer), and set at the HDA controller registers
(page 36 of the HDA Spec)

I was starting to think about how to store to audio and deal with
dynamic configuration. I guess one crazy idea could be to have the OS
create the audio files using text to speech, and maybe store them on
the EFI System Partition. For example when an OS installs it is going
to write the EFI Boot Variable that contains a Description of the boot
option. Maybe we could convert that to audio and add a nvram variable
that points to the given audio file?
Answer: This is one option. One question. How would the OS knows the bios
options and menus so it could create these files?
The EFI Boot options are NVRAM based, so we could use NVRAM for that. For
the HII (Setup) it may be possible to pass the data up to the OS and then
have it encoded. We might be able to make it visible to EFI via writing to
the disk on the EFI System Partition with an NVARM variable for any extra
information we needed. I see to remember we talked about passing data up to
the OS at one point so we could have an OS based setup utility. I don't
remember the details. Mike Rothman wrote the HII part of the spec so I can
hit him up for ideas when we try to figure this out.

Thanks,

Andrew Fish

Thanks and Regards
Rafael


Em seg, 23 de set de 2019 às 12:11, Andrew Fish <afish@apple.com
<mailto:afish@apple.com>> escreveu:
Rafael,

Did you do much research into CODECs? Like which one(s) should be
supported? I assume th CODEC implies the audio file formats that can be
decoded? Also how large are the audio files?

Is the CODEC protocol more of a plug-in for the Intel HDA? By that I mean
it only works on that hardware, or does it work generically on top of any
EFI_AUDIO_OUTPUT_PROTOCOL?

I was starting to think about how to store to audio and deal with dynamic
configuration. I guess one crazy idea could be to have the OS create the
audio files using text to speech, and maybe store them on the EFI System
Partition. For example when an OS installs it is going to write the EFI
Boot Variable that contains a Description of the boot option. Maybe we
could convert that to audio and add a nvram variable that points to the
given audio file?

Thanks,

Andrew Fish

On Sep 23, 2019, at 6:20 AM, Rafael Machado
<rafaelrodrigues.machado@gmail.com
<mailto:rafaelrodrigues.machado@gmail.com>> wrote:

Hi everyone.
So, based on everything was mentioned here.

The idea is to propose the creation of two protocols:

- EFI_AUDIO_OUTPUT_PROTOCOL: This protocol should be the responsible for
initializing the audio controller, that in the case of my MSc work does
initialization of the RING buffers, that are used as containers to the
audio streams that need to be processed.

- EFI_AUDIO_CODEC_PROTOCOL: This protocol should be responsible for
initializing the codec (each codec may need different init commands), and
also is used to control things like mute, volume level and this kind of
things. (can see the volume control actions at the last video I mentioned
on the previous e-mail)

Does this approach works at non-x86 platforms? (I don't have knowledge in
ARM platforms, so feedback from the community will be well received.)

Hope to hear some voices :)

Thanks and Regards
Rafael

Em sáb, 21 de set de 2019 às 09:36, Rafael Machado
<rafaelrodrigues.machado@gmail.com
<mailto:rafaelrodrigues.machado@gmail.com>> escreveu:
Hi Everyone
Sorry for the delay on the response, to many things happening at the same
time.
I will try to answer e-mails to this thread every Saturday or Sunday
morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I >>can also represent what ever we come up with at the
UEFI Spec Work Group.
During my MSc I had to study a lot the audio and BIOS architectures. The
idea was to eliminate the first barrier to the creation of a screen
reader for pre-OS environment, that was the lack of some open
implementation of audio control and actions at UEFI. To do that I studied
the Intel High Definition Audio Spec and a lot of UEFI specs to
understand better how to do that.
The initial target was to do all this development at OVMF, but as far as
I could get, the sound card is not available to OVMF. as Laszlo mentioned
at this e-mail there are some projects that may help on this, but at the
time I was working on my MSc I didn't find this, so I did everything on a
real system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and
working on a MSc degree at the same time having a 40hours/week job, being
a father and a husband is not an easy task, but it got to what I was
expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that
later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
<https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be>
2 - Some months later I started to work with the Ring Buffers and DMA
memory access. For the ones that have good years, it's possible to listen
some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
<https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be>
3 - Later, wen I was almost giving up, I noticed that the problem was
that one of my write operations was causing some misunderstanding between
the audio controller and the audio codec. The controller was sending
packets with 16bit depth, but the codec was processing them as 8bit
depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be
<https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be>

So the conclusion is that doing this at UEFI us much easier that doing at
the OS level.
The reference code, that is just a proof-of-concept, and that has several
things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility
<https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility>

Currently it is just an UEFI Application, but we can convert this to UEFI
drivers after some discussion. Everything is released as BDS so companies
can use without IP problems.
Just to give some more information about the need of this kind of
solution. There is a lot of blind people that work with hardware support,
so formatting disk, configuring RAID and booting dual-boot systems is
always a challenge to them. Even set the BIOS clock. How to do that
without the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given there
is no guideline on how a User Interface (UI) works. If accessibility
requires some from of hardware abstraction, like audio, then we could
likely get that into the UEFI Spec. What might be possible is an
EDK2 reference implementation of accessibility. Maybe we could use the
reference implementation to write a UEFI white paper on design >>for
accessibility? I there is an open source implementation, and an
official design guide this would make it much easier for advocacy
groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2 papers
is one of my targets since the beginning of my MSc almost 5 years ago.

I've got some experience with accessibility as the macOS EFI OS Loader
has a UI for the Full Disk Encryption password. If you press the power
button quickly 3 times at the disk encryption password prompt
accessibility is enabled and Voice Over gets turned on. You then get
localized voice prompts when you move between UI elements. Since this
is the OS loader all the resources are stored on the disk. You
quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize >>the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so
no audio files would need to be stored. There are some open-source
implementations that are not GPL.
This was described at my MSc as future works that can continue what I
have started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyone's
machines, so that would be really helpful for demos and design
review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible future
steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with
proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver to
produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also producing
EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application remains
functional.
- 4) Separate controller and codec code by creating an
EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate out
the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder
audio files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the need
of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my
thesis to English.
I wrote everything in Portuguese because there was not material about
UEFI to the Brazilian audience, and another target I have is to show
companies that we have people that can work at this kind of projects in
Brazil, bringing this kind of development to south america. (Yes, I have
complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@redhat.com
<mailto:lersek@redhat.com>> escreveu:
On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html
<https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html>

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html
<https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html>

(This is all I can mention right now.)

Thanks
Laszlo
--
Signed,
Ethin D. Probst


Rafael Machado <rafaelrodrigues.machado@...>
 

Hi Ethin.

The patent for MP3 expired a year or so ago, if memory serves. (I
don't know how long ago it expired but it definitely has expired, so
you don't need to worry about IP (I don't think).) An OS setup utility
would be nice if it weren't for the fact that it would prevent you
from modifying things like secure boot. Why can't we just either (1)
embed the audio files in NVRAM

Answer: Good point. But I think Andrew was thinking only on the boot
options, so the sound files would be generated by the OS, and not a OS
based solution that enables the BIOS settings to be changed. By the way, my
opinion is that the market would not like a OS solution that changes the
BIOS settings, due to security reasons.

Andrew
Is my understanding of your proposal correct?

or (2) embed a speech synthesizer
directly into the UEFI system (the synthesizer would not need to be
standardized, just easily embeddable) and then send PCM samples
directly to the HDA device using that synthesizer? ESpeak is one such
synthesizer; I don't know how we'd go about modifying it but given
enough time we could probably make it fit in the UEFI environment.

Answer: About porting eSpeak to UEFI, we need to check it's license first.
Considering the fact that it is GPL as presented below (from eSpeak 1.48.0
at https://sourceforge.net/projects/espeak/)






*" GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 Copyright (C)
2007 Free Software Foundation, Inc. <http://fsf.org/
<http://fsf.org/>> Everyone is permitted to copy and distribute verbatim
copies*
* of this license document, but changing it is not allowed. "*

I am not a lawyer and also have nothing against GPL and other licenses, but
would this mandate the BIOS vendors to open their BIOS code in case they
decide to use our eSpeak based synthesizer at their products?
My MSc's code was released as BSD to avoid this risk, but I am not sure it
this would be a real problem.

Thanks and Regards
Rafael R. Machado

Em qua, 25 de set de 2019 às 00:04, Ethin Probst <harlydavidsen@gmail.com>
escreveu:

The patent for MP3 expired a year or so ago, if memory serves. (I
don't know how long ago it expired but it definitely has expired, so
you don't need to worry about IP (I don't think).) An OS setup utility
would be nice if it weren't for the fact that it would prevent you
from modifying things like secure boot. Why can't we just either (1)
embed the audio files in NVRAM or (2) embed a speech synthesizer
directly into the UEFI system (the synthesizer would not need to be
standardized, just easily embeddable) and then send PCM samples
directly to the HDA device using that synthesizer? ESpeak is one such
synthesizer; I don't know how we'd go about modifying it but given
enough time we could probably make it fit in the UEFI environment.

On 9/24/19, Andrew Fish <afish@apple.com> wrote:


On Sep 24, 2019, at 5:06 AM, Rafael Machado
<rafaelrodrigues.machado@gmail.com> wrote:

Hi Everyone

Answering Ethin:
If other platforms are PCI-based (i.e. allow us to scan the PCI bus
and figure out where (in MMIO space) the HDA controller is mapped
to),
then it (theoretically) sould work. I don't know for sure though; I'm
not very knowledgeable in other CPU architectures
Answer: Neither do I. Lets wait some ARM expert to give some opinion :)
Rafael,

I'm not exactly the ARM expert, but what we have seen is as long as en
EFI
PCI driver follows the EFI rules for DMA it should work just fine on all
CPU
architectures. The historical problem is if you don't follow the rules on
x86 DMA still works since the memory coherency is maintained by
hardware on
x86. Since the coherency requires software on ARM, you have to do it
right.


Answering Andrew
Did you do much research into CODECs? Like which one(s) should be
supported? I assume the CODEC implies the audio file formats that can
be decoded? Also how large are the audio files?
Answer: During my research I have studied some codec specs, and the way
the codecs works is really similar. Normally they need to receive some
packets to initialize the nodes (node is the name of each internal
component of the codec), by receiving some commands (named verbs) that
are
defined at the HDASpec. For example the verb "power state" (0xF05) is
used to set the power state of each node on the codec, to really enable
the chip or put the chip to sleep.
The verbs can be found at this document
https://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/high-definition-audio-specification.pdf
<
https://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/high-definition-audio-specification.pdf
,
at page 216.
Not all verbs are standardized, for example the verb "Amplifier Gain
Mute"
(0xB--), each vendor can decide the command. In the case of the ASUS
system I have used, the codec is a Conexant CX20752
(https://www.datasheets360.com/pdf/7550682360275196829
<https://www.datasheets360.com/pdf/7550682360275196829>), and at the
datasheet we can see that the "Amplifier Gain Mute" verb is 0xB00 and
0xB20 (right and left channel in the case of a stereo stream) at page
33.
These verbs that are vendor defined creates the problems to think on a
generic solution.
About the second part of the question, the Codecs are independent, so
they
are only responsible to process signals that are passed to them in a
stream using a DMA buffer, or a command to change their behavior
(increase
/ decrease volume for example). So for the codec to process correctly
the
buffers managed by the audio controller, both need to be configured with
the same stream format, something like 2 channel 8 bit depth for
example.
At the second video I send previously
(https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
<https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be>) the
problem was that the controller was configured to manage a stream with 2
channel and 8 bits depth, but the codec was processing the data as 2
channel 16 bit depth.
About the audio files size, at my work I didn't use any compressed
format,
because I was running out of time to finish my project, so I was
processing raw audio data, that is much bigger than other formats like
MP3. In case we decide to use audio files at this first stage, maybe we
will need to port some audio format lib to UEFI also.
Thanks I tracked down a friend of mine who knows audio so I think I grok
it
better now. It seems HDA is a hardware standard. Seems like most of the
work
in an HDA driver is doing the setup to decode the data.

As far as a file format we might be able to use PCD, or as we know it a
wav
file. It might also be possible to use MP3, but there have been IP
issues in
the past encoding MP3 and I'm not sure how that will work out. The only
issue with WAV file is size, but we may be able to compress the WAV files
with a decompressor that already lives in the ROM. Saving file space is
probably as simple as wrapping the file with a header that describes the
file type and encryption scheme.

Is the CODEC protocol more of a plug-in for the Intel HDA? By that I
mean it only works on that hardware, or does it work generically on
top
of any EFI_AUDIO_OUTPUT_PROTOCOL?
Answer: My understanding is that the CODEC protocol will need the audio
controller to be already configured, because the communication with the
codec is done using the addresses that are related to the AUDIO
controller. So the access to the AUDIO Codec is managed by the Audio
controller. We need to write the commands at the address that was
previously allocated and set as the CORB/RIRB (Command Output Ring
Buffer
/ Response Input Ring Buffer), and set at the HDA controller registers
(page 36 of the HDA Spec)

I was starting to think about how to store to audio and deal with
dynamic configuration. I guess one crazy idea could be to have the OS
create the audio files using text to speech, and maybe store them on
the EFI System Partition. For example when an OS installs it is going
to write the EFI Boot Variable that contains a Description of the
boot
option. Maybe we could convert that to audio and add a nvram variable
that points to the given audio file?
Answer: This is one option. One question. How would the OS knows the
bios
options and menus so it could create these files?
The EFI Boot options are NVRAM based, so we could use NVRAM for that. For
the HII (Setup) it may be possible to pass the data up to the OS and then
have it encoded. We might be able to make it visible to EFI via writing
to
the disk on the EFI System Partition with an NVARM variable for any extra
information we needed. I see to remember we talked about passing data up
to
the OS at one point so we could have an OS based setup utility. I don't
remember the details. Mike Rothman wrote the HII part of the spec so I
can
hit him up for ideas when we try to figure this out.

Thanks,

Andrew Fish

Thanks and Regards
Rafael


Em seg, 23 de set de 2019 às 12:11, Andrew Fish <afish@apple.com
<mailto:afish@apple.com>> escreveu:
Rafael,

Did you do much research into CODECs? Like which one(s) should be
supported? I assume th CODEC implies the audio file formats that can be
decoded? Also how large are the audio files?

Is the CODEC protocol more of a plug-in for the Intel HDA? By that I
mean
it only works on that hardware, or does it work generically on top of
any
EFI_AUDIO_OUTPUT_PROTOCOL?

I was starting to think about how to store to audio and deal with
dynamic
configuration. I guess one crazy idea could be to have the OS create the
audio files using text to speech, and maybe store them on the EFI System
Partition. For example when an OS installs it is going to write the EFI
Boot Variable that contains a Description of the boot option. Maybe we
could convert that to audio and add a nvram variable that points to the
given audio file?

Thanks,

Andrew Fish

On Sep 23, 2019, at 6:20 AM, Rafael Machado
<rafaelrodrigues.machado@gmail.com
<mailto:rafaelrodrigues.machado@gmail.com>> wrote:

Hi everyone.
So, based on everything was mentioned here.

The idea is to propose the creation of two protocols:

- EFI_AUDIO_OUTPUT_PROTOCOL: This protocol should be the responsible
for
initializing the audio controller, that in the case of my MSc work does
initialization of the RING buffers, that are used as containers to the
audio streams that need to be processed.

- EFI_AUDIO_CODEC_PROTOCOL: This protocol should be responsible for
initializing the codec (each codec may need different init commands),
and
also is used to control things like mute, volume level and this kind of
things. (can see the volume control actions at the last video I
mentioned
on the previous e-mail)

Does this approach works at non-x86 platforms? (I don't have knowledge
in
ARM platforms, so feedback from the community will be well received.)

Hope to hear some voices :)

Thanks and Regards
Rafael

Em sáb, 21 de set de 2019 às 09:36, Rafael Machado
<rafaelrodrigues.machado@gmail.com
<mailto:rafaelrodrigues.machado@gmail.com>> escreveu:
Hi Everyone
Sorry for the delay on the response, to many things happening at the
same
time.
I will try to answer e-mails to this thread every Saturday or Sunday
morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I >>can also represent what ever we come up with at the
UEFI Spec Work Group.
During my MSc I had to study a lot the audio and BIOS architectures.
The
idea was to eliminate the first barrier to the creation of a screen
reader for pre-OS environment, that was the lack of some open
implementation of audio control and actions at UEFI. To do that I
studied
the Intel High Definition Audio Spec and a lot of UEFI specs to
understand better how to do that.
The initial target was to do all this development at OVMF, but as far
as
I could get, the sound card is not available to OVMF. as Laszlo
mentioned
at this e-mail there are some projects that may help on this, but at
the
time I was working on my MSc I didn't find this, so I did everything
on a
real system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and
working on a MSc degree at the same time having a 40hours/week job,
being
a father and a husband is not an easy task, but it got to what I was
expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that
later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
<https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be>
2 - Some months later I started to work with the Ring Buffers and DMA
memory access. For the ones that have good years, it's possible to
listen
some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
<https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be>
3 - Later, wen I was almost giving up, I noticed that the problem was
that one of my write operations was causing some misunderstanding
between
the audio controller and the audio codec. The controller was sending
packets with 16bit depth, but the codec was processing them as 8bit
depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be
<https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be>

So the conclusion is that doing this at UEFI us much easier that doing
at
the OS level.
The reference code, that is just a proof-of-concept, and that has
several
things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility
<https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility>

Currently it is just an UEFI Application, but we can convert this to
UEFI
drivers after some discussion. Everything is released as BDS so
companies
can use without IP problems.
Just to give some more information about the need of this kind of
solution. There is a lot of blind people that work with hardware
support,
so formatting disk, configuring RAID and booting dual-boot systems is
always a challenge to them. Even set the BIOS clock. How to do that
without the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given
there
is no guideline on how a User Interface (UI) works. If accessibility
requires some from of hardware abstraction, like audio, then we
could
likely get that into the UEFI Spec. What might be possible is an
EDK2 reference implementation of accessibility. Maybe we could use
the
reference implementation to write a UEFI white paper on design >>for
accessibility? I there is an open source implementation, and an
official design guide this would make it much easier for advocacy
groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2
papers
is one of my targets since the beginning of my MSc almost 5 years ago.

I've got some experience with accessibility as the macOS EFI OS
Loader
has a UI for the Full Disk Encryption password. If you press the
power
button quickly 3 times at the disk encryption password prompt
accessibility is enabled and Voice Over gets turned on. You then
get
localized voice prompts when you move between UI elements. Since
this
is the OS loader all the resources are stored on the disk. You
quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to
support),
Audio files are not small and firmware is size constrained, the need
to localize >>the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so
no audio files would need to be stored. There are some open-source
implementations that are not GPL.
This was described at my MSc as future works that can continue what I
have started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies
and
skills, because that challenge is one that we are willing to
accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on
everyone's
machines, so that would be really helpful for demos and design
review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible
future
steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with
proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver to
produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also producing
EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application remains
functional.
- 4) Separate controller and codec code by creating an
EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate
out
the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder
audio files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the
need
of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my
thesis to English.
I wrote everything in Portuguese because there was not material about
UEFI to the Brazilian audience, and another target I have is to show
companies that we have people that can work at this kind of projects in
Brazil, bringing this kind of development to south america. (Yes, I
have
complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@redhat.com
<mailto:lersek@redhat.com>> escreveu:
On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html
<https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as
of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html
<https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo

--
Signed,
Ethin D. Probst


Andrew Fish <afish@...>
 

On Sep 25, 2019, at 5:06 AM, Rafael Machado <rafaelrodrigues.machado@gmail.com> wrote:

Hi Ethin.

The patent for MP3 expired a year or so ago, if memory serves. (I
don't know how long ago it expired but it definitely has expired, so
you don't need to worry about IP (I don't think).) An OS setup utility
would be nice if it weren't for the fact that it would prevent you
from modifying things like secure boot. Why can't we just either (1)
embed the audio files in NVRAM

Answer: Good point. But I think Andrew was thinking only on the boot options, so the sound files would be generated by the OS, and not a OS based solution that enables the BIOS settings to be changed. By the way, my opinion is that the market would not like a OS solution that changes the BIOS settings, due to security reasons.

Andrew
Is my understanding of your proposal correct?
Answer: The EFI Firmware has some default boot options, but when an OS installs it writes the NVRAM variables that point to OS loader on the EFI System partition. Note the OS installer is the software that copied OS loader to the EFI System partition so this is logical for the flow. The NVRAM entry has a description and that is what gets displayed in a UI. So there could be a matching NVRAM variable that points to a sound file generated. For setup my thought was not to execute setup in the OS context, but to just use it to have the OS generate the audio files and place them some place, like the EFI System Partition, that the EFI HHI engine for setup could find them. This is really a use the OS synthesizer strategy.

Answer for MP3: Licensing is complicated, and the license for MP3 has been bad for a long time so I did not want to assume it was OK. We need to ask our Free Software friends to find out how OK it is to use, or how well it would be supported.

or (2) embed a speech synthesizer
directly into the UEFI system (the synthesizer would not need to be
standardized, just easily embeddable) and then send PCM samples
directly to the HDA device using that synthesizer? ESpeak is one such
synthesizer; I don't know how we'd go about modifying it but given
enough time we could probably make it fit in the UEFI environment.

Answer: About porting eSpeak to UEFI, we need to check it's license first.
Considering the fact that it is GPL as presented below (from eSpeak 1.48.0 at https://sourceforge.net/projects/espeak/)

" GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007

Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed. "

I am not a lawyer and also have nothing against GPL and other licenses, but would this mandate the BIOS vendors to open their BIOS code in case they decide to use our eSpeak based synthesizer at their products?
Answer: I was thinking that a speech synthesizer would be too complicated for EFI, but after talking to Santo I realized it might be possible if we ported an old obsolete version. It might sound out of date but that is probably the tradeoff we would need to make for not using a modern OS based speech synthesizer.

I'm not a lawyer either and GPL 3 usage can be complex for some. It would be possible to have a stand alone project that built the speech synthesizer as a stand alone binary driver. Even if the GPL 3 license causes issues for some a prototype of eSpeak ported to EFI would give us a size estimate and let us develop an EFI speech synthesizer Protocol that we could publish in the UEFI Spec.

I noticed an article @ the Intel Newsroom titled: "Stephen Hawking: Intel Helped Give Him His Voice". So maybe our Intel friends may know of some speech synthesizer options other than eSpeak?

Thanks,

Andrew Fish

My MSc's code was released as BSD to avoid this risk, but I am not sure it this would be a real problem.

Thanks and Regards
Rafael R. Machado

Em qua, 25 de set de 2019 às 00:04, Ethin Probst <harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com>> escreveu:
The patent for MP3 expired a year or so ago, if memory serves. (I
don't know how long ago it expired but it definitely has expired, so
you don't need to worry about IP (I don't think).) An OS setup utility
would be nice if it weren't for the fact that it would prevent you
from modifying things like secure boot. Why can't we just either (1)
embed the audio files in NVRAM or (2) embed a speech synthesizer
directly into the UEFI system (the synthesizer would not need to be
standardized, just easily embeddable) and then send PCM samples
directly to the HDA device using that synthesizer? ESpeak is one such
synthesizer; I don't know how we'd go about modifying it but given
enough time we could probably make it fit in the UEFI environment.

On 9/24/19, Andrew Fish <afish@apple.com <mailto:afish@apple.com>> wrote:


On Sep 24, 2019, at 5:06 AM, Rafael Machado
<rafaelrodrigues.machado@gmail.com <mailto:rafaelrodrigues.machado@gmail.com>> wrote:

Hi Everyone

Answering Ethin:
If other platforms are PCI-based (i.e. allow us to scan the PCI bus
and figure out where (in MMIO space) the HDA controller is mapped to),
then it (theoretically) sould work. I don't know for sure though; I'm
not very knowledgeable in other CPU architectures
Answer: Neither do I. Lets wait some ARM expert to give some opinion :)
Rafael,

I'm not exactly the ARM expert, but what we have seen is as long as en EFI
PCI driver follows the EFI rules for DMA it should work just fine on all CPU
architectures. The historical problem is if you don't follow the rules on
x86 DMA still works since the memory coherency is maintained by hardware on
x86. Since the coherency requires software on ARM, you have to do it right.


Answering Andrew
Did you do much research into CODECs? Like which one(s) should be
supported? I assume the CODEC implies the audio file formats that can
be decoded? Also how large are the audio files?
Answer: During my research I have studied some codec specs, and the way
the codecs works is really similar. Normally they need to receive some
packets to initialize the nodes (node is the name of each internal
component of the codec), by receiving some commands (named verbs) that are
defined at the HDASpec. For example the verb "power state" (0xF05) is
used to set the power state of each node on the codec, to really enable
the chip or put the chip to sleep.
The verbs can be found at this document
https://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/high-definition-audio-specification.pdf
<https://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/high-definition-audio-specification.pdf>,
at page 216.
Not all verbs are standardized, for example the verb "Amplifier Gain Mute"
(0xB--), each vendor can decide the command. In the case of the ASUS
system I have used, the codec is a Conexant CX20752
(https://www.datasheets360.com/pdf/7550682360275196829
<https://www.datasheets360.com/pdf/7550682360275196829>), and at the
datasheet we can see that the "Amplifier Gain Mute" verb is 0xB00 and
0xB20 (right and left channel in the case of a stereo stream) at page 33.
These verbs that are vendor defined creates the problems to think on a
generic solution.
About the second part of the question, the Codecs are independent, so they
are only responsible to process signals that are passed to them in a
stream using a DMA buffer, or a command to change their behavior (increase
/ decrease volume for example). So for the codec to process correctly the
buffers managed by the audio controller, both need to be configured with
the same stream format, something like 2 channel 8 bit depth for example.
At the second video I send previously
(https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
<https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be>) the
problem was that the controller was configured to manage a stream with 2
channel and 8 bits depth, but the codec was processing the data as 2
channel 16 bit depth.
About the audio files size, at my work I didn't use any compressed format,
because I was running out of time to finish my project, so I was
processing raw audio data, that is much bigger than other formats like
MP3. In case we decide to use audio files at this first stage, maybe we
will need to port some audio format lib to UEFI also.
Thanks I tracked down a friend of mine who knows audio so I think I grok it
better now. It seems HDA is a hardware standard. Seems like most of the work
in an HDA driver is doing the setup to decode the data.

As far as a file format we might be able to use PCD, or as we know it a wav
file. It might also be possible to use MP3, but there have been IP issues in
the past encoding MP3 and I'm not sure how that will work out. The only
issue with WAV file is size, but we may be able to compress the WAV files
with a decompressor that already lives in the ROM. Saving file space is
probably as simple as wrapping the file with a header that describes the
file type and encryption scheme.

Is the CODEC protocol more of a plug-in for the Intel HDA? By that I
mean it only works on that hardware, or does it work generically on top
of any EFI_AUDIO_OUTPUT_PROTOCOL?
Answer: My understanding is that the CODEC protocol will need the audio
controller to be already configured, because the communication with the
codec is done using the addresses that are related to the AUDIO
controller. So the access to the AUDIO Codec is managed by the Audio
controller. We need to write the commands at the address that was
previously allocated and set as the CORB/RIRB (Command Output Ring Buffer
/ Response Input Ring Buffer), and set at the HDA controller registers
(page 36 of the HDA Spec)

I was starting to think about how to store to audio and deal with
dynamic configuration. I guess one crazy idea could be to have the OS
create the audio files using text to speech, and maybe store them on
the EFI System Partition. For example when an OS installs it is going
to write the EFI Boot Variable that contains a Description of the boot
option. Maybe we could convert that to audio and add a nvram variable
that points to the given audio file?
Answer: This is one option. One question. How would the OS knows the bios
options and menus so it could create these files?
The EFI Boot options are NVRAM based, so we could use NVRAM for that. For
the HII (Setup) it may be possible to pass the data up to the OS and then
have it encoded. We might be able to make it visible to EFI via writing to
the disk on the EFI System Partition with an NVARM variable for any extra
information we needed. I see to remember we talked about passing data up to
the OS at one point so we could have an OS based setup utility. I don't
remember the details. Mike Rothman wrote the HII part of the spec so I can
hit him up for ideas when we try to figure this out.

Thanks,

Andrew Fish

Thanks and Regards
Rafael


Em seg, 23 de set de 2019 às 12:11, Andrew Fish <afish@apple.com <mailto:afish@apple.com>
<mailto:afish@apple.com <mailto:afish@apple.com>>> escreveu:
Rafael,

Did you do much research into CODECs? Like which one(s) should be
supported? I assume th CODEC implies the audio file formats that can be
decoded? Also how large are the audio files?

Is the CODEC protocol more of a plug-in for the Intel HDA? By that I mean
it only works on that hardware, or does it work generically on top of any
EFI_AUDIO_OUTPUT_PROTOCOL?

I was starting to think about how to store to audio and deal with dynamic
configuration. I guess one crazy idea could be to have the OS create the
audio files using text to speech, and maybe store them on the EFI System
Partition. For example when an OS installs it is going to write the EFI
Boot Variable that contains a Description of the boot option. Maybe we
could convert that to audio and add a nvram variable that points to the
given audio file?

Thanks,

Andrew Fish

On Sep 23, 2019, at 6:20 AM, Rafael Machado
<rafaelrodrigues.machado@gmail.com <mailto:rafaelrodrigues.machado@gmail.com>
<mailto:rafaelrodrigues.machado@gmail.com <mailto:rafaelrodrigues.machado@gmail.com>>> wrote:

Hi everyone.
So, based on everything was mentioned here.

The idea is to propose the creation of two protocols:

- EFI_AUDIO_OUTPUT_PROTOCOL: This protocol should be the responsible for
initializing the audio controller, that in the case of my MSc work does
initialization of the RING buffers, that are used as containers to the
audio streams that need to be processed.

- EFI_AUDIO_CODEC_PROTOCOL: This protocol should be responsible for
initializing the codec (each codec may need different init commands), and
also is used to control things like mute, volume level and this kind of
things. (can see the volume control actions at the last video I mentioned
on the previous e-mail)

Does this approach works at non-x86 platforms? (I don't have knowledge in
ARM platforms, so feedback from the community will be well received.)

Hope to hear some voices :)

Thanks and Regards
Rafael

Em sáb, 21 de set de 2019 às 09:36, Rafael Machado
<rafaelrodrigues.machado@gmail.com <mailto:rafaelrodrigues.machado@gmail.com>
<mailto:rafaelrodrigues.machado@gmail.com <mailto:rafaelrodrigues.machado@gmail.com>>> escreveu:
Hi Everyone
Sorry for the delay on the response, to many things happening at the same
time.
I will try to answer e-mails to this thread every Saturday or Sunday
morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I >>can also represent what ever we come up with at the
UEFI Spec Work Group.
During my MSc I had to study a lot the audio and BIOS architectures. The
idea was to eliminate the first barrier to the creation of a screen
reader for pre-OS environment, that was the lack of some open
implementation of audio control and actions at UEFI. To do that I studied
the Intel High Definition Audio Spec and a lot of UEFI specs to
understand better how to do that.
The initial target was to do all this development at OVMF, but as far as
I could get, the sound card is not available to OVMF. as Laszlo mentioned
at this e-mail there are some projects that may help on this, but at the
time I was working on my MSc I didn't find this, so I did everything on a
real system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and
working on a MSc degree at the same time having a 40hours/week job, being
a father and a husband is not an easy task, but it got to what I was
expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that
later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
<https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be>
2 - Some months later I started to work with the Ring Buffers and DMA
memory access. For the ones that have good years, it's possible to listen
some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
<https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be>
3 - Later, wen I was almost giving up, I noticed that the problem was
that one of my write operations was causing some misunderstanding between
the audio controller and the audio codec. The controller was sending
packets with 16bit depth, but the codec was processing them as 8bit
depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be
<https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be>

So the conclusion is that doing this at UEFI us much easier that doing at
the OS level.
The reference code, that is just a proof-of-concept, and that has several
things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility
<https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility>

Currently it is just an UEFI Application, but we can convert this to UEFI
drivers after some discussion. Everything is released as BDS so companies
can use without IP problems.
Just to give some more information about the need of this kind of
solution. There is a lot of blind people that work with hardware support,
so formatting disk, configuring RAID and booting dual-boot systems is
always a challenge to them. Even set the BIOS clock. How to do that
without the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given there
is no guideline on how a User Interface (UI) works. If accessibility
requires some from of hardware abstraction, like audio, then we could
likely get that into the UEFI Spec. What might be possible is an
EDK2 reference implementation of accessibility. Maybe we could use the
reference implementation to write a UEFI white paper on design >>for
accessibility? I there is an open source implementation, and an
official design guide this would make it much easier for advocacy
groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2 papers
is one of my targets since the beginning of my MSc almost 5 years ago.

I've got some experience with accessibility as the macOS EFI OS Loader
has a UI for the Full Disk Encryption password. If you press the power
button quickly 3 times at the disk encryption password prompt
accessibility is enabled and Voice Over gets turned on. You then get
localized voice prompts when you move between UI elements. Since this
is the OS loader all the resources are stored on the disk. You
quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize >>the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so
no audio files would need to be stored. There are some open-source
implementations that are not GPL.
This was described at my MSc as future works that can continue what I
have started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyone's
machines, so that would be really helpful for demos and design
review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible future
steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with
proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver to
produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also producing
EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application remains
functional.
- 4) Separate controller and codec code by creating an
EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate out
the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder
audio files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the need
of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my
thesis to English.
I wrote everything in Portuguese because there was not material about
UEFI to the Brazilian audience, and another target I have is to show
companies that we have people that can work at this kind of projects in
Brazil, bringing this kind of development to south america. (Yes, I have
complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@redhat.com <mailto:lersek@redhat.com>
<mailto:lersek@redhat.com <mailto:lersek@redhat.com>>> escreveu:
On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html
<https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html>

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html
<https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html>

(This is all I can mention right now.)

Thanks
Laszlo

--
Signed,
Ethin D. Probst