Date   

Re: UEFI accessibility mandate

Ethin Probst
 

This is incredible progress! I'd love to help where I can when you
release the src.

On 10/29/19, Andrew Fish <afish@...> wrote:
Ethin,

I've made progress using the EmulatorPkg. I've ported my test C command line
application into the EmulatorPkg Host (OS C command line application part
of the EmulatorPkg), so the the host now produces APIs that support Text to
Voice PCM files, and playing PCM, also called wave, files. There is a
EmulatorPkg driver to produce Text To PCM, and a driver to play PCM files. I
also wrote a test application. At this point the test application in the
emulator can use the two APIs to produce Text To Voice.

Next steps for me are to write up a Bugzilla, cleanup the code, and make
github branch to share the work.

Next steps for the community... From the EmulatorPkg it should be possible
to implement the UI work to add voice. it should also be possible to
implement a real EFI Voice to PCM driver. It should also be possible to use
the test application to test a real EFI audio driver.

Thanks,

Andrew Fish

On Oct 29, 2019, at 1:17 PM, Ethin Probst <harlydavidsen@...>
wrote:

Has there been any progress on this? I wish I could help, but I have
no experience in developing for EDK2...

On 10/1/19, Rafael Machado <rafaelrodrigues.machado@...> wrote:
Incredible!
I will find some time to try it at linux on the following weeks.
Thanks for the effort Andrew!

Rafael R. Machado

Em seg, 30 de set de 2019 às 22:18, Andrew Fish <afish@...>
escreveu:



On Sep 30, 2019, at 11:50 AM, Rafael Machado <
rafaelrodrigues.machado@...> wrote:

Hi Andrew

As you have mentioned:
" I might find some time to try to get the EmulatorPkg mocking
working."
Answer: This would be amazing! Thanks for trying to help!


Rafael,

I made a little progress this weekend watching American Football on TV,
and I had some time on an airplane today to add some comments.

I wrote a simple C command line program that does almost all the
primitive
audio functions we need to add Audio/Text-To-Speech to the EmulatorPkg.
Since ANSI C does not support audio I had to system() out to macOS
command
line tools to get some of the tasks complete. The only feature I'm
missing
is playing the PCM buffer asynchronously, and I may be able to add that
next weekend. I guessed at the Linux commands, it would be good if
someone
could debug the program on a Linux system, and also make sure I picked
the
right command line tools to use. I think for Windows there is going to
be
more coding involved, but I think there is at least a C API to play
audio,
but I only saw a C++ API for speech to text. But the good news is folks
can
use the simple C program to get the primitives working on Linux and
Windows
systems and it should be easy to port that work to the future
EmulatorPkg
work.

What I've got working on macOS is:
1) Mute/Unmute
2) GetVolume/SetVolume
3) Text to PCD buffer (Wave file format)
a) Supports word per minute rate
b) Supports localizing the voice based on RFC 4646 language code used
by
EFI (the program speaks in en, zh, and fr as a test)
4) PCD buffer to audio

This is how I was building on my Mac: clang -g -Wall -Werror audio.c

Short term anyone that has some free time if they could look at
testing/debugging audio.c on Linux and send feedback, also maybe some
one
could start playing around with getting audio.c ported to Windows?

After I figure out getting async audio output working in audio.c I
think
the next steps are working on the protocol/variable definitions. With
the
protocol/variable definitions we should be able to add the capability
to
the EmulatorPkg. At that point it should be possible for anyone to help
work on the accessibility UI. Obviously to get this working on a real
platform we will need a pure EFI Text to Speech driver (We could debug
that
in the EmulatorPkg). We would also need Intel HDA Audio drivers, and
Audio
drivers for OVFM, and other Audio hardware that is on peoples
platforms.

Thanks,

Andrew Fish

The plan is to try to get some students to help on this also at the
next
"google summer of code".
Not sure how it works, but since the idea is good, and it is a real
need,
I believe we have chances to have some more hands working on it.

Thanks and Regards
Rafael R. Machado



Em qui, 26 de set de 2019 às 20:44, Andrew Fish <afish@...>
escreveu:



On Sep 26, 2019, at 5:15 AM, Rafael Machado <
rafaelrodrigues.machado@...> wrote:

Hi everyone.

About Ethin's question:
As for the boot manager, yes, that would be covered by the speech
synthesizer protocol. Perhaps we could extend this to the setup
utility
too
(as well as all other things displayed on the screen)?
Answer: This is the target from my point of view. Since we have the
advantages of drivers at UEFI, there is no reason for not having
accessibility at pre-OS anymore. We could also think in future to add
other
kinds of accessibility, like the magnifying glass (that would be some
multi-layer version of the GOP driver), that would help low-vision
people.
Just one thing for everyone to think about. The "Convention on the
Rights
of Persons with Disabilities" say that a person has the right to
access
information. So if someone buys a notebook, all information accessible
to
"common" people must be available to all kinds of people, in a way
they
can
choose to use this information or not.
*" (e) Recognizing that disability is an evolving concept and that
disability results from the interaction between persons with
impairments
and attitudinal and environmental barriers that hinders their full and
effective participation in society on an equal basis with others,
..."*
Details can be found at the United Nations web site
https://www.un.org/disabilities/documents/convention/convention_accessible_pdf.pdf


About Andrew's comments and questions:
What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL
should contain?
Answer: From my perspective, this protocol would need at least the
following functions (names may change after more discussion.)


- EFI_AUDIO_OUTPUT_PROTOCOL.LoadStream()
This function is intended to copy some data to the dma buffers that
were allocated previously by the AudioController driver. The initial
idea
I
had was to add, in the case of my MSc, a HdaDriver.efi that is
responsible
to do all the HDA controller initialization (allocating dma buffers,
configuring the PCIe bus if needed). Not sure if this initialization
should
we added to the EFI_AUDIO_OUTPUT_PROTOCOL, because this could create
problems when using other controllers or concepts, like I believe ARM
platforms do.My understanding is that the platforms should have a
driver
to
initialize the audio controller/codec, so the
EFI_AUDIO_OUTPUT_PROTOCOL
is
just worried with loading and playing information.

- EFI_AUDIO_OUTPUT_PROTOCOL.PlayStream()
This function is responsible to play the stream. This is done by
setting one specific bit at the controllers configuration space
(pointed
by
the bar[0] pcie configuration register at the pcie config space.) At
this
project I don't think we need to be worried with implementing an sound
server to enable multiple streams at a time. To the actions done by
this
function are much simpler. Just play a stream.

- EFI_AUDIO_OUTPUT_PROTOCOL.StopStream()
This function is responsible to stop the stream play. The opposite
of
the previous one. Maybe we could use a single function that receives
some
parameter to start or stop the play.

- EFI_AUDIO_OUTPUT_PROTOCOL.SetVolume()
This function is responsible to set the volume to be used at all
nodes at the codec (so it depends on a EFI_AUDIO_CODEC_PROTOCOL that
knows
the commands a given codec needs to receive to process streams.)

- EFI_AUDIO_OUTPUT_PROTOCOL.SetStreamSpeed()
This function is responsible to set the speed a given stream should
be played on. Just to clarify, visual impaired people have an amazing
capacity of understanding sound streams in incredibly fast speeds. So
the
possibility of setting the stream processing speed is what makes them
be
as
productive as a person with usual vision capacities.

- EFI_AUDIO_OUTPUT_PROTOCOL.UnloadStream()
This function is responsible to clean the buffers and letting them
prepared to receive other audio streams.

Still about Andrews comments.
This gave me the idea of adding a EFI_AUDIO_OUTPUT_PROTOCOL and
EFI_TEXT_TO_SPEECH_PROTOCOL
driver to the EmulatorPkg that thunk down to the App to do the work.
With this we could prototype the UI part of accessibility.
Answer: This would be great, because we would split the development.
One
focusing on the HII and navigation (done normally using the tab key,
so
each element at the screen would need an tab sequence number and a
accessibility tag with what the speech software should say), and
another
focusing on the drivers/synthesizer/audio files depending on our
future
decisions.

So my understanding is that the list of things is increasing. It is
actually something like this (not ordered by priority):
- EFI_AUDIO_OUTPUT_PROTOCOL: Protocol used to process the streams
- EFI_AUDIO_CODEC_PROTOCOL: Protocol used to communicate with the
codec
- Audio configuration driver (HdaAudio.efi for example). Should this
create some other protocol, like EFI_AUDIO_CONTROLLER_PROTOCOL that
is
responsible to configure the controller?


Rafael,

Generally how it works for a PCI device is it gets its Start()
function
called to bind to the device. So the EFI_AUDIO_OUTPUT_PROTOCOL and
EFI_AUDIO_CODEC_PROTOCOL protocols would likely get installed on the
PCI
IO
handle of the HDA PCI hardware. Some basic init will happen during the
Start() and the driver could defer some config until one of the member
functions is called the 1st time.

So for example on a normal boot the Audio driver may not even get
started. So there may have to be an nvram variable to start the
driver.
We
might also need an NVRAM variable for the voice to text driver if it
is
stored on disk.

- EFI_TEXT_TO_SPEECH_PROTOCOL: Protocol responsible to convert text to
speech (using the EFI_AUDIO_OUTPUT_PROTOCOL )
- HII changes to add tab navigation capabilities and accessibility
tags
at each component on the screen

Are we aligned with the understanding?


I think so and if we can mock an implementation in the EmulatorPkg
that
would let the HII UI work happen independently of the text to speech,
and
Audio driver work.

I'm on some air plaines in the next weeks (I'm at the airport right
now),
and my wife is heading to a conference after I get back from my trips.
Given my dogs are not very talkative I might find some time to try to
get
the EmulatorPkg mocking working.

Thanks,

Andrew Fish

I believe soon we will be prepared to talk about this on a design
meeting.

Thanks and Regards
Rafael R. Machado



Em qua, 25 de set de 2019 às 22:08, Andrew Fish <afish@...>
escreveu:

Rafael,

What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL
should
contain?

I'm thinking if we had an EFI_TEXT_TO_SPEECH_PROTOCOL that driver
could
produce a PCM/Wave buffer, it could then use
EFI_AUDIO_OUTPUT_PROTOCOL
to play the sound.

I poked around my Mac at lunch and I can generate text to speech from
the command line into a wave file via the `say` command line tool. I
can
play the wave file from the command line via `afplay`. This gave me
the
idea of adding a EFI_AUDIO_OUTPUT_PROTOCOL and
EFI_TEXT_TO_SPEECH_PROTOCOL
driver to the EmulatorPkg that thunk down to the App to do the work.
With this we could prototype the UI part of accessibility. We could
also
debug the EFI EFI_TEXT_TO_SPEECH_PROTOCOL in this environment.

Thanks,

Andrew Fish

On Sep 21, 2019, at 5:36 AM, Rafael Machado <
rafaelrodrigues.machado@...> wrote:

Hi Everyone
Sorry for the delay on the response, to many things happening at the
same time.
I will try to answer e-mails to this thread every Saturday or Sunday
morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the
time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists,
and I >>can also represent what ever we come up with at the UEFI Spec
Work
Group.
During my MSc I had to study a lot the audio and BIOS architectures.
The
idea was to eliminate the first barrier to the creation of a screen
reader
for pre-OS environment, that was the lack of some open implementation
of
audio control and actions at UEFI. To do that I studied the Intel
High
Definition Audio Spec and a lot of UEFI specs to understand better
how
to
do that.
The initial target was to do all this development at OVMF, but as far
as
I could get, the sound card is not available to OVMF. as Laszlo
mentioned
at this e-mail there are some projects that may help on this, but at
the
time I was working on my MSc I didn't find this, so I did everything
on
a
real system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and
working on a MSc degree at the same time having a 40hours/week job,
being a
father and a husband is not an easy task, but it got to what I was
expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers",
that
later I noticed that are not mandatory. This is a simple C Major
scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA
memory access. For the ones that have good years, it's possible to
listen
some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was
that one of my write operations was causing some misunderstanding
between
the audio controller and the audio codec. The controller was sending
packets with 16bit depth, but the codec was processing them as 8bit
depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that
doing
at the OS level.
The reference code, that is just a proof-of-concept, and that has
several things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to
UEFI drivers after some discussion. Everything is released as BDS so
companies can use without IP problems.
Just to give some more information about the need of this kind of
solution. There is a lot of blind people that work with hardware
support,
so formatting disk, configuring RAID and booting dual-boot systems is
always a challenge to them. Even set the BIOS clock. How to do that
without
the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given
there
is no guideline on how a User Interface (UI) works. If accessibility
requires some from of hardware abstraction, like audio, then we could
likely get that into the UEFI Spec. What might be possible is an
EDK2
reference implementation of accessibility. Maybe we could use the
reference
implementation to write a UEFI white paper on design >>for
accessibility? I
there is an open source implementation, and an official design guide
this
would make it much easier for advocacy groups to lobby for this
feature.
I agree this is the way. Writing a white paper as an official EDK2
papers is one of my targets since the beginning of my MSc almost 5
years
ago.

I've got some experience with accessibility as the macOS EFI OS
Loader
has a UI for the Full Disk Encryption password. If you press the
power
button quickly 3 times at the disk encryption password prompt
accessibility is enabled and Voice Over gets turned on. You then
get
localized voice prompts when you move between UI elements. Since this
is
the OS loader all the resources are stored on the disk. You >>quickly
run
into a range of challenges since, audio is hard, abstracting audio is
hard
(what codec does firmware have to support), Audio files are not small
and
firmware is size constrained, the need to localize >>the audio
responses
causes even more size issues, the boot options are usually written by
an
OS
installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer,
so
no audio files would need to be stored. There are some open-source
implementations that are not GPL.
This was described at my MSc as future works that can continue what I
have started.

I listed a lot of reasons it is hard but as Kennedy stated in his
"We
choose to go to the Moon!" speech sometimes we chose to do things
"not
because they are easy, but because they are hard; because that >>goal
will
serve to organize and measure the best of our energies and skills,
because
that challenge is one that we are willing to accept". If we have a
design
that means we can break the problem up into >>smaller parts, and
maybe
we
can find people that have expertise in that part to build a chunk at
a
time. If we could implement the prototype in OVMF that would show how
it
works, but run on everyone's >>machines, so that would be really
helpful
for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible
future
steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with
proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver
to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also
producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application
remains functional.
- 4) Separate controller and codec code by creating
an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and
separate
out
the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder
audio files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the
need
of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my
thesis to English.
I wrote everything in Portuguese because there was not material about
UEFI to the Brazilian audience, and another target I have is to show
companies that we have people that can work at this kind of projects
in
Brazil, bringing this kind of development to south america. (Yes, I
have
complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@...>
escreveu:

On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the
time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the
UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like
audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility.
Maybe
we could use the reference implementation to write a UEFI white
paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it
much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the
disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to
support),
Audio files are not small and firmware is size constrained, the
need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his
"We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies
and
skills, because that challenge is one that we are willing to
accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on
everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard,
as
of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo


--
Signed,
Ethin D. Probst
--
Signed,
Ethin D. Probst


Re: UEFI accessibility mandate

Andrew Fish <afish@...>
 

Ethin,

I've made progress using the EmulatorPkg. I've ported my test C command line application into the EmulatorPkg Host (OS C command line application part of the EmulatorPkg), so the the host now produces APIs that support Text to Voice PCM files, and playing PCM, also called wave, files. There is a EmulatorPkg driver to produce Text To PCM, and a driver to play PCM files. I also wrote a test application. At this point the test application in the emulator can use the two APIs to produce Text To Voice.

Next steps for me are to write up a Bugzilla, cleanup the code, and make github branch to share the work.

Next steps for the community... From the EmulatorPkg it should be possible to implement the UI work to add voice. it should also be possible to implement a real EFI Voice to PCM driver. It should also be possible to use the test application to test a real EFI audio driver.

Thanks,

Andrew Fish

On Oct 29, 2019, at 1:17 PM, Ethin Probst <harlydavidsen@...> wrote:

Has there been any progress on this? I wish I could help, but I have
no experience in developing for EDK2...

On 10/1/19, Rafael Machado <rafaelrodrigues.machado@...> wrote:
Incredible!
I will find some time to try it at linux on the following weeks.
Thanks for the effort Andrew!

Rafael R. Machado

Em seg, 30 de set de 2019 às 22:18, Andrew Fish <afish@...> escreveu:



On Sep 30, 2019, at 11:50 AM, Rafael Machado <
rafaelrodrigues.machado@...> wrote:

Hi Andrew

As you have mentioned:
" I might find some time to try to get the EmulatorPkg mocking working."
Answer: This would be amazing! Thanks for trying to help!


Rafael,

I made a little progress this weekend watching American Football on TV,
and I had some time on an airplane today to add some comments.

I wrote a simple C command line program that does almost all the
primitive
audio functions we need to add Audio/Text-To-Speech to the EmulatorPkg.
Since ANSI C does not support audio I had to system() out to macOS
command
line tools to get some of the tasks complete. The only feature I'm
missing
is playing the PCM buffer asynchronously, and I may be able to add that
next weekend. I guessed at the Linux commands, it would be good if
someone
could debug the program on a Linux system, and also make sure I picked
the
right command line tools to use. I think for Windows there is going to
be
more coding involved, but I think there is at least a C API to play
audio,
but I only saw a C++ API for speech to text. But the good news is folks
can
use the simple C program to get the primitives working on Linux and
Windows
systems and it should be easy to port that work to the future EmulatorPkg
work.

What I've got working on macOS is:
1) Mute/Unmute
2) GetVolume/SetVolume
3) Text to PCD buffer (Wave file format)
a) Supports word per minute rate
b) Supports localizing the voice based on RFC 4646 language code used by
EFI (the program speaks in en, zh, and fr as a test)
4) PCD buffer to audio

This is how I was building on my Mac: clang -g -Wall -Werror audio.c

Short term anyone that has some free time if they could look at
testing/debugging audio.c on Linux and send feedback, also maybe some one
could start playing around with getting audio.c ported to Windows?

After I figure out getting async audio output working in audio.c I think
the next steps are working on the protocol/variable definitions. With the
protocol/variable definitions we should be able to add the capability to
the EmulatorPkg. At that point it should be possible for anyone to help
work on the accessibility UI. Obviously to get this working on a real
platform we will need a pure EFI Text to Speech driver (We could debug
that
in the EmulatorPkg). We would also need Intel HDA Audio drivers, and
Audio
drivers for OVFM, and other Audio hardware that is on peoples platforms.

Thanks,

Andrew Fish

The plan is to try to get some students to help on this also at the next
"google summer of code".
Not sure how it works, but since the idea is good, and it is a real need,
I believe we have chances to have some more hands working on it.

Thanks and Regards
Rafael R. Machado



Em qui, 26 de set de 2019 às 20:44, Andrew Fish <afish@...>
escreveu:



On Sep 26, 2019, at 5:15 AM, Rafael Machado <
rafaelrodrigues.machado@...> wrote:

Hi everyone.

About Ethin's question:
As for the boot manager, yes, that would be covered by the speech
synthesizer protocol. Perhaps we could extend this to the setup utility
too
(as well as all other things displayed on the screen)?
Answer: This is the target from my point of view. Since we have the
advantages of drivers at UEFI, there is no reason for not having
accessibility at pre-OS anymore. We could also think in future to add
other
kinds of accessibility, like the magnifying glass (that would be some
multi-layer version of the GOP driver), that would help low-vision
people.
Just one thing for everyone to think about. The "Convention on the
Rights
of Persons with Disabilities" say that a person has the right to access
information. So if someone buys a notebook, all information accessible
to
"common" people must be available to all kinds of people, in a way they
can
choose to use this information or not.
*" (e) Recognizing that disability is an evolving concept and that
disability results from the interaction between persons with impairments
and attitudinal and environmental barriers that hinders their full and
effective participation in society on an equal basis with others, ..."*
Details can be found at the United Nations web site
https://www.un.org/disabilities/documents/convention/convention_accessible_pdf.pdf


About Andrew's comments and questions:
What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL
should contain?
Answer: From my perspective, this protocol would need at least the
following functions (names may change after more discussion.)


- EFI_AUDIO_OUTPUT_PROTOCOL.LoadStream()
This function is intended to copy some data to the dma buffers that
were allocated previously by the AudioController driver. The initial idea
I
had was to add, in the case of my MSc, a HdaDriver.efi that is
responsible
to do all the HDA controller initialization (allocating dma buffers,
configuring the PCIe bus if needed). Not sure if this initialization
should
we added to the EFI_AUDIO_OUTPUT_PROTOCOL, because this could create
problems when using other controllers or concepts, like I believe ARM
platforms do.My understanding is that the platforms should have a driver
to
initialize the audio controller/codec, so the EFI_AUDIO_OUTPUT_PROTOCOL
is
just worried with loading and playing information.

- EFI_AUDIO_OUTPUT_PROTOCOL.PlayStream()
This function is responsible to play the stream. This is done by
setting one specific bit at the controllers configuration space (pointed
by
the bar[0] pcie configuration register at the pcie config space.) At
this
project I don't think we need to be worried with implementing an sound
server to enable multiple streams at a time. To the actions done by this
function are much simpler. Just play a stream.

- EFI_AUDIO_OUTPUT_PROTOCOL.StopStream()
This function is responsible to stop the stream play. The opposite
of
the previous one. Maybe we could use a single function that receives
some
parameter to start or stop the play.

- EFI_AUDIO_OUTPUT_PROTOCOL.SetVolume()
This function is responsible to set the volume to be used at all
nodes at the codec (so it depends on a EFI_AUDIO_CODEC_PROTOCOL that
knows
the commands a given codec needs to receive to process streams.)

- EFI_AUDIO_OUTPUT_PROTOCOL.SetStreamSpeed()
This function is responsible to set the speed a given stream should
be played on. Just to clarify, visual impaired people have an amazing
capacity of understanding sound streams in incredibly fast speeds. So
the
possibility of setting the stream processing speed is what makes them be
as
productive as a person with usual vision capacities.

- EFI_AUDIO_OUTPUT_PROTOCOL.UnloadStream()
This function is responsible to clean the buffers and letting them
prepared to receive other audio streams.

Still about Andrews comments.
This gave me the idea of adding a EFI_AUDIO_OUTPUT_PROTOCOL and
EFI_TEXT_TO_SPEECH_PROTOCOL
driver to the EmulatorPkg that thunk down to the App to do the work.
With this we could prototype the UI part of accessibility.
Answer: This would be great, because we would split the development. One
focusing on the HII and navigation (done normally using the tab key, so
each element at the screen would need an tab sequence number and a
accessibility tag with what the speech software should say), and another
focusing on the drivers/synthesizer/audio files depending on our future
decisions.

So my understanding is that the list of things is increasing. It is
actually something like this (not ordered by priority):
- EFI_AUDIO_OUTPUT_PROTOCOL: Protocol used to process the streams
- EFI_AUDIO_CODEC_PROTOCOL: Protocol used to communicate with the codec
- Audio configuration driver (HdaAudio.efi for example). Should this
create some other protocol, like EFI_AUDIO_CONTROLLER_PROTOCOL that is
responsible to configure the controller?


Rafael,

Generally how it works for a PCI device is it gets its Start() function
called to bind to the device. So the EFI_AUDIO_OUTPUT_PROTOCOL and
EFI_AUDIO_CODEC_PROTOCOL protocols would likely get installed on the PCI
IO
handle of the HDA PCI hardware. Some basic init will happen during the
Start() and the driver could defer some config until one of the member
functions is called the 1st time.

So for example on a normal boot the Audio driver may not even get
started. So there may have to be an nvram variable to start the driver.
We
might also need an NVRAM variable for the voice to text driver if it is
stored on disk.

- EFI_TEXT_TO_SPEECH_PROTOCOL: Protocol responsible to convert text to
speech (using the EFI_AUDIO_OUTPUT_PROTOCOL )
- HII changes to add tab navigation capabilities and accessibility tags
at each component on the screen

Are we aligned with the understanding?


I think so and if we can mock an implementation in the EmulatorPkg that
would let the HII UI work happen independently of the text to speech,
and
Audio driver work.

I'm on some air plaines in the next weeks (I'm at the airport right
now),
and my wife is heading to a conference after I get back from my trips.
Given my dogs are not very talkative I might find some time to try to
get
the EmulatorPkg mocking working.

Thanks,

Andrew Fish

I believe soon we will be prepared to talk about this on a design
meeting.

Thanks and Regards
Rafael R. Machado



Em qua, 25 de set de 2019 às 22:08, Andrew Fish <afish@...>
escreveu:

Rafael,

What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL
should
contain?

I'm thinking if we had an EFI_TEXT_TO_SPEECH_PROTOCOL that driver could
produce a PCM/Wave buffer, it could then use EFI_AUDIO_OUTPUT_PROTOCOL
to play the sound.

I poked around my Mac at lunch and I can generate text to speech from
the command line into a wave file via the `say` command line tool. I
can
play the wave file from the command line via `afplay`. This gave me the
idea of adding a EFI_AUDIO_OUTPUT_PROTOCOL and
EFI_TEXT_TO_SPEECH_PROTOCOL
driver to the EmulatorPkg that thunk down to the App to do the work.
With this we could prototype the UI part of accessibility. We could
also
debug the EFI EFI_TEXT_TO_SPEECH_PROTOCOL in this environment.

Thanks,

Andrew Fish

On Sep 21, 2019, at 5:36 AM, Rafael Machado <
rafaelrodrigues.machado@...> wrote:

Hi Everyone
Sorry for the delay on the response, to many things happening at the
same time.
I will try to answer e-mails to this thread every Saturday or Sunday
morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists,
and I >>can also represent what ever we come up with at the UEFI Spec
Work
Group.
During my MSc I had to study a lot the audio and BIOS architectures.
The
idea was to eliminate the first barrier to the creation of a screen
reader
for pre-OS environment, that was the lack of some open implementation
of
audio control and actions at UEFI. To do that I studied the Intel High
Definition Audio Spec and a lot of UEFI specs to understand better how
to
do that.
The initial target was to do all this development at OVMF, but as far
as
I could get, the sound card is not available to OVMF. as Laszlo
mentioned
at this e-mail there are some projects that may help on this, but at
the
time I was working on my MSc I didn't find this, so I did everything on
a
real system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and
working on a MSc degree at the same time having a 40hours/week job,
being a
father and a husband is not an easy task, but it got to what I was
expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that
later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA
memory access. For the ones that have good years, it's possible to
listen
some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was
that one of my write operations was causing some misunderstanding
between
the audio controller and the audio codec. The controller was sending
packets with 16bit depth, but the codec was processing them as 8bit
depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing
at the OS level.
The reference code, that is just a proof-of-concept, and that has
several things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to
UEFI drivers after some discussion. Everything is released as BDS so
companies can use without IP problems.
Just to give some more information about the need of this kind of
solution. There is a lot of blind people that work with hardware
support,
so formatting disk, configuring RAID and booting dual-boot systems is
always a challenge to them. Even set the BIOS clock. How to do that
without
the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given
there
is no guideline on how a User Interface (UI) works. If accessibility
requires some from of hardware abstraction, like audio, then we could
likely get that into the UEFI Spec. What might be possible is an EDK2
reference implementation of accessibility. Maybe we could use the
reference
implementation to write a UEFI white paper on design >>for
accessibility? I
there is an open source implementation, and an official design guide
this
would make it much easier for advocacy groups to lobby for this
feature.
I agree this is the way. Writing a white paper as an official EDK2
papers is one of my targets since the beginning of my MSc almost 5
years
ago.

I've got some experience with accessibility as the macOS EFI OS
Loader
has a UI for the Full Disk Encryption password. If you press the power
button quickly 3 times at the disk encryption password prompt
accessibility is enabled and Voice Over gets turned on. You then get
localized voice prompts when you move between UI elements. Since this
is
the OS loader all the resources are stored on the disk. You >>quickly
run
into a range of challenges since, audio is hard, abstracting audio is
hard
(what codec does firmware have to support), Audio files are not small
and
firmware is size constrained, the need to localize >>the audio
responses
causes even more size issues, the boot options are usually written by an
OS
installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so
no audio files would need to be stored. There are some open-source
implementations that are not GPL.
This was described at my MSc as future works that can continue what I
have started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things "not
because they are easy, but because they are hard; because that >>goal
will
serve to organize and measure the best of our energies and skills,
because
that challenge is one that we are willing to accept". If we have a
design
that means we can break the problem up into >>smaller parts, and maybe
we
can find people that have expertise in that part to build a chunk at a
time. If we could implement the prototype in OVMF that would show how
it
works, but run on everyone's >>machines, so that would be really
helpful
for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible
future
steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with
proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver
to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also
producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application
remains functional.
- 4) Separate controller and codec code by creating
an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate
out
the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder
audio files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the
need
of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my
thesis to English.
I wrote everything in Portuguese because there was not material about
UEFI to the Brazilian audience, and another target I have is to show
companies that we have people that can work at this kind of projects in
Brazil, bringing this kind of development to south america. (Yes, I
have
complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@...>
escreveu:

On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the
time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the
UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like
audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white
paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the
disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to
support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his
"We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to
accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as
of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo


--
Signed,
Ethin D. Probst


Re: UEFI accessibility mandate

Rafael Machado <rafaelrodrigues.machado@...>
 

Hi Ethin

I think you can start by studying the EmulatorPkg at the edk2 repository.

Try to compile and use it, and after that you could try the software
developed by Andrew to check if it works correctly at your system.

About the Uefi technology, in case you want to start studying it before
going to the EmulatorPkg, you can take a look at this training:
https://github.com/tianocore/tianocore.github.io/wiki/UEFI%20EDKII%20Learning%20Dev

Thanks and Regards
Rafael



Em ter, 29 de out de 2019 17:17, Ethin Probst <harlydavidsen@...>
escreveu:

Has there been any progress on this? I wish I could help, but I have
no experience in developing for EDK2...

On 10/1/19, Rafael Machado <rafaelrodrigues.machado@...> wrote:
Incredible!
I will find some time to try it at linux on the following weeks.
Thanks for the effort Andrew!

Rafael R. Machado

Em seg, 30 de set de 2019 às 22:18, Andrew Fish <afish@...>
escreveu:



On Sep 30, 2019, at 11:50 AM, Rafael Machado <
rafaelrodrigues.machado@...> wrote:

Hi Andrew

As you have mentioned:
" I might find some time to try to get the EmulatorPkg mocking working."
Answer: This would be amazing! Thanks for trying to help!


Rafael,

I made a little progress this weekend watching American Football on TV,
and I had some time on an airplane today to add some comments.

I wrote a simple C command line program that does almost all the
primitive
audio functions we need to add Audio/Text-To-Speech to the EmulatorPkg.
Since ANSI C does not support audio I had to system() out to macOS
command
line tools to get some of the tasks complete. The only feature I'm
missing
is playing the PCM buffer asynchronously, and I may be able to add that
next weekend. I guessed at the Linux commands, it would be good if
someone
could debug the program on a Linux system, and also make sure I picked
the
right command line tools to use. I think for Windows there is going to
be
more coding involved, but I think there is at least a C API to play
audio,
but I only saw a C++ API for speech to text. But the good news is folks
can
use the simple C program to get the primitives working on Linux and
Windows
systems and it should be easy to port that work to the future
EmulatorPkg
work.

What I've got working on macOS is:
1) Mute/Unmute
2) GetVolume/SetVolume
3) Text to PCD buffer (Wave file format)
a) Supports word per minute rate
b) Supports localizing the voice based on RFC 4646 language code used by
EFI (the program speaks in en, zh, and fr as a test)
4) PCD buffer to audio

This is how I was building on my Mac: clang -g -Wall -Werror audio.c

Short term anyone that has some free time if they could look at
testing/debugging audio.c on Linux and send feedback, also maybe some
one
could start playing around with getting audio.c ported to Windows?

After I figure out getting async audio output working in audio.c I think
the next steps are working on the protocol/variable definitions. With
the
protocol/variable definitions we should be able to add the capability to
the EmulatorPkg. At that point it should be possible for anyone to help
work on the accessibility UI. Obviously to get this working on a real
platform we will need a pure EFI Text to Speech driver (We could debug
that
in the EmulatorPkg). We would also need Intel HDA Audio drivers, and
Audio
drivers for OVFM, and other Audio hardware that is on peoples platforms.

Thanks,

Andrew Fish

The plan is to try to get some students to help on this also at the next
"google summer of code".
Not sure how it works, but since the idea is good, and it is a real
need,
I believe we have chances to have some more hands working on it.

Thanks and Regards
Rafael R. Machado



Em qui, 26 de set de 2019 às 20:44, Andrew Fish <afish@...>
escreveu:



On Sep 26, 2019, at 5:15 AM, Rafael Machado <
rafaelrodrigues.machado@...> wrote:

Hi everyone.

About Ethin's question:
As for the boot manager, yes, that would be covered by the speech
synthesizer protocol. Perhaps we could extend this to the setup utility
too
(as well as all other things displayed on the screen)?
Answer: This is the target from my point of view. Since we have the
advantages of drivers at UEFI, there is no reason for not having
accessibility at pre-OS anymore. We could also think in future to add
other
kinds of accessibility, like the magnifying glass (that would be some
multi-layer version of the GOP driver), that would help low-vision
people.
Just one thing for everyone to think about. The "Convention on the
Rights
of Persons with Disabilities" say that a person has the right to access
information. So if someone buys a notebook, all information accessible
to
"common" people must be available to all kinds of people, in a way they
can
choose to use this information or not.
*" (e) Recognizing that disability is an evolving concept and that
disability results from the interaction between persons with
impairments
and attitudinal and environmental barriers that hinders their full and
effective participation in society on an equal basis with others, ..."*
Details can be found at the United Nations web site
https://www.un.org/disabilities/documents/convention/convention_accessible_pdf.pdf


About Andrew's comments and questions:
What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL
should contain?
Answer: From my perspective, this protocol would need at least the
following functions (names may change after more discussion.)


- EFI_AUDIO_OUTPUT_PROTOCOL.LoadStream()
This function is intended to copy some data to the dma buffers that
were allocated previously by the AudioController driver. The initial
idea
I
had was to add, in the case of my MSc, a HdaDriver.efi that is
responsible
to do all the HDA controller initialization (allocating dma buffers,
configuring the PCIe bus if needed). Not sure if this initialization
should
we added to the EFI_AUDIO_OUTPUT_PROTOCOL, because this could create
problems when using other controllers or concepts, like I believe ARM
platforms do.My understanding is that the platforms should have a
driver
to
initialize the audio controller/codec, so the
EFI_AUDIO_OUTPUT_PROTOCOL
is
just worried with loading and playing information.

- EFI_AUDIO_OUTPUT_PROTOCOL.PlayStream()
This function is responsible to play the stream. This is done by
setting one specific bit at the controllers configuration space
(pointed
by
the bar[0] pcie configuration register at the pcie config space.) At
this
project I don't think we need to be worried with implementing an sound
server to enable multiple streams at a time. To the actions done by
this
function are much simpler. Just play a stream.

- EFI_AUDIO_OUTPUT_PROTOCOL.StopStream()
This function is responsible to stop the stream play. The opposite
of
the previous one. Maybe we could use a single function that receives
some
parameter to start or stop the play.

- EFI_AUDIO_OUTPUT_PROTOCOL.SetVolume()
This function is responsible to set the volume to be used at all
nodes at the codec (so it depends on a EFI_AUDIO_CODEC_PROTOCOL that
knows
the commands a given codec needs to receive to process streams.)

- EFI_AUDIO_OUTPUT_PROTOCOL.SetStreamSpeed()
This function is responsible to set the speed a given stream should
be played on. Just to clarify, visual impaired people have an amazing
capacity of understanding sound streams in incredibly fast speeds. So
the
possibility of setting the stream processing speed is what makes them
be
as
productive as a person with usual vision capacities.

- EFI_AUDIO_OUTPUT_PROTOCOL.UnloadStream()
This function is responsible to clean the buffers and letting them
prepared to receive other audio streams.

Still about Andrews comments.
This gave me the idea of adding a EFI_AUDIO_OUTPUT_PROTOCOL and
EFI_TEXT_TO_SPEECH_PROTOCOL
driver to the EmulatorPkg that thunk down to the App to do the work.
With this we could prototype the UI part of accessibility.
Answer: This would be great, because we would split the development.
One
focusing on the HII and navigation (done normally using the tab key, so
each element at the screen would need an tab sequence number and a
accessibility tag with what the speech software should say), and
another
focusing on the drivers/synthesizer/audio files depending on our future
decisions.

So my understanding is that the list of things is increasing. It is
actually something like this (not ordered by priority):
- EFI_AUDIO_OUTPUT_PROTOCOL: Protocol used to process the streams
- EFI_AUDIO_CODEC_PROTOCOL: Protocol used to communicate with the codec
- Audio configuration driver (HdaAudio.efi for example). Should this
create some other protocol, like EFI_AUDIO_CONTROLLER_PROTOCOL that is
responsible to configure the controller?


Rafael,

Generally how it works for a PCI device is it gets its Start() function
called to bind to the device. So the EFI_AUDIO_OUTPUT_PROTOCOL and
EFI_AUDIO_CODEC_PROTOCOL protocols would likely get installed on the
PCI
IO
handle of the HDA PCI hardware. Some basic init will happen during the
Start() and the driver could defer some config until one of the member
functions is called the 1st time.

So for example on a normal boot the Audio driver may not even get
started. So there may have to be an nvram variable to start the driver.
We
might also need an NVRAM variable for the voice to text driver if it is
stored on disk.

- EFI_TEXT_TO_SPEECH_PROTOCOL: Protocol responsible to convert text to
speech (using the EFI_AUDIO_OUTPUT_PROTOCOL )
- HII changes to add tab navigation capabilities and accessibility tags
at each component on the screen

Are we aligned with the understanding?


I think so and if we can mock an implementation in the EmulatorPkg that
would let the HII UI work happen independently of the text to speech,
and
Audio driver work.

I'm on some air plaines in the next weeks (I'm at the airport right
now),
and my wife is heading to a conference after I get back from my trips.
Given my dogs are not very talkative I might find some time to try to
get
the EmulatorPkg mocking working.

Thanks,

Andrew Fish

I believe soon we will be prepared to talk about this on a design
meeting.

Thanks and Regards
Rafael R. Machado



Em qua, 25 de set de 2019 às 22:08, Andrew Fish <afish@...>
escreveu:

Rafael,

What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL
should
contain?

I'm thinking if we had an EFI_TEXT_TO_SPEECH_PROTOCOL that driver
could
produce a PCM/Wave buffer, it could then use EFI_AUDIO_OUTPUT_PROTOCOL
to play the sound.

I poked around my Mac at lunch and I can generate text to speech from
the command line into a wave file via the `say` command line tool. I
can
play the wave file from the command line via `afplay`. This gave me
the
idea of adding a EFI_AUDIO_OUTPUT_PROTOCOL and
EFI_TEXT_TO_SPEECH_PROTOCOL
driver to the EmulatorPkg that thunk down to the App to do the work.
With this we could prototype the UI part of accessibility. We could
also
debug the EFI EFI_TEXT_TO_SPEECH_PROTOCOL in this environment.

Thanks,

Andrew Fish

On Sep 21, 2019, at 5:36 AM, Rafael Machado <
rafaelrodrigues.machado@...> wrote:

Hi Everyone
Sorry for the delay on the response, to many things happening at the
same time.
I will try to answer e-mails to this thread every Saturday or Sunday
morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the
time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists,
and I >>can also represent what ever we come up with at the UEFI Spec
Work
Group.
During my MSc I had to study a lot the audio and BIOS architectures.
The
idea was to eliminate the first barrier to the creation of a screen
reader
for pre-OS environment, that was the lack of some open implementation
of
audio control and actions at UEFI. To do that I studied the Intel High
Definition Audio Spec and a lot of UEFI specs to understand better how
to
do that.
The initial target was to do all this development at OVMF, but as far
as
I could get, the sound card is not available to OVMF. as Laszlo
mentioned
at this e-mail there are some projects that may help on this, but at
the
time I was working on my MSc I didn't find this, so I did everything
on
a
real system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and
working on a MSc degree at the same time having a 40hours/week job,
being a
father and a husband is not an easy task, but it got to what I was
expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that
later I noticed that are not mandatory. This is a simple C Major
scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA
memory access. For the ones that have good years, it's possible to
listen
some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was
that one of my write operations was causing some misunderstanding
between
the audio controller and the audio codec. The controller was sending
packets with 16bit depth, but the codec was processing them as 8bit
depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing
at the OS level.
The reference code, that is just a proof-of-concept, and that has
several things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to
UEFI drivers after some discussion. Everything is released as BDS so
companies can use without IP problems.
Just to give some more information about the need of this kind of
solution. There is a lot of blind people that work with hardware
support,
so formatting disk, configuring RAID and booting dual-boot systems is
always a challenge to them. Even set the BIOS clock. How to do that
without
the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given
there
is no guideline on how a User Interface (UI) works. If accessibility
requires some from of hardware abstraction, like audio, then we could
likely get that into the UEFI Spec. What might be possible is an
EDK2
reference implementation of accessibility. Maybe we could use the
reference
implementation to write a UEFI white paper on design >>for
accessibility? I
there is an open source implementation, and an official design guide
this
would make it much easier for advocacy groups to lobby for this
feature.
I agree this is the way. Writing a white paper as an official EDK2
papers is one of my targets since the beginning of my MSc almost 5
years
ago.

I've got some experience with accessibility as the macOS EFI OS
Loader
has a UI for the Full Disk Encryption password. If you press the power
button quickly 3 times at the disk encryption password prompt
accessibility is enabled and Voice Over gets turned on. You then get
localized voice prompts when you move between UI elements. Since this
is
the OS loader all the resources are stored on the disk. You >>quickly
run
into a range of challenges since, audio is hard, abstracting audio is
hard
(what codec does firmware have to support), Audio files are not small
and
firmware is size constrained, the need to localize >>the audio
responses
causes even more size issues, the boot options are usually written by
an
OS
installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer,
so
no audio files would need to be stored. There are some open-source
implementations that are not GPL.
This was described at my MSc as future works that can continue what I
have started.

I listed a lot of reasons it is hard but as Kennedy stated in his
"We
choose to go to the Moon!" speech sometimes we chose to do things
"not
because they are easy, but because they are hard; because that >>goal
will
serve to organize and measure the best of our energies and skills,
because
that challenge is one that we are willing to accept". If we have a
design
that means we can break the problem up into >>smaller parts, and maybe
we
can find people that have expertise in that part to build a chunk at a
time. If we could implement the prototype in OVMF that would show how
it
works, but run on everyone's >>machines, so that would be really
helpful
for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible
future
steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with
proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver
to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also
producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application
remains functional.
- 4) Separate controller and codec code by creating
an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and
separate
out
the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder
audio files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the
need
of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my
thesis to English.
I wrote everything in Portuguese because there was not material about
UEFI to the Brazilian audience, and another target I have is to show
companies that we have people that can work at this kind of projects
in
Brazil, bringing this kind of development to south america. (Yes, I
have
complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@...>
escreveu:

On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the
time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the
UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like
audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility.
Maybe
we could use the reference implementation to write a UEFI white
paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it
much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the
disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to
support),
Audio files are not small and firmware is size constrained, the
need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his
"We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies
and
skills, because that challenge is one that we are willing to
accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on
everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as
of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo


--
Signed,
Ethin D. Probst


Re: UEFI accessibility mandate

Ethin Probst
 

Has there been any progress on this? I wish I could help, but I have
no experience in developing for EDK2...

On 10/1/19, Rafael Machado <rafaelrodrigues.machado@...> wrote:
Incredible!
I will find some time to try it at linux on the following weeks.
Thanks for the effort Andrew!

Rafael R. Machado

Em seg, 30 de set de 2019 às 22:18, Andrew Fish <afish@...> escreveu:



On Sep 30, 2019, at 11:50 AM, Rafael Machado <
rafaelrodrigues.machado@...> wrote:

Hi Andrew

As you have mentioned:
" I might find some time to try to get the EmulatorPkg mocking working."
Answer: This would be amazing! Thanks for trying to help!


Rafael,

I made a little progress this weekend watching American Football on TV,
and I had some time on an airplane today to add some comments.

I wrote a simple C command line program that does almost all the
primitive
audio functions we need to add Audio/Text-To-Speech to the EmulatorPkg.
Since ANSI C does not support audio I had to system() out to macOS
command
line tools to get some of the tasks complete. The only feature I'm
missing
is playing the PCM buffer asynchronously, and I may be able to add that
next weekend. I guessed at the Linux commands, it would be good if
someone
could debug the program on a Linux system, and also make sure I picked
the
right command line tools to use. I think for Windows there is going to
be
more coding involved, but I think there is at least a C API to play
audio,
but I only saw a C++ API for speech to text. But the good news is folks
can
use the simple C program to get the primitives working on Linux and
Windows
systems and it should be easy to port that work to the future EmulatorPkg
work.

What I've got working on macOS is:
1) Mute/Unmute
2) GetVolume/SetVolume
3) Text to PCD buffer (Wave file format)
a) Supports word per minute rate
b) Supports localizing the voice based on RFC 4646 language code used by
EFI (the program speaks in en, zh, and fr as a test)
4) PCD buffer to audio

This is how I was building on my Mac: clang -g -Wall -Werror audio.c

Short term anyone that has some free time if they could look at
testing/debugging audio.c on Linux and send feedback, also maybe some one
could start playing around with getting audio.c ported to Windows?

After I figure out getting async audio output working in audio.c I think
the next steps are working on the protocol/variable definitions. With the
protocol/variable definitions we should be able to add the capability to
the EmulatorPkg. At that point it should be possible for anyone to help
work on the accessibility UI. Obviously to get this working on a real
platform we will need a pure EFI Text to Speech driver (We could debug
that
in the EmulatorPkg). We would also need Intel HDA Audio drivers, and
Audio
drivers for OVFM, and other Audio hardware that is on peoples platforms.

Thanks,

Andrew Fish

The plan is to try to get some students to help on this also at the next
"google summer of code".
Not sure how it works, but since the idea is good, and it is a real need,
I believe we have chances to have some more hands working on it.

Thanks and Regards
Rafael R. Machado



Em qui, 26 de set de 2019 às 20:44, Andrew Fish <afish@...>
escreveu:



On Sep 26, 2019, at 5:15 AM, Rafael Machado <
rafaelrodrigues.machado@...> wrote:

Hi everyone.

About Ethin's question:
As for the boot manager, yes, that would be covered by the speech
synthesizer protocol. Perhaps we could extend this to the setup utility
too
(as well as all other things displayed on the screen)?
Answer: This is the target from my point of view. Since we have the
advantages of drivers at UEFI, there is no reason for not having
accessibility at pre-OS anymore. We could also think in future to add
other
kinds of accessibility, like the magnifying glass (that would be some
multi-layer version of the GOP driver), that would help low-vision
people.
Just one thing for everyone to think about. The "Convention on the
Rights
of Persons with Disabilities" say that a person has the right to access
information. So if someone buys a notebook, all information accessible
to
"common" people must be available to all kinds of people, in a way they
can
choose to use this information or not.
*" (e) Recognizing that disability is an evolving concept and that
disability results from the interaction between persons with impairments
and attitudinal and environmental barriers that hinders their full and
effective participation in society on an equal basis with others, ..."*
Details can be found at the United Nations web site
https://www.un.org/disabilities/documents/convention/convention_accessible_pdf.pdf


About Andrew's comments and questions:
What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL
should contain?
Answer: From my perspective, this protocol would need at least the
following functions (names may change after more discussion.)


- EFI_AUDIO_OUTPUT_PROTOCOL.LoadStream()
This function is intended to copy some data to the dma buffers that
were allocated previously by the AudioController driver. The initial idea
I
had was to add, in the case of my MSc, a HdaDriver.efi that is
responsible
to do all the HDA controller initialization (allocating dma buffers,
configuring the PCIe bus if needed). Not sure if this initialization
should
we added to the EFI_AUDIO_OUTPUT_PROTOCOL, because this could create
problems when using other controllers or concepts, like I believe ARM
platforms do.My understanding is that the platforms should have a driver
to
initialize the audio controller/codec, so the EFI_AUDIO_OUTPUT_PROTOCOL
is
just worried with loading and playing information.

- EFI_AUDIO_OUTPUT_PROTOCOL.PlayStream()
This function is responsible to play the stream. This is done by
setting one specific bit at the controllers configuration space (pointed
by
the bar[0] pcie configuration register at the pcie config space.) At
this
project I don't think we need to be worried with implementing an sound
server to enable multiple streams at a time. To the actions done by this
function are much simpler. Just play a stream.

- EFI_AUDIO_OUTPUT_PROTOCOL.StopStream()
This function is responsible to stop the stream play. The opposite
of
the previous one. Maybe we could use a single function that receives
some
parameter to start or stop the play.

- EFI_AUDIO_OUTPUT_PROTOCOL.SetVolume()
This function is responsible to set the volume to be used at all
nodes at the codec (so it depends on a EFI_AUDIO_CODEC_PROTOCOL that
knows
the commands a given codec needs to receive to process streams.)

- EFI_AUDIO_OUTPUT_PROTOCOL.SetStreamSpeed()
This function is responsible to set the speed a given stream should
be played on. Just to clarify, visual impaired people have an amazing
capacity of understanding sound streams in incredibly fast speeds. So
the
possibility of setting the stream processing speed is what makes them be
as
productive as a person with usual vision capacities.

- EFI_AUDIO_OUTPUT_PROTOCOL.UnloadStream()
This function is responsible to clean the buffers and letting them
prepared to receive other audio streams.

Still about Andrews comments.
This gave me the idea of adding a EFI_AUDIO_OUTPUT_PROTOCOL and
EFI_TEXT_TO_SPEECH_PROTOCOL
driver to the EmulatorPkg that thunk down to the App to do the work.
With this we could prototype the UI part of accessibility.
Answer: This would be great, because we would split the development. One
focusing on the HII and navigation (done normally using the tab key, so
each element at the screen would need an tab sequence number and a
accessibility tag with what the speech software should say), and another
focusing on the drivers/synthesizer/audio files depending on our future
decisions.

So my understanding is that the list of things is increasing. It is
actually something like this (not ordered by priority):
- EFI_AUDIO_OUTPUT_PROTOCOL: Protocol used to process the streams
- EFI_AUDIO_CODEC_PROTOCOL: Protocol used to communicate with the codec
- Audio configuration driver (HdaAudio.efi for example). Should this
create some other protocol, like EFI_AUDIO_CONTROLLER_PROTOCOL that is
responsible to configure the controller?


Rafael,

Generally how it works for a PCI device is it gets its Start() function
called to bind to the device. So the EFI_AUDIO_OUTPUT_PROTOCOL and
EFI_AUDIO_CODEC_PROTOCOL protocols would likely get installed on the PCI
IO
handle of the HDA PCI hardware. Some basic init will happen during the
Start() and the driver could defer some config until one of the member
functions is called the 1st time.

So for example on a normal boot the Audio driver may not even get
started. So there may have to be an nvram variable to start the driver.
We
might also need an NVRAM variable for the voice to text driver if it is
stored on disk.

- EFI_TEXT_TO_SPEECH_PROTOCOL: Protocol responsible to convert text to
speech (using the EFI_AUDIO_OUTPUT_PROTOCOL )
- HII changes to add tab navigation capabilities and accessibility tags
at each component on the screen

Are we aligned with the understanding?


I think so and if we can mock an implementation in the EmulatorPkg that
would let the HII UI work happen independently of the text to speech,
and
Audio driver work.

I'm on some air plaines in the next weeks (I'm at the airport right
now),
and my wife is heading to a conference after I get back from my trips.
Given my dogs are not very talkative I might find some time to try to
get
the EmulatorPkg mocking working.

Thanks,

Andrew Fish

I believe soon we will be prepared to talk about this on a design
meeting.

Thanks and Regards
Rafael R. Machado



Em qua, 25 de set de 2019 às 22:08, Andrew Fish <afish@...>
escreveu:

Rafael,

What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL
should
contain?

I'm thinking if we had an EFI_TEXT_TO_SPEECH_PROTOCOL that driver could
produce a PCM/Wave buffer, it could then use EFI_AUDIO_OUTPUT_PROTOCOL
to play the sound.

I poked around my Mac at lunch and I can generate text to speech from
the command line into a wave file via the `say` command line tool. I
can
play the wave file from the command line via `afplay`. This gave me the
idea of adding a EFI_AUDIO_OUTPUT_PROTOCOL and
EFI_TEXT_TO_SPEECH_PROTOCOL
driver to the EmulatorPkg that thunk down to the App to do the work.
With this we could prototype the UI part of accessibility. We could
also
debug the EFI EFI_TEXT_TO_SPEECH_PROTOCOL in this environment.

Thanks,

Andrew Fish

On Sep 21, 2019, at 5:36 AM, Rafael Machado <
rafaelrodrigues.machado@...> wrote:

Hi Everyone
Sorry for the delay on the response, to many things happening at the
same time.
I will try to answer e-mails to this thread every Saturday or Sunday
morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists,
and I >>can also represent what ever we come up with at the UEFI Spec
Work
Group.
During my MSc I had to study a lot the audio and BIOS architectures.
The
idea was to eliminate the first barrier to the creation of a screen
reader
for pre-OS environment, that was the lack of some open implementation
of
audio control and actions at UEFI. To do that I studied the Intel High
Definition Audio Spec and a lot of UEFI specs to understand better how
to
do that.
The initial target was to do all this development at OVMF, but as far
as
I could get, the sound card is not available to OVMF. as Laszlo
mentioned
at this e-mail there are some projects that may help on this, but at
the
time I was working on my MSc I didn't find this, so I did everything on
a
real system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and
working on a MSc degree at the same time having a 40hours/week job,
being a
father and a husband is not an easy task, but it got to what I was
expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that
later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA
memory access. For the ones that have good years, it's possible to
listen
some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was
that one of my write operations was causing some misunderstanding
between
the audio controller and the audio codec. The controller was sending
packets with 16bit depth, but the codec was processing them as 8bit
depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing
at the OS level.
The reference code, that is just a proof-of-concept, and that has
several things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to
UEFI drivers after some discussion. Everything is released as BDS so
companies can use without IP problems.
Just to give some more information about the need of this kind of
solution. There is a lot of blind people that work with hardware
support,
so formatting disk, configuring RAID and booting dual-boot systems is
always a challenge to them. Even set the BIOS clock. How to do that
without
the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given
there
is no guideline on how a User Interface (UI) works. If accessibility
requires some from of hardware abstraction, like audio, then we could
likely get that into the UEFI Spec. What might be possible is an EDK2
reference implementation of accessibility. Maybe we could use the
reference
implementation to write a UEFI white paper on design >>for
accessibility? I
there is an open source implementation, and an official design guide
this
would make it much easier for advocacy groups to lobby for this
feature.
I agree this is the way. Writing a white paper as an official EDK2
papers is one of my targets since the beginning of my MSc almost 5
years
ago.

I've got some experience with accessibility as the macOS EFI OS
Loader
has a UI for the Full Disk Encryption password. If you press the power
button quickly 3 times at the disk encryption password prompt
accessibility is enabled and Voice Over gets turned on. You then get
localized voice prompts when you move between UI elements. Since this
is
the OS loader all the resources are stored on the disk. You >>quickly
run
into a range of challenges since, audio is hard, abstracting audio is
hard
(what codec does firmware have to support), Audio files are not small
and
firmware is size constrained, the need to localize >>the audio
responses
causes even more size issues, the boot options are usually written by an
OS
installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so
no audio files would need to be stored. There are some open-source
implementations that are not GPL.
This was described at my MSc as future works that can continue what I
have started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things "not
because they are easy, but because they are hard; because that >>goal
will
serve to organize and measure the best of our energies and skills,
because
that challenge is one that we are willing to accept". If we have a
design
that means we can break the problem up into >>smaller parts, and maybe
we
can find people that have expertise in that part to build a chunk at a
time. If we could implement the prototype in OVMF that would show how
it
works, but run on everyone's >>machines, so that would be really
helpful
for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible
future
steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with
proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver
to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also
producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application
remains functional.
- 4) Separate controller and codec code by creating
an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate
out
the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder
audio files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the
need
of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my
thesis to English.
I wrote everything in Portuguese because there was not material about
UEFI to the Brazilian audience, and another target I have is to show
companies that we have people that can work at this kind of projects in
Brazil, bringing this kind of development to south america. (Yes, I
have
complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@...>
escreveu:

On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the
time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the
UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like
audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white
paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the
disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to
support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his
"We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to
accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as
of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo

--
Signed,
Ethin D. Probst


Re: [edk2-devel] [Qemu-devel] [PATCH 1/2] q35: implement 128K SMRAM at default SMBASE address

Laszlo Ersek
 

On 10/04/19 13:31, Igor Mammedov wrote:
On Tue, 1 Oct 2019 20:03:20 +0200
"Laszlo Ersek" <lersek@...> wrote:
(1) What values to use.
SeaBIOS writes 0x00 into command port, but it seems that's taken by
EFI_SMM_COMMUNICATION_PROTOCOL. So we can use the next unused value
(lets say 0x4). We probably don't have to use status port or
EFI_SMM_COMMUNICATION_PROTOCOL, since the value of written into 0xB2
is sufficient to distinguish hotplug event.
Thanks. Can you please write a QEMU patch for the ACPI generator such
that hotplugging a VCPU writes value 4 to IO port 0xB2?

That will allow me to experiment with OVMF.

(I can experiment with some other parts in edk2 even before that.)

(2) How the parameters are passed.


(2a) For the new CPU, the SMI remains pending, until it gets an
INIT-SIPI-SIPI from one of the previously plugged CPUs (most likely, the
BSP). At that point, the new CPU will execute the "initial SMI handler
for hotplugged CPUs", at the default SMBASE.

That's a routine we'll have to write in assembly, from zero. In this
routine, we can read back IO ports 0xB2 and 0xB3. And QEMU will be happy
to provide the values last written (see apm_ioport_readb() in
"hw/isa/apm.c"). So we can receive the values in this routine. Alright.
Potentially we can can avoid writing custom SMI handler,
what do you think about following workflow:

on system boot after initial CPUs relocation, firmware set NOP SMI handler
at default SMBASE.
Then as reaction to GPE triggered SMI (on cpu hotplug), after SMI rendezvous,
a host cpu reads IO port 0xB2 and does hotplugged CPUs enumeration.

a) assuming we allow hotplug only in case of negotiated SMI broadcast
host CPU shoots down all in-flight INIT/SIPI/SIPI for hotpugged CPUs
to avoid race within relocation handler.
How is that "shootdown" possible?

After that host CPU in loop

b) it prepares/initializes necessary CPU structures for a hotplugged
CPU if necessary and replaces NOP SMI handler with the relocation
SMI handler that is used during system boot.

c) a host CPU sends NOP INIT/SIPI/SIPI to the hotplugged CPU

d) the woken up hotplugged CPU, jumps to default SMBASE and
executes hotplug relocation handler.

e) after the hotplugged CPU is relocated and if there are more
hotplugged CPUs, a host CPU repeats b-d steps for the next
hotplugged CPU.

f) after all CPUs are relocated, restore NOP SMI handler at default
SMBASE.
Thanks
Laszlo


Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

Sean
 

Rebecca,

I have not setup or integrated a Doxygen build as part of this. I think that is something that should happen and it should be done by a cloud build on a regular cadence but is independent of getting phase 1 CI up and running.

I would like to see a comprehensive plan for documentation. Spell check, image support, content other than code comments, and finally a defined expectation for contributions along with the code would be my initial requirements. Also hosting stable tag versions as well as master version. I think this can be its own RFC.

Thanks
Sean


Re: [edk2-devel] [Qemu-devel] [PATCH 1/2] q35: implement 128K SMRAM at default SMBASE address

Igor Mammedov <imammedo@...>
 

On Tue, 1 Oct 2019 20:03:20 +0200
"Laszlo Ersek" <lersek@...> wrote:

On 09/30/19 16:22, Yao, Jiewen wrote:

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Igor
Mammedov
Sent: Monday, September 30, 2019 8:37 PM
To: Laszlo Ersek <lersek@...>
To me it looks like we need to figure out how QEMU can make the OS call
into SMM (in the GPE cpu hotplug handler), passing in parameters and
such. This would be step (03).

Do you agree?

If so, I'll ask Jiewen about such OS->SMM calls separately, because I
seem to remember that there used to be an "SMM communcation table" of
sorts, for flexible OS->SMM calls. However, it appears to be deprecated
lately.
we can try to resurrect and put over it some kind of protocol
to describe which CPUs to where hotplugged.

or we could put a parameter into SMI status register (IO port 0xb3)
and the trigger SMI from GPE handler to tell SMI handler that cpu
hotplug happened and then use QEMU's cpu hotplug interface
to enumerate hotplugged CPUs for SMI handler.

The later is probably simpler as we won't need to reinvent the wheel
(just reuse the interface that's already in use by GPE handler).
Based on "docs/specs/acpi_cpu_hotplug.txt", this seems to boil down to a
bunch of IO port accesses at base 0x0cd8.

Is that correct?
yep, you can use it to iterate over hotplugged CPUs.
hw side (QEMU) uses cpu_hotplug_ops as IO write/read handlers
and firmware side (ACPI) scannig for hotplugged CPUs is implemented
in CPU_SCAN_METHOD.

What we can do on QEMU side is to write agreed upon value to command port (0xB2)
from CPU_SCAN_METHOD after taking ctrl_lock but before starting scan loop.
That way firmware will first bring up (from fw pov) all hotplugged CPUs
and then return control to OS to do the same from OS pov.



[Jiewen] The PI specification Volume 4 - SMM defines EFI_MM_COMMUNICATION_PROTOCOL.Communicate() - It can be used to communicate between OS and SMM handler. But it requires the runtime protocol call. I am not sure how OS loader passes this information to OS kernel.

As such, I think using ACPI SCI/GPE -> software SMI handler is an easier way to achieve this. I also recommend this way.
For parameter passing, we can use 1) Port B2 (1 byte), 2) Port B3 (1 byte), 3) chipset scratch register (4 bytes or 8 bytes, based upon scratch register size), 4) ACPI NVS OPREGION, if the data structure is complicated.
I'm confused about the details. In two categories:
(1) what values to use,
(2) how those values are passed.

Assume a CPU is hotpluged, QEMU injects an SCI, and the ACPI GPE handler
in the OS -- which also originates from QEMU -- writes a particular byte
to the Data port (0xB3), and then to the Control port (0xB2),
broadcasting an SMI.

(1) What values to use.

Note that values ICH9_APM_ACPI_ENABLE (2) and ICH9_APM_ACPI_DISABLE (3)
are already reserved in QEMU for IO port 0xB2, for different purposes.
So we can't use those in the GPE handler.
SeaBIOS writes 0x00 into command port, but it seems that's taken by
EFI_SMM_COMMUNICATION_PROTOCOL. So we can use the next unused value
(lets say 0x4). We probably don't have to use status port or
EFI_SMM_COMMUNICATION_PROTOCOL, since the value of written into 0xB2
is sufficient to distinguish hotplug event.

Furthermote, OVMF's EFI_SMM_CONTROL2_PROTOCOL.Trigger() implementation
(in "OvmfPkg/SmmControl2Dxe/SmmControl2Dxe.c") writes 0 to both ports,
as long as the caller does not specify them.

IoWrite8 (ICH9_APM_STS, DataPort == NULL ? 0 : *DataPort);
IoWrite8 (ICH9_APM_CNT, CommandPort == NULL ? 0 : *CommandPort);

And, there is only one Trigger() call site in edk2: namely in
"MdeModulePkg/Core/PiSmmCore/PiSmmIpl.c", in the
SmmCommunicationCommunicate() function.

(That function implements EFI_SMM_COMMUNICATION_PROTOCOL.Communicate().)
This call site passes NULL for both DataPort and CommandPort.

Yet further, EFI_SMM_COMMUNICATION_PROTOCOL.Communicate() is used for
example by the UEFI variable driver, for talking between the
unprivileged (runtime DXE) and privileged (SMM) half.

As a result, all of the software SMIs currently in use in OVMF, related
to actual firmware services, write 0 to both ports.

I guess we can choose new values, as long as we avoid 2 and 3 for the
control port (0xB2), because those are reserved in QEMU -- see
ich9_apm_ctrl_changed() in "hw/isa/lpc_ich9.c".


(2) How the parameters are passed.


(2a) For the new CPU, the SMI remains pending, until it gets an
INIT-SIPI-SIPI from one of the previously plugged CPUs (most likely, the
BSP). At that point, the new CPU will execute the "initial SMI handler
for hotplugged CPUs", at the default SMBASE.

That's a routine we'll have to write in assembly, from zero. In this
routine, we can read back IO ports 0xB2 and 0xB3. And QEMU will be happy
to provide the values last written (see apm_ioport_readb() in
"hw/isa/apm.c"). So we can receive the values in this routine. Alright.
Potentially we can can avoid writing custom SMI handler,
what do you think about following workflow:

on system boot after initial CPUs relocation, firmware set NOP SMI handler
at default SMBASE.
Then as reaction to GPE triggered SMI (on cpu hotplug), after SMI rendezvous,
a host cpu reads IO port 0xB2 and does hotplugged CPUs enumeration.

a) assuming we allow hotplug only in case of negotiated SMI broadcast
host CPU shoots down all in-flight INIT/SIPI/SIPI for hotpugged CPUs
to avoid race within relocation handler.

After that host CPU in loop

b) it prepares/initializes necessary CPU structures for a hotplugged
CPU if necessary and replaces NOP SMI handler with the relocation
SMI handler that is used during system boot.

c) a host CPU sends NOP INIT/SIPI/SIPI to the hotplugged CPU

d) the woken up hotplugged CPU, jumps to default SMBASE and
executes hotplug relocation handler.

e) after the hotplugged CPU is relocated and if there are more
hotplugged CPUs, a host CPU repeats b-d steps for the next
hotplugged CPU.

f) after all CPUs are relocated, restore NOP SMI handler at default
SMBASE.

(2b) On all other CPUs, the SMM foundation already accepts the SMI.

There point where it makes sense to start looking is SmmEntryPoint()
[MdeModulePkg/Core/PiSmmCore/PiSmmCore.c].

(2b1) This function first checks whether the SMI is synchronous. The
logic for determining that is based on
"gSmmCorePrivate->CommunicationBuffer" being non-NULL. This field is set
to non-NULL in SmmCommunicationCommunicate() -- see above, in (1).

In other words, the SMI is deemed synchronous if it was initiated with
EFI_SMM_COMMUNICATION_PROTOCOL.Communicate(). In that case, the
HandlerType GUID is extracted from the communication buffer, and passed
to SmiManage(). In turn, SmiManage() locates the SMI handler registered
with the same handler GUID, and delegates the SMI handling to that
specific handler.

This is how the UEFI variable driver is split in two halves:

- in "MdeModulePkg/Universal/Variable/RuntimeDxe/VariableSmm.c", we have
a call to gMmst->MmiHandlerRegister(), with HandlerType =
"gEfiSmmVariableProtocolGuid"

- in
"MdeModulePkg/Universal/Variable/RuntimeDxe/VariableSmmRuntimeDxe.c", we
format communication buffers with the header GUID set to the same
"gEfiSmmVariableProtocolGuid".

Of course, this is what does *not* apply to our use case, as the SMI is
raised by the OS (via an ACPI method), and *not* by a firmware agent
that calls EFI_SMM_COMMUNICATION_PROTOCOL.Communicate().

Therefore, we need to look further in SmmEntryPoint()
[MdeModulePkg/Core/PiSmmCore/PiSmmCore.c].

(2b2) What's left there is only the following:

//
// Process Asynchronous SMI sources
//
SmiManage (NULL, NULL, NULL, NULL);


So...

- Are we supposed to write a new DXE_SMM_DRIVER for OvmfPkg, and call
gMmst->MmiHandlerRegister() in it, with HandlerType=NULL? (I.e.,
register a "root SMI handler"?)

- And then in the handler, should we read IO ports 0xB2 / 0xB3?

- Also, is that handler where we'd somehow sync up with the hot-plugged
VCPU, and finally call EFI_SMM_CPU_SERVICE_PROTOCOL.SmmAddProcessor()?

- Does it matter what (pre-existent) CPU executes the handler? (IOW,
does it matter what the value of gMmst->CurrentlyExecutingCpu is?)

Thanks,
Laszlo



Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

rebecca@...
 

On 2019-10-02 15:47, Sean via Groups.Io wrote:
Please review and provide feedback. Build times are under 10 minutes for Ubuntu and 12 minutes for Windows.

One more side note: Details are in the readme on how to run a spell checker and there is an online build that can be reviewed. There is some "noise" with valid words but there is a massive number of misspelled words that would be great to get cleaned up and I would like to put this test into place to avoid future errors. Mike - maybe you can add to phase 2 wiki.
Do you have a location set up for where you'll publish the results of
the Doxygen builds?


--
Rebecca Cran


Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

Sean
 

All,

Phase 1 CI content ready for full review and I would propose that if no additional comments this RFC can be completed.

Master branch of my edk2 fork located here.
https://github.com/spbrogan/edk2

Build badges in the readme will take you to the individual builds.

Packages being tested, issues, and test details located here
https://github.com/spbrogan/edk2/blob/master/ci/readme.md. Also available thru a link on the readme.


There are 24 commits to the latest edk2 to enable this.

The first 7 commits are enabling edk2 to leverage PyTools.
* Add pip requirements file to track pip versions
* Add files describing tools, versions, and paths.
* Add new tools_def for vs2019 and 17 and gcc5
* Add new build_rule to support MSFT AARM64 assembler
* Add python file for ci build

There are also two commits that add new NULL libraries to speed up build and minimize unnecessary cross package dependencies.
* Add BaseCryptLibNull
* Add TlsLibNull

The remaining commits are changes to each package to describe the configuration for each test and minor tweaks to enable the packages to build with the tested toolchains and architectures.

Finally the last commit adds the AzurePipeline Support files into a new ci folder.

Please review and provide feedback. Build times are under 10 minutes for Ubuntu and 12 minutes for Windows.

One more side note: Details are in the readme on how to run a spell checker and there is an online build that can be reviewed. There is some "noise" with valid words but there is a massive number of misspelled words that would be great to get cleaned up and I would like to put this test into place to avoid future errors. Mike - maybe you can add to phase 2 wiki.

Thanks
Sean


Re: [edk2-devel] [Qemu-devel] [PATCH 1/2] q35: implement 128K SMRAM at default SMBASE address

Laszlo Ersek
 

On 09/30/19 16:22, Yao, Jiewen wrote:

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Igor
Mammedov
Sent: Monday, September 30, 2019 8:37 PM
To: Laszlo Ersek <lersek@...>
To me it looks like we need to figure out how QEMU can make the OS call
into SMM (in the GPE cpu hotplug handler), passing in parameters and
such. This would be step (03).

Do you agree?

If so, I'll ask Jiewen about such OS->SMM calls separately, because I
seem to remember that there used to be an "SMM communcation table" of
sorts, for flexible OS->SMM calls. However, it appears to be deprecated
lately.
we can try to resurrect and put over it some kind of protocol
to describe which CPUs to where hotplugged.

or we could put a parameter into SMI status register (IO port 0xb3)
and the trigger SMI from GPE handler to tell SMI handler that cpu
hotplug happened and then use QEMU's cpu hotplug interface
to enumerate hotplugged CPUs for SMI handler.

The later is probably simpler as we won't need to reinvent the wheel
(just reuse the interface that's already in use by GPE handler).
Based on "docs/specs/acpi_cpu_hotplug.txt", this seems to boil down to a
bunch of IO port accesses at base 0x0cd8.

Is that correct?

[Jiewen] The PI specification Volume 4 - SMM defines EFI_MM_COMMUNICATION_PROTOCOL.Communicate() - It can be used to communicate between OS and SMM handler. But it requires the runtime protocol call. I am not sure how OS loader passes this information to OS kernel.

As such, I think using ACPI SCI/GPE -> software SMI handler is an easier way to achieve this. I also recommend this way.
For parameter passing, we can use 1) Port B2 (1 byte), 2) Port B3 (1 byte), 3) chipset scratch register (4 bytes or 8 bytes, based upon scratch register size), 4) ACPI NVS OPREGION, if the data structure is complicated.
I'm confused about the details. In two categories:
(1) what values to use,
(2) how those values are passed.

Assume a CPU is hotpluged, QEMU injects an SCI, and the ACPI GPE handler
in the OS -- which also originates from QEMU -- writes a particular byte
to the Data port (0xB3), and then to the Control port (0xB2),
broadcasting an SMI.

(1) What values to use.

Note that values ICH9_APM_ACPI_ENABLE (2) and ICH9_APM_ACPI_DISABLE (3)
are already reserved in QEMU for IO port 0xB2, for different purposes.
So we can't use those in the GPE handler.

Furthermote, OVMF's EFI_SMM_CONTROL2_PROTOCOL.Trigger() implementation
(in "OvmfPkg/SmmControl2Dxe/SmmControl2Dxe.c") writes 0 to both ports,
as long as the caller does not specify them.

IoWrite8 (ICH9_APM_STS, DataPort == NULL ? 0 : *DataPort);
IoWrite8 (ICH9_APM_CNT, CommandPort == NULL ? 0 : *CommandPort);

And, there is only one Trigger() call site in edk2: namely in
"MdeModulePkg/Core/PiSmmCore/PiSmmIpl.c", in the
SmmCommunicationCommunicate() function.

(That function implements EFI_SMM_COMMUNICATION_PROTOCOL.Communicate().)
This call site passes NULL for both DataPort and CommandPort.

Yet further, EFI_SMM_COMMUNICATION_PROTOCOL.Communicate() is used for
example by the UEFI variable driver, for talking between the
unprivileged (runtime DXE) and privileged (SMM) half.

As a result, all of the software SMIs currently in use in OVMF, related
to actual firmware services, write 0 to both ports.

I guess we can choose new values, as long as we avoid 2 and 3 for the
control port (0xB2), because those are reserved in QEMU -- see
ich9_apm_ctrl_changed() in "hw/isa/lpc_ich9.c".


(2) How the parameters are passed.


(2a) For the new CPU, the SMI remains pending, until it gets an
INIT-SIPI-SIPI from one of the previously plugged CPUs (most likely, the
BSP). At that point, the new CPU will execute the "initial SMI handler
for hotplugged CPUs", at the default SMBASE.

That's a routine we'll have to write in assembly, from zero. In this
routine, we can read back IO ports 0xB2 and 0xB3. And QEMU will be happy
to provide the values last written (see apm_ioport_readb() in
"hw/isa/apm.c"). So we can receive the values in this routine. Alright.


(2b) On all other CPUs, the SMM foundation already accepts the SMI.

There point where it makes sense to start looking is SmmEntryPoint()
[MdeModulePkg/Core/PiSmmCore/PiSmmCore.c].

(2b1) This function first checks whether the SMI is synchronous. The
logic for determining that is based on
"gSmmCorePrivate->CommunicationBuffer" being non-NULL. This field is set
to non-NULL in SmmCommunicationCommunicate() -- see above, in (1).

In other words, the SMI is deemed synchronous if it was initiated with
EFI_SMM_COMMUNICATION_PROTOCOL.Communicate(). In that case, the
HandlerType GUID is extracted from the communication buffer, and passed
to SmiManage(). In turn, SmiManage() locates the SMI handler registered
with the same handler GUID, and delegates the SMI handling to that
specific handler.

This is how the UEFI variable driver is split in two halves:

- in "MdeModulePkg/Universal/Variable/RuntimeDxe/VariableSmm.c", we have
a call to gMmst->MmiHandlerRegister(), with HandlerType =
"gEfiSmmVariableProtocolGuid"

- in
"MdeModulePkg/Universal/Variable/RuntimeDxe/VariableSmmRuntimeDxe.c", we
format communication buffers with the header GUID set to the same
"gEfiSmmVariableProtocolGuid".

Of course, this is what does *not* apply to our use case, as the SMI is
raised by the OS (via an ACPI method), and *not* by a firmware agent
that calls EFI_SMM_COMMUNICATION_PROTOCOL.Communicate().

Therefore, we need to look further in SmmEntryPoint()
[MdeModulePkg/Core/PiSmmCore/PiSmmCore.c].

(2b2) What's left there is only the following:

//
// Process Asynchronous SMI sources
//
SmiManage (NULL, NULL, NULL, NULL);


So...

- Are we supposed to write a new DXE_SMM_DRIVER for OvmfPkg, and call
gMmst->MmiHandlerRegister() in it, with HandlerType=NULL? (I.e.,
register a "root SMI handler"?)

- And then in the handler, should we read IO ports 0xB2 / 0xB3?

- Also, is that handler where we'd somehow sync up with the hot-plugged
VCPU, and finally call EFI_SMM_CPU_SERVICE_PROTOCOL.SmmAddProcessor()?

- Does it matter what (pre-existent) CPU executes the handler? (IOW,
does it matter what the value of gMmst->CurrentlyExecutingCpu is?)

Thanks,
Laszlo


Re: UEFI accessibility mandate

Rafael Machado <rafaelrodrigues.machado@...>
 

Incredible!
I will find some time to try it at linux on the following weeks.
Thanks for the effort Andrew!

Rafael R. Machado

Em seg, 30 de set de 2019 às 22:18, Andrew Fish <afish@...> escreveu:



On Sep 30, 2019, at 11:50 AM, Rafael Machado <
rafaelrodrigues.machado@...> wrote:

Hi Andrew

As you have mentioned:
" I might find some time to try to get the EmulatorPkg mocking working."
Answer: This would be amazing! Thanks for trying to help!


Rafael,

I made a little progress this weekend watching American Football on TV,
and I had some time on an airplane today to add some comments.

I wrote a simple C command line program that does almost all the primitive
audio functions we need to add Audio/Text-To-Speech to the EmulatorPkg.
Since ANSI C does not support audio I had to system() out to macOS command
line tools to get some of the tasks complete. The only feature I'm missing
is playing the PCM buffer asynchronously, and I may be able to add that
next weekend. I guessed at the Linux commands, it would be good if someone
could debug the program on a Linux system, and also make sure I picked the
right command line tools to use. I think for Windows there is going to be
more coding involved, but I think there is at least a C API to play audio,
but I only saw a C++ API for speech to text. But the good news is folks can
use the simple C program to get the primitives working on Linux and Windows
systems and it should be easy to port that work to the future EmulatorPkg
work.

What I've got working on macOS is:
1) Mute/Unmute
2) GetVolume/SetVolume
3) Text to PCD buffer (Wave file format)
a) Supports word per minute rate
b) Supports localizing the voice based on RFC 4646 language code used by
EFI (the program speaks in en, zh, and fr as a test)
4) PCD buffer to audio

This is how I was building on my Mac: clang -g -Wall -Werror audio.c

Short term anyone that has some free time if they could look at
testing/debugging audio.c on Linux and send feedback, also maybe some one
could start playing around with getting audio.c ported to Windows?

After I figure out getting async audio output working in audio.c I think
the next steps are working on the protocol/variable definitions. With the
protocol/variable definitions we should be able to add the capability to
the EmulatorPkg. At that point it should be possible for anyone to help
work on the accessibility UI. Obviously to get this working on a real
platform we will need a pure EFI Text to Speech driver (We could debug that
in the EmulatorPkg). We would also need Intel HDA Audio drivers, and Audio
drivers for OVFM, and other Audio hardware that is on peoples platforms.

Thanks,

Andrew Fish

The plan is to try to get some students to help on this also at the next
"google summer of code".
Not sure how it works, but since the idea is good, and it is a real need,
I believe we have chances to have some more hands working on it.

Thanks and Regards
Rafael R. Machado



Em qui, 26 de set de 2019 às 20:44, Andrew Fish <afish@...>
escreveu:



On Sep 26, 2019, at 5:15 AM, Rafael Machado <
rafaelrodrigues.machado@...> wrote:

Hi everyone.

About Ethin's question:
As for the boot manager, yes, that would be covered by the speech
synthesizer protocol. Perhaps we could extend this to the setup utility too
(as well as all other things displayed on the screen)?
Answer: This is the target from my point of view. Since we have the
advantages of drivers at UEFI, there is no reason for not having
accessibility at pre-OS anymore. We could also think in future to add other
kinds of accessibility, like the magnifying glass (that would be some
multi-layer version of the GOP driver), that would help low-vision people.
Just one thing for everyone to think about. The "Convention on the Rights
of Persons with Disabilities" say that a person has the right to access
information. So if someone buys a notebook, all information accessible to
"common" people must be available to all kinds of people, in a way they can
choose to use this information or not.
*" (e) Recognizing that disability is an evolving concept and that
disability results from the interaction between persons with impairments
and attitudinal and environmental barriers that hinders their full and
effective participation in society on an equal basis with others, ..."*
Details can be found at the United Nations web site
https://www.un.org/disabilities/documents/convention/convention_accessible_pdf.pdf


About Andrew's comments and questions:
What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL
should contain?
Answer: From my perspective, this protocol would need at least the
following functions (names may change after more discussion.)


- EFI_AUDIO_OUTPUT_PROTOCOL.LoadStream()
This function is intended to copy some data to the dma buffers that
were allocated previously by the AudioController driver. The initial idea I
had was to add, in the case of my MSc, a HdaDriver.efi that is responsible
to do all the HDA controller initialization (allocating dma buffers,
configuring the PCIe bus if needed). Not sure if this initialization should
we added to the EFI_AUDIO_OUTPUT_PROTOCOL, because this could create
problems when using other controllers or concepts, like I believe ARM
platforms do.My understanding is that the platforms should have a driver to
initialize the audio controller/codec, so the EFI_AUDIO_OUTPUT_PROTOCOL is
just worried with loading and playing information.

- EFI_AUDIO_OUTPUT_PROTOCOL.PlayStream()
This function is responsible to play the stream. This is done by
setting one specific bit at the controllers configuration space (pointed by
the bar[0] pcie configuration register at the pcie config space.) At this
project I don't think we need to be worried with implementing an sound
server to enable multiple streams at a time. To the actions done by this
function are much simpler. Just play a stream.

- EFI_AUDIO_OUTPUT_PROTOCOL.StopStream()
This function is responsible to stop the stream play. The opposite of
the previous one. Maybe we could use a single function that receives some
parameter to start or stop the play.

- EFI_AUDIO_OUTPUT_PROTOCOL.SetVolume()
This function is responsible to set the volume to be used at all
nodes at the codec (so it depends on a EFI_AUDIO_CODEC_PROTOCOL that knows
the commands a given codec needs to receive to process streams.)

- EFI_AUDIO_OUTPUT_PROTOCOL.SetStreamSpeed()
This function is responsible to set the speed a given stream should
be played on. Just to clarify, visual impaired people have an amazing
capacity of understanding sound streams in incredibly fast speeds. So the
possibility of setting the stream processing speed is what makes them be as
productive as a person with usual vision capacities.

- EFI_AUDIO_OUTPUT_PROTOCOL.UnloadStream()
This function is responsible to clean the buffers and letting them
prepared to receive other audio streams.

Still about Andrews comments.
This gave me the idea of adding a EFI_AUDIO_OUTPUT_PROTOCOL and EFI_TEXT_TO_SPEECH_PROTOCOL
driver to the EmulatorPkg that thunk down to the App to do the work.
With this we could prototype the UI part of accessibility.
Answer: This would be great, because we would split the development. One
focusing on the HII and navigation (done normally using the tab key, so
each element at the screen would need an tab sequence number and a
accessibility tag with what the speech software should say), and another
focusing on the drivers/synthesizer/audio files depending on our future
decisions.

So my understanding is that the list of things is increasing. It is
actually something like this (not ordered by priority):
- EFI_AUDIO_OUTPUT_PROTOCOL: Protocol used to process the streams
- EFI_AUDIO_CODEC_PROTOCOL: Protocol used to communicate with the codec
- Audio configuration driver (HdaAudio.efi for example). Should this
create some other protocol, like EFI_AUDIO_CONTROLLER_PROTOCOL that is
responsible to configure the controller?


Rafael,

Generally how it works for a PCI device is it gets its Start() function
called to bind to the device. So the EFI_AUDIO_OUTPUT_PROTOCOL and
EFI_AUDIO_CODEC_PROTOCOL protocols would likely get installed on the PCI IO
handle of the HDA PCI hardware. Some basic init will happen during the
Start() and the driver could defer some config until one of the member
functions is called the 1st time.

So for example on a normal boot the Audio driver may not even get
started. So there may have to be an nvram variable to start the driver. We
might also need an NVRAM variable for the voice to text driver if it is
stored on disk.

- EFI_TEXT_TO_SPEECH_PROTOCOL: Protocol responsible to convert text to
speech (using the EFI_AUDIO_OUTPUT_PROTOCOL )
- HII changes to add tab navigation capabilities and accessibility tags
at each component on the screen

Are we aligned with the understanding?


I think so and if we can mock an implementation in the EmulatorPkg that
would let the HII UI work happen independently of the text to speech, and
Audio driver work.

I'm on some air plaines in the next weeks (I'm at the airport right now),
and my wife is heading to a conference after I get back from my trips.
Given my dogs are not very talkative I might find some time to try to get
the EmulatorPkg mocking working.

Thanks,

Andrew Fish

I believe soon we will be prepared to talk about this on a design meeting.

Thanks and Regards
Rafael R. Machado



Em qua, 25 de set de 2019 às 22:08, Andrew Fish <afish@...>
escreveu:

Rafael,

What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL should
contain?

I'm thinking if we had an EFI_TEXT_TO_SPEECH_PROTOCOL that driver could
produce a PCM/Wave buffer, it could then use EFI_AUDIO_OUTPUT_PROTOCOL
to play the sound.

I poked around my Mac at lunch and I can generate text to speech from
the command line into a wave file via the `say` command line tool. I can
play the wave file from the command line via `afplay`. This gave me the
idea of adding a EFI_AUDIO_OUTPUT_PROTOCOL and EFI_TEXT_TO_SPEECH_PROTOCOL
driver to the EmulatorPkg that thunk down to the App to do the work.
With this we could prototype the UI part of accessibility. We could also
debug the EFI EFI_TEXT_TO_SPEECH_PROTOCOL in this environment.

Thanks,

Andrew Fish

On Sep 21, 2019, at 5:36 AM, Rafael Machado <
rafaelrodrigues.machado@...> wrote:

Hi Everyone
Sorry for the delay on the response, to many things happening at the
same time.
I will try to answer e-mails to this thread every Saturday or Sunday
morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing lists,
and I >>can also represent what ever we come up with at the UEFI Spec Work
Group.
During my MSc I had to study a lot the audio and BIOS architectures. The
idea was to eliminate the first barrier to the creation of a screen reader
for pre-OS environment, that was the lack of some open implementation of
audio control and actions at UEFI. To do that I studied the Intel High
Definition Audio Spec and a lot of UEFI specs to understand better how to
do that.
The initial target was to do all this development at OVMF, but as far as
I could get, the sound card is not available to OVMF. as Laszlo mentioned
at this e-mail there are some projects that may help on this, but at the
time I was working on my MSc I didn't find this, so I did everything on a
real system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and
working on a MSc degree at the same time having a 40hours/week job, being a
father and a husband is not an easy task, but it got to what I was
expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that
later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA
memory access. For the ones that have good years, it's possible to listen
some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was
that one of my write operations was causing some misunderstanding between
the audio controller and the audio codec. The controller was sending
packets with 16bit depth, but the codec was processing them as 8bit depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing
at the OS level.
The reference code, that is just a proof-of-concept, and that has
several things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to
UEFI drivers after some discussion. Everything is released as BDS so
companies can use without IP problems.
Just to give some more information about the need of this kind of
solution. There is a lot of blind people that work with hardware support,
so formatting disk, configuring RAID and booting dual-boot systems is
always a challenge to them. Even set the BIOS clock. How to do that without
the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given there
is no guideline on how a User Interface (UI) works. If accessibility
requires some from of hardware abstraction, like audio, then we could
likely get that into the UEFI Spec. What might be possible is an EDK2
reference implementation of accessibility. Maybe we could use the reference
implementation to write a UEFI white paper on design >>for accessibility? I
there is an open source implementation, and an official design guide this
would make it much easier for advocacy groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2
papers is one of my targets since the beginning of my MSc almost 5 years
ago.

I've got some experience with accessibility as the macOS EFI OS Loader
has a UI for the Full Disk Encryption password. If you press the power
button quickly 3 times at the disk encryption password prompt
accessibility is enabled and Voice Over gets turned on. You then get
localized voice prompts when you move between UI elements. Since this is
the OS loader all the resources are stored on the disk. You >>quickly run
into a range of challenges since, audio is hard, abstracting audio is hard
(what codec does firmware have to support), Audio files are not small and
firmware is size constrained, the need to localize >>the audio responses
causes even more size issues, the boot options are usually written by an OS
installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so
no audio files would need to be stored. There are some open-source
implementations that are not GPL.
This was described at my MSc as future works that can continue what I
have started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things "not
because they are easy, but because they are hard; because that >>goal will
serve to organize and measure the best of our energies and skills, because
that challenge is one that we are willing to accept". If we have a design
that means we can break the problem up into >>smaller parts, and maybe we
can find people that have expertise in that part to build a chunk at a
time. If we could implement the prototype in OVMF that would show how it
works, but run on everyone's >>machines, so that would be really helpful
for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible future
steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with
proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver
to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also
producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application
remains functional.
- 4) Separate controller and codec code by creating
an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate out
the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder
audio files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the need
of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my
thesis to English.
I wrote everything in Portuguese because there was not material about
UEFI to the Brazilian audience, and another target I have is to show
companies that we have people that can work at this kind of projects in
Brazil, bringing this kind of development to south america. (Yes, I have
complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@...>
escreveu:

On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo


Re: UEFI accessibility mandate

Andrew Fish <afish@...>
 

On Sep 30, 2019, at 11:50 AM, Rafael Machado <rafaelrodrigues.machado@...> wrote:

Hi Andrew

As you have mentioned:
" I might find some time to try to get the EmulatorPkg mocking working."
Answer: This would be amazing! Thanks for trying to help!
Rafael,

I made a little progress this weekend watching American Football on TV, and I had some time on an airplane today to add some comments.

I wrote a simple C command line program that does almost all the primitive audio functions we need to add Audio/Text-To-Speech to the EmulatorPkg. Since ANSI C does not support audio I had to system() out to macOS command line tools to get some of the tasks complete. The only feature I'm missing is playing the PCM buffer asynchronously, and I may be able to add that next weekend. I guessed at the Linux commands, it would be good if someone could debug the program on a Linux system, and also make sure I picked the right command line tools to use. I think for Windows there is going to be more coding involved, but I think there is at least a C API to play audio, but I only saw a C++ API for speech to text. But the good news is folks can use the simple C program to get the primitives working on Linux and Windows systems and it should be easy to port that work to the future EmulatorPkg work.

What I've got working on macOS is:
1) Mute/Unmute
2) GetVolume/SetVolume
3) Text to PCD buffer (Wave file format)
a) Supports word per minute rate
b) Supports localizing the voice based on RFC 4646 language code used by EFI (the program speaks in en, zh, and fr as a test)
4) PCD buffer to audio

This is how I was building on my Mac: clang -g -Wall -Werror audio.c


Short term anyone that has some free time if they could look at testing/debugging audio.c on Linux and send feedback, also maybe some one could start playing around with getting audio.c ported to Windows?

After I figure out getting async audio output working in audio.c I think the next steps are working on the protocol/variable definitions. With the protocol/variable definitions we should be able to add the capability to the EmulatorPkg. At that point it should be possible for anyone to help work on the accessibility UI. Obviously to get this working on a real platform we will need a pure EFI Text to Speech driver (We could debug that in the EmulatorPkg). We would also need Intel HDA Audio drivers, and Audio drivers for OVFM, and other Audio hardware that is on peoples platforms.

Thanks,

Andrew Fish

The plan is to try to get some students to help on this also at the next "google summer of code".
Not sure how it works, but since the idea is good, and it is a real need, I believe we have chances to have some more hands working on it.

Thanks and Regards
Rafael R. Machado



Em qui, 26 de set de 2019 às 20:44, Andrew Fish <afish@... <mailto:afish@...>> escreveu:


On Sep 26, 2019, at 5:15 AM, Rafael Machado <rafaelrodrigues.machado@... <mailto:rafaelrodrigues.machado@...>> wrote:

Hi everyone.

About Ethin's question:
As for the boot manager, yes, that would be covered by the speech synthesizer protocol. Perhaps we could extend this to the setup utility too (as well as all other things displayed on the screen)?
Answer: This is the target from my point of view. Since we have the advantages of drivers at UEFI, there is no reason for not having accessibility at pre-OS anymore. We could also think in future to add other kinds of accessibility, like the magnifying glass (that would be some multi-layer version of the GOP driver), that would help low-vision people. Just one thing for everyone to think about. The "Convention on the Rights of Persons with Disabilities" say that a person has the right to access information. So if someone buys a notebook, all information accessible to "common" people must be available to all kinds of people, in a way they can choose to use this information or not.
" (e) Recognizing that disability is an evolving concept and that disability results from the interaction between persons with impairments and attitudinal and environmental barriers that hinders their full and effective participation in society on an equal basis with others, ..."
Details can be found at the United Nations web site https://www.un.org/disabilities/documents/convention/convention_accessible_pdf.pdf

About Andrew's comments and questions:
What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL should contain?
Answer: From my perspective, this protocol would need at least the following functions (names may change after more discussion.)
- EFI_AUDIO_OUTPUT_PROTOCOL.LoadStream()
This function is intended to copy some data to the dma buffers that were allocated previously by the AudioController driver. The initial idea I had was to add, in the case of my MSc, a HdaDriver.efi that is responsible to do all the HDA controller initialization (allocating dma buffers, configuring the PCIe bus if needed). Not sure if this initialization should we added to the EFI_AUDIO_OUTPUT_PROTOCOL, because this could create problems when using other controllers or concepts, like I believe ARM platforms do.My understanding is that the platforms should have a driver to initialize the audio controller/codec, so the EFI_AUDIO_OUTPUT_PROTOCOL is just worried with loading and playing information.

- EFI_AUDIO_OUTPUT_PROTOCOL.PlayStream()
This function is responsible to play the stream. This is done by setting one specific bit at the controllers configuration space (pointed by the bar[0] pcie configuration register at the pcie config space.) At this project I don't think we need to be worried with implementing an sound server to enable multiple streams at a time. To the actions done by this function are much simpler. Just play a stream.

- EFI_AUDIO_OUTPUT_PROTOCOL.StopStream()
This function is responsible to stop the stream play. The opposite of the previous one. Maybe we could use a single function that receives some parameter to start or stop the play.

- EFI_AUDIO_OUTPUT_PROTOCOL.SetVolume()
This function is responsible to set the volume to be used at all nodes at the codec (so it depends on a EFI_AUDIO_CODEC_PROTOCOL that knows the commands a given codec needs to receive to process streams.)

- EFI_AUDIO_OUTPUT_PROTOCOL.SetStreamSpeed()
This function is responsible to set the speed a given stream should be played on. Just to clarify, visual impaired people have an amazing capacity of understanding sound streams in incredibly fast speeds. So the possibility of setting the stream processing speed is what makes them be as productive as a person with usual vision capacities.

- EFI_AUDIO_OUTPUT_PROTOCOL.UnloadStream()
This function is responsible to clean the buffers and letting them prepared to receive other audio streams.

Still about Andrews comments.
This gave me the idea of adding a EFI_AUDIO_OUTPUT_PROTOCOL and EFI_TEXT_TO_SPEECH_PROTOCOL driver to the EmulatorPkg that thunk down to the App to do the work. With this we could prototype the UI part of accessibility.
Answer: This would be great, because we would split the development. One focusing on the HII and navigation (done normally using the tab key, so each element at the screen would need an tab sequence number and a accessibility tag with what the speech software should say), and another focusing on the drivers/synthesizer/audio files depending on our future decisions.

So my understanding is that the list of things is increasing. It is actually something like this (not ordered by priority):
- EFI_AUDIO_OUTPUT_PROTOCOL: Protocol used to process the streams
- EFI_AUDIO_CODEC_PROTOCOL: Protocol used to communicate with the codec
- Audio configuration driver (HdaAudio.efi for example). Should this create some other protocol, like EFI_AUDIO_CONTROLLER_PROTOCOL that is responsible to configure the controller?
Rafael,

Generally how it works for a PCI device is it gets its Start() function called to bind to the device. So the EFI_AUDIO_OUTPUT_PROTOCOL and EFI_AUDIO_CODEC_PROTOCOL protocols would likely get installed on the PCI IO handle of the HDA PCI hardware. Some basic init will happen during the Start() and the driver could defer some config until one of the member functions is called the 1st time.

So for example on a normal boot the Audio driver may not even get started. So there may have to be an nvram variable to start the driver. We might also need an NVRAM variable for the voice to text driver if it is stored on disk.

- EFI_TEXT_TO_SPEECH_PROTOCOL: Protocol responsible to convert text to speech (using the EFI_AUDIO_OUTPUT_PROTOCOL )
- HII changes to add tab navigation capabilities and accessibility tags at each component on the screen

Are we aligned with the understanding?
I think so and if we can mock an implementation in the EmulatorPkg that would let the HII UI work happen independently of the text to speech, and Audio driver work.

I'm on some air plaines in the next weeks (I'm at the airport right now), and my wife is heading to a conference after I get back from my trips. Given my dogs are not very talkative I might find some time to try to get the EmulatorPkg mocking working.

Thanks,

Andrew Fish

I believe soon we will be prepared to talk about this on a design meeting.

Thanks and Regards
Rafael R. Machado



Em qua, 25 de set de 2019 às 22:08, Andrew Fish <afish@... <mailto:afish@...>> escreveu:
Rafael,

What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL should contain?

I'm thinking if we had an EFI_TEXT_TO_SPEECH_PROTOCOL that driver could produce a PCM/Wave buffer, it could then use EFI_AUDIO_OUTPUT_PROTOCOL to play the sound.

I poked around my Mac at lunch and I can generate text to speech from the command line into a wave file via the `say` command line tool. I can play the wave file from the command line via `afplay`. This gave me the idea of adding a EFI_AUDIO_OUTPUT_PROTOCOL and EFI_TEXT_TO_SPEECH_PROTOCOL driver to the EmulatorPkg that thunk down to the App to do the work. With this we could prototype the UI part of accessibility. We could also debug the EFI EFI_TEXT_TO_SPEECH_PROTOCOL in this environment.

Thanks,

Andrew Fish

On Sep 21, 2019, at 5:36 AM, Rafael Machado <rafaelrodrigues.machado@... <mailto:rafaelrodrigues.machado@...>> wrote:

Hi Everyone
Sorry for the delay on the response, to many things happening at the same time.
I will try to answer e-mails to this thread every Saturday or Sunday morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time to help implement this feature, but I happy to help work on the architecture and design for UEFI accessibility on the edk2 mailing lists, and I >>can also represent what ever we come up with at the UEFI Spec Work Group.
During my MSc I had to study a lot the audio and BIOS architectures. The idea was to eliminate the first barrier to the creation of a screen reader for pre-OS environment, that was the lack of some open implementation of audio control and actions at UEFI. To do that I studied the Intel High Definition Audio Spec and a lot of UEFI specs to understand better how to do that.
The initial target was to do all this development at OVMF, but as far as I could get, the sound card is not available to OVMF. as Laszlo mentioned at this e-mail there are some projects that may help on this, but at the time I was working on my MSc I didn't find this, so I did everything on a real system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and working on a MSc degree at the same time having a 40hours/week job, being a father and a husband is not an easy task, but it got to what I was expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA memory access. For the ones that have good years, it's possible to listen some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was that one of my write operations was causing some misunderstanding between the audio controller and the audio codec. The controller was sending packets with 16bit depth, but the codec was processing them as 8bit depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing at the OS level.
The reference code, that is just a proof-of-concept, and that has several things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to UEFI drivers after some discussion. Everything is released as BDS so companies can use without IP problems.
Just to give some more information about the need of this kind of solution. There is a lot of blind people that work with hardware support, so formatting disk, configuring RAID and booting dual-boot systems is always a challenge to them. Even set the BIOS clock. How to do that without the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given there is no guideline on how a User Interface (UI) works. If accessibility requires some from of hardware abstraction, like audio, then we could >>likely get that into the UEFI Spec. What might be possible is an EDK2 reference implementation of accessibility. Maybe we could use the reference implementation to write a UEFI white paper on design >>for accessibility? I there is an open source implementation, and an official design guide this would make it much easier for advocacy groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2 papers is one of my targets since the beginning of my MSc almost 5 years ago.

I've got some experience with accessibility as the macOS EFI OS Loader has a UI for the Full Disk Encryption password. If you press the power button quickly 3 times at the disk encryption password prompt >>accessibility is enabled and Voice Over gets turned on. You then get localized voice prompts when you move between UI elements. Since this is the OS loader all the resources are stored on the disk. You >>quickly run into a range of challenges since, audio is hard, abstracting audio is hard (what codec does firmware have to support), Audio files are not small and firmware is size constrained, the need to localize >>the audio responses causes even more size issues, the boot options are usually written by an OS installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so no audio files would need to be stored. There are some open-source implementations that are not GPL.
This was described at my MSc as future works that can continue what I have started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We choose to go to the Moon!" speech sometimes we chose to do things "not because they are easy, but because they are hard; because that >>goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept". If we have a design that means we can break the problem up into >>smaller parts, and maybe we can find people that have expertise in that part to build a chunk at a time. If we could implement the prototype in OVMF that would show how it works, but run on everyone's >>machines, so that would be really helpful for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible future steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application remains functional.
- 4) Separate controller and codec code by creating an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate out the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder audio files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the need of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my thesis to English.
I wrote everything in Portuguese because there was not material about UEFI to the Brazilian audience, and another target I have is to show companies that we have people that can work at this kind of projects in Brazil, bringing this kind of development to south america. (Yes, I have complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@... <mailto:lersek@...>> escreveu:
On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo


Re: UEFI accessibility mandate

Rafael Machado <rafaelrodrigues.machado@...>
 

Hi Andrew

As you have mentioned:
" I might find some time to try to get the EmulatorPkg mocking working."
Answer: This would be amazing! Thanks for trying to help!

The plan is to try to get some students to help on this also at the next
"google summer of code".
Not sure how it works, but since the idea is good, and it is a real need, I
believe we have chances to have some more hands working on it.

Thanks and Regards
Rafael R. Machado



Em qui, 26 de set de 2019 às 20:44, Andrew Fish <afish@...> escreveu:



On Sep 26, 2019, at 5:15 AM, Rafael Machado <
rafaelrodrigues.machado@...> wrote:

Hi everyone.

About Ethin's question:
As for the boot manager, yes, that would be covered by the speech
synthesizer protocol. Perhaps we could extend this to the setup utility too
(as well as all other things displayed on the screen)?
Answer: This is the target from my point of view. Since we have the
advantages of drivers at UEFI, there is no reason for not having
accessibility at pre-OS anymore. We could also think in future to add other
kinds of accessibility, like the magnifying glass (that would be some
multi-layer version of the GOP driver), that would help low-vision people.
Just one thing for everyone to think about. The "Convention on the Rights
of Persons with Disabilities" say that a person has the right to access
information. So if someone buys a notebook, all information accessible to
"common" people must be available to all kinds of people, in a way they can
choose to use this information or not.
*" (e) Recognizing that disability is an evolving concept and that
disability results from the interaction between persons with impairments
and attitudinal and environmental barriers that hinders their full and
effective participation in society on an equal basis with others, ..."*
Details can be found at the United Nations web site
https://www.un.org/disabilities/documents/convention/convention_accessible_pdf.pdf


About Andrew's comments and questions:
What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL should
contain?
Answer: From my perspective, this protocol would need at least the
following functions (names may change after more discussion.)


- EFI_AUDIO_OUTPUT_PROTOCOL.LoadStream()
This function is intended to copy some data to the dma buffers that
were allocated previously by the AudioController driver. The initial idea I
had was to add, in the case of my MSc, a HdaDriver.efi that is responsible
to do all the HDA controller initialization (allocating dma buffers,
configuring the PCIe bus if needed). Not sure if this initialization should
we added to the EFI_AUDIO_OUTPUT_PROTOCOL, because this could create
problems when using other controllers or concepts, like I believe ARM
platforms do.My understanding is that the platforms should have a driver to
initialize the audio controller/codec, so the EFI_AUDIO_OUTPUT_PROTOCOL is
just worried with loading and playing information.

- EFI_AUDIO_OUTPUT_PROTOCOL.PlayStream()
This function is responsible to play the stream. This is done by
setting one specific bit at the controllers configuration space (pointed by
the bar[0] pcie configuration register at the pcie config space.) At this
project I don't think we need to be worried with implementing an sound
server to enable multiple streams at a time. To the actions done by this
function are much simpler. Just play a stream.

- EFI_AUDIO_OUTPUT_PROTOCOL.StopStream()
This function is responsible to stop the stream play. The opposite of
the previous one. Maybe we could use a single function that receives some
parameter to start or stop the play.

- EFI_AUDIO_OUTPUT_PROTOCOL.SetVolume()
This function is responsible to set the volume to be used at all nodes
at the codec (so it depends on a EFI_AUDIO_CODEC_PROTOCOL that knows the
commands a given codec needs to receive to process streams.)

- EFI_AUDIO_OUTPUT_PROTOCOL.SetStreamSpeed()
This function is responsible to set the speed a given stream should be
played on. Just to clarify, visual impaired people have an amazing capacity
of understanding sound streams in incredibly fast speeds. So the
possibility of setting the stream processing speed is what makes them be as
productive as a person with usual vision capacities.

- EFI_AUDIO_OUTPUT_PROTOCOL.UnloadStream()
This function is responsible to clean the buffers and letting them
prepared to receive other audio streams.

Still about Andrews comments.
This gave me the idea of adding a EFI_AUDIO_OUTPUT_PROTOCOL and EFI_TEXT_TO_SPEECH_PROTOCOL
driver to the EmulatorPkg that thunk down to the App to do the work.
With this we could prototype the UI part of accessibility.
Answer: This would be great, because we would split the development. One
focusing on the HII and navigation (done normally using the tab key, so
each element at the screen would need an tab sequence number and a
accessibility tag with what the speech software should say), and another
focusing on the drivers/synthesizer/audio files depending on our future
decisions.

So my understanding is that the list of things is increasing. It is
actually something like this (not ordered by priority):
- EFI_AUDIO_OUTPUT_PROTOCOL: Protocol used to process the streams
- EFI_AUDIO_CODEC_PROTOCOL: Protocol used to communicate with the codec
- Audio configuration driver (HdaAudio.efi for example). Should this
create some other protocol, like EFI_AUDIO_CONTROLLER_PROTOCOL that is
responsible to configure the controller?


Rafael,

Generally how it works for a PCI device is it gets its Start() function
called to bind to the device. So the EFI_AUDIO_OUTPUT_PROTOCOL and
EFI_AUDIO_CODEC_PROTOCOL protocols would likely get installed on the PCI IO
handle of the HDA PCI hardware. Some basic init will happen during the
Start() and the driver could defer some config until one of the member
functions is called the 1st time.

So for example on a normal boot the Audio driver may not even get started.
So there may have to be an nvram variable to start the driver. We might
also need an NVRAM variable for the voice to text driver if it is stored on
disk.

- EFI_TEXT_TO_SPEECH_PROTOCOL: Protocol responsible to convert text to
speech (using the EFI_AUDIO_OUTPUT_PROTOCOL )
- HII changes to add tab navigation capabilities and accessibility tags at
each component on the screen

Are we aligned with the understanding?


I think so and if we can mock an implementation in the EmulatorPkg that
would let the HII UI work happen independently of the text to speech, and
Audio driver work.

I'm on some air plaines in the next weeks (I'm at the airport right now),
and my wife is heading to a conference after I get back from my trips.
Given my dogs are not very talkative I might find some time to try to get
the EmulatorPkg mocking working.

Thanks,

Andrew Fish

I believe soon we will be prepared to talk about this on a design meeting.

Thanks and Regards
Rafael R. Machado



Em qua, 25 de set de 2019 às 22:08, Andrew Fish <afish@...>
escreveu:

Rafael,

What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL should
contain?

I'm thinking if we had an EFI_TEXT_TO_SPEECH_PROTOCOL that driver could
produce a PCM/Wave buffer, it could then use EFI_AUDIO_OUTPUT_PROTOCOL
to play the sound.

I poked around my Mac at lunch and I can generate text to speech from the
command line into a wave file via the `say` command line tool. I can play
the wave file from the command line via `afplay`. This gave me the idea of
adding a EFI_AUDIO_OUTPUT_PROTOCOL and EFI_TEXT_TO_SPEECH_PROTOCOL
driver to the EmulatorPkg that thunk down to the App to do the work.
With this we could prototype the UI part of accessibility. We could also
debug the EFI EFI_TEXT_TO_SPEECH_PROTOCOL in this environment.

Thanks,

Andrew Fish

On Sep 21, 2019, at 5:36 AM, Rafael Machado <
rafaelrodrigues.machado@...> wrote:

Hi Everyone
Sorry for the delay on the response, to many things happening at the same
time.
I will try to answer e-mails to this thread every Saturday or Sunday
morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing lists,
and I >>can also represent what ever we come up with at the UEFI Spec Work
Group.
During my MSc I had to study a lot the audio and BIOS architectures. The
idea was to eliminate the first barrier to the creation of a screen reader
for pre-OS environment, that was the lack of some open implementation of
audio control and actions at UEFI. To do that I studied the Intel High
Definition Audio Spec and a lot of UEFI specs to understand better how to
do that.
The initial target was to do all this development at OVMF, but as far as
I could get, the sound card is not available to OVMF. as Laszlo mentioned
at this e-mail there are some projects that may help on this, but at the
time I was working on my MSc I didn't find this, so I did everything on a
real system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and
working on a MSc degree at the same time having a 40hours/week job, being a
father and a husband is not an easy task, but it got to what I was
expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that
later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA
memory access. For the ones that have good years, it's possible to listen
some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was
that one of my write operations was causing some misunderstanding between
the audio controller and the audio codec. The controller was sending
packets with 16bit depth, but the codec was processing them as 8bit depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing at
the OS level.
The reference code, that is just a proof-of-concept, and that has several
things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to UEFI
drivers after some discussion. Everything is released as BDS so companies
can use without IP problems.
Just to give some more information about the need of this kind of
solution. There is a lot of blind people that work with hardware support,
so formatting disk, configuring RAID and booting dual-boot systems is
always a challenge to them. Even set the BIOS clock. How to do that without
the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given there
is no guideline on how a User Interface (UI) works. If accessibility
requires some from of hardware abstraction, like audio, then we could
likely get that into the UEFI Spec. What might be possible is an EDK2
reference implementation of accessibility. Maybe we could use the reference
implementation to write a UEFI white paper on design >>for accessibility? I
there is an open source implementation, and an official design guide this
would make it much easier for advocacy groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2 papers
is one of my targets since the beginning of my MSc almost 5 years ago.

I've got some experience with accessibility as the macOS EFI OS Loader
has a UI for the Full Disk Encryption password. If you press the power
button quickly 3 times at the disk encryption password prompt
accessibility is enabled and Voice Over gets turned on. You then get
localized voice prompts when you move between UI elements. Since this is
the OS loader all the resources are stored on the disk. You >>quickly run
into a range of challenges since, audio is hard, abstracting audio is hard
(what codec does firmware have to support), Audio files are not small and
firmware is size constrained, the need to localize >>the audio responses
causes even more size issues, the boot options are usually written by an OS
installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so
no audio files would need to be stored. There are some open-source
implementations that are not GPL.
This was described at my MSc as future works that can continue what I
have started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things "not
because they are easy, but because they are hard; because that >>goal will
serve to organize and measure the best of our energies and skills, because
that challenge is one that we are willing to accept". If we have a design
that means we can break the problem up into >>smaller parts, and maybe we
can find people that have expertise in that part to build a chunk at a
time. If we could implement the prototype in OVMF that would show how it
works, but run on everyone's >>machines, so that would be really helpful
for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible future
steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with
proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver
to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also
producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application
remains functional.
- 4) Separate controller and codec code by creating
an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate out
the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder
audio files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the need
of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my
thesis to English.
I wrote everything in Portuguese because there was not material about
UEFI to the Brazilian audience, and another target I have is to show
companies that we have people that can work at this kind of projects in
Brazil, bringing this kind of development to south america. (Yes, I have
complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@...>
escreveu:

On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo


Re: [edk2-devel] [Qemu-devel] [PATCH 1/2] q35: implement 128K SMRAM at default SMBASE address

Yao, Jiewen
 

below

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Igor
Mammedov
Sent: Monday, September 30, 2019 8:37 PM
To: Laszlo Ersek <lersek@...>
Cc: devel@edk2.groups.io; qemu-devel@...; Chen, Yingwen
<yingwen.chen@...>; phillip.goerl@...;
alex.williamson@...; Yao, Jiewen <jiewen.yao@...>; Nakajima,
Jun <jun.nakajima@...>; Kinney, Michael D
<michael.d.kinney@...>; pbonzini@...;
boris.ostrovsky@...; rfc@edk2.groups.io; joao.m.martins@...;
Brijesh Singh <brijesh.singh@...>
Subject: Re: [edk2-devel] [Qemu-devel] [PATCH 1/2] q35: implement 128K
SMRAM at default SMBASE address

On Mon, 30 Sep 2019 13:51:46 +0200
"Laszlo Ersek" <lersek@...> wrote:

Hi Igor,

On 09/24/19 13:19, Igor Mammedov wrote:
On Mon, 23 Sep 2019 20:35:02 +0200
"Laszlo Ersek" <lersek@...> wrote:
I've got good results. For this (1/2) QEMU patch:

Tested-by: Laszlo Ersek <lersek@...>

I tested the following scenarios. In every case, I verified the OVMF
log, and also the "info mtree" monitor command's result (i.e. whether
"smbase-blackhole" / "smbase-window" were disabled or enabled).
Mostly, I diffed these text files between the test scenarios (looking
for desired / undesired differences). In the Linux guests, I checked
/ compared the dmesg too (wrt. the UEFI memmap).

- unpatched OVMF (regression test), Fedora guest, normal boot and S3

- patched OVMF, but feature disabled with "-global
mch.smbase-smram=off" (another regression test), Fedora guest,
normal boot and S3

- patched OVMF, feature enabled, Fedora and various Windows guests
(win7, win8, win10 families, client/server), normal boot and S3

- a subset of the above guests, with S3 disabled (-global
ICH9-LPC.disable_s3=1), and obviously S3 resume not tested

SEV: used 5.2-ish Linux guest, with S3 disabled (no support under SEV
for that now):

- unpatched OVMF (regression test), normal boot

- patched OVMF but feature disabled on the QEMU cmdline (another
regression test), normal boot

- patched OVMF, feature enabled, normal boot.

I plan to post the OVMF patches tomorrow, for discussion.

(It's likely too early to push these QEMU / edk2 patches right now --
we don't know yet if this path will take us to the destination. For
now, it certainly looks great.)
Laszlo, thanks for trying it out.
It's nice to hear that approach is somewhat usable.
Hopefully we won't have to invent 'paused' cpu mode.

Pls CC me on your patches
(not that I qualify for reviewing,
but may be I could learn a thing or two from it)
Considering the plan at [1], the two patch sets [2] [3] should cover
step (01); at least as proof of concept.

[1] [edk2-devel] CPU hotplug using SMM with QEMU+OVMF
http://mid.mail-archive.com/20190830164802.1b17ff26@redhat.com

[2] The current thread:
[Qemu-devel] [PATCH 0/2] q35: mch: allow to lock down 128K RAM at
default SMBASE address
http://mid.mail-archive.com/20190917130708.10281-1-
imammedo@...

[3] [edk2-devel] [PATCH wave 1 00/10] support QEMU's "SMRAM at default
SMBASE" feature
http://mid.mail-archive.com/20190924113505.27272-1-lersek@redhat.com

(I'll have to figure out what SMI handler to put in place there, but I'd
like to experiment with that once we can cause a new CPU to start
executing code there, in SMM.)

So what's next?

To me it looks like we need to figure out how QEMU can make the OS call
into SMM (in the GPE cpu hotplug handler), passing in parameters and
such. This would be step (03).

Do you agree?

If so, I'll ask Jiewen about such OS->SMM calls separately, because I
seem to remember that there used to be an "SMM communcation table" of
sorts, for flexible OS->SMM calls. However, it appears to be deprecated
lately.
we can try to resurrect and put over it some kind of protocol
to describe which CPUs to where hotplugged.

or we could put a parameter into SMI status register (IO port 0xb3)
and the trigger SMI from GPE handler to tell SMI handler that cpu
hotplug happened and then use QEMU's cpu hotplug interface
to enumerate hotplugged CPUs for SMI handler.

The later is probably simpler as we won't need to reinvent the wheel
(just reuse the interface that's already in use by GPE handler).
[Jiewen] The PI specification Volume 4 - SMM defines EFI_MM_COMMUNICATION_PROTOCOL.Communicate() - It can be used to communicate between OS and SMM handler. But it requires the runtime protocol call. I am not sure how OS loader passes this information to OS kernel.

As such, I think using ACPI SCI/GPE -> software SMI handler is an easier way to achieve this. I also recommend this way.
For parameter passing, we can use 1) Port B2 (1 byte), 2) Port B3 (1 byte), 3) chipset scratch register (4 bytes or 8 bytes, based upon scratch register size), 4) ACPI NVS OPREGION, if the data structure is complicated.



Hmmm.... Yes, UEFI 2.8 has "Appendix O - UEFI ACPI Data Table", and it
writes (after defining the table format):

The first use of this UEFI ACPI table format is the SMM
Communication ACPI Table. This table describes a special software
SMI that can be used to initiate inter-mode communication in the OS
present environment by non-firmware agents with SMM code.

Note: The use of the SMM Communication ACPI table is deprecated in
UEFI spec. 2.7. This is due to the lack of a use case for
inter-mode communication by non-firmware agents with SMM code
and support for initiating this form of communication in
common OSes.

The changelog at the front of the UEFI spec also references the
Mantis#1691 spec ticket, "Remove/Deprecate SMM Communication ACPI
Table"
(addressed in UEFI 2.6B).

(I think that must have been a security ticket, because, while I
generally have access to Mantis tickets,
<https://mantis.uefi.org/mantis/view.php?id=1631> gives me "Access
Denied" :/ )

Thanks,
Laszlo




Re: [edk2-devel] [Qemu-devel] [PATCH 1/2] q35: implement 128K SMRAM at default SMBASE address

Igor Mammedov <imammedo@...>
 

On Mon, 30 Sep 2019 13:51:46 +0200
"Laszlo Ersek" <lersek@...> wrote:

Hi Igor,

On 09/24/19 13:19, Igor Mammedov wrote:
On Mon, 23 Sep 2019 20:35:02 +0200
"Laszlo Ersek" <lersek@...> wrote:
I've got good results. For this (1/2) QEMU patch:

Tested-by: Laszlo Ersek <lersek@...>

I tested the following scenarios. In every case, I verified the OVMF
log, and also the "info mtree" monitor command's result (i.e. whether
"smbase-blackhole" / "smbase-window" were disabled or enabled).
Mostly, I diffed these text files between the test scenarios (looking
for desired / undesired differences). In the Linux guests, I checked
/ compared the dmesg too (wrt. the UEFI memmap).

- unpatched OVMF (regression test), Fedora guest, normal boot and S3

- patched OVMF, but feature disabled with "-global
mch.smbase-smram=off" (another regression test), Fedora guest,
normal boot and S3

- patched OVMF, feature enabled, Fedora and various Windows guests
(win7, win8, win10 families, client/server), normal boot and S3

- a subset of the above guests, with S3 disabled (-global
ICH9-LPC.disable_s3=1), and obviously S3 resume not tested

SEV: used 5.2-ish Linux guest, with S3 disabled (no support under SEV
for that now):

- unpatched OVMF (regression test), normal boot

- patched OVMF but feature disabled on the QEMU cmdline (another
regression test), normal boot

- patched OVMF, feature enabled, normal boot.

I plan to post the OVMF patches tomorrow, for discussion.

(It's likely too early to push these QEMU / edk2 patches right now --
we don't know yet if this path will take us to the destination. For
now, it certainly looks great.)
Laszlo, thanks for trying it out.
It's nice to hear that approach is somewhat usable.
Hopefully we won't have to invent 'paused' cpu mode.

Pls CC me on your patches
(not that I qualify for reviewing,
but may be I could learn a thing or two from it)
Considering the plan at [1], the two patch sets [2] [3] should cover
step (01); at least as proof of concept.

[1] [edk2-devel] CPU hotplug using SMM with QEMU+OVMF
http://mid.mail-archive.com/20190830164802.1b17ff26@redhat.com

[2] The current thread:
[Qemu-devel] [PATCH 0/2] q35: mch: allow to lock down 128K RAM at default SMBASE address
http://mid.mail-archive.com/20190917130708.10281-1-imammedo@redhat.com

[3] [edk2-devel] [PATCH wave 1 00/10] support QEMU's "SMRAM at default SMBASE" feature
http://mid.mail-archive.com/20190924113505.27272-1-lersek@redhat.com

(I'll have to figure out what SMI handler to put in place there, but I'd
like to experiment with that once we can cause a new CPU to start
executing code there, in SMM.)

So what's next?

To me it looks like we need to figure out how QEMU can make the OS call
into SMM (in the GPE cpu hotplug handler), passing in parameters and
such. This would be step (03).

Do you agree?

If so, I'll ask Jiewen about such OS->SMM calls separately, because I
seem to remember that there used to be an "SMM communcation table" of
sorts, for flexible OS->SMM calls. However, it appears to be deprecated
lately.
we can try to resurrect and put over it some kind of protocol
to describe which CPUs to where hotplugged.

or we could put a parameter into SMI status register (IO port 0xb3)
and the trigger SMI from GPE handler to tell SMI handler that cpu
hotplug happened and then use QEMU's cpu hotplug interface
to enumerate hotplugged CPUs for SMI handler.

The later is probably simpler as we won't need to reinvent the wheel
(just reuse the interface that's already in use by GPE handler).

Hmmm.... Yes, UEFI 2.8 has "Appendix O - UEFI ACPI Data Table", and it
writes (after defining the table format):

The first use of this UEFI ACPI table format is the SMM
Communication ACPI Table. This table describes a special software
SMI that can be used to initiate inter-mode communication in the OS
present environment by non-firmware agents with SMM code.

Note: The use of the SMM Communication ACPI table is deprecated in
UEFI spec. 2.7. This is due to the lack of a use case for
inter-mode communication by non-firmware agents with SMM code
and support for initiating this form of communication in
common OSes.

The changelog at the front of the UEFI spec also references the
Mantis#1691 spec ticket, "Remove/Deprecate SMM Communication ACPI Table"
(addressed in UEFI 2.6B).

(I think that must have been a security ticket, because, while I
generally have access to Mantis tickets,
<https://mantis.uefi.org/mantis/view.php?id=1631> gives me "Access
Denied" :/ )

Thanks,
Laszlo



Re: [edk2-devel] [Qemu-devel] [PATCH 1/2] q35: implement 128K SMRAM at default SMBASE address

Laszlo Ersek
 

Hi Igor,

On 09/24/19 13:19, Igor Mammedov wrote:
On Mon, 23 Sep 2019 20:35:02 +0200
"Laszlo Ersek" <lersek@...> wrote:
I've got good results. For this (1/2) QEMU patch:

Tested-by: Laszlo Ersek <lersek@...>

I tested the following scenarios. In every case, I verified the OVMF
log, and also the "info mtree" monitor command's result (i.e. whether
"smbase-blackhole" / "smbase-window" were disabled or enabled).
Mostly, I diffed these text files between the test scenarios (looking
for desired / undesired differences). In the Linux guests, I checked
/ compared the dmesg too (wrt. the UEFI memmap).

- unpatched OVMF (regression test), Fedora guest, normal boot and S3

- patched OVMF, but feature disabled with "-global
mch.smbase-smram=off" (another regression test), Fedora guest,
normal boot and S3

- patched OVMF, feature enabled, Fedora and various Windows guests
(win7, win8, win10 families, client/server), normal boot and S3

- a subset of the above guests, with S3 disabled (-global
ICH9-LPC.disable_s3=1), and obviously S3 resume not tested

SEV: used 5.2-ish Linux guest, with S3 disabled (no support under SEV
for that now):

- unpatched OVMF (regression test), normal boot

- patched OVMF but feature disabled on the QEMU cmdline (another
regression test), normal boot

- patched OVMF, feature enabled, normal boot.

I plan to post the OVMF patches tomorrow, for discussion.

(It's likely too early to push these QEMU / edk2 patches right now --
we don't know yet if this path will take us to the destination. For
now, it certainly looks great.)
Laszlo, thanks for trying it out.
It's nice to hear that approach is somewhat usable.
Hopefully we won't have to invent 'paused' cpu mode.

Pls CC me on your patches
(not that I qualify for reviewing,
but may be I could learn a thing or two from it)
Considering the plan at [1], the two patch sets [2] [3] should cover
step (01); at least as proof of concept.

[1] [edk2-devel] CPU hotplug using SMM with QEMU+OVMF
http://mid.mail-archive.com/20190830164802.1b17ff26@redhat.com

[2] The current thread:
[Qemu-devel] [PATCH 0/2] q35: mch: allow to lock down 128K RAM at default SMBASE address
http://mid.mail-archive.com/20190917130708.10281-1-imammedo@redhat.com

[3] [edk2-devel] [PATCH wave 1 00/10] support QEMU's "SMRAM at default SMBASE" feature
http://mid.mail-archive.com/20190924113505.27272-1-lersek@redhat.com

(I'll have to figure out what SMI handler to put in place there, but I'd
like to experiment with that once we can cause a new CPU to start
executing code there, in SMM.)

So what's next?

To me it looks like we need to figure out how QEMU can make the OS call
into SMM (in the GPE cpu hotplug handler), passing in parameters and
such. This would be step (03).

Do you agree?

If so, I'll ask Jiewen about such OS->SMM calls separately, because I
seem to remember that there used to be an "SMM communcation table" of
sorts, for flexible OS->SMM calls. However, it appears to be deprecated
lately.

Hmmm.... Yes, UEFI 2.8 has "Appendix O - UEFI ACPI Data Table", and it
writes (after defining the table format):

The first use of this UEFI ACPI table format is the SMM
Communication ACPI Table. This table describes a special software
SMI that can be used to initiate inter-mode communication in the OS
present environment by non-firmware agents with SMM code.

Note: The use of the SMM Communication ACPI table is deprecated in
UEFI spec. 2.7. This is due to the lack of a use case for
inter-mode communication by non-firmware agents with SMM code
and support for initiating this form of communication in
common OSes.

The changelog at the front of the UEFI spec also references the
Mantis#1691 spec ticket, "Remove/Deprecate SMM Communication ACPI Table"
(addressed in UEFI 2.6B).

(I think that must have been a security ticket, because, while I
generally have access to Mantis tickets,
<https://mantis.uefi.org/mantis/view.php?id=1631> gives me "Access
Denied" :/ )

Thanks,
Laszlo


Re: UEFI accessibility mandate

Andrew Fish <afish@...>
 

On Sep 26, 2019, at 5:15 AM, Rafael Machado <rafaelrodrigues.machado@...> wrote:

Hi everyone.

About Ethin's question:
As for the boot manager, yes, that would be covered by the speech synthesizer protocol. Perhaps we could extend this to the setup utility too (as well as all other things displayed on the screen)?
Answer: This is the target from my point of view. Since we have the advantages of drivers at UEFI, there is no reason for not having accessibility at pre-OS anymore. We could also think in future to add other kinds of accessibility, like the magnifying glass (that would be some multi-layer version of the GOP driver), that would help low-vision people. Just one thing for everyone to think about. The "Convention on the Rights of Persons with Disabilities" say that a person has the right to access information. So if someone buys a notebook, all information accessible to "common" people must be available to all kinds of people, in a way they can choose to use this information or not.
" (e) Recognizing that disability is an evolving concept and that disability results from the interaction between persons with impairments and attitudinal and environmental barriers that hinders their full and effective participation in society on an equal basis with others, ..."
Details can be found at the United Nations web site https://www.un.org/disabilities/documents/convention/convention_accessible_pdf.pdf

About Andrew's comments and questions:
What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL should contain?
Answer: From my perspective, this protocol would need at least the following functions (names may change after more discussion.)
- EFI_AUDIO_OUTPUT_PROTOCOL.LoadStream()
This function is intended to copy some data to the dma buffers that were allocated previously by the AudioController driver. The initial idea I had was to add, in the case of my MSc, a HdaDriver.efi that is responsible to do all the HDA controller initialization (allocating dma buffers, configuring the PCIe bus if needed). Not sure if this initialization should we added to the EFI_AUDIO_OUTPUT_PROTOCOL, because this could create problems when using other controllers or concepts, like I believe ARM platforms do.My understanding is that the platforms should have a driver to initialize the audio controller/codec, so the EFI_AUDIO_OUTPUT_PROTOCOL is just worried with loading and playing information.

- EFI_AUDIO_OUTPUT_PROTOCOL.PlayStream()
This function is responsible to play the stream. This is done by setting one specific bit at the controllers configuration space (pointed by the bar[0] pcie configuration register at the pcie config space.) At this project I don't think we need to be worried with implementing an sound server to enable multiple streams at a time. To the actions done by this function are much simpler. Just play a stream.

- EFI_AUDIO_OUTPUT_PROTOCOL.StopStream()
This function is responsible to stop the stream play. The opposite of the previous one. Maybe we could use a single function that receives some parameter to start or stop the play.

- EFI_AUDIO_OUTPUT_PROTOCOL.SetVolume()
This function is responsible to set the volume to be used at all nodes at the codec (so it depends on a EFI_AUDIO_CODEC_PROTOCOL that knows the commands a given codec needs to receive to process streams.)

- EFI_AUDIO_OUTPUT_PROTOCOL.SetStreamSpeed()
This function is responsible to set the speed a given stream should be played on. Just to clarify, visual impaired people have an amazing capacity of understanding sound streams in incredibly fast speeds. So the possibility of setting the stream processing speed is what makes them be as productive as a person with usual vision capacities.

- EFI_AUDIO_OUTPUT_PROTOCOL.UnloadStream()
This function is responsible to clean the buffers and letting them prepared to receive other audio streams.

Still about Andrews comments.
This gave me the idea of adding a EFI_AUDIO_OUTPUT_PROTOCOL and EFI_TEXT_TO_SPEECH_PROTOCOL driver to the EmulatorPkg that thunk down to the App to do the work. With this we could prototype the UI part of accessibility.
Answer: This would be great, because we would split the development. One focusing on the HII and navigation (done normally using the tab key, so each element at the screen would need an tab sequence number and a accessibility tag with what the speech software should say), and another focusing on the drivers/synthesizer/audio files depending on our future decisions.

So my understanding is that the list of things is increasing. It is actually something like this (not ordered by priority):
- EFI_AUDIO_OUTPUT_PROTOCOL: Protocol used to process the streams
- EFI_AUDIO_CODEC_PROTOCOL: Protocol used to communicate with the codec
- Audio configuration driver (HdaAudio.efi for example). Should this create some other protocol, like EFI_AUDIO_CONTROLLER_PROTOCOL that is responsible to configure the controller?
Rafael,

Generally how it works for a PCI device is it gets its Start() function called to bind to the device. So the EFI_AUDIO_OUTPUT_PROTOCOL and EFI_AUDIO_CODEC_PROTOCOL protocols would likely get installed on the PCI IO handle of the HDA PCI hardware. Some basic init will happen during the Start() and the driver could defer some config until one of the member functions is called the 1st time.

So for example on a normal boot the Audio driver may not even get started. So there may have to be an nvram variable to start the driver. We might also need an NVRAM variable for the voice to text driver if it is stored on disk.

- EFI_TEXT_TO_SPEECH_PROTOCOL: Protocol responsible to convert text to speech (using the EFI_AUDIO_OUTPUT_PROTOCOL )
- HII changes to add tab navigation capabilities and accessibility tags at each component on the screen

Are we aligned with the understanding?
I think so and if we can mock an implementation in the EmulatorPkg that would let the HII UI work happen independently of the text to speech, and Audio driver work.

I'm on some air plaines in the next weeks (I'm at the airport right now), and my wife is heading to a conference after I get back from my trips. Given my dogs are not very talkative I might find some time to try to get the EmulatorPkg mocking working.

Thanks,

Andrew Fish

I believe soon we will be prepared to talk about this on a design meeting.

Thanks and Regards
Rafael R. Machado



Em qua, 25 de set de 2019 às 22:08, Andrew Fish <afish@... <mailto:afish@...>> escreveu:
Rafael,

What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL should contain?

I'm thinking if we had an EFI_TEXT_TO_SPEECH_PROTOCOL that driver could produce a PCM/Wave buffer, it could then use EFI_AUDIO_OUTPUT_PROTOCOL to play the sound.

I poked around my Mac at lunch and I can generate text to speech from the command line into a wave file via the `say` command line tool. I can play the wave file from the command line via `afplay`. This gave me the idea of adding a EFI_AUDIO_OUTPUT_PROTOCOL and EFI_TEXT_TO_SPEECH_PROTOCOL driver to the EmulatorPkg that thunk down to the App to do the work. With this we could prototype the UI part of accessibility. We could also debug the EFI EFI_TEXT_TO_SPEECH_PROTOCOL in this environment.

Thanks,

Andrew Fish

On Sep 21, 2019, at 5:36 AM, Rafael Machado <rafaelrodrigues.machado@... <mailto:rafaelrodrigues.machado@...>> wrote:

Hi Everyone
Sorry for the delay on the response, to many things happening at the same time.
I will try to answer e-mails to this thread every Saturday or Sunday morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time to help implement this feature, but I happy to help work on the architecture and design for UEFI accessibility on the edk2 mailing lists, and I >>can also represent what ever we come up with at the UEFI Spec Work Group.
During my MSc I had to study a lot the audio and BIOS architectures. The idea was to eliminate the first barrier to the creation of a screen reader for pre-OS environment, that was the lack of some open implementation of audio control and actions at UEFI. To do that I studied the Intel High Definition Audio Spec and a lot of UEFI specs to understand better how to do that.
The initial target was to do all this development at OVMF, but as far as I could get, the sound card is not available to OVMF. as Laszlo mentioned at this e-mail there are some projects that may help on this, but at the time I was working on my MSc I didn't find this, so I did everything on a real system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and working on a MSc degree at the same time having a 40hours/week job, being a father and a husband is not an easy task, but it got to what I was expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA memory access. For the ones that have good years, it's possible to listen some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was that one of my write operations was causing some misunderstanding between the audio controller and the audio codec. The controller was sending packets with 16bit depth, but the codec was processing them as 8bit depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing at the OS level.
The reference code, that is just a proof-of-concept, and that has several things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to UEFI drivers after some discussion. Everything is released as BDS so companies can use without IP problems.
Just to give some more information about the need of this kind of solution. There is a lot of blind people that work with hardware support, so formatting disk, configuring RAID and booting dual-boot systems is always a challenge to them. Even set the BIOS clock. How to do that without the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given there is no guideline on how a User Interface (UI) works. If accessibility requires some from of hardware abstraction, like audio, then we could >>likely get that into the UEFI Spec. What might be possible is an EDK2 reference implementation of accessibility. Maybe we could use the reference implementation to write a UEFI white paper on design >>for accessibility? I there is an open source implementation, and an official design guide this would make it much easier for advocacy groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2 papers is one of my targets since the beginning of my MSc almost 5 years ago.

I've got some experience with accessibility as the macOS EFI OS Loader has a UI for the Full Disk Encryption password. If you press the power button quickly 3 times at the disk encryption password prompt >>accessibility is enabled and Voice Over gets turned on. You then get localized voice prompts when you move between UI elements. Since this is the OS loader all the resources are stored on the disk. You >>quickly run into a range of challenges since, audio is hard, abstracting audio is hard (what codec does firmware have to support), Audio files are not small and firmware is size constrained, the need to localize >>the audio responses causes even more size issues, the boot options are usually written by an OS installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so no audio files would need to be stored. There are some open-source implementations that are not GPL.
This was described at my MSc as future works that can continue what I have started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We choose to go to the Moon!" speech sometimes we chose to do things "not because they are easy, but because they are hard; because that >>goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept". If we have a design that means we can break the problem up into >>smaller parts, and maybe we can find people that have expertise in that part to build a chunk at a time. If we could implement the prototype in OVMF that would show how it works, but run on everyone's >>machines, so that would be really helpful for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible future steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application remains functional.
- 4) Separate controller and codec code by creating an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate out the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder audio files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the need of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my thesis to English.
I wrote everything in Portuguese because there was not material about UEFI to the Brazilian audience, and another target I have is to show companies that we have people that can work at this kind of projects in Brazil, bringing this kind of development to south america. (Yes, I have complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@... <mailto:lersek@...>> escreveu:
On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo


Re: UEFI accessibility mandate

Ethin Probst
 

I'd like to point out that blind people do not just use speed
alterations to PCM data to achive such incredibly fast speeds; speed
alterations also come from the fact that the speech synthesizer is
capable of generating synthesis of slower and faster speeds depending
on settings. The initial design looks good; I like the idea of using a
speech synthesis protocol instead of recorded audio files.
Also, we don't just include tab navigation; we include the arrow keys
and any other keys that the firmware may use to facilitate
interaction.

On 9/26/19, Rafael Machado <rafaelrodrigues.machado@...> wrote:
Hi everyone.

About Ethin's question:
As for the boot manager, yes, that would be covered by the speech
synthesizer protocol. Perhaps we could extend this to the setup utility too
(as well as all other things displayed on the screen)?
Answer: This is the target from my point of view. Since we have the
advantages of drivers at UEFI, there is no reason for not having
accessibility at pre-OS anymore. We could also think in future to add other
kinds of accessibility, like the magnifying glass (that would be some
multi-layer version of the GOP driver), that would help low-vision people.
Just one thing for everyone to think about. The "Convention on the Rights
of Persons with Disabilities" say that a person has the right to access
information. So if someone buys a notebook, all information accessible to
"common" people must be available to all kinds of people, in a way they can
choose to use this information or not.
*" (e) Recognizing that disability is an evolving concept and that
disability results from the interaction between persons with impairments
and attitudinal and environmental barriers that hinders their full and
effective participation in society on an equal basis with others, ..."*
Details can be found at the United Nations web site
https://www.un.org/disabilities/documents/convention/convention_accessible_pdf.pdf


About Andrew's comments and questions:
What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL should
contain?
Answer: From my perspective, this protocol would need at least the
following functions (names may change after more discussion.)

- EFI_AUDIO_OUTPUT_PROTOCOL.LoadStream()
This function is intended to copy some data to the dma buffers that
were allocated previously by the AudioController driver. The initial idea I
had was to add, in the case of my MSc, a HdaDriver.efi that is responsible
to do all the HDA controller initialization (allocating dma buffers,
configuring the PCIe bus if needed). Not sure if this initialization should
we added to the EFI_AUDIO_OUTPUT_PROTOCOL, because this could create
problems when using other controllers or concepts, like I believe ARM
platforms do.My understanding is that the platforms should have a driver to
initialize the audio controller/codec, so the EFI_AUDIO_OUTPUT_PROTOCOL is
just worried with loading and playing information.

- EFI_AUDIO_OUTPUT_PROTOCOL.PlayStream()
This function is responsible to play the stream. This is done by
setting one specific bit at the controllers configuration space (pointed by
the bar[0] pcie configuration register at the pcie config space.) At this
project I don't think we need to be worried with implementing an sound
server to enable multiple streams at a time. To the actions done by this
function are much simpler. Just play a stream.

- EFI_AUDIO_OUTPUT_PROTOCOL.StopStream()
This function is responsible to stop the stream play. The opposite of
the previous one. Maybe we could use a single function that receives some
parameter to start or stop the play.

- EFI_AUDIO_OUTPUT_PROTOCOL.SetVolume()
This function is responsible to set the volume to be used at all nodes
at the codec (so it depends on a EFI_AUDIO_CODEC_PROTOCOL that knows the
commands a given codec needs to receive to process streams.)

- EFI_AUDIO_OUTPUT_PROTOCOL.SetStreamSpeed()
This function is responsible to set the speed a given stream should be
played on. Just to clarify, visual impaired people have an amazing capacity
of understanding sound streams in incredibly fast speeds. So the
possibility of setting the stream processing speed is what makes them be as
productive as a person with usual vision capacities.

- EFI_AUDIO_OUTPUT_PROTOCOL.UnloadStream()
This function is responsible to clean the buffers and letting them
prepared to receive other audio streams.

Still about Andrews comments.
This gave me the idea of adding a EFI_AUDIO_OUTPUT_PROTOCOL and
EFI_TEXT_TO_SPEECH_PROTOCOL
driver to the EmulatorPkg that thunk down to the App to do the work. With
this we could prototype the UI part of accessibility.
Answer: This would be great, because we would split the development. One
focusing on the HII and navigation (done normally using the tab key, so
each element at the screen would need an tab sequence number and a
accessibility tag with what the speech software should say), and another
focusing on the drivers/synthesizer/audio files depending on our future
decisions.

So my understanding is that the list of things is increasing. It is
actually something like this (not ordered by priority):
- EFI_AUDIO_OUTPUT_PROTOCOL: Protocol used to process the streams
- EFI_AUDIO_CODEC_PROTOCOL: Protocol used to communicate with the codec
- Audio configuration driver (HdaAudio.efi for example). Should this create
some other protocol, like EFI_AUDIO_CONTROLLER_PROTOCOL that is
responsible to configure the controller?
- EFI_TEXT_TO_SPEECH_PROTOCOL: Protocol responsible to convert text to
speech (using the EFI_AUDIO_OUTPUT_PROTOCOL )
- HII changes to add tab navigation capabilities and accessibility tags at
each component on the screen

Are we aligned with the understanding?

I believe soon we will be prepared to talk about this on a design meeting.

Thanks and Regards
Rafael R. Machado



Em qua, 25 de set de 2019 às 22:08, Andrew Fish <afish@...> escreveu:

Rafael,

What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL should
contain?

I'm thinking if we had an EFI_TEXT_TO_SPEECH_PROTOCOL that driver could
produce a PCM/Wave buffer, it could then use EFI_AUDIO_OUTPUT_PROTOCOL to
play the sound.

I poked around my Mac at lunch and I can generate text to speech from the
command line into a wave file via the `say` command line tool. I can play
the wave file from the command line via `afplay`. This gave me the idea
of
adding a EFI_AUDIO_OUTPUT_PROTOCOL and EFI_TEXT_TO_SPEECH_PROTOCOL
driver to the EmulatorPkg that thunk down to the App to do the work.
With this we could prototype the UI part of accessibility. We could also
debug the EFI EFI_TEXT_TO_SPEECH_PROTOCOL in this environment.

Thanks,

Andrew Fish

On Sep 21, 2019, at 5:36 AM, Rafael Machado <
rafaelrodrigues.machado@...> wrote:

Hi Everyone
Sorry for the delay on the response, to many things happening at the same
time.
I will try to answer e-mails to this thread every Saturday or Sunday
morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time
to
help implement this feature, but I happy to help work on the architecture
and design for UEFI accessibility on the edk2 mailing lists, and I >>can
also represent what ever we come up with at the UEFI Spec Work Group.
During my MSc I had to study a lot the audio and BIOS architectures. The
idea was to eliminate the first barrier to the creation of a screen
reader
for pre-OS environment, that was the lack of some open implementation of
audio control and actions at UEFI. To do that I studied the Intel High
Definition Audio Spec and a lot of UEFI specs to understand better how to
do that.
The initial target was to do all this development at OVMF, but as far as
I
could get, the sound card is not available to OVMF. as Laszlo mentioned
at
this e-mail there are some projects that may help on this, but at the
time
I was working on my MSc I didn't find this, so I did everything on a real
system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and
working on a MSc degree at the same time having a 40hours/week job, being
a
father and a husband is not an easy task, but it got to what I was
expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that
later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA
memory access. For the ones that have good years, it's possible to listen
some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was
that
one of my write operations was causing some misunderstanding between the
audio controller and the audio codec. The controller was sending packets
with 16bit depth, but the codec was processing them as 8bit depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing at
the OS level.
The reference code, that is just a proof-of-concept, and that has several
things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to UEFI
drivers after some discussion. Everything is released as BDS so companies
can use without IP problems.
Just to give some more information about the need of this kind of
solution. There is a lot of blind people that work with hardware support,
so formatting disk, configuring RAID and booting dual-boot systems is
always a challenge to them. Even set the BIOS clock. How to do that
without
the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given there
is no guideline on how a User Interface (UI) works. If accessibility
requires some from of hardware abstraction, like audio, then we could
likely get that into the UEFI Spec. What might be possible is an EDK2
reference implementation of accessibility. Maybe we could use the
reference
implementation to write a UEFI white paper on design >>for accessibility?
I
there is an open source implementation, and an official design guide this
would make it much easier for advocacy groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2 papers
is one of my targets since the beginning of my MSc almost 5 years ago.

I've got some experience with accessibility as the macOS EFI OS Loader
has a UI for the Full Disk Encryption password. If you press the power
button quickly 3 times at the disk encryption password prompt
accessibility is enabled and Voice Over gets turned on. You then get
localized voice prompts when you move between UI elements. Since this is
the OS loader all the resources are stored on the disk. You >>quickly run
into a range of challenges since, audio is hard, abstracting audio is
hard
(what codec does firmware have to support), Audio files are not small and
firmware is size constrained, the need to localize >>the audio responses
causes even more size issues, the boot options are usually written by an
OS
installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so
no
audio files would need to be stored. There are some open-source
implementations that are not GPL.
This was described at my MSc as future works that can continue what I
have
started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things "not
because they are easy, but because they are hard; because that >>goal
will
serve to organize and measure the best of our energies and skills,
because
that challenge is one that we are willing to accept". If we have a design
that means we can break the problem up into >>smaller parts, and maybe we
can find people that have expertise in that part to build a chunk at a
time. If we could implement the prototype in OVMF that would show how it
works, but run on everyone's >>machines, so that would be really helpful
for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible future
steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with
proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver
to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also
producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application
remains functional.
- 4) Separate controller and codec code by creating
an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate
out
the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder
audio
files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the need
of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my
thesis
to English.
I wrote everything in Portuguese because there was not material about
UEFI
to the Brazilian audience, and another target I have is to show companies
that we have people that can work at this kind of projects in Brazil,
bringing this kind of development to south america. (Yes, I have
complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@...>
escreveu:

On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo

--
Signed,
Ethin D. Probst


Re: UEFI accessibility mandate

Rafael Machado <rafaelrodrigues.machado@...>
 

Hi everyone.

About Ethin's question:
As for the boot manager, yes, that would be covered by the speech
synthesizer protocol. Perhaps we could extend this to the setup utility too
(as well as all other things displayed on the screen)?
Answer: This is the target from my point of view. Since we have the
advantages of drivers at UEFI, there is no reason for not having
accessibility at pre-OS anymore. We could also think in future to add other
kinds of accessibility, like the magnifying glass (that would be some
multi-layer version of the GOP driver), that would help low-vision people.
Just one thing for everyone to think about. The "Convention on the Rights
of Persons with Disabilities" say that a person has the right to access
information. So if someone buys a notebook, all information accessible to
"common" people must be available to all kinds of people, in a way they can
choose to use this information or not.
*" (e) Recognizing that disability is an evolving concept and that
disability results from the interaction between persons with impairments
and attitudinal and environmental barriers that hinders their full and
effective participation in society on an equal basis with others, ..."*
Details can be found at the United Nations web site
https://www.un.org/disabilities/documents/convention/convention_accessible_pdf.pdf


About Andrew's comments and questions:
What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL should
contain?
Answer: From my perspective, this protocol would need at least the
following functions (names may change after more discussion.)

- EFI_AUDIO_OUTPUT_PROTOCOL.LoadStream()
This function is intended to copy some data to the dma buffers that
were allocated previously by the AudioController driver. The initial idea I
had was to add, in the case of my MSc, a HdaDriver.efi that is responsible
to do all the HDA controller initialization (allocating dma buffers,
configuring the PCIe bus if needed). Not sure if this initialization should
we added to the EFI_AUDIO_OUTPUT_PROTOCOL, because this could create
problems when using other controllers or concepts, like I believe ARM
platforms do.My understanding is that the platforms should have a driver to
initialize the audio controller/codec, so the EFI_AUDIO_OUTPUT_PROTOCOL is
just worried with loading and playing information.

- EFI_AUDIO_OUTPUT_PROTOCOL.PlayStream()
This function is responsible to play the stream. This is done by
setting one specific bit at the controllers configuration space (pointed by
the bar[0] pcie configuration register at the pcie config space.) At this
project I don't think we need to be worried with implementing an sound
server to enable multiple streams at a time. To the actions done by this
function are much simpler. Just play a stream.

- EFI_AUDIO_OUTPUT_PROTOCOL.StopStream()
This function is responsible to stop the stream play. The opposite of
the previous one. Maybe we could use a single function that receives some
parameter to start or stop the play.

- EFI_AUDIO_OUTPUT_PROTOCOL.SetVolume()
This function is responsible to set the volume to be used at all nodes
at the codec (so it depends on a EFI_AUDIO_CODEC_PROTOCOL that knows the
commands a given codec needs to receive to process streams.)

- EFI_AUDIO_OUTPUT_PROTOCOL.SetStreamSpeed()
This function is responsible to set the speed a given stream should be
played on. Just to clarify, visual impaired people have an amazing capacity
of understanding sound streams in incredibly fast speeds. So the
possibility of setting the stream processing speed is what makes them be as
productive as a person with usual vision capacities.

- EFI_AUDIO_OUTPUT_PROTOCOL.UnloadStream()
This function is responsible to clean the buffers and letting them
prepared to receive other audio streams.

Still about Andrews comments.
This gave me the idea of adding a EFI_AUDIO_OUTPUT_PROTOCOL and EFI_TEXT_TO_SPEECH_PROTOCOL
driver to the EmulatorPkg that thunk down to the App to do the work. With
this we could prototype the UI part of accessibility.
Answer: This would be great, because we would split the development. One
focusing on the HII and navigation (done normally using the tab key, so
each element at the screen would need an tab sequence number and a
accessibility tag with what the speech software should say), and another
focusing on the drivers/synthesizer/audio files depending on our future
decisions.

So my understanding is that the list of things is increasing. It is
actually something like this (not ordered by priority):
- EFI_AUDIO_OUTPUT_PROTOCOL: Protocol used to process the streams
- EFI_AUDIO_CODEC_PROTOCOL: Protocol used to communicate with the codec
- Audio configuration driver (HdaAudio.efi for example). Should this create
some other protocol, like EFI_AUDIO_CONTROLLER_PROTOCOL that is
responsible to configure the controller?
- EFI_TEXT_TO_SPEECH_PROTOCOL: Protocol responsible to convert text to
speech (using the EFI_AUDIO_OUTPUT_PROTOCOL )
- HII changes to add tab navigation capabilities and accessibility tags at
each component on the screen

Are we aligned with the understanding?

I believe soon we will be prepared to talk about this on a design meeting.

Thanks and Regards
Rafael R. Machado



Em qua, 25 de set de 2019 às 22:08, Andrew Fish <afish@...> escreveu:

Rafael,

What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL should
contain?

I'm thinking if we had an EFI_TEXT_TO_SPEECH_PROTOCOL that driver could
produce a PCM/Wave buffer, it could then use EFI_AUDIO_OUTPUT_PROTOCOL to
play the sound.

I poked around my Mac at lunch and I can generate text to speech from the
command line into a wave file via the `say` command line tool. I can play
the wave file from the command line via `afplay`. This gave me the idea of
adding a EFI_AUDIO_OUTPUT_PROTOCOL and EFI_TEXT_TO_SPEECH_PROTOCOL
driver to the EmulatorPkg that thunk down to the App to do the work.
With this we could prototype the UI part of accessibility. We could also
debug the EFI EFI_TEXT_TO_SPEECH_PROTOCOL in this environment.

Thanks,

Andrew Fish

On Sep 21, 2019, at 5:36 AM, Rafael Machado <
rafaelrodrigues.machado@...> wrote:

Hi Everyone
Sorry for the delay on the response, to many things happening at the same
time.
I will try to answer e-mails to this thread every Saturday or Sunday
morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time to
help implement this feature, but I happy to help work on the architecture
and design for UEFI accessibility on the edk2 mailing lists, and I >>can
also represent what ever we come up with at the UEFI Spec Work Group.
During my MSc I had to study a lot the audio and BIOS architectures. The
idea was to eliminate the first barrier to the creation of a screen reader
for pre-OS environment, that was the lack of some open implementation of
audio control and actions at UEFI. To do that I studied the Intel High
Definition Audio Spec and a lot of UEFI specs to understand better how to
do that.
The initial target was to do all this development at OVMF, but as far as I
could get, the sound card is not available to OVMF. as Laszlo mentioned at
this e-mail there are some projects that may help on this, but at the time
I was working on my MSc I didn't find this, so I did everything on a real
system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and
working on a MSc degree at the same time having a 40hours/week job, being a
father and a husband is not an easy task, but it got to what I was
expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that
later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA
memory access. For the ones that have good years, it's possible to listen
some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was that
one of my write operations was causing some misunderstanding between the
audio controller and the audio codec. The controller was sending packets
with 16bit depth, but the codec was processing them as 8bit depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing at
the OS level.
The reference code, that is just a proof-of-concept, and that has several
things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to UEFI
drivers after some discussion. Everything is released as BDS so companies
can use without IP problems.
Just to give some more information about the need of this kind of
solution. There is a lot of blind people that work with hardware support,
so formatting disk, configuring RAID and booting dual-boot systems is
always a challenge to them. Even set the BIOS clock. How to do that without
the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given there
is no guideline on how a User Interface (UI) works. If accessibility
requires some from of hardware abstraction, like audio, then we could
likely get that into the UEFI Spec. What might be possible is an EDK2
reference implementation of accessibility. Maybe we could use the reference
implementation to write a UEFI white paper on design >>for accessibility? I
there is an open source implementation, and an official design guide this
would make it much easier for advocacy groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2 papers
is one of my targets since the beginning of my MSc almost 5 years ago.

I've got some experience with accessibility as the macOS EFI OS Loader
has a UI for the Full Disk Encryption password. If you press the power
button quickly 3 times at the disk encryption password prompt
accessibility is enabled and Voice Over gets turned on. You then get
localized voice prompts when you move between UI elements. Since this is
the OS loader all the resources are stored on the disk. You >>quickly run
into a range of challenges since, audio is hard, abstracting audio is hard
(what codec does firmware have to support), Audio files are not small and
firmware is size constrained, the need to localize >>the audio responses
causes even more size issues, the boot options are usually written by an OS
installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so no
audio files would need to be stored. There are some open-source
implementations that are not GPL.
This was described at my MSc as future works that can continue what I have
started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things "not
because they are easy, but because they are hard; because that >>goal will
serve to organize and measure the best of our energies and skills, because
that challenge is one that we are willing to accept". If we have a design
that means we can break the problem up into >>smaller parts, and maybe we
can find people that have expertise in that part to build a chunk at a
time. If we could implement the prototype in OVMF that would show how it
works, but run on everyone's >>machines, so that would be really helpful
for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible future
steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with
proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver
to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also
producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application
remains functional.
- 4) Separate controller and codec code by creating
an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate out
the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder audio
files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the need
of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my thesis
to English.
I wrote everything in Portuguese because there was not material about UEFI
to the Brazilian audience, and another target I have is to show companies
that we have people that can work at this kind of projects in Brazil,
bringing this kind of development to south america. (Yes, I have
complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@...>
escreveu:

On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo


Re: UEFI accessibility mandate

Andrew Fish <afish@...>
 

Rafael,

What member functions to you think that EFI_AUDIO_OUTPUT_PROTOCOL should contain?

I'm thinking if we had an EFI_TEXT_TO_SPEECH_PROTOCOL that driver could produce a PCM/Wave buffer, it could then use EFI_AUDIO_OUTPUT_PROTOCOL to play the sound.

I poked around my Mac at lunch and I can generate text to speech from the command line into a wave file via the `say` command line tool. I can play the wave file from the command line via `afplay`. This gave me the idea of adding a EFI_AUDIO_OUTPUT_PROTOCOL and EFI_TEXT_TO_SPEECH_PROTOCOL driver to the EmulatorPkg that thunk down to the App to do the work. With this we could prototype the UI part of accessibility. We could also debug the EFI EFI_TEXT_TO_SPEECH_PROTOCOL in this environment.

Thanks,

Andrew Fish

On Sep 21, 2019, at 5:36 AM, Rafael Machado <rafaelrodrigues.machado@... <mailto:rafaelrodrigues.machado@...>> wrote:

Hi Everyone
Sorry for the delay on the response, to many things happening at the same time.
I will try to answer e-mails to this thread every Saturday or Sunday morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time to help implement this feature, but I happy to help work on the architecture and design for UEFI accessibility on the edk2 mailing lists, and I >>can also represent what ever we come up with at the UEFI Spec Work Group.
During my MSc I had to study a lot the audio and BIOS architectures. The idea was to eliminate the first barrier to the creation of a screen reader for pre-OS environment, that was the lack of some open implementation of audio control and actions at UEFI. To do that I studied the Intel High Definition Audio Spec and a lot of UEFI specs to understand better how to do that.
The initial target was to do all this development at OVMF, but as far as I could get, the sound card is not available to OVMF. as Laszlo mentioned at this e-mail there are some projects that may help on this, but at the time I was working on my MSc I didn't find this, so I did everything on a real system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and working on a MSc degree at the same time having a 40hours/week job, being a father and a husband is not an easy task, but it got to what I was expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA memory access. For the ones that have good years, it's possible to listen some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was that one of my write operations was causing some misunderstanding between the audio controller and the audio codec. The controller was sending packets with 16bit depth, but the codec was processing them as 8bit depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing at the OS level.
The reference code, that is just a proof-of-concept, and that has several things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to UEFI drivers after some discussion. Everything is released as BDS so companies can use without IP problems.
Just to give some more information about the need of this kind of solution. There is a lot of blind people that work with hardware support, so formatting disk, configuring RAID and booting dual-boot systems is always a challenge to them. Even set the BIOS clock. How to do that without the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given there is no guideline on how a User Interface (UI) works. If accessibility requires some from of hardware abstraction, like audio, then we could >>likely get that into the UEFI Spec. What might be possible is an EDK2 reference implementation of accessibility. Maybe we could use the reference implementation to write a UEFI white paper on design >>for accessibility? I there is an open source implementation, and an official design guide this would make it much easier for advocacy groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2 papers is one of my targets since the beginning of my MSc almost 5 years ago.

I've got some experience with accessibility as the macOS EFI OS Loader has a UI for the Full Disk Encryption password. If you press the power button quickly 3 times at the disk encryption password prompt >>accessibility is enabled and Voice Over gets turned on. You then get localized voice prompts when you move between UI elements. Since this is the OS loader all the resources are stored on the disk. You >>quickly run into a range of challenges since, audio is hard, abstracting audio is hard (what codec does firmware have to support), Audio files are not small and firmware is size constrained, the need to localize >>the audio responses causes even more size issues, the boot options are usually written by an OS installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so no audio files would need to be stored. There are some open-source implementations that are not GPL.
This was described at my MSc as future works that can continue what I have started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We choose to go to the Moon!" speech sometimes we chose to do things "not because they are easy, but because they are hard; because that >>goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept". If we have a design that means we can break the problem up into >>smaller parts, and maybe we can find people that have expertise in that part to build a chunk at a time. If we could implement the prototype in OVMF that would show how it works, but run on everyone's >>machines, so that would be really helpful for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible future steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application remains functional.
- 4) Separate controller and codec code by creating an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate out the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder audio files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the need of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my thesis to English.
I wrote everything in Portuguese because there was not material about UEFI to the Brazilian audience, and another target I have is to show companies that we have people that can work at this kind of projects in Brazil, bringing this kind of development to south america. (Yes, I have complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@... <mailto:lersek@...>> escreveu:
On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo

581 - 600 of 772