Date   

Re: [edk2-devel] [Qemu-devel] [PATCH 1/2] q35: implement 128K SMRAM at default SMBASE address

Paolo Bonzini <pbonzini@...>
 

On 20/09/19 11:28, Laszlo Ersek wrote:
On QEMU side, we can drop black-hole approach and allocate
dedicated SMRAM region, which explicitly gets mapped into
RAM address space and after SMI hanlder initialization, gets
unmapped (locked). So that SMRAM would be accessible only
from SMM context. That way RAM at 0x30000 could be used as
normal when SMRAM is unmapped.
I prefer the black-hole approach, introduced in your current patch
series, if it can work. Way less opportunity for confusion.
Another possibility would be to alias the 0xA0000..0xBFFFF SMRAM to
0x30000..0x4FFFF (only when in SMM).

I'm not super enthusiastic about adding this kind of QEMU-only feature.
The alternative would be to implement VT-d range locking through the
intel-iommu device's PCI configuration space (which includes _adding_
the configuration space, i.e. making the IOMMU a PCI device in the first
place, and the support to the firmware for configuring the VT-d BAR at
0xfed90000). This would be the right way to do it, but it would entail
a lot of work throughout the stack. :( So I guess some variant of this
would be okay, as long as it's peppered with "this is not how real
hardware does it" comments in both QEMU and EDK2.

Thanks,

Paolo

I've started work on the counterpart OVMF patches; I'll report back.


Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

Liming Gao
 

Mike:

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Michael D Kinney
Sent: Tuesday, September 24, 2019 1:44 AM
To: Sean Brogan <sean.brogan@...>; devel@edk2.groups.io; rfc@edk2.groups.io; Kinney, Michael D
<michael.d.kinney@...>
Cc: Bret Barkelew <Bret.Barkelew@...>
Subject: Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

Hi Sean,

For host based tests, I agree that VS2017 or VS2019
would be a good choice. Pick the one with the best
coverage and easiest for developers to get feedback on
the test results and test coverage. That may be sufficient
for automated CI tests. Enabling other tool chains for
host based testing will be required to support developers
that implement unit tests and want to test them locally
before adding to the automated CI tests.

For build tests, I think we need VS2015, VS2017, GCC,
and XCODE5. We can add VS2019 if there is a request to make
that one of the fully validated tool chains for EDK II.
VS2019 tool chain has been added for IA32/X64/ARM/AARH64/EBC archs.
CLANG9 tool chain for IA32/X64 will be added. I request to add CLANG9 build test.

Thanks
Liming
For pre-commit gates, we could choose to do build tests of
only the libs/modules touched by the patch series. Then
perform complete package/platform build tests in a
Daily/Weekly/Release scope.

We need to update the RFC with the specific tool chains and
tool versions for both Host Based tests and Code Compilation
Tests. They are not the same.

Thanks,

Mike

-----Original Message-----
From: Sean Brogan <sean.brogan@...>
Sent: Friday, September 20, 2019 2:30 PM
To: Kinney, Michael D <michael.d.kinney@...>;
devel@edk2.groups.io; rfc@edk2.groups.io; Kinney,
Michael D <michael.d.kinney@...>
Cc: Bret Barkelew <Bret.Barkelew@...>
Subject: RE: [edk2-devel] [RFC] EDK II Continuous
Integration Phase 1

Currently it is building on Windows with VS2019.
VS2017 would be trivial but not worth it in my
perspective given how aligned the two are. If you
really wanted to do a weekly or nightly build it could
be added to that but I have been focused on a PR build.
I have a pipeline for GCC on linux. It doesn't'
support host based unit tests but I am working to get
the rest of the tests running.
I do not have a pipeline for Mac or LLVM/Clang. That
would be next and I think your files below should help.

To date we use PYTOOLs features to install our extra
tools. This makes it super easy and works for both
local and server based builds. It gives strong
versioning and management of that. It also tracks
those versions and you can see them on every build in
the Built Tool Report artifact. It can be downloaded
here for html version.
https://dev.azure.com/tianocore/edk2-ci-
play/_build/results?buildId=803&view=artifacts.
BUILD_TOOL_REPORT.html

Here it is pasted as plain text below. You can see the
iasl and nasm versions here.

Key Value Type
TOOL_CHAIN_TAG VS2019 TOOL
VC Version 14.22.27905 TOOL
d:\a\1\s\Conf\build_rule.txt 1.04 INFO
d:\a\1\s\Conf\target.txt 1.03 INFO
d:\a\1\s\Conf\tools_def.txt 1.22 INFO
iasl 20190215.0.0 INFO
Mu-Basetools 2019.03.1 INFO
mu_nasm 2.14.02 INFO


Hope that helps.

I would be happy to queue up this topic for next design
meeting and open it up to questions? I hope others
would take a look prior to meeting.

Thanks
Sean



-----Original Message-----
From: Kinney, Michael D <michael.d.kinney@...>
Sent: Thursday, September 19, 2019 2:56 PM
To: devel@edk2.groups.io; Sean Brogan
<sean.brogan@...>; rfc@edk2.groups.io;
Kinney, Michael D <michael.d.kinney@...>
Cc: Bret Barkelew <Bret.Barkelew@...>
Subject: RE: [edk2-devel] [RFC] EDK II Continuous
Integration Phase 1

Hi Sean,

Which OS/Compiler configurations are currently enabled
for the Code Compilation Test?

I have been working on enabling multiple OS/Compiler
configurations in Azure Pipelines. There are some
tools that need to be installed for each of these
environments.
Examples include NASM, iASL, Python.

For the work you have done, how are these extra tools
installed? Is it in the YML files or in the Python
scripts.

One critical task is to identify the tools and their
specific versions that the CI system is configured to
use.
These configurations should be documented in a Wiki
page and updated as new tools are released and adopted
by EDK II.
The inventory of tools used to validate a release
should Also be documented in a release notes for a
stable tag.

Here are the YML files that install the additional
tools required to support EDK II builds. I need the
source and versions of these tools to be reviewed and
approved.

https://nam06.safelinks.protection.outlook.com/?url=htt
ps%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-
ci%2Fblob%2Fmaster%2FAzurePipelines%2FWindowsPrerequisi
tes.yml&amp;data=02%7C01%7Csean.brogan%40microsoft.com%
7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91
ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=h
M4AKoMutISI5Oc%2FeVMevx%2FVRUCjJHuhEwqom0R30Ak%3D&amp;r
eserved=0
https://nam06.safelinks.protection.outlook.com/?url=htt
ps%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-
ci%2Fblob%2Fmaster%2FAzurePipelines%2FUbuntuPrerequisit
es.yml&amp;data=02%7C01%7Csean.brogan%40microsoft.com%7
C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91a
b2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=KQ
2zas2bJ%2FCjSRWGyMrxq5Rk4cW5lOgXQNR99QJbEKY%3D&amp;rese
rved=0
https://nam06.safelinks.protection.outlook.com/?url=htt
ps%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-
ci%2Fblob%2Fmaster%2FAzurePipelines%2FMacOsPrerequisite
s.yml&amp;data=02%7C01%7Csean.brogan%40microsoft.com%7C
9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91ab
2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=OzA
mzYkRJ3rQOHEDgKTREz%2BNp7acOWUACh1s%2Fb4UPGk%3D&amp;res
erved=0

Thanks,

Mike

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On
Behalf Of Sean
via Groups.Io
Sent: Thursday, August 29, 2019 7:22 PM
To: rfc@edk2.groups.io; Kinney, Michael D
<michael.d.kinney@...>; devel@edk2.groups.io
Cc: Bret Barkelew <Bret.Barkelew@...>
Subject: Re: [edk2-devel] [RFC] EDK II Continuous
Integration Phase 1

Mike, as you mentioned we have been working towards
enabling a
practical and extensible CI for Edk2 using Azure dev
ops and the
recently added edk2-pytool infrastructure. We have
been using similar
CI for Project Mu for the last few years.

Our approach is a little different in that we focus
on validating the
whole code base rather than just the incoming patch.
We do this
because we have found unexpected consequences of
patches and overall
we want all code to be compliant not just new
additions. We have
found the time to test the whole tree is not much
longer than only the
parts impacted by a code change (except maybe when
running the entire
compile test on every package). This obviously comes
with an initial
tax of needing to get the codebase into compliant
form.
Anyway we have prepared an RFC in addition to yours
and would like to
see these two efforts merged together.

We are still working on making a few optimizations.
Currently if the full set of tests are run we take
about 20 minutes.
This is because compiling MdeModulePkg for debug,
release, and host
based tests take a while. Most other packages are in
the 10 minute
range. We do have easy ways to disable or limit
certain tests as well
as expand the matrix to leverage more cloud resources
(more parallel
builds).


Content is best viewed online with links to helpful
content but is
also attached below:
https://nam06.safelinks.protection.outlook.com/?url=htt
ps%3A%2F%2Fgith
ub.com%2Fspbrogan%2Fedk2-staging%2Fblob%2Fedk2-
&amp;data=02%7C01%7Csea
n.brogan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c
1e1b%7C72f988bf
86f141af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&a
mp;sdata=xOiPHH
VbMhFBsIpj%2F9mhOhrkhdsF2Gbehd6%2Frk0XYgM%3D&amp;reserv
ed=0
stuart-ci-latest/Readme-CI-RFC.md

# CI and PR Gates

## Background

Historically, while the TianoCore maintainers and
stewards have done a
fantastic job of keeping contribution policies
consistent and
contributions clean and well-documented, there have
been few processes
that ran to verify the sanity, cleanliness, and
efficacy of the
codebase, and even fewer that publicly published
their results for the
community at large. This has caused inconsistancies
and issues within
the codebase from time to time.

Adding continuous integration (and potentially PR
gates) to the checkin process ensures that simple
errors like these
are caught and can be fixed on a regular basis.

## Strategy

While a number of CI solutions exist, this proposal
will focus on the
usage of Azure Dev Ops and Build Pipelines. For
demonstration, a
sample [TianoCore
repo](https://nam06.safelinks.protection.outlook.com/?u
rl=https%3A%2F%
2Fgithub.com%2Fspbrogan%2Fedk2-
staging.git&amp;data=02%7C01%7Csean.bro
gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%
7C72f988bf86f14
1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sd
ata=QmzQhemLUNB
7FGJwoGeeewExThrUjUArxnkmtro1b8A%3D&amp;reserved=0)
(branch edk2-stuart-ci-latest) and [Dev Ops
Pipeline](https://nam06.safelinks.protection.outlook.co
m/?url=https%3A
%2F%2Fdev.azure.com%2Ftianocore%2Fedk2-ci-
&amp;data=02%7C01%7Csean.bro
gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%
7C72f988bf86f14
1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sd
ata=rS%2FrjIFkS
uEhI%2FDWyslf34i711%2FbY8Gw%2Byexq%2FwydjU%3D&amp;reser
ved=0
play/_build?definitionId=12) have been set up.

Furthermore, this proposal will leverage the
TianoCore python tools
PIP modules:
[library](https://nam06.safelinks.protection.outlook.co
m/?url=https%3A
%2F%2Fpypi.org%2Fproject%2Fedk2-pytool-
&amp;data=02%7C01%7Csean.brogan
%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C7
2f988bf86f141af
91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata
=jzM6MKZFFAEvyR
4h0%2Bkf5pUcBSsoVzkIHFFi1Bd6Il4%3D&amp;reserved=0
library/) and
[extensions](https://nam06.safelinks.protection.outlook
.com/?url=https
%3A%2F%2Fpypi.org%2Fproject%2Fedk2-pytool-
&amp;data=02%7C01%7Csean.bro
gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%
7C72f988bf86f14
1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sd
ata=jzM6MKZFFAE
vyR4h0%2Bkf5pUcBSsoVzkIHFFi1Bd6Il4%3D&amp;reserved=0
extensions/) (with repos located
[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F
%2Fgithub.com%2Ftianocore%2Fedk2-pytool-
&amp;data=02%7C01%7Csean.broga
n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C
72f988bf86f141a
f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdat
a=OwT%2BiNTN9Rv
r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
library) and
[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F
%2Fgithub.com%2Ftianocore%2Fedk2-
&amp;data=02%7C01%7Csean.brogan%40mic
rosoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988b
f86f141af91ab2d
7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=tInVw
GwCXXGBgcmHQ%2B
0UwRaEbtZWRy8hTp5wpRtN0R0%3D&amp;reserved=0
pytool-extensions)).

The primary execution flows can be found in the
`azure-
pipelines-pr-gate.yml` and `azure-pipelines-pr-gate-
linux.yml` files.
These YAML files are consumed by the Azure Dev Ops
Build Pipeline and
dictate what server resources should be used, how
they should be
configured, and what processes should be run on them.
An overview of this schema can be found
[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F
%2Fdocs.microsoft.com%2Fen-
&amp;data=02%7C01%7Csean.brogan%40microsoft
.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f14
1af91ab2d7cd011
db47%7C1%7C0%7C637045269452466989&amp;sdata=j8nyYG3Vniz
wKFZ8l1BWoLcPy%
2BuaSoVT0jN2Esi7wH8%3D&amp;reserved=0
us/azure/devops/pipelines/yaml-schema?view=azure-
devops&tabs=schema).

Inspection of these files reveals the EDKII Tools
commands that make
up the primary processes for the CI
build: 'stuart_setup', 'stuart_update', and
'stuart_ci_build'. These
commands come from the EDKII Tools PIP modules and
are configured as
described below. More documentation on the stuart
tools can be found
[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F
%2Fgithub.com%2Ftianocore%2Fedk2-pytool-
&amp;data=02%7C01%7Csean.broga
n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C
72f988bf86f141a
f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdat
a=OwT%2BiNTN9Rv
r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
extensions/blob/master/docs/using.md) and
[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F
%2Fgithub.com%2Ftianocore%2Fedk2-pytool-
&amp;data=02%7C01%7Csean.broga
n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C
72f988bf86f141a
f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdat
a=OwT%2BiNTN9Rv
r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
extensions/blob/master/docs/features/feature_invocables
.md).

## Configuration

Configuration of the CI process consists of (in order
of precedence):
* command-line arguments passed in via the Pipeline
YAML
* a per-package configuration file (e.g. `<package-
name>.mu.yaml`) that is detected by the CI system in
EDKII Tools.
* a global configuration Python module (e.g.
`CISetting.py`) passed in via the command-line

The global configuration file is described in [this
readme](https://nam06.safelinks.protection.outlook.com/
?url=https%3A%2
F%2Fgithub.com%2Ftianocore%2Fedk2-pytool-
&amp;data=02%7C01%7Csean.brog
an%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7
C72f988bf86f141
af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sda
ta=OwT%2BiNTN9R
vr14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
extensions/blob/master/docs/usability/using_settings_ma
nager.md) from the EDKII Tools documentation. This
configuration is
written as a Python module so that decisions can be
made dynamically
based on command line parameters and codebase state.

The per-package configuration file can override most
settings in the
global configuration file, but is not dynamic. This
file can be used
to skip or customize tests that may be incompatible
with a specific
package.
By default, the global configuration will try to run
all tests on all
packages.

## CI Test Types

All CI tests are instances of EDKII Tools plugins.
Documentation on the plugin system can be found
[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F
%2Fgithub.com%2Ftianocore%2Fedk2-pytool-
&amp;data=02%7C01%7Csean.broga
n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C
72f988bf86f141a
f91ab2d7cd011db47%7C1%7C0%7C637045269452476994&amp;sdat
a=1AM4geY%2BEFh
ekun7oQOYyVVt%2Bwn6CAF2hhtkS0sgwPU%3D&amp;reserved=0
extensions/blob/master/docs/usability/using_plugin_mana
ger.md) and
[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F
%2Fgithub.com%2Ftianocore%2Fedk2-
&amp;data=02%7C01%7Csean.brogan%40mic
rosoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988b
f86f141af91ab2d
7cd011db47%7C1%7C0%7C637045269452476994&amp;sdata=3vyPM
vrfG3V4doNM0%2B
cKTpDSHEEEvkfxKAtw3eiJWsY%3D&amp;reserved=0
pytool-
extensions/blob/master/docs/features/feature_plugin_man
ager.md). Upon invocation, each plugin will be passed
the path to the
current package under test and a dictionary
containing its targeted
configuration, as assembled from the command line,
per-package
configuration, and global configuration.

Note: CI plugins are considered unique from build
plugins and helper
plugins, even though some CI plugins may execute
steps of a build.

In the example, these plugins live alongside the code
under test (in
the `BaseTools` directory), but may be moved to the
'edk2-test' repo
if that location makes more sense for the community.

### Module Inclusion Test - DscCompleteCheck

This test scans all available modules (via INF files)
and compares
them to the package-level DSC file for the package
each module is
contained within. The test considers it an error if
any module does
not appear in the `Components` section of at least
one package-level
DSC (indicating that it would not be built if the
package were built).

### Code Compilation Test - CompilerPlugin

Once the Module Inclusion Test has verified that all
modules would be
built if all package-level DSCs were built, the Code
Compilation Test
simply runs through and builds every package-level
DSC on every
toolchain and for every architecture that is
supported. Any module
that fails to build is considered an error.

### Host-Based UnitTests - HostUnitTestCompilerPlugin
and
HostUnitTestDscCompleteCheck

The [Testing RFC doc](Readme-Testing-RFC.md) has much
more detail on
this, but the basic idea is that host- based unit
tests can be
compiled against individual modules and libraries and
run on the build
agent (in this case, the Dev Ops build server). The
successful and
failing test case results are collected and included
in the final
build report.

### GUID Uniqueness Test - GuidCheck

This test works on the collection of all packages
rather than an
individual package. It looks at all FILE_GUIDs and
GUIDs declared in
DEC files and ensures that they are unique for the
codebase. This
prevents, for example, accidental duplication of
GUIDs when using an
existing INF as a template for a new module.

### Cross-Package Dependency Test - DependencyCheck

This test compares the list of all packages used in
INFs files for a
given package against a list of "allowed
dependencies" in plugin
configuration for that package. Any module that
depends on a
disallowed package will cause a test failure.

### Library Declaration Test - LibraryClassCheck

This test looks at all library header files found in
a package's
`Include/Library` directory and ensures that all
files have a matching
LibraryClass declaration in the DEC file for the
package. Any missing
declarations will cause a failure.

### Invalid Character Test - CharEncodingCheck

This test scans all files in a package to make sure
that there are no
invalid Unicode characters that may cause build
errors in some
character sets/localizations.

## Next Steps

* Receive community feedback on RFC.
* Determine where this phase makes sense given
existing RFCs from
other TianoCore contributors.
* Optimize testing beharior.
* Only run a subset of tests on PRs or individual
commits.
* Run full testing either once per day or once
every several
commits.
* Add more tests/capabilities.
* Continue to improve results formatting.
* Continue to improve CI documentation.
* Much of this documentation effort is pending
community feedback on
which parts are needed and what phases are
priorities.

Thanks


-----Original Message-----
From: rfc@edk2.groups.io <rfc@edk2.groups.io> On
Behalf Of Michael D
Kinney via Groups.Io
Sent: Thursday, August 29, 2019 1:23 PM
To: devel@edk2.groups.io; rfc@edk2.groups.io
Subject: [edk2-rfc] [RFC] EDK II Continuous
Integration Phase 1

Hello,

This is a proposal for a first step towards
continuous integration for
all TianoCore repositories to help improve to quality
of commits and
automate testing and release processes for all EDK II
packages and
platforms.

This is based on work from a number of EDK II
community members that
have provide valuable input and evaluations.

* Rebecca Cran <mailto:rebecca@...> Jenkins
evaluation
* Laszlo Ersek <mailto:lersek@...> GitLab
evaluation
* Philippe Mathieu-Daudé <mailto:philmd@...>
GitLab evaluation
* Sean Brogan <mailto:sean.brogan@...>
Azure Pipelines and
HBFA
* Bret Barkelew <mailto:Bret.Barkelew@...>
Azure Pipelines and HBFA
* Jiewen Yao <mailto:jiewen.yao@...> HBFA

The following link is a link to an EDK II WIKI page
that contains a
summary of the work to date. Please provide feedback
in the EDK II
mailing lists. The WIKI pages will be updated with
input from the
entire EDK II community.


https://nam06.safelinks.protection.outlook.com/?url=htt
ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
%2Fwiki%2FEDK-II-Continuous-
Integration&amp;data=02%7C01%7Csean.brogan%40microsoft.
com%7C6f67f169a6c746b4288608d72cbea7b6%7C72f988bf86f141
af91ab2d7cd011db47%7C1%7C0%7C637027069686644659&amp;sda
ta=GR9wN6gP3mJlCTopAaQ2rlzhby1nuF%2BwDVsfFIQAZjA%3D&amp
;reserved=0

Proposal
========
Phase 1 of adding continuous integration is limited
to the
edk2 repository. Additional repositories will be
added later.

The following changes are proposed:
* Remove EDK II Maintainers write access to edk2
repository.
Only EDK II Administrators will continue to have
write
access, and that should only be used to handle
extraordinary
events.
* EDK II Maintainers use a GitHub Pull Request
instead of push
to commit a patch series to the edk2 repository.
There are
no other changes to the development and review
process. The
patch series is prepared in an EDK II maintainer
branch with
all commit message requirements met on each patch
in the series.
* The edk2 repository only accepts Pull Requests from
members
of the EDK II Maintainers team. Pull Requests from
anyone else
are rejected.
* Run pre-commit checks using Azure Pipelines
* If all pre-commit checks pass, then the patch
series is auto
committed. The result of this commit must match
the contents
and commit history that would have occurred using
the previous
push operation.
* If any pre-commit checks fail, then notify the
submitter.
A typical reason for a failure would be a merge
conflict with
another pull request that was just processed.
* Limit pre-commit checks execution time to 10
minutes.
* Provide on-demand builds to EDK II Maintainers that
to allow
EDK II Maintainers to submit a branch through for
the same
set of pre-commit checks without submitting a pull
request.

## Pre-Commit Checks in Phase 1
* Run and pass PatchCheck.py with no errors

=====================================================

The following are some additional pre-commit check
ideas that could be
quickly added once the initial version using
PatchCheck.py is fully
functional.
Please provide feedback on the ones you like and
additional ones you
think may improve the quality of the commits to the
edk2 repository.

## Proposed Pre-Commit Checks in Phase 2
* Verify Reviewed-by and Acked-by tags are present
with
correct maintainer email addresses
* Verify no non-ASCII characters in modified files
* Verify no binary files in set of modified files
* Verify package dependency rules in modified files

## Proposed Pre-Commit Checks in Phase 3
* Run ECC on modified files
* Verify modified modules/libs build
* Run available host based tests (HBFA) against
modified
modules/libs

Best regards,

Mike











Re: UEFI accessibility mandate

Rafael Machado <rafaelrodrigues.machado@...>
 

Hi Everyone

Answering Ethin:
If other platforms are PCI-based (i.e. allow us to scan the PCI bus
and figure out where (in MMIO space) the HDA controller is mapped to),
then it (theoretically) sould work. I don't know for sure though; I'm
not very knowledgeable in other CPU architectures
Answer: Neither do I. Lets wait some ARM expert to give some opinion :)

Answering Andrew
Did you do much research into CODECs? Like which one(s) should be
supported? I assume the CODEC implies the audio file formats that can be
decoded? Also how large are the audio files?
Answer: During my research I have studied some codec specs, and the way the
codecs works is really similar. Normally they need to receive some packets
to initialize the nodes (node is the name of each internal component of the
codec), by receiving some commands (named verbs) that are defined at the
HDASpec. For example the verb "power state" (0xF05) is used to set the
power state of each node on the codec, to really enable the chip or put the
chip to sleep.
The verbs can be found at this document
https://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/high-definition-audio-specification.pdf,
at page 216.
Not all verbs are standardized, for example the verb "Amplifier Gain Mute"
(0xB--), each vendor can decide the command. In the case of the ASUS system
I have used, the codec is a Conexant CX20752 (
https://www.datasheets360.com/pdf/7550682360275196829), and at the
datasheet we can see that the "Amplifier Gain Mute" verb is 0xB00 and 0xB20
(right and left channel in the case of a stereo stream) at page 33. These
verbs that are vendor defined creates the problems to think on a generic
solution.
About the second part of the question, the Codecs are independent, so they
are only responsible to process signals that are passed to them in a stream
using a DMA buffer, or a command to change their behavior (increase /
decrease volume for example). So for the codec to process correctly the
buffers managed by the audio controller, both need to be configured with
the same stream format, something like 2 channel 8 bit depth for example.
At the second video I send previously (
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be) the problem
was that the controller was configured to manage a stream with 2 channel
and 8 bits depth, but the codec was processing the data as 2 channel 16 bit
depth.
About the audio files size, at my work I didn't use any compressed format,
because I was running out of time to finish my project, so I was processing
raw audio data, that is much bigger than other formats like MP3. In case we
decide to use audio files at this first stage, maybe we will need to port
some audio format lib to UEFI also.

Is the CODEC protocol more of a plug-in for the Intel HDA? By that I mean
it only works on that hardware, or does it work generically on top of
any EFI_AUDIO_OUTPUT_PROTOCOL?
Answer: My understanding is that the CODEC protocol will need the audio
controller to be already configured, because the communication with the
codec is done using the addresses that are related to the AUDIO controller.
So the access to the AUDIO Codec is managed by the Audio controller. We
need to write the commands at the address that was previously allocated and
set as the CORB/RIRB (Command Output Ring Buffer / Response Input Ring
Buffer), and set at the HDA controller registers (page 36 of the HDA Spec)

I was starting to think about how to store to audio and deal with dynamic
configuration. I guess one crazy idea could be to have the OS create the
audio files using text to speech, and maybe store them on the EFI System
Partition. For example when an OS installs it is going to write the EFI
Boot Variable that contains a Description of the boot option. Maybe we
could convert that to audio and add a nvram variable that points to the
given audio file?
Answer: This is one option. One question. How would the OS knows the bios
options and menus so it could create these files?

Thanks and Regards
Rafael


Em seg, 23 de set de 2019 às 12:11, Andrew Fish <afish@...> escreveu:

Rafael,

Did you do much research into CODECs? Like which one(s) should be
supported? I assume th CODEC implies the audio file formats that can be
decoded? Also how large are the audio files?

Is the CODEC protocol more of a plug-in for the Intel HDA? By that I mean
it only works on that hardware, or does it work generically on top of
any EFI_AUDIO_OUTPUT_PROTOCOL?

I was starting to think about how to store to audio and deal with dynamic
configuration. I guess one crazy idea could be to have the OS create the
audio files using text to speech, and maybe store them on the EFI System
Partition. For example when an OS installs it is going to write the EFI
Boot Variable that contains a Description of the boot option. Maybe we
could convert that to audio and add a nvram variable that points to the
given audio file?

Thanks,

Andrew Fish

On Sep 23, 2019, at 6:20 AM, Rafael Machado <
rafaelrodrigues.machado@...> wrote:

Hi everyone.
So, based on everything was mentioned here.

The idea is to propose the creation of two protocols:

- EFI_AUDIO_OUTPUT_PROTOCOL: This protocol should be the responsible for
initializing the audio controller, that in the case of my MSc work does
initialization of the RING buffers, that are used as containers to the
audio streams that need to be processed.

- EFI_AUDIO_CODEC_PROTOCOL: This protocol should be responsible for
initializing the codec (each codec may need different init commands), and
also is used to control things like mute, volume level and this kind of
things. (can see the volume control actions at the last video I mentioned
on the previous e-mail)

Does this approach works at non-x86 platforms? (I don't have knowledge in
ARM platforms, so feedback from the community will be well received.)

Hope to hear some voices :)

Thanks and Regards
Rafael

Em sáb, 21 de set de 2019 às 09:36, Rafael Machado <
rafaelrodrigues.machado@...> escreveu:

Hi Everyone
Sorry for the delay on the response, to many things happening at the same
time.
I will try to answer e-mails to this thread every Saturday or Sunday
morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing lists,
and I >>can also represent what ever we come up with at the UEFI Spec Work
Group.
During my MSc I had to study a lot the audio and BIOS architectures. The
idea was to eliminate the first barrier to the creation of a screen reader
for pre-OS environment, that was the lack of some open implementation of
audio control and actions at UEFI. To do that I studied the Intel High
Definition Audio Spec and a lot of UEFI specs to understand better how to
do that.
The initial target was to do all this development at OVMF, but as far as
I could get, the sound card is not available to OVMF. as Laszlo mentioned
at this e-mail there are some projects that may help on this, but at the
time I was working on my MSc I didn't find this, so I did everything on a
real system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and
working on a MSc degree at the same time having a 40hours/week job, being a
father and a husband is not an easy task, but it got to what I was
expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that
later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA
memory access. For the ones that have good years, it's possible to listen
some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was
that one of my write operations was causing some misunderstanding between
the audio controller and the audio codec. The controller was sending
packets with 16bit depth, but the codec was processing them as 8bit depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing at
the OS level.
The reference code, that is just a proof-of-concept, and that has several
things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to UEFI
drivers after some discussion. Everything is released as BDS so companies
can use without IP problems.
Just to give some more information about the need of this kind of
solution. There is a lot of blind people that work with hardware support,
so formatting disk, configuring RAID and booting dual-boot systems is
always a challenge to them. Even set the BIOS clock. How to do that without
the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given there
is no guideline on how a User Interface (UI) works. If accessibility
requires some from of hardware abstraction, like audio, then we could
likely get that into the UEFI Spec. What might be possible is an EDK2
reference implementation of accessibility. Maybe we could use the reference
implementation to write a UEFI white paper on design >>for accessibility? I
there is an open source implementation, and an official design guide this
would make it much easier for advocacy groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2 papers
is one of my targets since the beginning of my MSc almost 5 years ago.

I've got some experience with accessibility as the macOS EFI OS Loader
has a UI for the Full Disk Encryption password. If you press the power
button quickly 3 times at the disk encryption password prompt
accessibility is enabled and Voice Over gets turned on. You then get
localized voice prompts when you move between UI elements. Since this is
the OS loader all the resources are stored on the disk. You >>quickly run
into a range of challenges since, audio is hard, abstracting audio is hard
(what codec does firmware have to support), Audio files are not small and
firmware is size constrained, the need to localize >>the audio responses
causes even more size issues, the boot options are usually written by an OS
installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so
no audio files would need to be stored. There are some open-source
implementations that are not GPL.
This was described at my MSc as future works that can continue what I
have started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things "not
because they are easy, but because they are hard; because that >>goal will
serve to organize and measure the best of our energies and skills, because
that challenge is one that we are willing to accept". If we have a design
that means we can break the problem up into >>smaller parts, and maybe we
can find people that have expertise in that part to build a chunk at a
time. If we could implement the prototype in OVMF that would show how it
works, but run on everyone's >>machines, so that would be really helpful
for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible future
steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with
proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver
to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also
producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application
remains functional.
- 4) Separate controller and codec code by creating
an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate out
the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder
audio files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the need
of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my
thesis to English.
I wrote everything in Portuguese because there was not material about
UEFI to the Brazilian audience, and another target I have is to show
companies that we have people that can work at this kind of projects in
Brazil, bringing this kind of development to south america. (Yes, I have
complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@...>
escreveu:

On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo


Re: [edk2-devel] [Qemu-devel] [PATCH 1/2] q35: implement 128K SMRAM at default SMBASE address

Igor Mammedov <imammedo@...>
 

On Mon, 23 Sep 2019 20:35:02 +0200
"Laszlo Ersek" <lersek@...> wrote:

On 09/20/19 11:28, Laszlo Ersek wrote:
On 09/20/19 10:28, Igor Mammedov wrote:
On Thu, 19 Sep 2019 19:02:07 +0200
"Laszlo Ersek" <lersek@...> wrote:

Hi Igor,

(+Brijesh)

long-ish pondering ahead, with a question at the end.
[...]

Finally: can you please remind me why we lock down 128KB (32 pages) at
0x3_0000, and not just half of that? What do we need the range at
[0x4_0000..0x4_FFFF] for?

If I recall correctly, CPU consumes 64K of save/restore area.
The rest 64K are temporary RAM for using in SMI relocation handler,
if it's possible to get away without it then we can drop it and
lock only 64K required for CPU state. It won't help with SEV
conflict though as it's in the first 64K.
OK. Let's go with 128KB for now. Shrinking the area is always easier
than growing it.

On QEMU side, we can drop black-hole approach and allocate
dedicated SMRAM region, which explicitly gets mapped into
RAM address space and after SMI hanlder initialization, gets
unmapped (locked). So that SMRAM would be accessible only
from SMM context. That way RAM at 0x30000 could be used as
normal when SMRAM is unmapped.
I prefer the black-hole approach, introduced in your current patch
series, if it can work. Way less opportunity for confusion.

I've started work on the counterpart OVMF patches; I'll report back.
I've got good results. For this (1/2) QEMU patch:

Tested-by: Laszlo Ersek <lersek@...>

I tested the following scenarios. In every case, I verified the OVMF
log, and also the "info mtree" monitor command's result (i.e. whether
"smbase-blackhole" / "smbase-window" were disabled or enabled). Mostly,
I diffed these text files between the test scenarios (looking for
desired / undesired differences). In the Linux guests, I checked /
compared the dmesg too (wrt. the UEFI memmap).

- unpatched OVMF (regression test), Fedora guest, normal boot and S3

- patched OVMF, but feature disabled with "-global mch.smbase-smram=off"
(another regression test), Fedora guest, normal boot and S3

- patched OVMF, feature enabled, Fedora and various Windows guests
(win7, win8, win10 families, client/server), normal boot and S3

- a subset of the above guests, with S3 disabled (-global
ICH9-LPC.disable_s3=1), and obviously S3 resume not tested

SEV: used 5.2-ish Linux guest, with S3 disabled (no support under SEV
for that now):

- unpatched OVMF (regression test), normal boot

- patched OVMF but feature disabled on the QEMU cmdline (another
regression test), normal boot

- patched OVMF, feature enabled, normal boot.

I plan to post the OVMF patches tomorrow, for discussion.

(It's likely too early to push these QEMU / edk2 patches right now -- we
don't know yet if this path will take us to the destination. For now, it
certainly looks great.)
Laszlo, thanks for trying it out.
It's nice to hear that approach is somewhat usable.
Hopefully we won't have to invent 'paused' cpu mode.

Pls CC me on your patches
(not that I qualify for reviewing,
but may be I could learn a thing or two from it)

Thanks
Laszlo



Re: [edk2-devel] [Qemu-devel] [PATCH 1/2] q35: implement 128K SMRAM at default SMBASE address

Laszlo Ersek
 

On 09/20/19 11:28, Laszlo Ersek wrote:
On 09/20/19 10:28, Igor Mammedov wrote:
On Thu, 19 Sep 2019 19:02:07 +0200
"Laszlo Ersek" <lersek@...> wrote:

Hi Igor,

(+Brijesh)

long-ish pondering ahead, with a question at the end.
[...]

Finally: can you please remind me why we lock down 128KB (32 pages) at
0x3_0000, and not just half of that? What do we need the range at
[0x4_0000..0x4_FFFF] for?

If I recall correctly, CPU consumes 64K of save/restore area.
The rest 64K are temporary RAM for using in SMI relocation handler,
if it's possible to get away without it then we can drop it and
lock only 64K required for CPU state. It won't help with SEV
conflict though as it's in the first 64K.
OK. Let's go with 128KB for now. Shrinking the area is always easier
than growing it.

On QEMU side, we can drop black-hole approach and allocate
dedicated SMRAM region, which explicitly gets mapped into
RAM address space and after SMI hanlder initialization, gets
unmapped (locked). So that SMRAM would be accessible only
from SMM context. That way RAM at 0x30000 could be used as
normal when SMRAM is unmapped.
I prefer the black-hole approach, introduced in your current patch
series, if it can work. Way less opportunity for confusion.

I've started work on the counterpart OVMF patches; I'll report back.
I've got good results. For this (1/2) QEMU patch:

Tested-by: Laszlo Ersek <lersek@...>

I tested the following scenarios. In every case, I verified the OVMF
log, and also the "info mtree" monitor command's result (i.e. whether
"smbase-blackhole" / "smbase-window" were disabled or enabled). Mostly,
I diffed these text files between the test scenarios (looking for
desired / undesired differences). In the Linux guests, I checked /
compared the dmesg too (wrt. the UEFI memmap).

- unpatched OVMF (regression test), Fedora guest, normal boot and S3

- patched OVMF, but feature disabled with "-global mch.smbase-smram=off"
(another regression test), Fedora guest, normal boot and S3

- patched OVMF, feature enabled, Fedora and various Windows guests
(win7, win8, win10 families, client/server), normal boot and S3

- a subset of the above guests, with S3 disabled (-global
ICH9-LPC.disable_s3=1), and obviously S3 resume not tested

SEV: used 5.2-ish Linux guest, with S3 disabled (no support under SEV
for that now):

- unpatched OVMF (regression test), normal boot

- patched OVMF but feature disabled on the QEMU cmdline (another
regression test), normal boot

- patched OVMF, feature enabled, normal boot.

I plan to post the OVMF patches tomorrow, for discussion.

(It's likely too early to push these QEMU / edk2 patches right now -- we
don't know yet if this path will take us to the destination. For now, it
certainly looks great.)

Thanks
Laszlo


Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

Michael D Kinney
 

Hi Sean,

For host based tests, I agree that VS2017 or VS2019
would be a good choice. Pick the one with the best
coverage and easiest for developers to get feedback on
the test results and test coverage. That may be sufficient
for automated CI tests. Enabling other tool chains for
host based testing will be required to support developers
that implement unit tests and want to test them locally
before adding to the automated CI tests.

For build tests, I think we need VS2015, VS2017, GCC,
and XCODE5. We can add VS2019 if there is a request to make
that one of the fully validated tool chains for EDK II.
For pre-commit gates, we could choose to do build tests of
only the libs/modules touched by the patch series. Then
perform complete package/platform build tests in a
Daily/Weekly/Release scope.

We need to update the RFC with the specific tool chains and
tool versions for both Host Based tests and Code Compilation
Tests. They are not the same.

Thanks,

Mike

-----Original Message-----
From: Sean Brogan <sean.brogan@...>
Sent: Friday, September 20, 2019 2:30 PM
To: Kinney, Michael D <michael.d.kinney@...>;
devel@edk2.groups.io; rfc@edk2.groups.io; Kinney,
Michael D <michael.d.kinney@...>
Cc: Bret Barkelew <Bret.Barkelew@...>
Subject: RE: [edk2-devel] [RFC] EDK II Continuous
Integration Phase 1

Currently it is building on Windows with VS2019.
VS2017 would be trivial but not worth it in my
perspective given how aligned the two are. If you
really wanted to do a weekly or nightly build it could
be added to that but I have been focused on a PR build.
I have a pipeline for GCC on linux. It doesn't'
support host based unit tests but I am working to get
the rest of the tests running.
I do not have a pipeline for Mac or LLVM/Clang. That
would be next and I think your files below should help.

To date we use PYTOOLs features to install our extra
tools. This makes it super easy and works for both
local and server based builds. It gives strong
versioning and management of that. It also tracks
those versions and you can see them on every build in
the Built Tool Report artifact. It can be downloaded
here for html version.
https://dev.azure.com/tianocore/edk2-ci-
play/_build/results?buildId=803&view=artifacts.
BUILD_TOOL_REPORT.html

Here it is pasted as plain text below. You can see the
iasl and nasm versions here.

Key Value Type
TOOL_CHAIN_TAG VS2019 TOOL
VC Version 14.22.27905 TOOL
d:\a\1\s\Conf\build_rule.txt 1.04 INFO
d:\a\1\s\Conf\target.txt 1.03 INFO
d:\a\1\s\Conf\tools_def.txt 1.22 INFO
iasl 20190215.0.0 INFO
Mu-Basetools 2019.03.1 INFO
mu_nasm 2.14.02 INFO


Hope that helps.

I would be happy to queue up this topic for next design
meeting and open it up to questions? I hope others
would take a look prior to meeting.

Thanks
Sean



-----Original Message-----
From: Kinney, Michael D <michael.d.kinney@...>
Sent: Thursday, September 19, 2019 2:56 PM
To: devel@edk2.groups.io; Sean Brogan
<sean.brogan@...>; rfc@edk2.groups.io;
Kinney, Michael D <michael.d.kinney@...>
Cc: Bret Barkelew <Bret.Barkelew@...>
Subject: RE: [edk2-devel] [RFC] EDK II Continuous
Integration Phase 1

Hi Sean,

Which OS/Compiler configurations are currently enabled
for the Code Compilation Test?

I have been working on enabling multiple OS/Compiler
configurations in Azure Pipelines. There are some
tools that need to be installed for each of these
environments.
Examples include NASM, iASL, Python.

For the work you have done, how are these extra tools
installed? Is it in the YML files or in the Python
scripts.

One critical task is to identify the tools and their
specific versions that the CI system is configured to
use.
These configurations should be documented in a Wiki
page and updated as new tools are released and adopted
by EDK II.
The inventory of tools used to validate a release
should Also be documented in a release notes for a
stable tag.

Here are the YML files that install the additional
tools required to support EDK II builds. I need the
source and versions of these tools to be reviewed and
approved.

https://nam06.safelinks.protection.outlook.com/?url=htt
ps%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-
ci%2Fblob%2Fmaster%2FAzurePipelines%2FWindowsPrerequisi
tes.yml&amp;data=02%7C01%7Csean.brogan%40microsoft.com%
7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91
ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=h
M4AKoMutISI5Oc%2FeVMevx%2FVRUCjJHuhEwqom0R30Ak%3D&amp;r
eserved=0
https://nam06.safelinks.protection.outlook.com/?url=htt
ps%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-
ci%2Fblob%2Fmaster%2FAzurePipelines%2FUbuntuPrerequisit
es.yml&amp;data=02%7C01%7Csean.brogan%40microsoft.com%7
C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91a
b2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=KQ
2zas2bJ%2FCjSRWGyMrxq5Rk4cW5lOgXQNR99QJbEKY%3D&amp;rese
rved=0
https://nam06.safelinks.protection.outlook.com/?url=htt
ps%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-
ci%2Fblob%2Fmaster%2FAzurePipelines%2FMacOsPrerequisite
s.yml&amp;data=02%7C01%7Csean.brogan%40microsoft.com%7C
9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91ab
2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=OzA
mzYkRJ3rQOHEDgKTREz%2BNp7acOWUACh1s%2Fb4UPGk%3D&amp;res
erved=0

Thanks,

Mike

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On
Behalf Of Sean
via Groups.Io
Sent: Thursday, August 29, 2019 7:22 PM
To: rfc@edk2.groups.io; Kinney, Michael D
<michael.d.kinney@...>; devel@edk2.groups.io
Cc: Bret Barkelew <Bret.Barkelew@...>
Subject: Re: [edk2-devel] [RFC] EDK II Continuous
Integration Phase 1

Mike, as you mentioned we have been working towards
enabling a
practical and extensible CI for Edk2 using Azure dev
ops and the
recently added edk2-pytool infrastructure. We have
been using similar
CI for Project Mu for the last few years.

Our approach is a little different in that we focus
on validating the
whole code base rather than just the incoming patch.
We do this
because we have found unexpected consequences of
patches and overall
we want all code to be compliant not just new
additions. We have
found the time to test the whole tree is not much
longer than only the
parts impacted by a code change (except maybe when
running the entire
compile test on every package). This obviously comes
with an initial
tax of needing to get the codebase into compliant
form.
Anyway we have prepared an RFC in addition to yours
and would like to
see these two efforts merged together.

We are still working on making a few optimizations.
Currently if the full set of tests are run we take
about 20 minutes.
This is because compiling MdeModulePkg for debug,
release, and host
based tests take a while. Most other packages are in
the 10 minute
range. We do have easy ways to disable or limit
certain tests as well
as expand the matrix to leverage more cloud resources
(more parallel
builds).


Content is best viewed online with links to helpful
content but is
also attached below:
https://nam06.safelinks.protection.outlook.com/?url=htt
ps%3A%2F%2Fgith
ub.com%2Fspbrogan%2Fedk2-staging%2Fblob%2Fedk2-
&amp;data=02%7C01%7Csea
n.brogan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c
1e1b%7C72f988bf
86f141af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&a
mp;sdata=xOiPHH
VbMhFBsIpj%2F9mhOhrkhdsF2Gbehd6%2Frk0XYgM%3D&amp;reserv
ed=0
stuart-ci-latest/Readme-CI-RFC.md

# CI and PR Gates

## Background

Historically, while the TianoCore maintainers and
stewards have done a
fantastic job of keeping contribution policies
consistent and
contributions clean and well-documented, there have
been few processes
that ran to verify the sanity, cleanliness, and
efficacy of the
codebase, and even fewer that publicly published
their results for the
community at large. This has caused inconsistancies
and issues within
the codebase from time to time.

Adding continuous integration (and potentially PR
gates) to the checkin process ensures that simple
errors like these
are caught and can be fixed on a regular basis.

## Strategy

While a number of CI solutions exist, this proposal
will focus on the
usage of Azure Dev Ops and Build Pipelines. For
demonstration, a
sample [TianoCore
repo](https://nam06.safelinks.protection.outlook.com/?u
rl=https%3A%2F%
2Fgithub.com%2Fspbrogan%2Fedk2-
staging.git&amp;data=02%7C01%7Csean.bro
gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%
7C72f988bf86f14
1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sd
ata=QmzQhemLUNB
7FGJwoGeeewExThrUjUArxnkmtro1b8A%3D&amp;reserved=0)
(branch edk2-stuart-ci-latest) and [Dev Ops
Pipeline](https://nam06.safelinks.protection.outlook.co
m/?url=https%3A
%2F%2Fdev.azure.com%2Ftianocore%2Fedk2-ci-
&amp;data=02%7C01%7Csean.bro
gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%
7C72f988bf86f14
1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sd
ata=rS%2FrjIFkS
uEhI%2FDWyslf34i711%2FbY8Gw%2Byexq%2FwydjU%3D&amp;reser
ved=0
play/_build?definitionId=12) have been set up.

Furthermore, this proposal will leverage the
TianoCore python tools
PIP modules:
[library](https://nam06.safelinks.protection.outlook.co
m/?url=https%3A
%2F%2Fpypi.org%2Fproject%2Fedk2-pytool-
&amp;data=02%7C01%7Csean.brogan
%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C7
2f988bf86f141af
91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata
=jzM6MKZFFAEvyR
4h0%2Bkf5pUcBSsoVzkIHFFi1Bd6Il4%3D&amp;reserved=0
library/) and
[extensions](https://nam06.safelinks.protection.outlook
.com/?url=https
%3A%2F%2Fpypi.org%2Fproject%2Fedk2-pytool-
&amp;data=02%7C01%7Csean.bro
gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%
7C72f988bf86f14
1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sd
ata=jzM6MKZFFAE
vyR4h0%2Bkf5pUcBSsoVzkIHFFi1Bd6Il4%3D&amp;reserved=0
extensions/) (with repos located
[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F
%2Fgithub.com%2Ftianocore%2Fedk2-pytool-
&amp;data=02%7C01%7Csean.broga
n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C
72f988bf86f141a
f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdat
a=OwT%2BiNTN9Rv
r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
library) and
[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F
%2Fgithub.com%2Ftianocore%2Fedk2-
&amp;data=02%7C01%7Csean.brogan%40mic
rosoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988b
f86f141af91ab2d
7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=tInVw
GwCXXGBgcmHQ%2B
0UwRaEbtZWRy8hTp5wpRtN0R0%3D&amp;reserved=0
pytool-extensions)).

The primary execution flows can be found in the
`azure-
pipelines-pr-gate.yml` and `azure-pipelines-pr-gate-
linux.yml` files.
These YAML files are consumed by the Azure Dev Ops
Build Pipeline and
dictate what server resources should be used, how
they should be
configured, and what processes should be run on them.
An overview of this schema can be found
[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F
%2Fdocs.microsoft.com%2Fen-
&amp;data=02%7C01%7Csean.brogan%40microsoft
.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f14
1af91ab2d7cd011
db47%7C1%7C0%7C637045269452466989&amp;sdata=j8nyYG3Vniz
wKFZ8l1BWoLcPy%
2BuaSoVT0jN2Esi7wH8%3D&amp;reserved=0
us/azure/devops/pipelines/yaml-schema?view=azure-
devops&tabs=schema).

Inspection of these files reveals the EDKII Tools
commands that make
up the primary processes for the CI
build: 'stuart_setup', 'stuart_update', and
'stuart_ci_build'. These
commands come from the EDKII Tools PIP modules and
are configured as
described below. More documentation on the stuart
tools can be found
[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F
%2Fgithub.com%2Ftianocore%2Fedk2-pytool-
&amp;data=02%7C01%7Csean.broga
n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C
72f988bf86f141a
f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdat
a=OwT%2BiNTN9Rv
r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
extensions/blob/master/docs/using.md) and
[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F
%2Fgithub.com%2Ftianocore%2Fedk2-pytool-
&amp;data=02%7C01%7Csean.broga
n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C
72f988bf86f141a
f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdat
a=OwT%2BiNTN9Rv
r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
extensions/blob/master/docs/features/feature_invocables
.md).

## Configuration

Configuration of the CI process consists of (in order
of precedence):
* command-line arguments passed in via the Pipeline
YAML
* a per-package configuration file (e.g. `<package-
name>.mu.yaml`) that is detected by the CI system in
EDKII Tools.
* a global configuration Python module (e.g.
`CISetting.py`) passed in via the command-line

The global configuration file is described in [this
readme](https://nam06.safelinks.protection.outlook.com/
?url=https%3A%2
F%2Fgithub.com%2Ftianocore%2Fedk2-pytool-
&amp;data=02%7C01%7Csean.brog
an%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7
C72f988bf86f141
af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sda
ta=OwT%2BiNTN9R
vr14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
extensions/blob/master/docs/usability/using_settings_ma
nager.md) from the EDKII Tools documentation. This
configuration is
written as a Python module so that decisions can be
made dynamically
based on command line parameters and codebase state.

The per-package configuration file can override most
settings in the
global configuration file, but is not dynamic. This
file can be used
to skip or customize tests that may be incompatible
with a specific
package.
By default, the global configuration will try to run
all tests on all
packages.

## CI Test Types

All CI tests are instances of EDKII Tools plugins.
Documentation on the plugin system can be found
[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F
%2Fgithub.com%2Ftianocore%2Fedk2-pytool-
&amp;data=02%7C01%7Csean.broga
n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C
72f988bf86f141a
f91ab2d7cd011db47%7C1%7C0%7C637045269452476994&amp;sdat
a=1AM4geY%2BEFh
ekun7oQOYyVVt%2Bwn6CAF2hhtkS0sgwPU%3D&amp;reserved=0
extensions/blob/master/docs/usability/using_plugin_mana
ger.md) and
[here](https://nam06.safelinks.protection.outlook.com/?
url=https%3A%2F
%2Fgithub.com%2Ftianocore%2Fedk2-
&amp;data=02%7C01%7Csean.brogan%40mic
rosoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988b
f86f141af91ab2d
7cd011db47%7C1%7C0%7C637045269452476994&amp;sdata=3vyPM
vrfG3V4doNM0%2B
cKTpDSHEEEvkfxKAtw3eiJWsY%3D&amp;reserved=0
pytool-
extensions/blob/master/docs/features/feature_plugin_man
ager.md). Upon invocation, each plugin will be passed
the path to the
current package under test and a dictionary
containing its targeted
configuration, as assembled from the command line,
per-package
configuration, and global configuration.

Note: CI plugins are considered unique from build
plugins and helper
plugins, even though some CI plugins may execute
steps of a build.

In the example, these plugins live alongside the code
under test (in
the `BaseTools` directory), but may be moved to the
'edk2-test' repo
if that location makes more sense for the community.

### Module Inclusion Test - DscCompleteCheck

This test scans all available modules (via INF files)
and compares
them to the package-level DSC file for the package
each module is
contained within. The test considers it an error if
any module does
not appear in the `Components` section of at least
one package-level
DSC (indicating that it would not be built if the
package were built).

### Code Compilation Test - CompilerPlugin

Once the Module Inclusion Test has verified that all
modules would be
built if all package-level DSCs were built, the Code
Compilation Test
simply runs through and builds every package-level
DSC on every
toolchain and for every architecture that is
supported. Any module
that fails to build is considered an error.

### Host-Based UnitTests - HostUnitTestCompilerPlugin
and
HostUnitTestDscCompleteCheck

The [Testing RFC doc](Readme-Testing-RFC.md) has much
more detail on
this, but the basic idea is that host- based unit
tests can be
compiled against individual modules and libraries and
run on the build
agent (in this case, the Dev Ops build server). The
successful and
failing test case results are collected and included
in the final
build report.

### GUID Uniqueness Test - GuidCheck

This test works on the collection of all packages
rather than an
individual package. It looks at all FILE_GUIDs and
GUIDs declared in
DEC files and ensures that they are unique for the
codebase. This
prevents, for example, accidental duplication of
GUIDs when using an
existing INF as a template for a new module.

### Cross-Package Dependency Test - DependencyCheck

This test compares the list of all packages used in
INFs files for a
given package against a list of "allowed
dependencies" in plugin
configuration for that package. Any module that
depends on a
disallowed package will cause a test failure.

### Library Declaration Test - LibraryClassCheck

This test looks at all library header files found in
a package's
`Include/Library` directory and ensures that all
files have a matching
LibraryClass declaration in the DEC file for the
package. Any missing
declarations will cause a failure.

### Invalid Character Test - CharEncodingCheck

This test scans all files in a package to make sure
that there are no
invalid Unicode characters that may cause build
errors in some
character sets/localizations.

## Next Steps

* Receive community feedback on RFC.
* Determine where this phase makes sense given
existing RFCs from
other TianoCore contributors.
* Optimize testing beharior.
* Only run a subset of tests on PRs or individual
commits.
* Run full testing either once per day or once
every several
commits.
* Add more tests/capabilities.
* Continue to improve results formatting.
* Continue to improve CI documentation.
* Much of this documentation effort is pending
community feedback on
which parts are needed and what phases are
priorities.

Thanks


-----Original Message-----
From: rfc@edk2.groups.io <rfc@edk2.groups.io> On
Behalf Of Michael D
Kinney via Groups.Io
Sent: Thursday, August 29, 2019 1:23 PM
To: devel@edk2.groups.io; rfc@edk2.groups.io
Subject: [edk2-rfc] [RFC] EDK II Continuous
Integration Phase 1

Hello,

This is a proposal for a first step towards
continuous integration for
all TianoCore repositories to help improve to quality
of commits and
automate testing and release processes for all EDK II
packages and
platforms.

This is based on work from a number of EDK II
community members that
have provide valuable input and evaluations.

* Rebecca Cran <mailto:rebecca@...> Jenkins
evaluation
* Laszlo Ersek <mailto:lersek@...> GitLab
evaluation
* Philippe Mathieu-Daudé <mailto:philmd@...>
GitLab evaluation
* Sean Brogan <mailto:sean.brogan@...>
Azure Pipelines and
HBFA
* Bret Barkelew <mailto:Bret.Barkelew@...>
Azure Pipelines and HBFA
* Jiewen Yao <mailto:jiewen.yao@...> HBFA

The following link is a link to an EDK II WIKI page
that contains a
summary of the work to date. Please provide feedback
in the EDK II
mailing lists. The WIKI pages will be updated with
input from the
entire EDK II community.


https://nam06.safelinks.protection.outlook.com/?url=htt
ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
%2Fwiki%2FEDK-II-Continuous-
Integration&amp;data=02%7C01%7Csean.brogan%40microsoft.
com%7C6f67f169a6c746b4288608d72cbea7b6%7C72f988bf86f141
af91ab2d7cd011db47%7C1%7C0%7C637027069686644659&amp;sda
ta=GR9wN6gP3mJlCTopAaQ2rlzhby1nuF%2BwDVsfFIQAZjA%3D&amp
;reserved=0

Proposal
========
Phase 1 of adding continuous integration is limited
to the
edk2 repository. Additional repositories will be
added later.

The following changes are proposed:
* Remove EDK II Maintainers write access to edk2
repository.
Only EDK II Administrators will continue to have
write
access, and that should only be used to handle
extraordinary
events.
* EDK II Maintainers use a GitHub Pull Request
instead of push
to commit a patch series to the edk2 repository.
There are
no other changes to the development and review
process. The
patch series is prepared in an EDK II maintainer
branch with
all commit message requirements met on each patch
in the series.
* The edk2 repository only accepts Pull Requests from
members
of the EDK II Maintainers team. Pull Requests from
anyone else
are rejected.
* Run pre-commit checks using Azure Pipelines
* If all pre-commit checks pass, then the patch
series is auto
committed. The result of this commit must match
the contents
and commit history that would have occurred using
the previous
push operation.
* If any pre-commit checks fail, then notify the
submitter.
A typical reason for a failure would be a merge
conflict with
another pull request that was just processed.
* Limit pre-commit checks execution time to 10
minutes.
* Provide on-demand builds to EDK II Maintainers that
to allow
EDK II Maintainers to submit a branch through for
the same
set of pre-commit checks without submitting a pull
request.

## Pre-Commit Checks in Phase 1
* Run and pass PatchCheck.py with no errors

=====================================================

The following are some additional pre-commit check
ideas that could be
quickly added once the initial version using
PatchCheck.py is fully
functional.
Please provide feedback on the ones you like and
additional ones you
think may improve the quality of the commits to the
edk2 repository.

## Proposed Pre-Commit Checks in Phase 2
* Verify Reviewed-by and Acked-by tags are present
with
correct maintainer email addresses
* Verify no non-ASCII characters in modified files
* Verify no binary files in set of modified files
* Verify package dependency rules in modified files

## Proposed Pre-Commit Checks in Phase 3
* Run ECC on modified files
* Verify modified modules/libs build
* Run available host based tests (HBFA) against
modified
modules/libs

Best regards,

Mike










Re: UEFI accessibility mandate

Andrew Fish <afish@...>
 

Rafael,

Did you do much research into CODECs? Like which one(s) should be supported? I assume th CODEC implies the audio file formats that can be decoded? Also how large are the audio files?

Is the CODEC protocol more of a plug-in for the Intel HDA? By that I mean it only works on that hardware, or does it work generically on top of any EFI_AUDIO_OUTPUT_PROTOCOL?

I was starting to think about how to store to audio and deal with dynamic configuration. I guess one crazy idea could be to have the OS create the audio files using text to speech, and maybe store them on the EFI System Partition. For example when an OS installs it is going to write the EFI Boot Variable that contains a Description of the boot option. Maybe we could convert that to audio and add a nvram variable that points to the given audio file?

Thanks,

Andrew Fish

On Sep 23, 2019, at 6:20 AM, Rafael Machado <rafaelrodrigues.machado@...> wrote:

Hi everyone.
So, based on everything was mentioned here.

The idea is to propose the creation of two protocols:

- EFI_AUDIO_OUTPUT_PROTOCOL: This protocol should be the responsible for initializing the audio controller, that in the case of my MSc work does initialization of the RING buffers, that are used as containers to the audio streams that need to be processed.

- EFI_AUDIO_CODEC_PROTOCOL: This protocol should be responsible for initializing the codec (each codec may need different init commands), and also is used to control things like mute, volume level and this kind of things. (can see the volume control actions at the last video I mentioned on the previous e-mail)

Does this approach works at non-x86 platforms? (I don't have knowledge in ARM platforms, so feedback from the community will be well received.)

Hope to hear some voices :)

Thanks and Regards
Rafael

Em sáb, 21 de set de 2019 às 09:36, Rafael Machado <rafaelrodrigues.machado@... <mailto:rafaelrodrigues.machado@...>> escreveu:
Hi Everyone
Sorry for the delay on the response, to many things happening at the same time.
I will try to answer e-mails to this thread every Saturday or Sunday morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time to help implement this feature, but I happy to help work on the architecture and design for UEFI accessibility on the edk2 mailing lists, and I >>can also represent what ever we come up with at the UEFI Spec Work Group.
During my MSc I had to study a lot the audio and BIOS architectures. The idea was to eliminate the first barrier to the creation of a screen reader for pre-OS environment, that was the lack of some open implementation of audio control and actions at UEFI. To do that I studied the Intel High Definition Audio Spec and a lot of UEFI specs to understand better how to do that.
The initial target was to do all this development at OVMF, but as far as I could get, the sound card is not available to OVMF. as Laszlo mentioned at this e-mail there are some projects that may help on this, but at the time I was working on my MSc I didn't find this, so I did everything on a real system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and working on a MSc degree at the same time having a 40hours/week job, being a father and a husband is not an easy task, but it got to what I was expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA memory access. For the ones that have good years, it's possible to listen some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was that one of my write operations was causing some misunderstanding between the audio controller and the audio codec. The controller was sending packets with 16bit depth, but the codec was processing them as 8bit depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing at the OS level.
The reference code, that is just a proof-of-concept, and that has several things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to UEFI drivers after some discussion. Everything is released as BDS so companies can use without IP problems.
Just to give some more information about the need of this kind of solution. There is a lot of blind people that work with hardware support, so formatting disk, configuring RAID and booting dual-boot systems is always a challenge to them. Even set the BIOS clock. How to do that without the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given there is no guideline on how a User Interface (UI) works. If accessibility requires some from of hardware abstraction, like audio, then we could >>likely get that into the UEFI Spec. What might be possible is an EDK2 reference implementation of accessibility. Maybe we could use the reference implementation to write a UEFI white paper on design >>for accessibility? I there is an open source implementation, and an official design guide this would make it much easier for advocacy groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2 papers is one of my targets since the beginning of my MSc almost 5 years ago.

I've got some experience with accessibility as the macOS EFI OS Loader has a UI for the Full Disk Encryption password. If you press the power button quickly 3 times at the disk encryption password prompt >>accessibility is enabled and Voice Over gets turned on. You then get localized voice prompts when you move between UI elements. Since this is the OS loader all the resources are stored on the disk. You >>quickly run into a range of challenges since, audio is hard, abstracting audio is hard (what codec does firmware have to support), Audio files are not small and firmware is size constrained, the need to localize >>the audio responses causes even more size issues, the boot options are usually written by an OS installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so no audio files would need to be stored. There are some open-source implementations that are not GPL.
This was described at my MSc as future works that can continue what I have started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We choose to go to the Moon!" speech sometimes we chose to do things "not because they are easy, but because they are hard; because that >>goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept". If we have a design that means we can break the problem up into >>smaller parts, and maybe we can find people that have expertise in that part to build a chunk at a time. If we could implement the prototype in OVMF that would show how it works, but run on everyone's >>machines, so that would be really helpful for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible future steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application remains functional.
- 4) Separate controller and codec code by creating an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate out the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder audio files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the need of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my thesis to English.
I wrote everything in Portuguese because there was not material about UEFI to the Brazilian audience, and another target I have is to show companies that we have people that can work at this kind of projects in Brazil, bringing this kind of development to south america. (Yes, I have complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@... <mailto:lersek@...>> escreveu:
On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo


Re: UEFI accessibility mandate

Ethin Probst
 

If other platforms are PCI-based (i.e. allow us to scan the PCI bus
and figure out where (in MMIO space) the HDA controller is mapped to),
then it (theoretically) sould work. I don't know for sure though; I'm
not very knowledgeable in other CPU architectures.

On 9/23/19, Rafael Machado <rafaelrodrigues.machado@...> wrote:
Hi everyone.
So, based on everything was mentioned here.

The idea is to propose the creation of two protocols:

- EFI_AUDIO_OUTPUT_PROTOCOL: This protocol should be the responsible for
initializing the audio controller, that in the case of my MSc work does
initialization of the RING buffers, that are used as containers to the
audio streams that need to be processed.

- EFI_AUDIO_CODEC_PROTOCOL: This protocol should be responsible for
initializing the codec (each codec may need different init commands), and
also is used to control things like mute, volume level and this kind of
things. (can see the volume control actions at the last video I mentioned
on the previous e-mail)

Does this approach works at non-x86 platforms? (I don't have knowledge in
ARM platforms, so feedback from the community will be well received.)

Hope to hear some voices :)

Thanks and Regards
Rafael

Em sáb, 21 de set de 2019 às 09:36, Rafael Machado <
rafaelrodrigues.machado@...> escreveu:

Hi Everyone
Sorry for the delay on the response, to many things happening at the same
time.
I will try to answer e-mails to this thread every Saturday or Sunday
morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time
to
help implement this feature, but I happy to help work on the architecture
and design for UEFI accessibility on the edk2 mailing lists, and I >>can
also represent what ever we come up with at the UEFI Spec Work Group.
During my MSc I had to study a lot the audio and BIOS architectures. The
idea was to eliminate the first barrier to the creation of a screen
reader
for pre-OS environment, that was the lack of some open implementation of
audio control and actions at UEFI. To do that I studied the Intel High
Definition Audio Spec and a lot of UEFI specs to understand better how to
do that.
The initial target was to do all this development at OVMF, but as far as
I
could get, the sound card is not available to OVMF. as Laszlo mentioned
at
this e-mail there are some projects that may help on this, but at the
time
I was working on my MSc I didn't find this, so I did everything on a real
system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and
working on a MSc degree at the same time having a 40hours/week job, being
a
father and a husband is not an easy task, but it got to what I was
expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that
later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA
memory access. For the ones that have good years, it's possible to listen
some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was
that
one of my write operations was causing some misunderstanding between the
audio controller and the audio codec. The controller was sending packets
with 16bit depth, but the codec was processing them as 8bit depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing at
the OS level.
The reference code, that is just a proof-of-concept, and that has several
things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to UEFI
drivers after some discussion. Everything is released as BDS so companies
can use without IP problems.
Just to give some more information about the need of this kind of
solution. There is a lot of blind people that work with hardware support,
so formatting disk, configuring RAID and booting dual-boot systems is
always a challenge to them. Even set the BIOS clock. How to do that
without
the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given there
is no guideline on how a User Interface (UI) works. If accessibility
requires some from of hardware abstraction, like audio, then we could
likely get that into the UEFI Spec. What might be possible is an EDK2
reference implementation of accessibility. Maybe we could use the
reference
implementation to write a UEFI white paper on design >>for accessibility?
I
there is an open source implementation, and an official design guide this
would make it much easier for advocacy groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2 papers
is one of my targets since the beginning of my MSc almost 5 years ago.

I've got some experience with accessibility as the macOS EFI OS Loader
has a UI for the Full Disk Encryption password. If you press the power
button quickly 3 times at the disk encryption password prompt
accessibility is enabled and Voice Over gets turned on. You then get
localized voice prompts when you move between UI elements. Since this is
the OS loader all the resources are stored on the disk. You >>quickly run
into a range of challenges since, audio is hard, abstracting audio is
hard
(what codec does firmware have to support), Audio files are not small and
firmware is size constrained, the need to localize >>the audio responses
causes even more size issues, the boot options are usually written by an
OS
installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so
no
audio files would need to be stored. There are some open-source
implementations that are not GPL.
This was described at my MSc as future works that can continue what I
have
started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things "not
because they are easy, but because they are hard; because that >>goal
will
serve to organize and measure the best of our energies and skills,
because
that challenge is one that we are willing to accept". If we have a design
that means we can break the problem up into >>smaller parts, and maybe we
can find people that have expertise in that part to build a chunk at a
time. If we could implement the prototype in OVMF that would show how it
works, but run on everyone's >>machines, so that would be really helpful
for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible future
steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with
proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver
to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also
producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application
remains functional.
- 4) Separate controller and codec code by creating
an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate
out
the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder
audio
files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the need
of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my
thesis
to English.
I wrote everything in Portuguese because there was not material about
UEFI
to the Brazilian audience, and another target I have is to show companies
that we have people that can work at this kind of projects in Brazil,
bringing this kind of development to south america. (Yes, I have
complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@...>
escreveu:

On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo
--
Signed,
Ethin D. Probst


Re: UEFI accessibility mandate

Rafael Machado <rafaelrodrigues.machado@...>
 

Hi everyone.
So, based on everything was mentioned here.

The idea is to propose the creation of two protocols:

- EFI_AUDIO_OUTPUT_PROTOCOL: This protocol should be the responsible for
initializing the audio controller, that in the case of my MSc work does
initialization of the RING buffers, that are used as containers to the
audio streams that need to be processed.

- EFI_AUDIO_CODEC_PROTOCOL: This protocol should be responsible for
initializing the codec (each codec may need different init commands), and
also is used to control things like mute, volume level and this kind of
things. (can see the volume control actions at the last video I mentioned
on the previous e-mail)

Does this approach works at non-x86 platforms? (I don't have knowledge in
ARM platforms, so feedback from the community will be well received.)

Hope to hear some voices :)

Thanks and Regards
Rafael

Em sáb, 21 de set de 2019 às 09:36, Rafael Machado <
rafaelrodrigues.machado@...> escreveu:

Hi Everyone
Sorry for the delay on the response, to many things happening at the same
time.
I will try to answer e-mails to this thread every Saturday or Sunday
morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time to
help implement this feature, but I happy to help work on the architecture
and design for UEFI accessibility on the edk2 mailing lists, and I >>can
also represent what ever we come up with at the UEFI Spec Work Group.
During my MSc I had to study a lot the audio and BIOS architectures. The
idea was to eliminate the first barrier to the creation of a screen reader
for pre-OS environment, that was the lack of some open implementation of
audio control and actions at UEFI. To do that I studied the Intel High
Definition Audio Spec and a lot of UEFI specs to understand better how to
do that.
The initial target was to do all this development at OVMF, but as far as I
could get, the sound card is not available to OVMF. as Laszlo mentioned at
this e-mail there are some projects that may help on this, but at the time
I was working on my MSc I didn't find this, so I did everything on a real
system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and
working on a MSc degree at the same time having a 40hours/week job, being a
father and a husband is not an easy task, but it got to what I was
expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that
later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA
memory access. For the ones that have good years, it's possible to listen
some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was that
one of my write operations was causing some misunderstanding between the
audio controller and the audio codec. The controller was sending packets
with 16bit depth, but the codec was processing them as 8bit depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing at
the OS level.
The reference code, that is just a proof-of-concept, and that has several
things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to UEFI
drivers after some discussion. Everything is released as BDS so companies
can use without IP problems.
Just to give some more information about the need of this kind of
solution. There is a lot of blind people that work with hardware support,
so formatting disk, configuring RAID and booting dual-boot systems is
always a challenge to them. Even set the BIOS clock. How to do that without
the system's feedback?

It would be hard to have a UEFI mandate for accessibility, given there
is no guideline on how a User Interface (UI) works. If accessibility
requires some from of hardware abstraction, like audio, then we could
likely get that into the UEFI Spec. What might be possible is an EDK2
reference implementation of accessibility. Maybe we could use the reference
implementation to write a UEFI white paper on design >>for accessibility? I
there is an open source implementation, and an official design guide this
would make it much easier for advocacy groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2 papers
is one of my targets since the beginning of my MSc almost 5 years ago.

I've got some experience with accessibility as the macOS EFI OS Loader
has a UI for the Full Disk Encryption password. If you press the power
button quickly 3 times at the disk encryption password prompt
accessibility is enabled and Voice Over gets turned on. You then get
localized voice prompts when you move between UI elements. Since this is
the OS loader all the resources are stored on the disk. You >>quickly run
into a range of challenges since, audio is hard, abstracting audio is hard
(what codec does firmware have to support), Audio files are not small and
firmware is size constrained, the need to localize >>the audio responses
causes even more size issues, the boot options are usually written by an OS
installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so no
audio files would need to be stored. There are some open-source
implementations that are not GPL.
This was described at my MSc as future works that can continue what I have
started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things "not
because they are easy, but because they are hard; because that >>goal will
serve to organize and measure the best of our energies and skills, because
that challenge is one that we are willing to accept". If we have a design
that means we can break the problem up into >>smaller parts, and maybe we
can find people that have expertise in that part to build a chunk at a
time. If we could implement the prototype in OVMF that would show how it
works, but run on everyone's >>machines, so that would be really helpful
for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible future
steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with
proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver
to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also
producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application
remains functional.
- 4) Separate controller and codec code by creating
an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate out
the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder audio
files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the need
of audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my thesis
to English.
I wrote everything in Portuguese because there was not material about UEFI
to the Brazilian audience, and another target I have is to show companies
that we have people that can work at this kind of projects in Brazil,
bringing this kind of development to south america. (Yes, I have
complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@...>
escreveu:

On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo


Re: UEFI accessibility mandate

Rafael Machado <rafaelrodrigues.machado@...>
 

Hi Everyone
Sorry for the delay on the response, to many things happening at the same
time.
I will try to answer e-mails to this thread every Saturday or Sunday
morning at least.
About Andrew's and Laszlo's comments and questions

Please let us know what you find out. I probably don''t have the time to
help implement this feature, but I happy to help work on the architecture
and design for UEFI accessibility on the edk2 mailing lists, and I >>can
also represent what ever we come up with at the UEFI Spec Work Group.
During my MSc I had to study a lot the audio and BIOS architectures. The
idea was to eliminate the first barrier to the creation of a screen reader
for pre-OS environment, that was the lack of some open implementation of
audio control and actions at UEFI. To do that I studied the Intel High
Definition Audio Spec and a lot of UEFI specs to understand better how to
do that.
The initial target was to do all this development at OVMF, but as far as I
could get, the sound card is not available to OVMF. as Laszlo mentioned at
this e-mail there are some projects that may help on this, but at the time
I was working on my MSc I didn't find this, so I did everything on a real
system (a ASUS notebook).
It took me 2 years of work, because I didn't know a lot of things and
working on a MSc degree at the same time having a 40hours/week job, being a
father and a husband is not an easy task, but it got to what I was
expecting.
The evolution of the project was this:
1 - First tests using some registers named "Immediate Registers", that
later I noticed that are not mandatory. This is a simple C Major scale:
https://www.youtube.com/watch?v=I-mgzcOnRCg&feature=youtu.be
2 - Some months later I started to work with the Ring Buffers and DMA
memory access. For the ones that have good years, it's possible to listen
some music behing the noise.
https://www.youtube.com/watch?v=6ED2BSc89-Y&feature=youtu.be
3 - Later, wen I was almost giving up, I noticed that the problem was that
one of my write operations was causing some misunderstanding between the
audio controller and the audio codec. The controller was sending packets
with 16bit depth, but the codec was processing them as 8bit depth
https://www.youtube.com/watch?v=2De9dI9WbwM&feature=youtu.be

So the conclusion is that doing this at UEFI us much easier that doing at
the OS level.
The reference code, that is just a proof-of-concept, and that has several
things to be improved, can be found here:
https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Currently it is just an UEFI Application, but we can convert this to UEFI
drivers after some discussion. Everything is released as BDS so companies
can use without IP problems.
Just to give some more information about the need of this kind of solution.
There is a lot of blind people that work with hardware support, so
formatting disk, configuring RAID and booting dual-boot systems is always a
challenge to them. Even set the BIOS clock. How to do that without the
system's feedback?

It would be hard to have a UEFI mandate for accessibility, given there is
no guideline on how a User Interface (UI) works. If accessibility requires
some from of hardware abstraction, like audio, then we could >>likely get
that into the UEFI Spec. What might be possible is an EDK2 reference
implementation of accessibility. Maybe we could use the reference
implementation to write a UEFI white paper on design >>for accessibility? I
there is an open source implementation, and an official design guide this
would make it much easier for advocacy groups to lobby for this feature.
I agree this is the way. Writing a white paper as an official EDK2 papers
is one of my targets since the beginning of my MSc almost 5 years ago.

I've got some experience with accessibility as the macOS EFI OS Loader
has a UI for the Full Disk Encryption password. If you press the power
button quickly 3 times at the disk encryption password prompt
accessibility is enabled and Voice Over gets turned on. You then get
localized voice prompts when you move between UI elements. Since this is
the OS loader all the resources are stored on the disk. You >>quickly run
into a range of challenges since, audio is hard, abstracting audio is hard
(what codec does firmware have to support), Audio files are not small and
firmware is size constrained, the need to localize >>the audio responses
causes even more size issues, the boot options are usually written by an OS
installer so how would firmware know what to call them?
The solution to this would be the porting of some voice synthesizer, so no
audio files would need to be stored. There are some open-source
implementations that are not GPL.
This was described at my MSc as future works that can continue what I have
started.

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things "not
because they are easy, but because they are hard; because that >>goal will
serve to organize and measure the best of our energies and skills, because
that challenge is one that we are willing to accept". If we have a design
that means we can break the problem up into >>smaller parts, and maybe we
can find people that have expertise in that part to build a chunk at a
time. If we could implement the prototype in OVMF that would show how it
works, but run on everyone's >>machines, so that would be really helpful
for demos and design review.
I totally agree. Amazing words that I didn't have heard yet. Thanks!
As far as I could understand, and with Leif's help, some possible future
steps could be (not at this specific order):
- 1) Convert proof-of-concept HDA driver to UEFI driver model with
proper PCI discovery.
- 2) Design a prototype EFI_AUDIO_OUTPUT_PROTOCOL, rework driver
to produce this and application to discover and consume it.
- 3) Implement a USB Audio Class driver also
producing EFI_AUDIO_OUTPUT_PROTOCOL and ensure test application
remains functional.
- 4) Separate controller and codec code by creating
an EFI_AUDIO_CODEC_PROTOCOL, implement this in HDA driver, and separate out
the codec support into individual drivers.
- 5) Prototype audio output additions to HII. (using pre-recorder audio
files)
- 6) Porting of some voice synthesizer to UEFI. (eliminating the need of
audio files)

Beyond this, there are other things we should look at adding, like
- EFI_AUDIO_INPUT_PROTOCOL.
- Audio input additions to HII.

It's a lot of work, but I accept the challenge.
It may take a long time, but it is possible.

I am still trying to find some time to finish the translation of my thesis
to English.
I wrote everything in Portuguese because there was not material about UEFI
to the Brazilian audience, and another target I have is to show companies
that we have people that can work at this kind of projects in Brazil,
bringing this kind of development to south america. (Yes, I have
complicated target, but I like the challenge :) )

Thanks and Regards
Rafael R. Machado

Em qui, 19 de set de 2019 às 14:45, Laszlo Ersek <lersek@...>
escreveu:

On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo


Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

Sean
 

Currently it is building on Windows with VS2019. VS2017 would be trivial but not worth it in my perspective given how aligned the two are. If you really wanted to do a weekly or nightly build it could be added to that but I have been focused on a PR build.
I have a pipeline for GCC on linux. It doesn't' support host based unit tests but I am working to get the rest of the tests running.
I do not have a pipeline for Mac or LLVM/Clang. That would be next and I think your files below should help.

To date we use PYTOOLs features to install our extra tools. This makes it super easy and works for both local and server based builds. It gives strong versioning and management of that. It also tracks those versions and you can see them on every build in the Built Tool Report artifact. It can be downloaded here for html version. https://dev.azure.com/tianocore/edk2-ci-play/_build/results?buildId=803&view=artifacts. BUILD_TOOL_REPORT.html

Here it is pasted as plain text below. You can see the iasl and nasm versions here.

Key Value Type
TOOL_CHAIN_TAG VS2019 TOOL
VC Version 14.22.27905 TOOL
d:\a\1\s\Conf\build_rule.txt 1.04 INFO
d:\a\1\s\Conf\target.txt 1.03 INFO
d:\a\1\s\Conf\tools_def.txt 1.22 INFO
iasl 20190215.0.0 INFO
Mu-Basetools 2019.03.1 INFO
mu_nasm 2.14.02 INFO


Hope that helps.

I would be happy to queue up this topic for next design meeting and open it up to questions? I hope others would take a look prior to meeting.

Thanks
Sean

-----Original Message-----
From: Kinney, Michael D <michael.d.kinney@...>
Sent: Thursday, September 19, 2019 2:56 PM
To: devel@edk2.groups.io; Sean Brogan <sean.brogan@...>; rfc@edk2.groups.io; Kinney, Michael D <michael.d.kinney@...>
Cc: Bret Barkelew <Bret.Barkelew@...>
Subject: RE: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

Hi Sean,

Which OS/Compiler configurations are currently enabled for the Code Compilation Test?

I have been working on enabling multiple OS/Compiler configurations in Azure Pipelines. There are some tools that need to be installed for each of these environments.
Examples include NASM, iASL, Python.

For the work you have done, how are these extra tools installed? Is it in the YML files or in the Python scripts.

One critical task is to identify the tools and their specific versions that the CI system is configured to use.
These configurations should be documented in a Wiki page and updated as new tools are released and adopted by EDK II.
The inventory of tools used to validate a release should Also be documented in a release notes for a stable tag.

Here are the YML files that install the additional tools required to support EDK II builds. I need the source and versions of these tools to be reviewed and approved.

https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-ci%2Fblob%2Fmaster%2FAzurePipelines%2FWindowsPrerequisites.yml&;data=02%7C01%7Csean.brogan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=hM4AKoMutISI5Oc%2FeVMevx%2FVRUCjJHuhEwqom0R30Ak%3D&amp;reserved=0
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-ci%2Fblob%2Fmaster%2FAzurePipelines%2FUbuntuPrerequisites.yml&;data=02%7C01%7Csean.brogan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=KQ2zas2bJ%2FCjSRWGyMrxq5Rk4cW5lOgXQNR99QJbEKY%3D&amp;reserved=0
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2-ci%2Fblob%2Fmaster%2FAzurePipelines%2FMacOsPrerequisites.yml&;data=02%7C01%7Csean.brogan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=OzAmzYkRJ3rQOHEDgKTREz%2BNp7acOWUACh1s%2Fb4UPGk%3D&amp;reserved=0

Thanks,

Mike

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Sean
via Groups.Io
Sent: Thursday, August 29, 2019 7:22 PM
To: rfc@edk2.groups.io; Kinney, Michael D
<michael.d.kinney@...>; devel@edk2.groups.io
Cc: Bret Barkelew <Bret.Barkelew@...>
Subject: Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

Mike, as you mentioned we have been working towards enabling a
practical and extensible CI for Edk2 using Azure dev ops and the
recently added edk2-pytool infrastructure. We have been using similar
CI for Project Mu for the last few years.

Our approach is a little different in that we focus on validating the
whole code base rather than just the incoming patch. We do this
because we have found unexpected consequences of patches and overall
we want all code to be compliant not just new additions. We have
found the time to test the whole tree is not much longer than only the
parts impacted by a code change (except maybe when running the entire
compile test on every package). This obviously comes with an initial
tax of needing to get the codebase into compliant form.
Anyway we have prepared an RFC in addition to yours and would like to
see these two efforts merged together.

We are still working on making a few optimizations.
Currently if the full set of tests are run we take about 20 minutes.
This is because compiling MdeModulePkg for debug, release, and host
based tests take a while. Most other packages are in the 10 minute
range. We do have easy ways to disable or limit certain tests as well
as expand the matrix to leverage more cloud resources (more parallel
builds).


Content is best viewed online with links to helpful content but is
also attached below:
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
ub.com%2Fspbrogan%2Fedk2-staging%2Fblob%2Fedk2-&amp;data=02%7C01%7Csea
n.brogan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf
86f141af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=xOiPHH
VbMhFBsIpj%2F9mhOhrkhdsF2Gbehd6%2Frk0XYgM%3D&amp;reserved=0
stuart-ci-latest/Readme-CI-RFC.md

# CI and PR Gates

## Background

Historically, while the TianoCore maintainers and stewards have done a
fantastic job of keeping contribution policies consistent and
contributions clean and well-documented, there have been few processes
that ran to verify the sanity, cleanliness, and efficacy of the
codebase, and even fewer that publicly published their results for the
community at large. This has caused inconsistancies and issues within
the codebase from time to time.

Adding continuous integration (and potentially PR
gates) to the checkin process ensures that simple errors like these
are caught and can be fixed on a regular basis.

## Strategy

While a number of CI solutions exist, this proposal will focus on the
usage of Azure Dev Ops and Build Pipelines. For demonstration, a
sample [TianoCore
repo](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%
2Fgithub.com%2Fspbrogan%2Fedk2-staging.git&amp;data=02%7C01%7Csean.bro
gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f14
1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=QmzQhemLUNB
7FGJwoGeeewExThrUjUArxnkmtro1b8A%3D&amp;reserved=0)
(branch edk2-stuart-ci-latest) and [Dev Ops
Pipeline](https://nam06.safelinks.protection.outlook.com/?url=https%3A
%2F%2Fdev.azure.com%2Ftianocore%2Fedk2-ci-&amp;data=02%7C01%7Csean.bro
gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f14
1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=rS%2FrjIFkS
uEhI%2FDWyslf34i711%2FbY8Gw%2Byexq%2FwydjU%3D&amp;reserved=0
play/_build?definitionId=12) have been set up.

Furthermore, this proposal will leverage the TianoCore python tools
PIP modules:
[library](https://nam06.safelinks.protection.outlook.com/?url=https%3A
%2F%2Fpypi.org%2Fproject%2Fedk2-pytool-&amp;data=02%7C01%7Csean.brogan
%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af
91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=jzM6MKZFFAEvyR
4h0%2Bkf5pUcBSsoVzkIHFFi1Bd6Il4%3D&amp;reserved=0
library/) and
[extensions](https://nam06.safelinks.protection.outlook.com/?url=https
%3A%2F%2Fpypi.org%2Fproject%2Fedk2-pytool-&amp;data=02%7C01%7Csean.bro
gan%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f14
1af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=jzM6MKZFFAE
vyR4h0%2Bkf5pUcBSsoVzkIHFFi1Bd6Il4%3D&amp;reserved=0
extensions/) (with repos located
[here](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F
%2Fgithub.com%2Ftianocore%2Fedk2-pytool-&amp;data=02%7C01%7Csean.broga
n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141a
f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=OwT%2BiNTN9Rv
r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
library) and
[here](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F
%2Fgithub.com%2Ftianocore%2Fedk2-&amp;data=02%7C01%7Csean.brogan%40mic
rosoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91ab2d
7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=tInVwGwCXXGBgcmHQ%2B
0UwRaEbtZWRy8hTp5wpRtN0R0%3D&amp;reserved=0
pytool-extensions)).

The primary execution flows can be found in the `azure-
pipelines-pr-gate.yml` and `azure-pipelines-pr-gate- linux.yml` files.
These YAML files are consumed by the Azure Dev Ops Build Pipeline and
dictate what server resources should be used, how they should be
configured, and what processes should be run on them.
An overview of this schema can be found
[here](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F
%2Fdocs.microsoft.com%2Fen-&amp;data=02%7C01%7Csean.brogan%40microsoft
.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91ab2d7cd011
db47%7C1%7C0%7C637045269452466989&amp;sdata=j8nyYG3VnizwKFZ8l1BWoLcPy%
2BuaSoVT0jN2Esi7wH8%3D&amp;reserved=0
us/azure/devops/pipelines/yaml-schema?view=azure-
devops&tabs=schema).

Inspection of these files reveals the EDKII Tools commands that make
up the primary processes for the CI
build: 'stuart_setup', 'stuart_update', and 'stuart_ci_build'. These
commands come from the EDKII Tools PIP modules and are configured as
described below. More documentation on the stuart tools can be found
[here](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F
%2Fgithub.com%2Ftianocore%2Fedk2-pytool-&amp;data=02%7C01%7Csean.broga
n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141a
f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=OwT%2BiNTN9Rv
r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
extensions/blob/master/docs/using.md) and
[here](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F
%2Fgithub.com%2Ftianocore%2Fedk2-pytool-&amp;data=02%7C01%7Csean.broga
n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141a
f91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=OwT%2BiNTN9Rv
r14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
extensions/blob/master/docs/features/feature_invocables
.md).

## Configuration

Configuration of the CI process consists of (in order of precedence):
* command-line arguments passed in via the Pipeline YAML
* a per-package configuration file (e.g. `<package-
name>.mu.yaml`) that is detected by the CI system in
EDKII Tools.
* a global configuration Python module (e.g.
`CISetting.py`) passed in via the command-line

The global configuration file is described in [this
readme](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2
F%2Fgithub.com%2Ftianocore%2Fedk2-pytool-&amp;data=02%7C01%7Csean.brog
an%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141
af91ab2d7cd011db47%7C1%7C0%7C637045269452466989&amp;sdata=OwT%2BiNTN9R
vr14Oj67IdDlK8WGJClTIsmEzdyrBt2Ho%3D&amp;reserved=0
extensions/blob/master/docs/usability/using_settings_ma
nager.md) from the EDKII Tools documentation. This configuration is
written as a Python module so that decisions can be made dynamically
based on command line parameters and codebase state.

The per-package configuration file can override most settings in the
global configuration file, but is not dynamic. This file can be used
to skip or customize tests that may be incompatible with a specific
package.
By default, the global configuration will try to run all tests on all
packages.

## CI Test Types

All CI tests are instances of EDKII Tools plugins.
Documentation on the plugin system can be found
[here](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F
%2Fgithub.com%2Ftianocore%2Fedk2-pytool-&amp;data=02%7C01%7Csean.broga
n%40microsoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141a
f91ab2d7cd011db47%7C1%7C0%7C637045269452476994&amp;sdata=1AM4geY%2BEFh
ekun7oQOYyVVt%2Bwn6CAF2hhtkS0sgwPU%3D&amp;reserved=0
extensions/blob/master/docs/usability/using_plugin_mana
ger.md) and
[here](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F
%2Fgithub.com%2Ftianocore%2Fedk2-&amp;data=02%7C01%7Csean.brogan%40mic
rosoft.com%7C9e7d685fa1764cf500af08d73d4c1e1b%7C72f988bf86f141af91ab2d
7cd011db47%7C1%7C0%7C637045269452476994&amp;sdata=3vyPMvrfG3V4doNM0%2B
cKTpDSHEEEvkfxKAtw3eiJWsY%3D&amp;reserved=0
pytool-
extensions/blob/master/docs/features/feature_plugin_man
ager.md). Upon invocation, each plugin will be passed the path to the
current package under test and a dictionary containing its targeted
configuration, as assembled from the command line, per-package
configuration, and global configuration.

Note: CI plugins are considered unique from build plugins and helper
plugins, even though some CI plugins may execute steps of a build.

In the example, these plugins live alongside the code under test (in
the `BaseTools` directory), but may be moved to the 'edk2-test' repo
if that location makes more sense for the community.

### Module Inclusion Test - DscCompleteCheck

This test scans all available modules (via INF files) and compares
them to the package-level DSC file for the package each module is
contained within. The test considers it an error if any module does
not appear in the `Components` section of at least one package-level
DSC (indicating that it would not be built if the package were built).

### Code Compilation Test - CompilerPlugin

Once the Module Inclusion Test has verified that all modules would be
built if all package-level DSCs were built, the Code Compilation Test
simply runs through and builds every package-level DSC on every
toolchain and for every architecture that is supported. Any module
that fails to build is considered an error.

### Host-Based UnitTests - HostUnitTestCompilerPlugin and
HostUnitTestDscCompleteCheck

The [Testing RFC doc](Readme-Testing-RFC.md) has much more detail on
this, but the basic idea is that host- based unit tests can be
compiled against individual modules and libraries and run on the build
agent (in this case, the Dev Ops build server). The successful and
failing test case results are collected and included in the final
build report.

### GUID Uniqueness Test - GuidCheck

This test works on the collection of all packages rather than an
individual package. It looks at all FILE_GUIDs and GUIDs declared in
DEC files and ensures that they are unique for the codebase. This
prevents, for example, accidental duplication of GUIDs when using an
existing INF as a template for a new module.

### Cross-Package Dependency Test - DependencyCheck

This test compares the list of all packages used in INFs files for a
given package against a list of "allowed dependencies" in plugin
configuration for that package. Any module that depends on a
disallowed package will cause a test failure.

### Library Declaration Test - LibraryClassCheck

This test looks at all library header files found in a package's
`Include/Library` directory and ensures that all files have a matching
LibraryClass declaration in the DEC file for the package. Any missing
declarations will cause a failure.

### Invalid Character Test - CharEncodingCheck

This test scans all files in a package to make sure that there are no
invalid Unicode characters that may cause build errors in some
character sets/localizations.

## Next Steps

* Receive community feedback on RFC.
* Determine where this phase makes sense given existing RFCs from
other TianoCore contributors.
* Optimize testing beharior.
* Only run a subset of tests on PRs or individual commits.
* Run full testing either once per day or once every several
commits.
* Add more tests/capabilities.
* Continue to improve results formatting.
* Continue to improve CI documentation.
* Much of this documentation effort is pending community feedback on
which parts are needed and what phases are priorities.

Thanks


-----Original Message-----
From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf Of Michael D
Kinney via Groups.Io
Sent: Thursday, August 29, 2019 1:23 PM
To: devel@edk2.groups.io; rfc@edk2.groups.io
Subject: [edk2-rfc] [RFC] EDK II Continuous Integration Phase 1

Hello,

This is a proposal for a first step towards continuous integration for
all TianoCore repositories to help improve to quality of commits and
automate testing and release processes for all EDK II packages and
platforms.

This is based on work from a number of EDK II community members that
have provide valuable input and evaluations.

* Rebecca Cran <mailto:rebecca@...> Jenkins evaluation
* Laszlo Ersek <mailto:lersek@...> GitLab evaluation
* Philippe Mathieu-Daudé <mailto:philmd@...> GitLab evaluation
* Sean Brogan <mailto:sean.brogan@...> Azure Pipelines and
HBFA
* Bret Barkelew <mailto:Bret.Barkelew@...>
Azure Pipelines and HBFA
* Jiewen Yao <mailto:jiewen.yao@...> HBFA

The following link is a link to an EDK II WIKI page that contains a
summary of the work to date. Please provide feedback in the EDK II
mailing lists. The WIKI pages will be updated with input from the
entire EDK II community.


https://nam06.safelinks.protection.outlook.com/?url=htt
ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
%2Fwiki%2FEDK-II-Continuous-
Integration&amp;data=02%7C01%7Csean.brogan%40microsoft.
com%7C6f67f169a6c746b4288608d72cbea7b6%7C72f988bf86f141
af91ab2d7cd011db47%7C1%7C0%7C637027069686644659&amp;sda
ta=GR9wN6gP3mJlCTopAaQ2rlzhby1nuF%2BwDVsfFIQAZjA%3D&amp
;reserved=0

Proposal
========
Phase 1 of adding continuous integration is limited to the
edk2 repository. Additional repositories will be added later.

The following changes are proposed:
* Remove EDK II Maintainers write access to edk2 repository.
Only EDK II Administrators will continue to have write
access, and that should only be used to handle extraordinary
events.
* EDK II Maintainers use a GitHub Pull Request instead of push
to commit a patch series to the edk2 repository.
There are
no other changes to the development and review process. The
patch series is prepared in an EDK II maintainer branch with
all commit message requirements met on each patch in the series.
* The edk2 repository only accepts Pull Requests from members
of the EDK II Maintainers team. Pull Requests from anyone else
are rejected.
* Run pre-commit checks using Azure Pipelines
* If all pre-commit checks pass, then the patch series is auto
committed. The result of this commit must match the contents
and commit history that would have occurred using the previous
push operation.
* If any pre-commit checks fail, then notify the submitter.
A typical reason for a failure would be a merge conflict with
another pull request that was just processed.
* Limit pre-commit checks execution time to 10 minutes.
* Provide on-demand builds to EDK II Maintainers that to allow
EDK II Maintainers to submit a branch through for the same
set of pre-commit checks without submitting a pull request.

## Pre-Commit Checks in Phase 1
* Run and pass PatchCheck.py with no errors

=====================================================

The following are some additional pre-commit check
ideas that could be quickly added once the initial
version using PatchCheck.py is fully functional.
Please provide feedback on the ones you like and
additional ones you think may improve the quality of
the commits to the edk2 repository.

## Proposed Pre-Commit Checks in Phase 2
* Verify Reviewed-by and Acked-by tags are present with
correct maintainer email addresses
* Verify no non-ASCII characters in modified files
* Verify no binary files in set of modified files
* Verify package dependency rules in modified files

## Proposed Pre-Commit Checks in Phase 3
* Run ECC on modified files
* Verify modified modules/libs build
* Run available host based tests (HBFA) against
modified
modules/libs

Best regards,

Mike










Re: [edk2-devel] [Qemu-devel] [PATCH 1/2] q35: implement 128K SMRAM at default SMBASE address

Laszlo Ersek
 

On 09/20/19 10:28, Igor Mammedov wrote:
On Thu, 19 Sep 2019 19:02:07 +0200
"Laszlo Ersek" <lersek@...> wrote:

Hi Igor,

(+Brijesh)

long-ish pondering ahead, with a question at the end.
[...]

Finally: can you please remind me why we lock down 128KB (32 pages) at
0x3_0000, and not just half of that? What do we need the range at
[0x4_0000..0x4_FFFF] for?

If I recall correctly, CPU consumes 64K of save/restore area.
The rest 64K are temporary RAM for using in SMI relocation handler,
if it's possible to get away without it then we can drop it and
lock only 64K required for CPU state. It won't help with SEV
conflict though as it's in the first 64K.
OK. Let's go with 128KB for now. Shrinking the area is always easier
than growing it.

On QEMU side, we can drop black-hole approach and allocate
dedicated SMRAM region, which explicitly gets mapped into
RAM address space and after SMI hanlder initialization, gets
unmapped (locked). So that SMRAM would be accessible only
from SMM context. That way RAM at 0x30000 could be used as
normal when SMRAM is unmapped.
I prefer the black-hole approach, introduced in your current patch
series, if it can work. Way less opportunity for confusion.

I've started work on the counterpart OVMF patches; I'll report back.

Thanks
Laszlo


Re: [edk2-devel] [Qemu-devel] [PATCH 1/2] q35: implement 128K SMRAM at default SMBASE address

Igor Mammedov <imammedo@...>
 

On Thu, 19 Sep 2019 19:02:07 +0200
"Laszlo Ersek" <lersek@...> wrote:

Hi Igor,

(+Brijesh)

long-ish pondering ahead, with a question at the end.
[...]

Finally: can you please remind me why we lock down 128KB (32 pages) at
0x3_0000, and not just half of that? What do we need the range at
[0x4_0000..0x4_FFFF] for?

If I recall correctly, CPU consumes 64K of save/restore area.
The rest 64K are temporary RAM for using in SMI relocation handler,
if it's possible to get away without it then we can drop it and
lock only 64K required for CPU state. It won't help with SEV
conflict though as it's in the first 64K.

On QEMU side, we can drop black-hole approach and allocate
dedicated SMRAM region, which explicitly gets mapped into
RAM address space and after SMI hanlder initialization, gets
unmapped (locked). So that SMRAM would be accessible only
from SMM context. That way RAM at 0x30000 could be used as
normal when SMRAM is unmapped.


Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

Michael D Kinney
 

Hi Sean,

Which OS/Compiler configurations are currently enabled
for the Code Compilation Test?

I have been working on enabling multiple OS/Compiler
configurations in Azure Pipelines. There are some tools
that need to be installed for each of these environments.
Examples include NASM, iASL, Python.

For the work you have done, how are these extra tools
installed? Is it in the YML files or in the Python
scripts.

One critical task is to identify the tools and their
specific versions that the CI system is configured to use.
These configurations should be documented in a Wiki page
and updated as new tools are released and adopted by EDK II.
The inventory of tools used to validate a release should
Also be documented in a release notes for a stable tag.

Here are the YML files that install the additional tools
required to support EDK II builds. I need the source and
versions of these tools to be reviewed and approved.

https://github.com/mdkinney/edk2-ci/blob/master/AzurePipelines/WindowsPrerequisites.yml
https://github.com/mdkinney/edk2-ci/blob/master/AzurePipelines/UbuntuPrerequisites.yml
https://github.com/mdkinney/edk2-ci/blob/master/AzurePipelines/MacOsPrerequisites.yml

Thanks,

Mike

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On
Behalf Of Sean via Groups.Io
Sent: Thursday, August 29, 2019 7:22 PM
To: rfc@edk2.groups.io; Kinney, Michael D
<michael.d.kinney@...>; devel@edk2.groups.io
Cc: Bret Barkelew <Bret.Barkelew@...>
Subject: Re: [edk2-devel] [RFC] EDK II Continuous
Integration Phase 1

Mike, as you mentioned we have been working towards
enabling a practical and extensible CI for Edk2 using
Azure dev ops and the recently added edk2-pytool
infrastructure. We have been using similar CI for
Project Mu for the last few years.

Our approach is a little different in that we focus on
validating the whole code base rather than just the
incoming patch. We do this because we have found
unexpected consequences of patches and overall we want
all code to be compliant not just new additions. We
have found the time to test the whole tree is not much
longer than only the parts impacted by a code change
(except maybe when running the entire compile test on
every package). This obviously comes with an initial
tax of needing to get the codebase into compliant form.
Anyway we have prepared an RFC in addition to yours and
would like to see these two efforts merged together.

We are still working on making a few optimizations.
Currently if the full set of tests are run we take
about 20 minutes. This is because compiling
MdeModulePkg for debug, release, and host based tests
take a while. Most other packages are in the 10 minute
range. We do have easy ways to disable or limit
certain tests as well as expand the matrix to leverage
more cloud resources (more parallel builds).


Content is best viewed online with links to helpful
content but is also attached below:
https://github.com/spbrogan/edk2-staging/blob/edk2-
stuart-ci-latest/Readme-CI-RFC.md

# CI and PR Gates

## Background

Historically, while the TianoCore maintainers and
stewards have done a fantastic job of keeping
contribution policies consistent and contributions
clean and well-documented, there have been few
processes that ran to verify the sanity, cleanliness,
and efficacy of the codebase, and even fewer that
publicly published their results for the community at
large. This has caused inconsistancies and issues
within the codebase from time to time.

Adding continuous integration (and potentially PR
gates) to the checkin process ensures that simple
errors like these are caught and can be fixed on a
regular basis.

## Strategy

While a number of CI solutions exist, this proposal
will focus on the usage of Azure Dev Ops and Build
Pipelines. For demonstration, a sample [TianoCore
repo](https://github.com/spbrogan/edk2-staging.git)
(branch edk2-stuart-ci-latest) and [Dev Ops
Pipeline](https://dev.azure.com/tianocore/edk2-ci-
play/_build?definitionId=12) have been set up.

Furthermore, this proposal will leverage the TianoCore
python tools PIP modules:
[library](https://pypi.org/project/edk2-pytool-
library/) and
[extensions](https://pypi.org/project/edk2-pytool-
extensions/) (with repos located
[here](https://github.com/tianocore/edk2-pytool-
library) and [here](https://github.com/tianocore/edk2-
pytool-extensions)).

The primary execution flows can be found in the `azure-
pipelines-pr-gate.yml` and `azure-pipelines-pr-gate-
linux.yml` files. These YAML files are consumed by the
Azure Dev Ops Build Pipeline and dictate what server
resources should be used, how they should be
configured, and what processes should be run on them.
An overview of this schema can be found
[here](https://docs.microsoft.com/en-
us/azure/devops/pipelines/yaml-schema?view=azure-
devops&tabs=schema).

Inspection of these files reveals the EDKII Tools
commands that make up the primary processes for the CI
build: 'stuart_setup', 'stuart_update', and
'stuart_ci_build'. These commands come from the EDKII
Tools PIP modules and are configured as described
below. More documentation on the stuart tools can be
found [here](https://github.com/tianocore/edk2-pytool-
extensions/blob/master/docs/using.md) and
[here](https://github.com/tianocore/edk2-pytool-
extensions/blob/master/docs/features/feature_invocables
.md).

## Configuration

Configuration of the CI process consists of (in order
of precedence):
* command-line arguments passed in via the Pipeline
YAML
* a per-package configuration file (e.g. `<package-
name>.mu.yaml`) that is detected by the CI system in
EDKII Tools.
* a global configuration Python module (e.g.
`CISetting.py`) passed in via the command-line

The global configuration file is described in [this
readme](https://github.com/tianocore/edk2-pytool-
extensions/blob/master/docs/usability/using_settings_ma
nager.md) from the EDKII Tools documentation. This
configuration is written as a Python module so that
decisions can be made dynamically based on command line
parameters and codebase state.

The per-package configuration file can override most
settings in the global configuration file, but is not
dynamic. This file can be used to skip or customize
tests that may be incompatible with a specific package.
By default, the global configuration will try to run
all tests on all packages.

## CI Test Types

All CI tests are instances of EDKII Tools plugins.
Documentation on the plugin system can be found
[here](https://github.com/tianocore/edk2-pytool-
extensions/blob/master/docs/usability/using_plugin_mana
ger.md) and [here](https://github.com/tianocore/edk2-
pytool-
extensions/blob/master/docs/features/feature_plugin_man
ager.md). Upon invocation, each plugin will be passed
the path to the current package under test and a
dictionary containing its targeted configuration, as
assembled from the command line, per-package
configuration, and global configuration.

Note: CI plugins are considered unique from build
plugins and helper plugins, even though some CI plugins
may execute steps of a build.

In the example, these plugins live alongside the code
under test (in the `BaseTools` directory), but may be
moved to the 'edk2-test' repo if that location makes
more sense for the community.

### Module Inclusion Test - DscCompleteCheck

This test scans all available modules (via INF files)
and compares them to the package-level DSC file for the
package each module is contained within. The test
considers it an error if any module does not appear in
the `Components` section of at least one package-level
DSC (indicating that it would not be built if the
package were built).

### Code Compilation Test - CompilerPlugin

Once the Module Inclusion Test has verified that all
modules would be built if all package-level DSCs were
built, the Code Compilation Test simply runs through
and builds every package-level DSC on every toolchain
and for every architecture that is supported. Any
module that fails to build is considered an error.

### Host-Based UnitTests - HostUnitTestCompilerPlugin
and HostUnitTestDscCompleteCheck

The [Testing RFC doc](Readme-Testing-RFC.md) has much
more detail on this, but the basic idea is that host-
based unit tests can be compiled against individual
modules and libraries and run on the build agent (in
this case, the Dev Ops build server). The successful
and failing test case results are collected and
included in the final build report.

### GUID Uniqueness Test - GuidCheck

This test works on the collection of all packages
rather than an individual package. It looks at all
FILE_GUIDs and GUIDs declared in DEC files and ensures
that they are unique for the codebase. This prevents,
for example, accidental duplication of GUIDs when using
an existing INF as a template for a new module.

### Cross-Package Dependency Test - DependencyCheck

This test compares the list of all packages used in
INFs files for a given package against a list of
"allowed dependencies" in plugin configuration for that
package. Any module that depends on a disallowed
package will cause a test failure.

### Library Declaration Test - LibraryClassCheck

This test looks at all library header files found in a
package's `Include/Library` directory and ensures that
all files have a matching LibraryClass declaration in
the DEC file for the package. Any missing declarations
will cause a failure.

### Invalid Character Test - CharEncodingCheck

This test scans all files in a package to make sure
that there are no invalid Unicode characters that may
cause build errors in some character
sets/localizations.

## Next Steps

* Receive community feedback on RFC.
* Determine where this phase makes sense given existing
RFCs from other TianoCore contributors.
* Optimize testing beharior.
* Only run a subset of tests on PRs or individual
commits.
* Run full testing either once per day or once every
several commits.
* Add more tests/capabilities.
* Continue to improve results formatting.
* Continue to improve CI documentation.
* Much of this documentation effort is pending
community feedback on which parts are needed and what
phases are priorities.

Thanks


-----Original Message-----
From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf
Of Michael D Kinney via Groups.Io
Sent: Thursday, August 29, 2019 1:23 PM
To: devel@edk2.groups.io; rfc@edk2.groups.io
Subject: [edk2-rfc] [RFC] EDK II Continuous Integration
Phase 1

Hello,

This is a proposal for a first step towards continuous
integration for all TianoCore repositories to help
improve to quality of commits and automate testing and
release processes for all EDK II packages and
platforms.

This is based on work from a number of EDK II community
members that have provide valuable input and
evaluations.

* Rebecca Cran <mailto:rebecca@...> Jenkins
evaluation
* Laszlo Ersek <mailto:lersek@...> GitLab
evaluation
* Philippe Mathieu-Daudé <mailto:philmd@...>
GitLab evaluation
* Sean Brogan <mailto:sean.brogan@...> Azure
Pipelines and HBFA
* Bret Barkelew <mailto:Bret.Barkelew@...>
Azure Pipelines and HBFA
* Jiewen Yao <mailto:jiewen.yao@...> HBFA

The following link is a link to an EDK II WIKI page
that contains a summary of the work to date. Please
provide feedback in the EDK II mailing lists. The WIKI
pages will be updated with input from the entire EDK II
community.


https://nam06.safelinks.protection.outlook.com/?url=htt
ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
%2Fwiki%2FEDK-II-Continuous-
Integration&amp;data=02%7C01%7Csean.brogan%40microsoft.
com%7C6f67f169a6c746b4288608d72cbea7b6%7C72f988bf86f141
af91ab2d7cd011db47%7C1%7C0%7C637027069686644659&amp;sda
ta=GR9wN6gP3mJlCTopAaQ2rlzhby1nuF%2BwDVsfFIQAZjA%3D&amp
;reserved=0

Proposal
========
Phase 1 of adding continuous integration is limited to
the
edk2 repository. Additional repositories will be added
later.

The following changes are proposed:
* Remove EDK II Maintainers write access to edk2
repository.
Only EDK II Administrators will continue to have
write
access, and that should only be used to handle
extraordinary
events.
* EDK II Maintainers use a GitHub Pull Request instead
of push
to commit a patch series to the edk2 repository.
There are
no other changes to the development and review
process. The
patch series is prepared in an EDK II maintainer
branch with
all commit message requirements met on each patch in
the series.
* The edk2 repository only accepts Pull Requests from
members
of the EDK II Maintainers team. Pull Requests from
anyone else
are rejected.
* Run pre-commit checks using Azure Pipelines
* If all pre-commit checks pass, then the patch series
is auto
committed. The result of this commit must match the
contents
and commit history that would have occurred using the
previous
push operation.
* If any pre-commit checks fail, then notify the
submitter.
A typical reason for a failure would be a merge
conflict with
another pull request that was just processed.
* Limit pre-commit checks execution time to 10 minutes.
* Provide on-demand builds to EDK II Maintainers that
to allow
EDK II Maintainers to submit a branch through for the
same
set of pre-commit checks without submitting a pull
request.

## Pre-Commit Checks in Phase 1
* Run and pass PatchCheck.py with no errors

=====================================================

The following are some additional pre-commit check
ideas that could be quickly added once the initial
version using PatchCheck.py is fully functional.
Please provide feedback on the ones you like and
additional ones you think may improve the quality of
the commits to the edk2 repository.

## Proposed Pre-Commit Checks in Phase 2
* Verify Reviewed-by and Acked-by tags are present with
correct maintainer email addresses
* Verify no non-ASCII characters in modified files
* Verify no binary files in set of modified files
* Verify package dependency rules in modified files

## Proposed Pre-Commit Checks in Phase 3
* Run ECC on modified files
* Verify modified modules/libs build
* Run available host based tests (HBFA) against
modified
modules/libs

Best regards,

Mike










Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

Michael D Kinney
 

Hi Sean,

This looks really good and I agree we can combine the
RFCs and enable more pre-commits tests.

Additional responses below.

Mike

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On
Behalf Of Sean via Groups.Io
Sent: Thursday, August 29, 2019 7:22 PM
To: rfc@edk2.groups.io; Kinney, Michael D
<michael.d.kinney@...>; devel@edk2.groups.io
Cc: Bret Barkelew <Bret.Barkelew@...>
Subject: Re: [edk2-devel] [RFC] EDK II Continuous
Integration Phase 1

Mike, as you mentioned we have been working towards
enabling a practical and extensible CI for Edk2 using
Azure dev ops and the recently added edk2-pytool
infrastructure. We have been using similar CI for
Project Mu for the last few years.

Our approach is a little different in that we focus on
validating the whole code base rather than just the
incoming patch. We do this because we have found
unexpected consequences of patches and overall we want
all code to be compliant not just new additions. We
have found the time to test the whole tree is not much
longer than only the parts impacted by a code change
(except maybe when running the entire compile test on
every package). This obviously comes with an initial
tax of needing to get the codebase into compliant form.
Anyway we have prepared an RFC in addition to yours and
would like to see these two efforts merged together.
I am in favor of complete code base testing as long as the
time to complete the tests is reasonable. We can do both
patch level testing to make sure the patches are well
formatted. Focused testing on the components modified by
the patches and some amount of complete testing to find any
unexpected side effects.


We are still working on making a few optimizations.
Currently if the full set of tests are run we take
about 20 minutes. This is because compiling
MdeModulePkg for debug, release, and host based tests
take a while. Most other packages are in the 10 minute
range. We do have easy ways to disable or limit
certain tests as well as expand the matrix to leverage
more cloud resources (more parallel builds).


Content is best viewed online with links to helpful
content but is also attached below:
https://github.com/spbrogan/edk2-staging/blob/edk2-
stuart-ci-latest/Readme-CI-RFC.md

# CI and PR Gates

## Background

Historically, while the TianoCore maintainers and
stewards have done a fantastic job of keeping
contribution policies consistent and contributions
clean and well-documented, there have been few
processes that ran to verify the sanity, cleanliness,
and efficacy of the codebase, and even fewer that
publicly published their results for the community at
large. This has caused inconsistancies and issues
within the codebase from time to time.

Adding continuous integration (and potentially PR
gates) to the checkin process ensures that simple
errors like these are caught and can be fixed on a
regular basis.

## Strategy

While a number of CI solutions exist, this proposal
will focus on the usage of Azure Dev Ops and Build
Pipelines. For demonstration, a sample [TianoCore
repo](https://github.com/spbrogan/edk2-staging.git)
(branch edk2-stuart-ci-latest) and [Dev Ops
Pipeline](https://dev.azure.com/tianocore/edk2-ci-
play/_build?definitionId=12) have been set up.

Furthermore, this proposal will leverage the TianoCore
python tools PIP modules:
[library](https://pypi.org/project/edk2-pytool-
library/) and
[extensions](https://pypi.org/project/edk2-pytool-
extensions/) (with repos located
[here](https://github.com/tianocore/edk2-pytool-
library) and [here](https://github.com/tianocore/edk2-
pytool-extensions)).

The primary execution flows can be found in the `azure-
pipelines-pr-gate.yml` and `azure-pipelines-pr-gate-
linux.yml` files. These YAML files are consumed by the
Azure Dev Ops Build Pipeline and dictate what server
resources should be used, how they should be
configured, and what processes should be run on them.
An overview of this schema can be found
[here](https://docs.microsoft.com/en-
us/azure/devops/pipelines/yaml-schema?view=azure-
devops&tabs=schema).

Inspection of these files reveals the EDKII Tools
commands that make up the primary processes for the CI
build: 'stuart_setup', 'stuart_update', and
'stuart_ci_build'. These commands come from the EDKII
Tools PIP modules and are configured as described
below. More documentation on the stuart tools can be
found [here](https://github.com/tianocore/edk2-pytool-
extensions/blob/master/docs/using.md) and
[here](https://github.com/tianocore/edk2-pytool-
extensions/blob/master/docs/features/feature_invocables
.md).

## Configuration

Configuration of the CI process consists of (in order
of precedence):
* command-line arguments passed in via the Pipeline
YAML
* a per-package configuration file (e.g. `<package-
name>.mu.yaml`) that is detected by the CI system in
EDKII Tools.
* a global configuration Python module (e.g.
`CISetting.py`) passed in via the command-line

The global configuration file is described in [this
readme](https://github.com/tianocore/edk2-pytool-
extensions/blob/master/docs/usability/using_settings_ma
nager.md) from the EDKII Tools documentation. This
configuration is written as a Python module so that
decisions can be made dynamically based on command line
parameters and codebase state.

The per-package configuration file can override most
settings in the global configuration file, but is not
dynamic. This file can be used to skip or customize
tests that may be incompatible with a specific package.
By default, the global configuration will try to run
all tests on all packages.

## CI Test Types

All CI tests are instances of EDKII Tools plugins.
Documentation on the plugin system can be found
[here](https://github.com/tianocore/edk2-pytool-
extensions/blob/master/docs/usability/using_plugin_mana
ger.md) and [here](https://github.com/tianocore/edk2-
pytool-
extensions/blob/master/docs/features/feature_plugin_man
ager.md). Upon invocation, each plugin will be passed
the path to the current package under test and a
dictionary containing its targeted configuration, as
assembled from the command line, per-package
configuration, and global configuration.

Note: CI plugins are considered unique from build
plugins and helper plugins, even though some CI plugins
may execute steps of a build.

In the example, these plugins live alongside the code
under test (in the `BaseTools` directory), but may be
moved to the 'edk2-test' repo if that location makes
more sense for the community.
Which of the tests below are passing for all edk2 packages?
Leif has been working on fixing some of the non-ASCII files
and line ending issues. I have been working on some of the
UNI file format issues. If there is a small set of tasks that
would allow more of these tests to be enabled quickly, then
let's get those entered into TianoCore Bugzilla.

### Module Inclusion Test - DscCompleteCheck

This test scans all available modules (via INF files)
and compares them to the package-level DSC file for the
package each module is contained within. The test
considers it an error if any module does not appear in
the `Components` section of at least one package-level
DSC (indicating that it would not be built if the
package were built).

### Code Compilation Test - CompilerPlugin

Once the Module Inclusion Test has verified that all
modules would be built if all package-level DSCs were
built, the Code Compilation Test simply runs through
and builds every package-level DSC on every toolchain
and for every architecture that is supported. Any
module that fails to build is considered an error.

### Host-Based UnitTests - HostUnitTestCompilerPlugin
and HostUnitTestDscCompleteCheck

The [Testing RFC doc](Readme-Testing-RFC.md) has much
more detail on this, but the basic idea is that host-
based unit tests can be compiled against individual
modules and libraries and run on the build agent (in
this case, the Dev Ops build server). The successful
and failing test case results are collected and
included in the final build report.

### GUID Uniqueness Test - GuidCheck

This test works on the collection of all packages
rather than an individual package. It looks at all
FILE_GUIDs and GUIDs declared in DEC files and ensures
that they are unique for the codebase. This prevents,
for example, accidental duplication of GUIDs when using
an existing INF as a template for a new module.

### Cross-Package Dependency Test - DependencyCheck

This test compares the list of all packages used in
INFs files for a given package against a list of
"allowed dependencies" in plugin configuration for that
package. Any module that depends on a disallowed
package will cause a test failure.

### Library Declaration Test - LibraryClassCheck

This test looks at all library header files found in a
package's `Include/Library` directory and ensures that
all files have a matching LibraryClass declaration in
the DEC file for the package. Any missing declarations
will cause a failure.

### Invalid Character Test - CharEncodingCheck

This test scans all files in a package to make sure
that there are no invalid Unicode characters that may
cause build errors in some character
sets/localizations.

## Next Steps

* Receive community feedback on RFC.
* Determine where this phase makes sense given existing
RFCs from other TianoCore contributors.
* Optimize testing beharior.
* Only run a subset of tests on PRs or individual
commits.
* Run full testing either once per day or once every
several commits.
* Add more tests/capabilities.
* Continue to improve results formatting.
* Continue to improve CI documentation.
* Much of this documentation effort is pending
community feedback on which parts are needed and what
phases are priorities.

Thanks


-----Original Message-----
From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf
Of Michael D Kinney via Groups.Io
Sent: Thursday, August 29, 2019 1:23 PM
To: devel@edk2.groups.io; rfc@edk2.groups.io
Subject: [edk2-rfc] [RFC] EDK II Continuous Integration
Phase 1

Hello,

This is a proposal for a first step towards continuous
integration for all TianoCore repositories to help
improve to quality of commits and automate testing and
release processes for all EDK II packages and
platforms.

This is based on work from a number of EDK II community
members that have provide valuable input and
evaluations.

* Rebecca Cran <mailto:rebecca@...> Jenkins
evaluation
* Laszlo Ersek <mailto:lersek@...> GitLab
evaluation
* Philippe Mathieu-Daudé <mailto:philmd@...>
GitLab evaluation
* Sean Brogan <mailto:sean.brogan@...> Azure
Pipelines and HBFA
* Bret Barkelew <mailto:Bret.Barkelew@...>
Azure Pipelines and HBFA
* Jiewen Yao <mailto:jiewen.yao@...> HBFA

The following link is a link to an EDK II WIKI page
that contains a summary of the work to date. Please
provide feedback in the EDK II mailing lists. The WIKI
pages will be updated with input from the entire EDK II
community.


https://nam06.safelinks.protection.outlook.com/?url=htt
ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
%2Fwiki%2FEDK-II-Continuous-
Integration&amp;data=02%7C01%7Csean.brogan%40microsoft.
com%7C6f67f169a6c746b4288608d72cbea7b6%7C72f988bf86f141
af91ab2d7cd011db47%7C1%7C0%7C637027069686644659&amp;sda
ta=GR9wN6gP3mJlCTopAaQ2rlzhby1nuF%2BwDVsfFIQAZjA%3D&amp
;reserved=0

Proposal
========
Phase 1 of adding continuous integration is limited to
the
edk2 repository. Additional repositories will be added
later.

The following changes are proposed:
* Remove EDK II Maintainers write access to edk2
repository.
Only EDK II Administrators will continue to have
write
access, and that should only be used to handle
extraordinary
events.
* EDK II Maintainers use a GitHub Pull Request instead
of push
to commit a patch series to the edk2 repository.
There are
no other changes to the development and review
process. The
patch series is prepared in an EDK II maintainer
branch with
all commit message requirements met on each patch in
the series.
* The edk2 repository only accepts Pull Requests from
members
of the EDK II Maintainers team. Pull Requests from
anyone else
are rejected.
* Run pre-commit checks using Azure Pipelines
* If all pre-commit checks pass, then the patch series
is auto
committed. The result of this commit must match the
contents
and commit history that would have occurred using the
previous
push operation.
* If any pre-commit checks fail, then notify the
submitter.
A typical reason for a failure would be a merge
conflict with
another pull request that was just processed.
* Limit pre-commit checks execution time to 10 minutes.
* Provide on-demand builds to EDK II Maintainers that
to allow
EDK II Maintainers to submit a branch through for the
same
set of pre-commit checks without submitting a pull
request.

## Pre-Commit Checks in Phase 1
* Run and pass PatchCheck.py with no errors

=====================================================

The following are some additional pre-commit check
ideas that could be quickly added once the initial
version using PatchCheck.py is fully functional.
Please provide feedback on the ones you like and
additional ones you think may improve the quality of
the commits to the edk2 repository.

## Proposed Pre-Commit Checks in Phase 2
* Verify Reviewed-by and Acked-by tags are present with
correct maintainer email addresses
* Verify no non-ASCII characters in modified files
* Verify no binary files in set of modified files
* Verify package dependency rules in modified files

## Proposed Pre-Commit Checks in Phase 3
* Run ECC on modified files
* Verify modified modules/libs build
* Run available host based tests (HBFA) against
modified
modules/libs

Best regards,

Mike










Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

Michael D Kinney
 

Sean,

There may be many ways to improve the process and reduce
the work maintainers perform. So these are ideas we can
explore further going forward. I will add the concept
of a non-maintainer opening the PR to the Wiki.

In order to get the improvements to code quality from CI
enabled as quickly as possible, I recommend we limit PRs
to maintainers.

Thanks,

Mike

-----Original Message-----
From: Laszlo Ersek <lersek@...>
Sent: Tuesday, September 3, 2019 10:46 AM
To: Sean Brogan <sean.brogan@...>;
rfc@edk2.groups.io; Ni, Ray <ray.ni@...>;
devel@edk2.groups.io; Gao, Liming
<liming.gao@...>; Kinney, Michael D
<michael.d.kinney@...>
Subject: Re: [edk2-rfc] [edk2-devel] [RFC] EDK II
Continuous Integration Phase 1

On 09/03/19 19:09, Sean Brogan wrote:
Laszlo/Mike,

The idea that the maintainer must create the PR is
fighting the
optimized github PR flow. Github and PRs process is
optimized for
letting everyone contribute from "their" fork while
still protecting
and putting process in place for the "upstream".

Why not use github to assign maintainers to each
package or filepath
and then let contributors submit their own PR to edk2
after the
mailing list has approved. Then the PR has a policy
that someone that
is the maintainer of all files changed must approve
before the PR can
be completed (or you could even set it so that
maintainer must
complete PR). This would have the benefit of less
monotonous work
for the maintainers and on rejected PRs the
contributor could easily
update their branch and resubmit their PR.
I'll let Mike respond.

Thanks
Laszlo


Re: [RFC] EDK II Continuous Integration Phase 1

Michael D Kinney
 

Responses below.

Mike

-----Original Message-----
From: Gao, Liming <liming.gao@...>
Sent: Friday, August 30, 2019 1:44 AM
To: rfc@edk2.groups.io; Kinney, Michael D
<michael.d.kinney@...>; devel@edk2.groups.io
Subject: RE: [RFC] EDK II Continuous Integration Phase
1

Mike:
I add my comments.

-----Original Message-----
From: rfc@edk2.groups.io [mailto:rfc@edk2.groups.io]
On Behalf Of
Michael D Kinney
Sent: Friday, August 30, 2019 4:23 AM
To: devel@edk2.groups.io; rfc@edk2.groups.io
Subject: [edk2-rfc] [RFC] EDK II Continuous
Integration Phase 1

Hello,

This is a proposal for a first step towards continuous
integration for
all TianoCore repositories to help improve to quality
of commits and
automate testing and release processes for all EDK II
packages and
platforms.

This is based on work from a number of EDK II
community members that
have provide valuable input and evaluations.

* Rebecca Cran <rebecca@...> Jenkins evaluation
* Laszlo Ersek <lersek@...> GitLab evaluation
* Philippe Mathieu-Daudé <philmd@...> GitLab
evaluation
* Sean Brogan <sean.brogan@...> Azure
Pipelines and HBFA
* Bret Barkelew <Bret.Barkelew@...> Azure
Pipelines and HBFA
* Jiewen Yao <jiewen.yao@...> HBFA

The following link is a link to an EDK II WIKI page
that contains a
summary of the work to date. Please provide feedback
in the EDK II
mailing lists. The WIKI pages will be updated with
input from the
entire EDK II community.


https://github.com/tianocore/tianocore.github.io/wiki/
EDK-II-Continuous
-
Integration

Proposal
========
Phase 1 of adding continuous integration is limited to
the
edk2 repository. Additional repositories will be
added later.

The following changes are proposed:
* Remove EDK II Maintainers write access to edk2
repository.
Only EDK II Administrators will continue to have
write
access, and that should only be used to handle
extraordinary
events.
* EDK II Maintainers use a GitHub Pull Request instead
of push
to commit a patch series to the edk2 repository.
There are
no other changes to the development and review
process. The
patch series is prepared in an EDK II maintainer
branch with
all commit message requirements met on each patch in
the series.

Will the maintainer manually provide pull request after
the patch passes the review?
Can the script scan the mail list and auto trig pull
request once the patch gets Reviewed-by or Acked-by
from Package maintainers?
The maintainer will do this step manually. After the
required Reviewed-by and Acked-by responses have been
received, the commit messages in the patch series must
be updated to include the Reviewed-by and Acked-by tags.
I do not see an easy way to automate the editing of the
commit messages and guarantee it is always correct.


* The edk2 repository only accepts Pull Requests from
members
of the EDK II Maintainers team. Pull Requests from
anyone else
are rejected.
* Run pre-commit checks using Azure Pipelines
The maintainer manually trig pre-commit check or auto
trig pre-commit?
By default, pre-commit should be auto trigged based on
pull request.
Automatically triggered.

I would be good to provide an on-demand service for maintainers
to run the exact same pre-commit checks on a branch to see
if there are issues that need to be resolved.


* If all pre-commit checks pass, then the patch series
is auto
committed. The result of this commit must match the
contents
and commit history that would have occurred using
the previous
push operation.
* If any pre-commit checks fail, then notify the
submitter.

Will Pre-commit check fail send the mail to the patch
submitter?
Yes.

The patch submitter need update the patch and go
through review process again.
Potentially. If it is a very simple merge conflict with
no functional impacts, then the maintainer should be able
to resolve and resubmit without additional code review.

If there are functional changes to resolve merge conflict,
then another review cycle would be required.


A typical reason for a failure would be a merge
conflict with
another pull request that was just processed.
* Limit pre-commit checks execution time to 10
minutes.
* Provide on-demand builds to EDK II Maintainers that
to allow
EDK II Maintainers to submit a branch through for
the same
set of pre-commit checks without submitting a pull
request.

## Pre-Commit Checks in Phase 1
* Run and pass PatchCheck.py with no errors

=====================================================

The following are some additional pre-commit check
ideas that could be
quickly added once the initial version using
PatchCheck.py is fully
functional. Please provide feedback on the ones you
like and
additional ones you think may improve the quality of
the commits to the
edk2 repository.

## Proposed Pre-Commit Checks in Phase 2
* Verify Reviewed-by and Acked-by tags are present
with
correct maintainer email addresses
* Verify no non-ASCII characters in modified files
* Verify no binary files in set of modified files
Now, Edk2 has few binary files, like logo.bmp.
I see one BZ to request logo bmp update.
(BZ
https://bugzilla.tianocore.org/show_bug.cgi?id=2050)
So, we need the exception way for it.

* Verify package dependency rules in modified files

## Proposed Pre-Commit Checks in Phase 3
* Run ECC on modified files
* Verify modified modules/libs build
* Run available host based tests (HBFA) against
modified
modules/libs
I request boot test on Emulator and Ovmf in the daily
and weekly scope.
Daily can cover boot to Shell.
Weekly can cover more boot functionality.

Thanks
Liming
Best regards,

Mike







Re: UEFI accessibility mandate

Laszlo Ersek
 

On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo


Re: [Qemu-devel] [PATCH 1/2] q35: implement 128K SMRAM at default SMBASE address

Laszlo Ersek
 

Hi Igor,

(+Brijesh)

long-ish pondering ahead, with a question at the end.

On 09/17/19 15:07, Igor Mammedov wrote:
Use commit (2f295167e0 q35/mch: implement extended TSEG sizes) for
inspiration and (ab)use reserved register in config space at 0x9c
offset [*] to extend q35 pci-host with ability to use 128K at
0x30000 as SMRAM and hide it (like TSEG) from non-SMM context.

Usage:
1: write 0xff in the register
2: if the feature is supported, follow up read from the register
should return 0x01. At this point RAM at 0x30000 is still
available for SMI handler configuration from non-SMM context
3: writing 0x02 in the register, locks SMBASE area, making its contents
available only from SMM context. In non-SMM context, reads return
0xff and writes are ignored. Further writes into the register are
ignored until the system reset.
I haven't written any OVMF code for this yet, but I've spent a few hours
thinking about it. Progress! :)

So, this looks really promising.

I'm commenting now for summarizing my thoughts before I write the --
initially minimal -- counterpart patches in OVMF.

There is one complication that deserves this separate email, and that's
SEV's effect on the SMM save state map. It is the topic of the following
edk2 commit range:

1 61a044c6c15f OvmfPkg/MemEncryptSevLib: find pages of initial SMRAM
save state map
2 86defc2c2575 OvmfPkg/PlatformPei: SEV: allocate pages of initial
SMRAM save state map
3 5ef3b66fec13 OvmfPkg/SmmCpuFeaturesLib: SEV: encrypt+free pages of
init. save state map
4 5e2e5647b9fb OvmfPkg/AmdSevDxe: decrypt the pages of the initial
SMRAM save state map

For performing the initial SMBASE relocation, QEMU+KVM have to read the
save state map from guest RAM. For that, under SEV, the guest RAM has to
be decrypted before, and re-encrypted after. Meanwhile, the guest RAM
(while decrypted) should not leak other (unrelated) information to the
hypervisor.

Therefore the above edk2 commits implement the following procedure:

(a.1) in OvmfPkg/PlatformPei, the page in which the save state map
exists, is allocated away from other firmware components

(a.2) in AmdSevDxe, which runs early in the DXE phase (via APRIORI DXE
file), we decrypt the page in question

(a.3) PiSmmCpuDxeSmm, after it performs the initial SMBASE relocation,
calls a hook in SmmCpuFeaturesLib -- and in that hook, we
re-encrypt the page, and also release (free) it for other uses
(firmware and OS both)

Clearly, the above conflicts (or at least intersects) with reserving
128KB (32 pages) at 0x3_0000, i.e. at [0x3_0000..0x4_FFFF], forever,
from the OS. (That area is going to be locked to SMM, and so the OS
should see from the UEFI memmap that it should not touch it.) The
conflict exists because the save state map is in the last kilobyte of
page#15 (from said pages #0..#31) -- the save state map is at
[0x3_FC00..0x3_FFFF].

So here's my plan:

(b.1) In PlatformPei, probe the QEMU feature by writing 0xFF to register
0x9C, and reading back 0x01. If the feature is available, then:

(b.1.1) set a new dynamic PCD to TRUE

(b.1.2) allocate away (as reserved memory) the range
[0x3_0000..0x4_FFFF]

(b.2) In PlatformPei, conditionalize the current, SEV-specific,
allocation of the save state map, on the PCD being FALSE.
(Modifying (a.1).) This will prevent a conflicting
double-allocation (double coverage by memory allocation HOBs)

(b.3) In AmdSevDxe, don't touch the decryption of the save state map
(a.2)

(b.4) in the SmmCpuFeaturesLib hook, preserve the re-encryption of the
save state map (part of (a.3))

(b.5) in the SmmCpuFeaturesLib hook, conditionalize the freeing of the
save state map, on the PCD being FALSE. (Modifying (a.3).) This
will prevent a hole from being punched in the allocation that
covers [0x3_0000..0x4_FFFF], and the entire allocation will
survive into OS runtime.

The above steps ensure that, while the decrypt/encrypt steps prevail,
the save state map allocations will be ingested by the larger, and
longer-term, [0x3_0000..0x4_FFFF] allocation.

Furthermore:

(b.6) Extend SmmAccessDxe to write 0x02 to register 0x9C, in an
EFI_DXE_MM_READY_TO_LOCK_PROTOCOL notification callback, if the
PCD is true. This will cause the [0x3_0000..0x4_FFFF] area to be
locked down at the same time with the rest of SMRAM (i.e., TSEG).

(b.7) Extend SmmAccessDxe to save S3 boot script opcodes for writing
0x02 to register 0x9C, if the PCD is true. This makes sure that
the above lockdown will occur also during S3 resume, before the OS
is reached. (The S3 boot script is executed *after* SMBASE
relocation and TSEG locking, during S3 resume.)

Considering (b.6) and (b.7), the "lockdown on normal boot" (from (b.6))
could actually be merged into (b.5) -- we don't necessarily have to
delay the normal boot lockdown of [0x3_0000..0x4_FFFF] until platform
BDS signals EFI_DXE_MM_READY_TO_LOCK_PROTOCOL; we could do it right
after initial SMBASE relocation.

However: I'd really like to keep (b.6) and (b.7) together, in the same
driver, and (b.7) is really inappropriate for PiSmmCpuDxeSmm (into which
SmmCpuFeaturesLib is linked). For writing S3 boot script opcodes in
(b.7), we need another protocol notify (on EFI_S3_SAVE_STATE_PROTOCOL),
and sneaking that kind of callback into PiSmmCpuDxeSmm sounds really
bad. Hence I plan to add both (b.6) and (b.7) to SmmAccessDxe.

I'll report back with my findings; just wanted to give a heads-up (to
myself as well :))

Finally: can you please remind me why we lock down 128KB (32 pages) at
0x3_0000, and not just half of that? What do we need the range at
[0x4_0000..0x4_FFFF] for?

Thanks!
Laszlo



*) https://www.mail-archive.com/qemu-devel@nongnu.org/msg455991.html

Signed-off-by: Igor Mammedov <imammedo@...>
---
include/hw/pci-host/q35.h | 10 +++++
hw/i386/pc.c | 4 +-
hw/pci-host/q35.c | 80 +++++++++++++++++++++++++++++++++++----
3 files changed, 86 insertions(+), 8 deletions(-)

diff --git a/include/hw/pci-host/q35.h b/include/hw/pci-host/q35.h
index b3bcf2e632..976fbae599 100644
--- a/include/hw/pci-host/q35.h
+++ b/include/hw/pci-host/q35.h
@@ -32,6 +32,7 @@
#include "hw/acpi/ich9.h"
#include "hw/pci-host/pam.h"
#include "hw/i386/intel_iommu.h"
+#include "qemu/units.h"

#define TYPE_Q35_HOST_DEVICE "q35-pcihost"
#define Q35_HOST_DEVICE(obj) \
@@ -54,6 +55,8 @@ typedef struct MCHPCIState {
MemoryRegion smram_region, open_high_smram;
MemoryRegion smram, low_smram, high_smram;
MemoryRegion tseg_blackhole, tseg_window;
+ MemoryRegion smbase_blackhole, smbase_window;
+ bool has_smram_at_smbase;
Range pci_hole;
uint64_t below_4g_mem_size;
uint64_t above_4g_mem_size;
@@ -97,6 +100,13 @@ typedef struct Q35PCIHost {
#define MCH_HOST_BRIDGE_EXT_TSEG_MBYTES_QUERY 0xffff
#define MCH_HOST_BRIDGE_EXT_TSEG_MBYTES_MAX 0xfff

+#define MCH_HOST_BRIDGE_SMBASE_SIZE (128 * KiB)
+#define MCH_HOST_BRIDGE_SMBASE_ADDR 0x30000
+#define MCH_HOST_BRIDGE_F_SMBASE 0x9c
+#define MCH_HOST_BRIDGE_F_SMBASE_QUERY 0xff
+#define MCH_HOST_BRIDGE_F_SMBASE_IN_RAM 0x01
+#define MCH_HOST_BRIDGE_F_SMBASE_LCK 0x02
+
#define MCH_HOST_BRIDGE_PCIEXBAR 0x60 /* 64bit register */
#define MCH_HOST_BRIDGE_PCIEXBAR_SIZE 8 /* 64bit register */
#define MCH_HOST_BRIDGE_PCIEXBAR_DEFAULT 0xb0000000
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index bad866fe44..288d28358a 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -119,7 +119,9 @@ struct hpet_fw_config hpet_cfg = {.count = UINT8_MAX};
/* Physical Address of PVH entry point read from kernel ELF NOTE */
static size_t pvh_start_addr;

-GlobalProperty pc_compat_4_1[] = {};
+GlobalProperty pc_compat_4_1[] = {
+ { "mch", "smbase-smram", "off" },
+};
const size_t pc_compat_4_1_len = G_N_ELEMENTS(pc_compat_4_1);

GlobalProperty pc_compat_4_0[] = {};
diff --git a/hw/pci-host/q35.c b/hw/pci-host/q35.c
index 158d270b9f..c1bd9f78ae 100644
--- a/hw/pci-host/q35.c
+++ b/hw/pci-host/q35.c
@@ -275,20 +275,20 @@ static const TypeInfo q35_host_info = {
* MCH D0:F0
*/

-static uint64_t tseg_blackhole_read(void *ptr, hwaddr reg, unsigned size)
+static uint64_t blackhole_read(void *ptr, hwaddr reg, unsigned size)
{
return 0xffffffff;
}

-static void tseg_blackhole_write(void *opaque, hwaddr addr, uint64_t val,
- unsigned width)
+static void blackhole_write(void *opaque, hwaddr addr, uint64_t val,
+ unsigned width)
{
/* nothing */
}

-static const MemoryRegionOps tseg_blackhole_ops = {
- .read = tseg_blackhole_read,
- .write = tseg_blackhole_write,
+static const MemoryRegionOps blackhole_ops = {
+ .read = blackhole_read,
+ .write = blackhole_write,
.endianness = DEVICE_NATIVE_ENDIAN,
.valid.min_access_size = 1,
.valid.max_access_size = 4,
@@ -430,6 +430,46 @@ static void mch_update_ext_tseg_mbytes(MCHPCIState *mch)
}
}

+static void mch_update_smbase_smram(MCHPCIState *mch)
+{
+ PCIDevice *pd = PCI_DEVICE(mch);
+ uint8_t *reg = pd->config + MCH_HOST_BRIDGE_F_SMBASE;
+ bool lck;
+
+ if (!mch->has_smram_at_smbase) {
+ return;
+ }
+
+ if (*reg == MCH_HOST_BRIDGE_F_SMBASE_QUERY) {
+ pd->wmask[MCH_HOST_BRIDGE_F_SMBASE] =
+ MCH_HOST_BRIDGE_F_SMBASE_LCK;
+ *reg = MCH_HOST_BRIDGE_F_SMBASE_IN_RAM;
+ return;
+ }
+
+ /*
+ * default/reset state, discard written value
+ * which will disable SMRAM balackhole at SMBASE
+ */
+ if (pd->wmask[MCH_HOST_BRIDGE_F_SMBASE] == 0xff) {
+ *reg = 0x00;
+ }
+
+ memory_region_transaction_begin();
+ if (*reg & MCH_HOST_BRIDGE_F_SMBASE_LCK) {
+ /* disable all writes */
+ pd->wmask[MCH_HOST_BRIDGE_F_SMBASE] &=
+ ~MCH_HOST_BRIDGE_F_SMBASE_LCK;
+ *reg = MCH_HOST_BRIDGE_F_SMBASE_LCK;
+ lck = true;
+ } else {
+ lck = false;
+ }
+ memory_region_set_enabled(&mch->smbase_blackhole, lck);
+ memory_region_set_enabled(&mch->smbase_window, lck);
+ memory_region_transaction_commit();
+}
+
static void mch_write_config(PCIDevice *d,
uint32_t address, uint32_t val, int len)
{
@@ -456,6 +496,10 @@ static void mch_write_config(PCIDevice *d,
MCH_HOST_BRIDGE_EXT_TSEG_MBYTES_SIZE)) {
mch_update_ext_tseg_mbytes(mch);
}
+
+ if (ranges_overlap(address, len, MCH_HOST_BRIDGE_F_SMBASE, 1)) {
+ mch_update_smbase_smram(mch);
+ }
}

static void mch_update(MCHPCIState *mch)
@@ -464,6 +508,7 @@ static void mch_update(MCHPCIState *mch)
mch_update_pam(mch);
mch_update_smram(mch);
mch_update_ext_tseg_mbytes(mch);
+ mch_update_smbase_smram(mch);

/*
* pci hole goes from end-of-low-ram to io-apic.
@@ -514,6 +559,9 @@ static void mch_reset(DeviceState *qdev)
MCH_HOST_BRIDGE_EXT_TSEG_MBYTES_QUERY);
}

+ d->config[MCH_HOST_BRIDGE_F_SMBASE] = 0;
+ d->wmask[MCH_HOST_BRIDGE_F_SMBASE] = 0xff;
+
mch_update(mch);
}

@@ -563,7 +611,7 @@ static void mch_realize(PCIDevice *d, Error **errp)
memory_region_add_subregion(&mch->smram, 0xfeda0000, &mch->high_smram);

memory_region_init_io(&mch->tseg_blackhole, OBJECT(mch),
- &tseg_blackhole_ops, NULL,
+ &blackhole_ops, NULL,
"tseg-blackhole", 0);
memory_region_set_enabled(&mch->tseg_blackhole, false);
memory_region_add_subregion_overlap(mch->system_memory,
@@ -575,6 +623,23 @@ static void mch_realize(PCIDevice *d, Error **errp)
memory_region_set_enabled(&mch->tseg_window, false);
memory_region_add_subregion(&mch->smram, mch->below_4g_mem_size,
&mch->tseg_window);
+
+ memory_region_init_io(&mch->smbase_blackhole, OBJECT(mch), &blackhole_ops,
+ NULL, "smbase-blackhole",
+ MCH_HOST_BRIDGE_SMBASE_SIZE);
+ memory_region_set_enabled(&mch->smbase_blackhole, false);
+ memory_region_add_subregion_overlap(mch->system_memory,
+ MCH_HOST_BRIDGE_SMBASE_ADDR,
+ &mch->smbase_blackhole, 1);
+
+ memory_region_init_alias(&mch->smbase_window, OBJECT(mch),
+ "smbase-window", mch->ram_memory,
+ MCH_HOST_BRIDGE_SMBASE_ADDR,
+ MCH_HOST_BRIDGE_SMBASE_SIZE);
+ memory_region_set_enabled(&mch->smbase_window, false);
+ memory_region_add_subregion(&mch->smram, MCH_HOST_BRIDGE_SMBASE_ADDR,
+ &mch->smbase_window);
+
object_property_add_const_link(qdev_get_machine(), "smram",
OBJECT(&mch->smram), &error_abort);

@@ -601,6 +666,7 @@ uint64_t mch_mcfg_base(void)
static Property mch_props[] = {
DEFINE_PROP_UINT16("extended-tseg-mbytes", MCHPCIState, ext_tseg_mbytes,
16),
+ DEFINE_PROP_BOOL("smbase-smram", MCHPCIState, has_smram_at_smbase, true),
DEFINE_PROP_END_OF_LIST(),
};


Re: UEFI accessibility mandate

Andrew Fish <afish@...>
 

Rafael,

Please let us know what you find out. I probably don''t have the time to help implement this feature, but I happy to help work on the architecture and design for UEFI accessibility on the edk2 mailing lists, and I can also represent what ever we come up with at the UEFI Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given there is no guideline on how a User Interface (UI) works. If accessibility requires some from of hardware abstraction, like audio, then we could likely get that into the UEFI Spec. What might be possible is an EDK2 reference implementation of accessibility. Maybe we could use the reference implementation to write a UEFI white paper on design for accessibility? I there is an open source implementation, and an official design guide this would make it much easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS Loader has a UI for the Full Disk Encryption password. If you press the power button quickly 3 times at the disk encryption password prompt accessibility is enabled and Voice Over gets turned on. You then get localized voice prompts when you move between UI elements. Since this is the OS loader all the resources are stored on the disk. You quickly run into a range of challenges since, audio is hard, abstracting audio is hard (what codec does firmware have to support), Audio files are not small and firmware is size constrained, the need to localize the audio responses causes even more size issues, the boot options are usually written by an OS installer so how would firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We choose to go to the Moon!" speech sometimes we chose to do things "not because they are easy, but because they are hard; because that goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept". If we have a design that means we can break the problem up into smaller parts, and maybe we can find people that have expertise in that part to build a chunk at a time. If we could implement the prototype in OVMF that would show how it works, but run on everyones machines, so that would be really helpful for demos and design review.

Thanks,

Andrew Fish

On Sep 13, 2019, at 4:16 AM, Rafael Machado <rafaelrodrigues.machado@...> wrote:

Hi Ethin
I will contact the community you mentioned to get feedback and align expectations.

Hope to hear comments and feedback of other member of discussion (here at EDK2, that is where the work will happen).

Thanks and Regards
Rafael

Em ter, 10 de set de 2019 às 20:50, Ethin Probst <harlydavidsen@... <mailto:harlydavidsen@...>> escreveu:
I've posted this to the largest forum of blind visually impaired
people I know of to date and have gotten some excellent feedback from
them. I also provided in that post a link to this discussion. However,
I can't possibly forward all their questions onto this group (and I
doubt I can answer them all). If you wish to join the discussion over
there, its over at https://forum.audiogames.net/post/461041/#p461041.
(If we can find a way to merge the two discussions, that would make
everything easier. If we can't, I'm fine with that.)

On 9/10/19, Rafael Machado <rafaelrodrigues.machado@... <mailto:rafaelrodrigues.machado@...>> wrote:
Hi Ethin and all

As Leif mentioned, my mastering thesys was the implementation of a
reference code so a driver for audio at pre-OS would be possible,
eliminating the first barrier for the creation of a screen reader at UEFI,
that is the lack of audio drivers.

My project was based on Intel High Definition Audio, so it is not a generic
code that would fit at all scenarios, but it a first step.
The idea was to proceed this work using the Google Summer of Code
initiative,and I will try my best to make this happen in future editions of
this initiative.
Unfortunately my thesys was written in portuguese, because there was almost
nothing related to BIOS in Portuguese, so the decision was to write in
Portuguese to help the community to attract brasilian developers. As soon
as I find some slot, I'll translate to english.
But you are not alone. Just in Brazil I discovered around 30 blind
developers that area waiting for accessibility at BIOS. Also had some
contacts from US and Ireland.
So it is a market to be explored. Everyone will win.
The companies, because they will attract more people and their families,
and the ones who need accessibility.
By the way, most of the population will need accessible solutions if they
live enough. This includes the uefi community members, that today do not
have special needs, but that may require it in future.
Accessibility is not a wast of money.
This ia what I would like to show to the UEFI community.
Hope to have the opportunity to present this in some future event.

Since this discussion was started, who knows what are the next steps?
My code is available at github if someone gets interested (ps.: The code
has failures. It was just a Prof of Concept)

https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Lets keep the discussion, because it is important.

Thanks and Regards
Rafael R. Machado


Em ter, 10 de set de 2019 17:00, Ethin Probst <harlydavidsen@... <mailto:harlydavidsen@...>>
escreveu:

Woops, I just realized while reading my message after I'd sent it that
I had linked to the wrong document. The priority system is outlined
over here (https://freebsoft.org/doc/speechd/ssip.html#Top). This
document discusses SsIP -- the Speech Synthesis Interface Protocol.

On 9/10/19, Ethin Probst <harlydavidsen@... <mailto:harlydavidsen@...>> wrote:
While conquering refreshable braille displays would be nice, I think
we should take it in stages. (Some blind people would want this in
along with the accessibility subsystems, however I do not believe that
would be the wisest approach.) Instead, I think we should do what you
suggest: implement protocols for communicating with audio devices
(both input and output) and add audio hooks for the HII. (We could
even make those audio hooks just general hooks/events instead of
partitioning them for a particular purpose.)
You mentioned that not all devices 'have a keyboard'. This is most
definitely true, but not all devices have audio outputs and/or inputs
either. Therefore, if we can't mandate it for *all* devices, perhaps
we could only mandate it (or at least somehow infer it as a
requirement) for devices that both have an interaction mechanism (any
form of interacting with the system directly via the HII and not, say,
over the wire, because that would be superfluous) and have
mechanisms/devices/drivers/... for audio output at minimum. (Audio
input is not necessarily something that is required unless the device
is controlled primarily by voice, but I think having voice controls is
a bit beyond the UEFI specification and is not something UEFI should
worry about. Voice control is not something you should have in the
preboot environment anyway, correct?)
So, I think the first stage we should work on should include:
- Implementing protocols for communicating with audio outputs (don't
worry about inputs right now)
- Implementing protocols/hooks/events that can be intercepted in the
HII by a foreground/background UEFI driver/application
- Implementing some kind of method to allow the accessibility
protocols to read what is drawn to the text when navigating around
The third item -- when implemented properly -- will prevent the
accessibility subsystem from reading everything on the screen every
time the "cursor" moves. Perhaps another good idea would be to
implement some kind of priority system for text-to-speech, like speech
dispatcher does
(
https://docs.microsoft.com/en-us/cortana/skills/speech-synthesis-markup-language
).

On 9/10/19, Leif Lindholm <leif.lindholm@... <mailto:leif.lindholm@...>> wrote:
Hi Ethin,

Apologies for the delay in responding - I personally don't tend to
read the RFC list (and I know that applies to some other developers
who have now subscribed).

On Sun, Aug 11, 2019 at 06:15:18PM -0700, Ethin Probst wrote:
Hello all,

I'm new here, and was recommended to the TianoCore project by
someone over at the UEFI forum. I've run across TianoCore before,
and like the project.
Before anyone gets worried by the subject line, no, this is not any
kind of legal thing. Its just something I believe needs to
happen. :)
Back in 2016-2017 I contacted the UEFI forum about two problems, one
of which was the format of the spec, which I figured out on my
own. The other problem was not so easily dealt with. Te other
problem relates to accessibility of UEFI-compliant systems and
platform firmware to persons with disabilities. As it currently
stands, such a thing is nonexistent. To be fair, I completely
understand the difficulty that such a thing would require, and I
would fully agree if we still used the PC-AT BIOS systems -- yes,
indeed, I would never suggest this kind of thing on such a system
given that there was no actual standard of any kind for
BIOSes. However, now that UEFI is here, we have such a possibility.
As it currently stands, people with low vision or blind people have
access to their computers in the general sense (I am blind
myself). We can do pretty much anything anyone else could do. We can
code, play games, all that. There are few things that we cannot
do. One of those things is managing our systems firmware in the
preboot environment.
As it stands now, I can only boot other OSes or disks via
memorization. While that worked on BIOS machines (I have, or had, an
old Toshiba laptop that was BIOS-based), it no longer works because
UEFI is mercurial. When I access the boot menu now, I play a game of
chance. If the cards are in my favor, the OS I want to boot boots,
and I can go on my way. But if the cards aren't in my favor, I end
up making something happen that was unintended, and, worst of all, I
have no idea what I did.
However, the boot menu is only one portion of a platform firmware
UI. What about the setup utility or other utilities offered by
computer manufacturers? What about diagnostic utilities,
bootloaders, etc? What do I do with those? Well, I only have one
option -- sited assistance. If I go into my computers setup utility,
I cannot trust myself and say to myself, "OK, I know what I'm
doing. All I need to do is change x and save and quit." No, I can't
do that, because memorizing such a complex interface is extremely
difficult, and its something I wouldn't expect anyone to do.
My proposal is simple, and I'm posting it here because I'd like
comments and feedback before it actually gets implemented (it will
take a lot of time, I'm sure): mandate, in the UEFI specification,
that accessibility features for persons with disabilities must be
implemented and documented, and, if such features are not
implemented, then that vendor is not compliant with the
specification. Place strict minimum requirements for what the
accessibility features should do and how they should work.
So, first of all, I need to point out that what you suggest would be
very difficult for the UEFI Forum to mandate. *But* I don't think this
needs to be a problem.

UEFI primarily exists as a means through which interoperability can be
guaranteed. This means it does not tend to so much mandate what is
actually implemented in a platform, as it defines how it behaves if it
is. This is not 100% true, but suffice to say that mandating the
functionality you are asking for *without* having a reference
implementation available would likely simply be ignored.

If you want to turn thumbscrews, get Microsoft to add the requirement
to their windows logo requirements :)

However, as much as it is difficult to force people to develop this
support, if a functional reference implementation was published under
a permissive open source license, it would take a pretty spectacularly
incompetent product owner not to incorporate it in their products.

Now, I'm sure someone out there will ask me how this can be
done. Well, that's why I've joined the group -- though as I
familiarize myself with EDK2 development and all that I may actually
be able to participate as more than just an accessibility expert, of
sorts.
I have added Rafael Machado on cc. He did a PhD on firmware
accessibility, including a prototype implementing an audio driver for
EDK2. The resulting thesis can be found at
https://arxiv.org/abs/1712.03186 .

As a side note, I have been blind all my life. I was born with
retinopathy of prematurity (ROP), which resulted because I was born
at 26 weeks. My retina was detached, and, though the doctors
attempted to fix it, it would not remain attached, and there is no
chance of it getting fixed now. I would neither want nor care for
such a cure, however. I have lived my entire life blind, and while
the thought of gaining site back is appealing, I am unwilling to go
through the years and years of rewiring and reconditioning of my
brain that would be required for me to survive with site. To me, it
is simply not worth the cost.
But back to the discussion at hand: I would be happy to discuss how
the accessibility features would work and what would be
required. Even standardizing, through the specification, a key
combination to toggle the accessibility features would be nice, as
that would alleviate the major problem of a blind person buying a
new computer and not knowing how to enable the accessibility
features. The overarching goal would be to make the preboot
environment (including applications run within it) accessible and
usable by blind and visually impaired people as a boot service
only.
Yes, I agree, operating systems are already doing their things.

Again, mandating such a key combination is not necessarily something
that the UEFI specification can do. Some devices won't necessarily
have a keyboard.

Some things the UEFI specification could do is:
- Add protocols for Audio output (and possibly input) and talking to
audio codecs.
- Add audio hooks for the Human Interaction Interface subsystem (HII).

EDK2 could then start adding device drivers producing these protocols,
and enable the support for some reference platforms.

Would we need any specific support for things like refreshable braille
displays?

It would be superfluous to make this a runtime service, as all
major OSes already have accessibility features. Plus, managing such
a thing would be impossible to do.
Agreed.

This email has gotten quite long, so I will suspend the discussion
of functionality and how I would like it to work for a future email
once everyone has gotten on board.
Best Regards,

Leif


Thank you for your time and consideration.



--
Signed,
Ethin D. Probst

--
Signed,
Ethin D. Probst



--
Signed,
Ethin D. Probst

621 - 640 of 786