Date   

Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

Michael D Kinney
 

Hi Sean,

Which OS/Compiler configurations are currently enabled
for the Code Compilation Test?

I have been working on enabling multiple OS/Compiler
configurations in Azure Pipelines. There are some tools
that need to be installed for each of these environments.
Examples include NASM, iASL, Python.

For the work you have done, how are these extra tools
installed? Is it in the YML files or in the Python
scripts.

One critical task is to identify the tools and their
specific versions that the CI system is configured to use.
These configurations should be documented in a Wiki page
and updated as new tools are released and adopted by EDK II.
The inventory of tools used to validate a release should
Also be documented in a release notes for a stable tag.

Here are the YML files that install the additional tools
required to support EDK II builds. I need the source and
versions of these tools to be reviewed and approved.

https://github.com/mdkinney/edk2-ci/blob/master/AzurePipelines/WindowsPrerequisites.yml
https://github.com/mdkinney/edk2-ci/blob/master/AzurePipelines/UbuntuPrerequisites.yml
https://github.com/mdkinney/edk2-ci/blob/master/AzurePipelines/MacOsPrerequisites.yml

Thanks,

Mike

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On
Behalf Of Sean via Groups.Io
Sent: Thursday, August 29, 2019 7:22 PM
To: rfc@edk2.groups.io; Kinney, Michael D
<michael.d.kinney@intel.com>; devel@edk2.groups.io
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Subject: Re: [edk2-devel] [RFC] EDK II Continuous
Integration Phase 1

Mike, as you mentioned we have been working towards
enabling a practical and extensible CI for Edk2 using
Azure dev ops and the recently added edk2-pytool
infrastructure. We have been using similar CI for
Project Mu for the last few years.

Our approach is a little different in that we focus on
validating the whole code base rather than just the
incoming patch. We do this because we have found
unexpected consequences of patches and overall we want
all code to be compliant not just new additions. We
have found the time to test the whole tree is not much
longer than only the parts impacted by a code change
(except maybe when running the entire compile test on
every package). This obviously comes with an initial
tax of needing to get the codebase into compliant form.
Anyway we have prepared an RFC in addition to yours and
would like to see these two efforts merged together.

We are still working on making a few optimizations.
Currently if the full set of tests are run we take
about 20 minutes. This is because compiling
MdeModulePkg for debug, release, and host based tests
take a while. Most other packages are in the 10 minute
range. We do have easy ways to disable or limit
certain tests as well as expand the matrix to leverage
more cloud resources (more parallel builds).


Content is best viewed online with links to helpful
content but is also attached below:
https://github.com/spbrogan/edk2-staging/blob/edk2-
stuart-ci-latest/Readme-CI-RFC.md

# CI and PR Gates

## Background

Historically, while the TianoCore maintainers and
stewards have done a fantastic job of keeping
contribution policies consistent and contributions
clean and well-documented, there have been few
processes that ran to verify the sanity, cleanliness,
and efficacy of the codebase, and even fewer that
publicly published their results for the community at
large. This has caused inconsistancies and issues
within the codebase from time to time.

Adding continuous integration (and potentially PR
gates) to the checkin process ensures that simple
errors like these are caught and can be fixed on a
regular basis.

## Strategy

While a number of CI solutions exist, this proposal
will focus on the usage of Azure Dev Ops and Build
Pipelines. For demonstration, a sample [TianoCore
repo](https://github.com/spbrogan/edk2-staging.git)
(branch edk2-stuart-ci-latest) and [Dev Ops
Pipeline](https://dev.azure.com/tianocore/edk2-ci-
play/_build?definitionId=12) have been set up.

Furthermore, this proposal will leverage the TianoCore
python tools PIP modules:
[library](https://pypi.org/project/edk2-pytool-
library/) and
[extensions](https://pypi.org/project/edk2-pytool-
extensions/) (with repos located
[here](https://github.com/tianocore/edk2-pytool-
library) and [here](https://github.com/tianocore/edk2-
pytool-extensions)).

The primary execution flows can be found in the `azure-
pipelines-pr-gate.yml` and `azure-pipelines-pr-gate-
linux.yml` files. These YAML files are consumed by the
Azure Dev Ops Build Pipeline and dictate what server
resources should be used, how they should be
configured, and what processes should be run on them.
An overview of this schema can be found
[here](https://docs.microsoft.com/en-
us/azure/devops/pipelines/yaml-schema?view=azure-
devops&tabs=schema).

Inspection of these files reveals the EDKII Tools
commands that make up the primary processes for the CI
build: 'stuart_setup', 'stuart_update', and
'stuart_ci_build'. These commands come from the EDKII
Tools PIP modules and are configured as described
below. More documentation on the stuart tools can be
found [here](https://github.com/tianocore/edk2-pytool-
extensions/blob/master/docs/using.md) and
[here](https://github.com/tianocore/edk2-pytool-
extensions/blob/master/docs/features/feature_invocables
.md).

## Configuration

Configuration of the CI process consists of (in order
of precedence):
* command-line arguments passed in via the Pipeline
YAML
* a per-package configuration file (e.g. `<package-
name>.mu.yaml`) that is detected by the CI system in
EDKII Tools.
* a global configuration Python module (e.g.
`CISetting.py`) passed in via the command-line

The global configuration file is described in [this
readme](https://github.com/tianocore/edk2-pytool-
extensions/blob/master/docs/usability/using_settings_ma
nager.md) from the EDKII Tools documentation. This
configuration is written as a Python module so that
decisions can be made dynamically based on command line
parameters and codebase state.

The per-package configuration file can override most
settings in the global configuration file, but is not
dynamic. This file can be used to skip or customize
tests that may be incompatible with a specific package.
By default, the global configuration will try to run
all tests on all packages.

## CI Test Types

All CI tests are instances of EDKII Tools plugins.
Documentation on the plugin system can be found
[here](https://github.com/tianocore/edk2-pytool-
extensions/blob/master/docs/usability/using_plugin_mana
ger.md) and [here](https://github.com/tianocore/edk2-
pytool-
extensions/blob/master/docs/features/feature_plugin_man
ager.md). Upon invocation, each plugin will be passed
the path to the current package under test and a
dictionary containing its targeted configuration, as
assembled from the command line, per-package
configuration, and global configuration.

Note: CI plugins are considered unique from build
plugins and helper plugins, even though some CI plugins
may execute steps of a build.

In the example, these plugins live alongside the code
under test (in the `BaseTools` directory), but may be
moved to the 'edk2-test' repo if that location makes
more sense for the community.

### Module Inclusion Test - DscCompleteCheck

This test scans all available modules (via INF files)
and compares them to the package-level DSC file for the
package each module is contained within. The test
considers it an error if any module does not appear in
the `Components` section of at least one package-level
DSC (indicating that it would not be built if the
package were built).

### Code Compilation Test - CompilerPlugin

Once the Module Inclusion Test has verified that all
modules would be built if all package-level DSCs were
built, the Code Compilation Test simply runs through
and builds every package-level DSC on every toolchain
and for every architecture that is supported. Any
module that fails to build is considered an error.

### Host-Based UnitTests - HostUnitTestCompilerPlugin
and HostUnitTestDscCompleteCheck

The [Testing RFC doc](Readme-Testing-RFC.md) has much
more detail on this, but the basic idea is that host-
based unit tests can be compiled against individual
modules and libraries and run on the build agent (in
this case, the Dev Ops build server). The successful
and failing test case results are collected and
included in the final build report.

### GUID Uniqueness Test - GuidCheck

This test works on the collection of all packages
rather than an individual package. It looks at all
FILE_GUIDs and GUIDs declared in DEC files and ensures
that they are unique for the codebase. This prevents,
for example, accidental duplication of GUIDs when using
an existing INF as a template for a new module.

### Cross-Package Dependency Test - DependencyCheck

This test compares the list of all packages used in
INFs files for a given package against a list of
"allowed dependencies" in plugin configuration for that
package. Any module that depends on a disallowed
package will cause a test failure.

### Library Declaration Test - LibraryClassCheck

This test looks at all library header files found in a
package's `Include/Library` directory and ensures that
all files have a matching LibraryClass declaration in
the DEC file for the package. Any missing declarations
will cause a failure.

### Invalid Character Test - CharEncodingCheck

This test scans all files in a package to make sure
that there are no invalid Unicode characters that may
cause build errors in some character
sets/localizations.

## Next Steps

* Receive community feedback on RFC.
* Determine where this phase makes sense given existing
RFCs from other TianoCore contributors.
* Optimize testing beharior.
* Only run a subset of tests on PRs or individual
commits.
* Run full testing either once per day or once every
several commits.
* Add more tests/capabilities.
* Continue to improve results formatting.
* Continue to improve CI documentation.
* Much of this documentation effort is pending
community feedback on which parts are needed and what
phases are priorities.

Thanks


-----Original Message-----
From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf
Of Michael D Kinney via Groups.Io
Sent: Thursday, August 29, 2019 1:23 PM
To: devel@edk2.groups.io; rfc@edk2.groups.io
Subject: [edk2-rfc] [RFC] EDK II Continuous Integration
Phase 1

Hello,

This is a proposal for a first step towards continuous
integration for all TianoCore repositories to help
improve to quality of commits and automate testing and
release processes for all EDK II packages and
platforms.

This is based on work from a number of EDK II community
members that have provide valuable input and
evaluations.

* Rebecca Cran <mailto:rebecca@bsdio.com> Jenkins
evaluation
* Laszlo Ersek <mailto:lersek@redhat.com> GitLab
evaluation
* Philippe Mathieu-Daudé <mailto:philmd@redhat.com>
GitLab evaluation
* Sean Brogan <mailto:sean.brogan@microsoft.com> Azure
Pipelines and HBFA
* Bret Barkelew <mailto:Bret.Barkelew@microsoft.com>
Azure Pipelines and HBFA
* Jiewen Yao <mailto:jiewen.yao@intel.com> HBFA

The following link is a link to an EDK II WIKI page
that contains a summary of the work to date. Please
provide feedback in the EDK II mailing lists. The WIKI
pages will be updated with input from the entire EDK II
community.


https://nam06.safelinks.protection.outlook.com/?url=htt
ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
%2Fwiki%2FEDK-II-Continuous-
Integration&amp;data=02%7C01%7Csean.brogan%40microsoft.
com%7C6f67f169a6c746b4288608d72cbea7b6%7C72f988bf86f141
af91ab2d7cd011db47%7C1%7C0%7C637027069686644659&amp;sda
ta=GR9wN6gP3mJlCTopAaQ2rlzhby1nuF%2BwDVsfFIQAZjA%3D&amp
;reserved=0

Proposal
========
Phase 1 of adding continuous integration is limited to
the
edk2 repository. Additional repositories will be added
later.

The following changes are proposed:
* Remove EDK II Maintainers write access to edk2
repository.
Only EDK II Administrators will continue to have
write
access, and that should only be used to handle
extraordinary
events.
* EDK II Maintainers use a GitHub Pull Request instead
of push
to commit a patch series to the edk2 repository.
There are
no other changes to the development and review
process. The
patch series is prepared in an EDK II maintainer
branch with
all commit message requirements met on each patch in
the series.
* The edk2 repository only accepts Pull Requests from
members
of the EDK II Maintainers team. Pull Requests from
anyone else
are rejected.
* Run pre-commit checks using Azure Pipelines
* If all pre-commit checks pass, then the patch series
is auto
committed. The result of this commit must match the
contents
and commit history that would have occurred using the
previous
push operation.
* If any pre-commit checks fail, then notify the
submitter.
A typical reason for a failure would be a merge
conflict with
another pull request that was just processed.
* Limit pre-commit checks execution time to 10 minutes.
* Provide on-demand builds to EDK II Maintainers that
to allow
EDK II Maintainers to submit a branch through for the
same
set of pre-commit checks without submitting a pull
request.

## Pre-Commit Checks in Phase 1
* Run and pass PatchCheck.py with no errors

=====================================================

The following are some additional pre-commit check
ideas that could be quickly added once the initial
version using PatchCheck.py is fully functional.
Please provide feedback on the ones you like and
additional ones you think may improve the quality of
the commits to the edk2 repository.

## Proposed Pre-Commit Checks in Phase 2
* Verify Reviewed-by and Acked-by tags are present with
correct maintainer email addresses
* Verify no non-ASCII characters in modified files
* Verify no binary files in set of modified files
* Verify package dependency rules in modified files

## Proposed Pre-Commit Checks in Phase 3
* Run ECC on modified files
* Verify modified modules/libs build
* Run available host based tests (HBFA) against
modified
modules/libs

Best regards,

Mike










Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

Michael D Kinney
 

Hi Sean,

This looks really good and I agree we can combine the
RFCs and enable more pre-commits tests.

Additional responses below.

Mike

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On
Behalf Of Sean via Groups.Io
Sent: Thursday, August 29, 2019 7:22 PM
To: rfc@edk2.groups.io; Kinney, Michael D
<michael.d.kinney@intel.com>; devel@edk2.groups.io
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Subject: Re: [edk2-devel] [RFC] EDK II Continuous
Integration Phase 1

Mike, as you mentioned we have been working towards
enabling a practical and extensible CI for Edk2 using
Azure dev ops and the recently added edk2-pytool
infrastructure. We have been using similar CI for
Project Mu for the last few years.

Our approach is a little different in that we focus on
validating the whole code base rather than just the
incoming patch. We do this because we have found
unexpected consequences of patches and overall we want
all code to be compliant not just new additions. We
have found the time to test the whole tree is not much
longer than only the parts impacted by a code change
(except maybe when running the entire compile test on
every package). This obviously comes with an initial
tax of needing to get the codebase into compliant form.
Anyway we have prepared an RFC in addition to yours and
would like to see these two efforts merged together.
I am in favor of complete code base testing as long as the
time to complete the tests is reasonable. We can do both
patch level testing to make sure the patches are well
formatted. Focused testing on the components modified by
the patches and some amount of complete testing to find any
unexpected side effects.


We are still working on making a few optimizations.
Currently if the full set of tests are run we take
about 20 minutes. This is because compiling
MdeModulePkg for debug, release, and host based tests
take a while. Most other packages are in the 10 minute
range. We do have easy ways to disable or limit
certain tests as well as expand the matrix to leverage
more cloud resources (more parallel builds).


Content is best viewed online with links to helpful
content but is also attached below:
https://github.com/spbrogan/edk2-staging/blob/edk2-
stuart-ci-latest/Readme-CI-RFC.md

# CI and PR Gates

## Background

Historically, while the TianoCore maintainers and
stewards have done a fantastic job of keeping
contribution policies consistent and contributions
clean and well-documented, there have been few
processes that ran to verify the sanity, cleanliness,
and efficacy of the codebase, and even fewer that
publicly published their results for the community at
large. This has caused inconsistancies and issues
within the codebase from time to time.

Adding continuous integration (and potentially PR
gates) to the checkin process ensures that simple
errors like these are caught and can be fixed on a
regular basis.

## Strategy

While a number of CI solutions exist, this proposal
will focus on the usage of Azure Dev Ops and Build
Pipelines. For demonstration, a sample [TianoCore
repo](https://github.com/spbrogan/edk2-staging.git)
(branch edk2-stuart-ci-latest) and [Dev Ops
Pipeline](https://dev.azure.com/tianocore/edk2-ci-
play/_build?definitionId=12) have been set up.

Furthermore, this proposal will leverage the TianoCore
python tools PIP modules:
[library](https://pypi.org/project/edk2-pytool-
library/) and
[extensions](https://pypi.org/project/edk2-pytool-
extensions/) (with repos located
[here](https://github.com/tianocore/edk2-pytool-
library) and [here](https://github.com/tianocore/edk2-
pytool-extensions)).

The primary execution flows can be found in the `azure-
pipelines-pr-gate.yml` and `azure-pipelines-pr-gate-
linux.yml` files. These YAML files are consumed by the
Azure Dev Ops Build Pipeline and dictate what server
resources should be used, how they should be
configured, and what processes should be run on them.
An overview of this schema can be found
[here](https://docs.microsoft.com/en-
us/azure/devops/pipelines/yaml-schema?view=azure-
devops&tabs=schema).

Inspection of these files reveals the EDKII Tools
commands that make up the primary processes for the CI
build: 'stuart_setup', 'stuart_update', and
'stuart_ci_build'. These commands come from the EDKII
Tools PIP modules and are configured as described
below. More documentation on the stuart tools can be
found [here](https://github.com/tianocore/edk2-pytool-
extensions/blob/master/docs/using.md) and
[here](https://github.com/tianocore/edk2-pytool-
extensions/blob/master/docs/features/feature_invocables
.md).

## Configuration

Configuration of the CI process consists of (in order
of precedence):
* command-line arguments passed in via the Pipeline
YAML
* a per-package configuration file (e.g. `<package-
name>.mu.yaml`) that is detected by the CI system in
EDKII Tools.
* a global configuration Python module (e.g.
`CISetting.py`) passed in via the command-line

The global configuration file is described in [this
readme](https://github.com/tianocore/edk2-pytool-
extensions/blob/master/docs/usability/using_settings_ma
nager.md) from the EDKII Tools documentation. This
configuration is written as a Python module so that
decisions can be made dynamically based on command line
parameters and codebase state.

The per-package configuration file can override most
settings in the global configuration file, but is not
dynamic. This file can be used to skip or customize
tests that may be incompatible with a specific package.
By default, the global configuration will try to run
all tests on all packages.

## CI Test Types

All CI tests are instances of EDKII Tools plugins.
Documentation on the plugin system can be found
[here](https://github.com/tianocore/edk2-pytool-
extensions/blob/master/docs/usability/using_plugin_mana
ger.md) and [here](https://github.com/tianocore/edk2-
pytool-
extensions/blob/master/docs/features/feature_plugin_man
ager.md). Upon invocation, each plugin will be passed
the path to the current package under test and a
dictionary containing its targeted configuration, as
assembled from the command line, per-package
configuration, and global configuration.

Note: CI plugins are considered unique from build
plugins and helper plugins, even though some CI plugins
may execute steps of a build.

In the example, these plugins live alongside the code
under test (in the `BaseTools` directory), but may be
moved to the 'edk2-test' repo if that location makes
more sense for the community.
Which of the tests below are passing for all edk2 packages?
Leif has been working on fixing some of the non-ASCII files
and line ending issues. I have been working on some of the
UNI file format issues. If there is a small set of tasks that
would allow more of these tests to be enabled quickly, then
let's get those entered into TianoCore Bugzilla.

### Module Inclusion Test - DscCompleteCheck

This test scans all available modules (via INF files)
and compares them to the package-level DSC file for the
package each module is contained within. The test
considers it an error if any module does not appear in
the `Components` section of at least one package-level
DSC (indicating that it would not be built if the
package were built).

### Code Compilation Test - CompilerPlugin

Once the Module Inclusion Test has verified that all
modules would be built if all package-level DSCs were
built, the Code Compilation Test simply runs through
and builds every package-level DSC on every toolchain
and for every architecture that is supported. Any
module that fails to build is considered an error.

### Host-Based UnitTests - HostUnitTestCompilerPlugin
and HostUnitTestDscCompleteCheck

The [Testing RFC doc](Readme-Testing-RFC.md) has much
more detail on this, but the basic idea is that host-
based unit tests can be compiled against individual
modules and libraries and run on the build agent (in
this case, the Dev Ops build server). The successful
and failing test case results are collected and
included in the final build report.

### GUID Uniqueness Test - GuidCheck

This test works on the collection of all packages
rather than an individual package. It looks at all
FILE_GUIDs and GUIDs declared in DEC files and ensures
that they are unique for the codebase. This prevents,
for example, accidental duplication of GUIDs when using
an existing INF as a template for a new module.

### Cross-Package Dependency Test - DependencyCheck

This test compares the list of all packages used in
INFs files for a given package against a list of
"allowed dependencies" in plugin configuration for that
package. Any module that depends on a disallowed
package will cause a test failure.

### Library Declaration Test - LibraryClassCheck

This test looks at all library header files found in a
package's `Include/Library` directory and ensures that
all files have a matching LibraryClass declaration in
the DEC file for the package. Any missing declarations
will cause a failure.

### Invalid Character Test - CharEncodingCheck

This test scans all files in a package to make sure
that there are no invalid Unicode characters that may
cause build errors in some character
sets/localizations.

## Next Steps

* Receive community feedback on RFC.
* Determine where this phase makes sense given existing
RFCs from other TianoCore contributors.
* Optimize testing beharior.
* Only run a subset of tests on PRs or individual
commits.
* Run full testing either once per day or once every
several commits.
* Add more tests/capabilities.
* Continue to improve results formatting.
* Continue to improve CI documentation.
* Much of this documentation effort is pending
community feedback on which parts are needed and what
phases are priorities.

Thanks


-----Original Message-----
From: rfc@edk2.groups.io <rfc@edk2.groups.io> On Behalf
Of Michael D Kinney via Groups.Io
Sent: Thursday, August 29, 2019 1:23 PM
To: devel@edk2.groups.io; rfc@edk2.groups.io
Subject: [edk2-rfc] [RFC] EDK II Continuous Integration
Phase 1

Hello,

This is a proposal for a first step towards continuous
integration for all TianoCore repositories to help
improve to quality of commits and automate testing and
release processes for all EDK II packages and
platforms.

This is based on work from a number of EDK II community
members that have provide valuable input and
evaluations.

* Rebecca Cran <mailto:rebecca@bsdio.com> Jenkins
evaluation
* Laszlo Ersek <mailto:lersek@redhat.com> GitLab
evaluation
* Philippe Mathieu-Daudé <mailto:philmd@redhat.com>
GitLab evaluation
* Sean Brogan <mailto:sean.brogan@microsoft.com> Azure
Pipelines and HBFA
* Bret Barkelew <mailto:Bret.Barkelew@microsoft.com>
Azure Pipelines and HBFA
* Jiewen Yao <mailto:jiewen.yao@intel.com> HBFA

The following link is a link to an EDK II WIKI page
that contains a summary of the work to date. Please
provide feedback in the EDK II mailing lists. The WIKI
pages will be updated with input from the entire EDK II
community.


https://nam06.safelinks.protection.outlook.com/?url=htt
ps%3A%2F%2Fgithub.com%2Ftianocore%2Ftianocore.github.io
%2Fwiki%2FEDK-II-Continuous-
Integration&amp;data=02%7C01%7Csean.brogan%40microsoft.
com%7C6f67f169a6c746b4288608d72cbea7b6%7C72f988bf86f141
af91ab2d7cd011db47%7C1%7C0%7C637027069686644659&amp;sda
ta=GR9wN6gP3mJlCTopAaQ2rlzhby1nuF%2BwDVsfFIQAZjA%3D&amp
;reserved=0

Proposal
========
Phase 1 of adding continuous integration is limited to
the
edk2 repository. Additional repositories will be added
later.

The following changes are proposed:
* Remove EDK II Maintainers write access to edk2
repository.
Only EDK II Administrators will continue to have
write
access, and that should only be used to handle
extraordinary
events.
* EDK II Maintainers use a GitHub Pull Request instead
of push
to commit a patch series to the edk2 repository.
There are
no other changes to the development and review
process. The
patch series is prepared in an EDK II maintainer
branch with
all commit message requirements met on each patch in
the series.
* The edk2 repository only accepts Pull Requests from
members
of the EDK II Maintainers team. Pull Requests from
anyone else
are rejected.
* Run pre-commit checks using Azure Pipelines
* If all pre-commit checks pass, then the patch series
is auto
committed. The result of this commit must match the
contents
and commit history that would have occurred using the
previous
push operation.
* If any pre-commit checks fail, then notify the
submitter.
A typical reason for a failure would be a merge
conflict with
another pull request that was just processed.
* Limit pre-commit checks execution time to 10 minutes.
* Provide on-demand builds to EDK II Maintainers that
to allow
EDK II Maintainers to submit a branch through for the
same
set of pre-commit checks without submitting a pull
request.

## Pre-Commit Checks in Phase 1
* Run and pass PatchCheck.py with no errors

=====================================================

The following are some additional pre-commit check
ideas that could be quickly added once the initial
version using PatchCheck.py is fully functional.
Please provide feedback on the ones you like and
additional ones you think may improve the quality of
the commits to the edk2 repository.

## Proposed Pre-Commit Checks in Phase 2
* Verify Reviewed-by and Acked-by tags are present with
correct maintainer email addresses
* Verify no non-ASCII characters in modified files
* Verify no binary files in set of modified files
* Verify package dependency rules in modified files

## Proposed Pre-Commit Checks in Phase 3
* Run ECC on modified files
* Verify modified modules/libs build
* Run available host based tests (HBFA) against
modified
modules/libs

Best regards,

Mike










Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

Michael D Kinney
 

Sean,

There may be many ways to improve the process and reduce
the work maintainers perform. So these are ideas we can
explore further going forward. I will add the concept
of a non-maintainer opening the PR to the Wiki.

In order to get the improvements to code quality from CI
enabled as quickly as possible, I recommend we limit PRs
to maintainers.

Thanks,

Mike

-----Original Message-----
From: Laszlo Ersek <lersek@redhat.com>
Sent: Tuesday, September 3, 2019 10:46 AM
To: Sean Brogan <sean.brogan@microsoft.com>;
rfc@edk2.groups.io; Ni, Ray <ray.ni@intel.com>;
devel@edk2.groups.io; Gao, Liming
<liming.gao@intel.com>; Kinney, Michael D
<michael.d.kinney@intel.com>
Subject: Re: [edk2-rfc] [edk2-devel] [RFC] EDK II
Continuous Integration Phase 1

On 09/03/19 19:09, Sean Brogan wrote:
Laszlo/Mike,

The idea that the maintainer must create the PR is
fighting the
optimized github PR flow. Github and PRs process is
optimized for
letting everyone contribute from "their" fork while
still protecting
and putting process in place for the "upstream".

Why not use github to assign maintainers to each
package or filepath
and then let contributors submit their own PR to edk2
after the
mailing list has approved. Then the PR has a policy
that someone that
is the maintainer of all files changed must approve
before the PR can
be completed (or you could even set it so that
maintainer must
complete PR). This would have the benefit of less
monotonous work
for the maintainers and on rejected PRs the
contributor could easily
update their branch and resubmit their PR.
I'll let Mike respond.

Thanks
Laszlo


Re: [RFC] EDK II Continuous Integration Phase 1

Michael D Kinney
 

Responses below.

Mike

-----Original Message-----
From: Gao, Liming <liming.gao@intel.com>
Sent: Friday, August 30, 2019 1:44 AM
To: rfc@edk2.groups.io; Kinney, Michael D
<michael.d.kinney@intel.com>; devel@edk2.groups.io
Subject: RE: [RFC] EDK II Continuous Integration Phase
1

Mike:
I add my comments.

-----Original Message-----
From: rfc@edk2.groups.io [mailto:rfc@edk2.groups.io]
On Behalf Of
Michael D Kinney
Sent: Friday, August 30, 2019 4:23 AM
To: devel@edk2.groups.io; rfc@edk2.groups.io
Subject: [edk2-rfc] [RFC] EDK II Continuous
Integration Phase 1

Hello,

This is a proposal for a first step towards continuous
integration for
all TianoCore repositories to help improve to quality
of commits and
automate testing and release processes for all EDK II
packages and
platforms.

This is based on work from a number of EDK II
community members that
have provide valuable input and evaluations.

* Rebecca Cran <rebecca@bsdio.com> Jenkins evaluation
* Laszlo Ersek <lersek@redhat.com> GitLab evaluation
* Philippe Mathieu-Daudé <philmd@redhat.com> GitLab
evaluation
* Sean Brogan <sean.brogan@microsoft.com> Azure
Pipelines and HBFA
* Bret Barkelew <Bret.Barkelew@microsoft.com> Azure
Pipelines and HBFA
* Jiewen Yao <jiewen.yao@intel.com> HBFA

The following link is a link to an EDK II WIKI page
that contains a
summary of the work to date. Please provide feedback
in the EDK II
mailing lists. The WIKI pages will be updated with
input from the
entire EDK II community.


https://github.com/tianocore/tianocore.github.io/wiki/
EDK-II-Continuous
-
Integration

Proposal
========
Phase 1 of adding continuous integration is limited to
the
edk2 repository. Additional repositories will be
added later.

The following changes are proposed:
* Remove EDK II Maintainers write access to edk2
repository.
Only EDK II Administrators will continue to have
write
access, and that should only be used to handle
extraordinary
events.
* EDK II Maintainers use a GitHub Pull Request instead
of push
to commit a patch series to the edk2 repository.
There are
no other changes to the development and review
process. The
patch series is prepared in an EDK II maintainer
branch with
all commit message requirements met on each patch in
the series.

Will the maintainer manually provide pull request after
the patch passes the review?
Can the script scan the mail list and auto trig pull
request once the patch gets Reviewed-by or Acked-by
from Package maintainers?
The maintainer will do this step manually. After the
required Reviewed-by and Acked-by responses have been
received, the commit messages in the patch series must
be updated to include the Reviewed-by and Acked-by tags.
I do not see an easy way to automate the editing of the
commit messages and guarantee it is always correct.


* The edk2 repository only accepts Pull Requests from
members
of the EDK II Maintainers team. Pull Requests from
anyone else
are rejected.
* Run pre-commit checks using Azure Pipelines
The maintainer manually trig pre-commit check or auto
trig pre-commit?
By default, pre-commit should be auto trigged based on
pull request.
Automatically triggered.

I would be good to provide an on-demand service for maintainers
to run the exact same pre-commit checks on a branch to see
if there are issues that need to be resolved.


* If all pre-commit checks pass, then the patch series
is auto
committed. The result of this commit must match the
contents
and commit history that would have occurred using
the previous
push operation.
* If any pre-commit checks fail, then notify the
submitter.

Will Pre-commit check fail send the mail to the patch
submitter?
Yes.

The patch submitter need update the patch and go
through review process again.
Potentially. If it is a very simple merge conflict with
no functional impacts, then the maintainer should be able
to resolve and resubmit without additional code review.

If there are functional changes to resolve merge conflict,
then another review cycle would be required.


A typical reason for a failure would be a merge
conflict with
another pull request that was just processed.
* Limit pre-commit checks execution time to 10
minutes.
* Provide on-demand builds to EDK II Maintainers that
to allow
EDK II Maintainers to submit a branch through for
the same
set of pre-commit checks without submitting a pull
request.

## Pre-Commit Checks in Phase 1
* Run and pass PatchCheck.py with no errors

=====================================================

The following are some additional pre-commit check
ideas that could be
quickly added once the initial version using
PatchCheck.py is fully
functional. Please provide feedback on the ones you
like and
additional ones you think may improve the quality of
the commits to the
edk2 repository.

## Proposed Pre-Commit Checks in Phase 2
* Verify Reviewed-by and Acked-by tags are present
with
correct maintainer email addresses
* Verify no non-ASCII characters in modified files
* Verify no binary files in set of modified files
Now, Edk2 has few binary files, like logo.bmp.
I see one BZ to request logo bmp update.
(BZ
https://bugzilla.tianocore.org/show_bug.cgi?id=2050)
So, we need the exception way for it.

* Verify package dependency rules in modified files

## Proposed Pre-Commit Checks in Phase 3
* Run ECC on modified files
* Verify modified modules/libs build
* Run available host based tests (HBFA) against
modified
modules/libs
I request boot test on Emulator and Ovmf in the daily
and weekly scope.
Daily can cover boot to Shell.
Weekly can cover more boot functionality.

Thanks
Liming
Best regards,

Mike







Re: UEFI accessibility mandate

Laszlo Ersek
 

On 09/18/19 19:57, Andrew Fish wrote:
Rafael,

Please let us know what you find out. I probably don''t have the time
to help implement this feature, but I happy to help work on the
architecture and design for UEFI accessibility on the edk2 mailing
lists, and I can also represent what ever we come up with at the UEFI
Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given
there is no guideline on how a User Interface (UI) works. If
accessibility requires some from of hardware abstraction, like audio,
then we could likely get that into the UEFI Spec. What might be
possible is an EDK2 reference implementation of accessibility. Maybe
we could use the reference implementation to write a UEFI white paper
on design for accessibility? I there is an open source
implementation, and an official design guide this would make it much
easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS
Loader has a UI for the Full Disk Encryption password. If you press
the power button quickly 3 times at the disk encryption password
prompt accessibility is enabled and Voice Over gets turned on. You
then get localized voice prompts when you move between UI elements.
Since this is the OS loader all the resources are stored on the disk.
You quickly run into a range of challenges since, audio is hard,
abstracting audio is hard (what codec does firmware have to support),
Audio files are not small and firmware is size constrained, the need
to localize the audio responses causes even more size issues, the
boot options are usually written by an OS installer so how would
firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We
choose to go to the Moon!" speech sometimes we chose to do things
"not because they are easy, but because they are hard; because that
goal will serve to organize and measure the best of our energies and
skills, because that challenge is one that we are willing to accept".
If we have a design that means we can break the problem up into
smaller parts, and maybe we can find people that have expertise in
that part to build a chunk at a time. If we could implement the
prototype in OVMF that would show how it works, but run on everyones
machines, so that would be really helpful for demos and design
review.
Somewhat related, in April there was a thread on virtio-dev that
suggests there is interest in a virtio-audio device model:

https://lists.oasis-open.org/archives/virtio-dev/201904/msg00049.html

It looks like the ACRN project already implements a (non-standard, as of
now) virtio-audio device already:

https://lists.oasis-open.org/archives/virtio-dev/201907/msg00061.html

(This is all I can mention right now.)

Thanks
Laszlo


Re: [Qemu-devel] [PATCH 1/2] q35: implement 128K SMRAM at default SMBASE address

Laszlo Ersek
 

Hi Igor,

(+Brijesh)

long-ish pondering ahead, with a question at the end.

On 09/17/19 15:07, Igor Mammedov wrote:
Use commit (2f295167e0 q35/mch: implement extended TSEG sizes) for
inspiration and (ab)use reserved register in config space at 0x9c
offset [*] to extend q35 pci-host with ability to use 128K at
0x30000 as SMRAM and hide it (like TSEG) from non-SMM context.

Usage:
1: write 0xff in the register
2: if the feature is supported, follow up read from the register
should return 0x01. At this point RAM at 0x30000 is still
available for SMI handler configuration from non-SMM context
3: writing 0x02 in the register, locks SMBASE area, making its contents
available only from SMM context. In non-SMM context, reads return
0xff and writes are ignored. Further writes into the register are
ignored until the system reset.
I haven't written any OVMF code for this yet, but I've spent a few hours
thinking about it. Progress! :)

So, this looks really promising.

I'm commenting now for summarizing my thoughts before I write the --
initially minimal -- counterpart patches in OVMF.

There is one complication that deserves this separate email, and that's
SEV's effect on the SMM save state map. It is the topic of the following
edk2 commit range:

1 61a044c6c15f OvmfPkg/MemEncryptSevLib: find pages of initial SMRAM
save state map
2 86defc2c2575 OvmfPkg/PlatformPei: SEV: allocate pages of initial
SMRAM save state map
3 5ef3b66fec13 OvmfPkg/SmmCpuFeaturesLib: SEV: encrypt+free pages of
init. save state map
4 5e2e5647b9fb OvmfPkg/AmdSevDxe: decrypt the pages of the initial
SMRAM save state map

For performing the initial SMBASE relocation, QEMU+KVM have to read the
save state map from guest RAM. For that, under SEV, the guest RAM has to
be decrypted before, and re-encrypted after. Meanwhile, the guest RAM
(while decrypted) should not leak other (unrelated) information to the
hypervisor.

Therefore the above edk2 commits implement the following procedure:

(a.1) in OvmfPkg/PlatformPei, the page in which the save state map
exists, is allocated away from other firmware components

(a.2) in AmdSevDxe, which runs early in the DXE phase (via APRIORI DXE
file), we decrypt the page in question

(a.3) PiSmmCpuDxeSmm, after it performs the initial SMBASE relocation,
calls a hook in SmmCpuFeaturesLib -- and in that hook, we
re-encrypt the page, and also release (free) it for other uses
(firmware and OS both)

Clearly, the above conflicts (or at least intersects) with reserving
128KB (32 pages) at 0x3_0000, i.e. at [0x3_0000..0x4_FFFF], forever,
from the OS. (That area is going to be locked to SMM, and so the OS
should see from the UEFI memmap that it should not touch it.) The
conflict exists because the save state map is in the last kilobyte of
page#15 (from said pages #0..#31) -- the save state map is at
[0x3_FC00..0x3_FFFF].

So here's my plan:

(b.1) In PlatformPei, probe the QEMU feature by writing 0xFF to register
0x9C, and reading back 0x01. If the feature is available, then:

(b.1.1) set a new dynamic PCD to TRUE

(b.1.2) allocate away (as reserved memory) the range
[0x3_0000..0x4_FFFF]

(b.2) In PlatformPei, conditionalize the current, SEV-specific,
allocation of the save state map, on the PCD being FALSE.
(Modifying (a.1).) This will prevent a conflicting
double-allocation (double coverage by memory allocation HOBs)

(b.3) In AmdSevDxe, don't touch the decryption of the save state map
(a.2)

(b.4) in the SmmCpuFeaturesLib hook, preserve the re-encryption of the
save state map (part of (a.3))

(b.5) in the SmmCpuFeaturesLib hook, conditionalize the freeing of the
save state map, on the PCD being FALSE. (Modifying (a.3).) This
will prevent a hole from being punched in the allocation that
covers [0x3_0000..0x4_FFFF], and the entire allocation will
survive into OS runtime.

The above steps ensure that, while the decrypt/encrypt steps prevail,
the save state map allocations will be ingested by the larger, and
longer-term, [0x3_0000..0x4_FFFF] allocation.

Furthermore:

(b.6) Extend SmmAccessDxe to write 0x02 to register 0x9C, in an
EFI_DXE_MM_READY_TO_LOCK_PROTOCOL notification callback, if the
PCD is true. This will cause the [0x3_0000..0x4_FFFF] area to be
locked down at the same time with the rest of SMRAM (i.e., TSEG).

(b.7) Extend SmmAccessDxe to save S3 boot script opcodes for writing
0x02 to register 0x9C, if the PCD is true. This makes sure that
the above lockdown will occur also during S3 resume, before the OS
is reached. (The S3 boot script is executed *after* SMBASE
relocation and TSEG locking, during S3 resume.)

Considering (b.6) and (b.7), the "lockdown on normal boot" (from (b.6))
could actually be merged into (b.5) -- we don't necessarily have to
delay the normal boot lockdown of [0x3_0000..0x4_FFFF] until platform
BDS signals EFI_DXE_MM_READY_TO_LOCK_PROTOCOL; we could do it right
after initial SMBASE relocation.

However: I'd really like to keep (b.6) and (b.7) together, in the same
driver, and (b.7) is really inappropriate for PiSmmCpuDxeSmm (into which
SmmCpuFeaturesLib is linked). For writing S3 boot script opcodes in
(b.7), we need another protocol notify (on EFI_S3_SAVE_STATE_PROTOCOL),
and sneaking that kind of callback into PiSmmCpuDxeSmm sounds really
bad. Hence I plan to add both (b.6) and (b.7) to SmmAccessDxe.

I'll report back with my findings; just wanted to give a heads-up (to
myself as well :))

Finally: can you please remind me why we lock down 128KB (32 pages) at
0x3_0000, and not just half of that? What do we need the range at
[0x4_0000..0x4_FFFF] for?

Thanks!
Laszlo



*) https://www.mail-archive.com/qemu-devel@nongnu.org/msg455991.html

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
include/hw/pci-host/q35.h | 10 +++++
hw/i386/pc.c | 4 +-
hw/pci-host/q35.c | 80 +++++++++++++++++++++++++++++++++++----
3 files changed, 86 insertions(+), 8 deletions(-)

diff --git a/include/hw/pci-host/q35.h b/include/hw/pci-host/q35.h
index b3bcf2e632..976fbae599 100644
--- a/include/hw/pci-host/q35.h
+++ b/include/hw/pci-host/q35.h
@@ -32,6 +32,7 @@
#include "hw/acpi/ich9.h"
#include "hw/pci-host/pam.h"
#include "hw/i386/intel_iommu.h"
+#include "qemu/units.h"

#define TYPE_Q35_HOST_DEVICE "q35-pcihost"
#define Q35_HOST_DEVICE(obj) \
@@ -54,6 +55,8 @@ typedef struct MCHPCIState {
MemoryRegion smram_region, open_high_smram;
MemoryRegion smram, low_smram, high_smram;
MemoryRegion tseg_blackhole, tseg_window;
+ MemoryRegion smbase_blackhole, smbase_window;
+ bool has_smram_at_smbase;
Range pci_hole;
uint64_t below_4g_mem_size;
uint64_t above_4g_mem_size;
@@ -97,6 +100,13 @@ typedef struct Q35PCIHost {
#define MCH_HOST_BRIDGE_EXT_TSEG_MBYTES_QUERY 0xffff
#define MCH_HOST_BRIDGE_EXT_TSEG_MBYTES_MAX 0xfff

+#define MCH_HOST_BRIDGE_SMBASE_SIZE (128 * KiB)
+#define MCH_HOST_BRIDGE_SMBASE_ADDR 0x30000
+#define MCH_HOST_BRIDGE_F_SMBASE 0x9c
+#define MCH_HOST_BRIDGE_F_SMBASE_QUERY 0xff
+#define MCH_HOST_BRIDGE_F_SMBASE_IN_RAM 0x01
+#define MCH_HOST_BRIDGE_F_SMBASE_LCK 0x02
+
#define MCH_HOST_BRIDGE_PCIEXBAR 0x60 /* 64bit register */
#define MCH_HOST_BRIDGE_PCIEXBAR_SIZE 8 /* 64bit register */
#define MCH_HOST_BRIDGE_PCIEXBAR_DEFAULT 0xb0000000
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index bad866fe44..288d28358a 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -119,7 +119,9 @@ struct hpet_fw_config hpet_cfg = {.count = UINT8_MAX};
/* Physical Address of PVH entry point read from kernel ELF NOTE */
static size_t pvh_start_addr;

-GlobalProperty pc_compat_4_1[] = {};
+GlobalProperty pc_compat_4_1[] = {
+ { "mch", "smbase-smram", "off" },
+};
const size_t pc_compat_4_1_len = G_N_ELEMENTS(pc_compat_4_1);

GlobalProperty pc_compat_4_0[] = {};
diff --git a/hw/pci-host/q35.c b/hw/pci-host/q35.c
index 158d270b9f..c1bd9f78ae 100644
--- a/hw/pci-host/q35.c
+++ b/hw/pci-host/q35.c
@@ -275,20 +275,20 @@ static const TypeInfo q35_host_info = {
* MCH D0:F0
*/

-static uint64_t tseg_blackhole_read(void *ptr, hwaddr reg, unsigned size)
+static uint64_t blackhole_read(void *ptr, hwaddr reg, unsigned size)
{
return 0xffffffff;
}

-static void tseg_blackhole_write(void *opaque, hwaddr addr, uint64_t val,
- unsigned width)
+static void blackhole_write(void *opaque, hwaddr addr, uint64_t val,
+ unsigned width)
{
/* nothing */
}

-static const MemoryRegionOps tseg_blackhole_ops = {
- .read = tseg_blackhole_read,
- .write = tseg_blackhole_write,
+static const MemoryRegionOps blackhole_ops = {
+ .read = blackhole_read,
+ .write = blackhole_write,
.endianness = DEVICE_NATIVE_ENDIAN,
.valid.min_access_size = 1,
.valid.max_access_size = 4,
@@ -430,6 +430,46 @@ static void mch_update_ext_tseg_mbytes(MCHPCIState *mch)
}
}

+static void mch_update_smbase_smram(MCHPCIState *mch)
+{
+ PCIDevice *pd = PCI_DEVICE(mch);
+ uint8_t *reg = pd->config + MCH_HOST_BRIDGE_F_SMBASE;
+ bool lck;
+
+ if (!mch->has_smram_at_smbase) {
+ return;
+ }
+
+ if (*reg == MCH_HOST_BRIDGE_F_SMBASE_QUERY) {
+ pd->wmask[MCH_HOST_BRIDGE_F_SMBASE] =
+ MCH_HOST_BRIDGE_F_SMBASE_LCK;
+ *reg = MCH_HOST_BRIDGE_F_SMBASE_IN_RAM;
+ return;
+ }
+
+ /*
+ * default/reset state, discard written value
+ * which will disable SMRAM balackhole at SMBASE
+ */
+ if (pd->wmask[MCH_HOST_BRIDGE_F_SMBASE] == 0xff) {
+ *reg = 0x00;
+ }
+
+ memory_region_transaction_begin();
+ if (*reg & MCH_HOST_BRIDGE_F_SMBASE_LCK) {
+ /* disable all writes */
+ pd->wmask[MCH_HOST_BRIDGE_F_SMBASE] &=
+ ~MCH_HOST_BRIDGE_F_SMBASE_LCK;
+ *reg = MCH_HOST_BRIDGE_F_SMBASE_LCK;
+ lck = true;
+ } else {
+ lck = false;
+ }
+ memory_region_set_enabled(&mch->smbase_blackhole, lck);
+ memory_region_set_enabled(&mch->smbase_window, lck);
+ memory_region_transaction_commit();
+}
+
static void mch_write_config(PCIDevice *d,
uint32_t address, uint32_t val, int len)
{
@@ -456,6 +496,10 @@ static void mch_write_config(PCIDevice *d,
MCH_HOST_BRIDGE_EXT_TSEG_MBYTES_SIZE)) {
mch_update_ext_tseg_mbytes(mch);
}
+
+ if (ranges_overlap(address, len, MCH_HOST_BRIDGE_F_SMBASE, 1)) {
+ mch_update_smbase_smram(mch);
+ }
}

static void mch_update(MCHPCIState *mch)
@@ -464,6 +508,7 @@ static void mch_update(MCHPCIState *mch)
mch_update_pam(mch);
mch_update_smram(mch);
mch_update_ext_tseg_mbytes(mch);
+ mch_update_smbase_smram(mch);

/*
* pci hole goes from end-of-low-ram to io-apic.
@@ -514,6 +559,9 @@ static void mch_reset(DeviceState *qdev)
MCH_HOST_BRIDGE_EXT_TSEG_MBYTES_QUERY);
}

+ d->config[MCH_HOST_BRIDGE_F_SMBASE] = 0;
+ d->wmask[MCH_HOST_BRIDGE_F_SMBASE] = 0xff;
+
mch_update(mch);
}

@@ -563,7 +611,7 @@ static void mch_realize(PCIDevice *d, Error **errp)
memory_region_add_subregion(&mch->smram, 0xfeda0000, &mch->high_smram);

memory_region_init_io(&mch->tseg_blackhole, OBJECT(mch),
- &tseg_blackhole_ops, NULL,
+ &blackhole_ops, NULL,
"tseg-blackhole", 0);
memory_region_set_enabled(&mch->tseg_blackhole, false);
memory_region_add_subregion_overlap(mch->system_memory,
@@ -575,6 +623,23 @@ static void mch_realize(PCIDevice *d, Error **errp)
memory_region_set_enabled(&mch->tseg_window, false);
memory_region_add_subregion(&mch->smram, mch->below_4g_mem_size,
&mch->tseg_window);
+
+ memory_region_init_io(&mch->smbase_blackhole, OBJECT(mch), &blackhole_ops,
+ NULL, "smbase-blackhole",
+ MCH_HOST_BRIDGE_SMBASE_SIZE);
+ memory_region_set_enabled(&mch->smbase_blackhole, false);
+ memory_region_add_subregion_overlap(mch->system_memory,
+ MCH_HOST_BRIDGE_SMBASE_ADDR,
+ &mch->smbase_blackhole, 1);
+
+ memory_region_init_alias(&mch->smbase_window, OBJECT(mch),
+ "smbase-window", mch->ram_memory,
+ MCH_HOST_BRIDGE_SMBASE_ADDR,
+ MCH_HOST_BRIDGE_SMBASE_SIZE);
+ memory_region_set_enabled(&mch->smbase_window, false);
+ memory_region_add_subregion(&mch->smram, MCH_HOST_BRIDGE_SMBASE_ADDR,
+ &mch->smbase_window);
+
object_property_add_const_link(qdev_get_machine(), "smram",
OBJECT(&mch->smram), &error_abort);

@@ -601,6 +666,7 @@ uint64_t mch_mcfg_base(void)
static Property mch_props[] = {
DEFINE_PROP_UINT16("extended-tseg-mbytes", MCHPCIState, ext_tseg_mbytes,
16),
+ DEFINE_PROP_BOOL("smbase-smram", MCHPCIState, has_smram_at_smbase, true),
DEFINE_PROP_END_OF_LIST(),
};


Re: UEFI accessibility mandate

Andrew Fish <afish@...>
 

Rafael,

Please let us know what you find out. I probably don''t have the time to help implement this feature, but I happy to help work on the architecture and design for UEFI accessibility on the edk2 mailing lists, and I can also represent what ever we come up with at the UEFI Spec Work Group.

It would be hard to have a UEFI mandate for accessibility, given there is no guideline on how a User Interface (UI) works. If accessibility requires some from of hardware abstraction, like audio, then we could likely get that into the UEFI Spec. What might be possible is an EDK2 reference implementation of accessibility. Maybe we could use the reference implementation to write a UEFI white paper on design for accessibility? I there is an open source implementation, and an official design guide this would make it much easier for advocacy groups to lobby for this feature.

I've got some experience with accessibility as the macOS EFI OS Loader has a UI for the Full Disk Encryption password. If you press the power button quickly 3 times at the disk encryption password prompt accessibility is enabled and Voice Over gets turned on. You then get localized voice prompts when you move between UI elements. Since this is the OS loader all the resources are stored on the disk. You quickly run into a range of challenges since, audio is hard, abstracting audio is hard (what codec does firmware have to support), Audio files are not small and firmware is size constrained, the need to localize the audio responses causes even more size issues, the boot options are usually written by an OS installer so how would firmware know what to call them?

I listed a lot of reasons it is hard but as Kennedy stated in his "We choose to go to the Moon!" speech sometimes we chose to do things "not because they are easy, but because they are hard; because that goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept". If we have a design that means we can break the problem up into smaller parts, and maybe we can find people that have expertise in that part to build a chunk at a time. If we could implement the prototype in OVMF that would show how it works, but run on everyones machines, so that would be really helpful for demos and design review.

Thanks,

Andrew Fish

On Sep 13, 2019, at 4:16 AM, Rafael Machado <rafaelrodrigues.machado@gmail.com> wrote:

Hi Ethin
I will contact the community you mentioned to get feedback and align expectations.

Hope to hear comments and feedback of other member of discussion (here at EDK2, that is where the work will happen).

Thanks and Regards
Rafael

Em ter, 10 de set de 2019 às 20:50, Ethin Probst <harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com>> escreveu:
I've posted this to the largest forum of blind visually impaired
people I know of to date and have gotten some excellent feedback from
them. I also provided in that post a link to this discussion. However,
I can't possibly forward all their questions onto this group (and I
doubt I can answer them all). If you wish to join the discussion over
there, its over at https://forum.audiogames.net/post/461041/#p461041.
(If we can find a way to merge the two discussions, that would make
everything easier. If we can't, I'm fine with that.)

On 9/10/19, Rafael Machado <rafaelrodrigues.machado@gmail.com <mailto:rafaelrodrigues.machado@gmail.com>> wrote:
Hi Ethin and all

As Leif mentioned, my mastering thesys was the implementation of a
reference code so a driver for audio at pre-OS would be possible,
eliminating the first barrier for the creation of a screen reader at UEFI,
that is the lack of audio drivers.

My project was based on Intel High Definition Audio, so it is not a generic
code that would fit at all scenarios, but it a first step.
The idea was to proceed this work using the Google Summer of Code
initiative,and I will try my best to make this happen in future editions of
this initiative.
Unfortunately my thesys was written in portuguese, because there was almost
nothing related to BIOS in Portuguese, so the decision was to write in
Portuguese to help the community to attract brasilian developers. As soon
as I find some slot, I'll translate to english.
But you are not alone. Just in Brazil I discovered around 30 blind
developers that area waiting for accessibility at BIOS. Also had some
contacts from US and Ireland.
So it is a market to be explored. Everyone will win.
The companies, because they will attract more people and their families,
and the ones who need accessibility.
By the way, most of the population will need accessible solutions if they
live enough. This includes the uefi community members, that today do not
have special needs, but that may require it in future.
Accessibility is not a wast of money.
This ia what I would like to show to the UEFI community.
Hope to have the opportunity to present this in some future event.

Since this discussion was started, who knows what are the next steps?
My code is available at github if someone gets interested (ps.: The code
has failures. It was just a Prof of Concept)

https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Lets keep the discussion, because it is important.

Thanks and Regards
Rafael R. Machado


Em ter, 10 de set de 2019 17:00, Ethin Probst <harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com>>
escreveu:

Woops, I just realized while reading my message after I'd sent it that
I had linked to the wrong document. The priority system is outlined
over here (https://freebsoft.org/doc/speechd/ssip.html#Top). This
document discusses SsIP -- the Speech Synthesis Interface Protocol.

On 9/10/19, Ethin Probst <harlydavidsen@gmail.com <mailto:harlydavidsen@gmail.com>> wrote:
While conquering refreshable braille displays would be nice, I think
we should take it in stages. (Some blind people would want this in
along with the accessibility subsystems, however I do not believe that
would be the wisest approach.) Instead, I think we should do what you
suggest: implement protocols for communicating with audio devices
(both input and output) and add audio hooks for the HII. (We could
even make those audio hooks just general hooks/events instead of
partitioning them for a particular purpose.)
You mentioned that not all devices 'have a keyboard'. This is most
definitely true, but not all devices have audio outputs and/or inputs
either. Therefore, if we can't mandate it for *all* devices, perhaps
we could only mandate it (or at least somehow infer it as a
requirement) for devices that both have an interaction mechanism (any
form of interacting with the system directly via the HII and not, say,
over the wire, because that would be superfluous) and have
mechanisms/devices/drivers/... for audio output at minimum. (Audio
input is not necessarily something that is required unless the device
is controlled primarily by voice, but I think having voice controls is
a bit beyond the UEFI specification and is not something UEFI should
worry about. Voice control is not something you should have in the
preboot environment anyway, correct?)
So, I think the first stage we should work on should include:
- Implementing protocols for communicating with audio outputs (don't
worry about inputs right now)
- Implementing protocols/hooks/events that can be intercepted in the
HII by a foreground/background UEFI driver/application
- Implementing some kind of method to allow the accessibility
protocols to read what is drawn to the text when navigating around
The third item -- when implemented properly -- will prevent the
accessibility subsystem from reading everything on the screen every
time the "cursor" moves. Perhaps another good idea would be to
implement some kind of priority system for text-to-speech, like speech
dispatcher does
(
https://docs.microsoft.com/en-us/cortana/skills/speech-synthesis-markup-language
).

On 9/10/19, Leif Lindholm <leif.lindholm@linaro.org <mailto:leif.lindholm@linaro.org>> wrote:
Hi Ethin,

Apologies for the delay in responding - I personally don't tend to
read the RFC list (and I know that applies to some other developers
who have now subscribed).

On Sun, Aug 11, 2019 at 06:15:18PM -0700, Ethin Probst wrote:
Hello all,

I'm new here, and was recommended to the TianoCore project by
someone over at the UEFI forum. I've run across TianoCore before,
and like the project.
Before anyone gets worried by the subject line, no, this is not any
kind of legal thing. Its just something I believe needs to
happen. :)
Back in 2016-2017 I contacted the UEFI forum about two problems, one
of which was the format of the spec, which I figured out on my
own. The other problem was not so easily dealt with. Te other
problem relates to accessibility of UEFI-compliant systems and
platform firmware to persons with disabilities. As it currently
stands, such a thing is nonexistent. To be fair, I completely
understand the difficulty that such a thing would require, and I
would fully agree if we still used the PC-AT BIOS systems -- yes,
indeed, I would never suggest this kind of thing on such a system
given that there was no actual standard of any kind for
BIOSes. However, now that UEFI is here, we have such a possibility.
As it currently stands, people with low vision or blind people have
access to their computers in the general sense (I am blind
myself). We can do pretty much anything anyone else could do. We can
code, play games, all that. There are few things that we cannot
do. One of those things is managing our systems firmware in the
preboot environment.
As it stands now, I can only boot other OSes or disks via
memorization. While that worked on BIOS machines (I have, or had, an
old Toshiba laptop that was BIOS-based), it no longer works because
UEFI is mercurial. When I access the boot menu now, I play a game of
chance. If the cards are in my favor, the OS I want to boot boots,
and I can go on my way. But if the cards aren't in my favor, I end
up making something happen that was unintended, and, worst of all, I
have no idea what I did.
However, the boot menu is only one portion of a platform firmware
UI. What about the setup utility or other utilities offered by
computer manufacturers? What about diagnostic utilities,
bootloaders, etc? What do I do with those? Well, I only have one
option -- sited assistance. If I go into my computers setup utility,
I cannot trust myself and say to myself, "OK, I know what I'm
doing. All I need to do is change x and save and quit." No, I can't
do that, because memorizing such a complex interface is extremely
difficult, and its something I wouldn't expect anyone to do.
My proposal is simple, and I'm posting it here because I'd like
comments and feedback before it actually gets implemented (it will
take a lot of time, I'm sure): mandate, in the UEFI specification,
that accessibility features for persons with disabilities must be
implemented and documented, and, if such features are not
implemented, then that vendor is not compliant with the
specification. Place strict minimum requirements for what the
accessibility features should do and how they should work.
So, first of all, I need to point out that what you suggest would be
very difficult for the UEFI Forum to mandate. *But* I don't think this
needs to be a problem.

UEFI primarily exists as a means through which interoperability can be
guaranteed. This means it does not tend to so much mandate what is
actually implemented in a platform, as it defines how it behaves if it
is. This is not 100% true, but suffice to say that mandating the
functionality you are asking for *without* having a reference
implementation available would likely simply be ignored.

If you want to turn thumbscrews, get Microsoft to add the requirement
to their windows logo requirements :)

However, as much as it is difficult to force people to develop this
support, if a functional reference implementation was published under
a permissive open source license, it would take a pretty spectacularly
incompetent product owner not to incorporate it in their products.

Now, I'm sure someone out there will ask me how this can be
done. Well, that's why I've joined the group -- though as I
familiarize myself with EDK2 development and all that I may actually
be able to participate as more than just an accessibility expert, of
sorts.
I have added Rafael Machado on cc. He did a PhD on firmware
accessibility, including a prototype implementing an audio driver for
EDK2. The resulting thesis can be found at
https://arxiv.org/abs/1712.03186 .

As a side note, I have been blind all my life. I was born with
retinopathy of prematurity (ROP), which resulted because I was born
at 26 weeks. My retina was detached, and, though the doctors
attempted to fix it, it would not remain attached, and there is no
chance of it getting fixed now. I would neither want nor care for
such a cure, however. I have lived my entire life blind, and while
the thought of gaining site back is appealing, I am unwilling to go
through the years and years of rewiring and reconditioning of my
brain that would be required for me to survive with site. To me, it
is simply not worth the cost.
But back to the discussion at hand: I would be happy to discuss how
the accessibility features would work and what would be
required. Even standardizing, through the specification, a key
combination to toggle the accessibility features would be nice, as
that would alleviate the major problem of a blind person buying a
new computer and not knowing how to enable the accessibility
features. The overarching goal would be to make the preboot
environment (including applications run within it) accessible and
usable by blind and visually impaired people as a boot service
only.
Yes, I agree, operating systems are already doing their things.

Again, mandating such a key combination is not necessarily something
that the UEFI specification can do. Some devices won't necessarily
have a keyboard.

Some things the UEFI specification could do is:
- Add protocols for Audio output (and possibly input) and talking to
audio codecs.
- Add audio hooks for the Human Interaction Interface subsystem (HII).

EDK2 could then start adding device drivers producing these protocols,
and enable the support for some reference platforms.

Would we need any specific support for things like refreshable braille
displays?

It would be superfluous to make this a runtime service, as all
major OSes already have accessibility features. Plus, managing such
a thing would be impossible to do.
Agreed.

This email has gotten quite long, so I will suspend the discussion
of functionality and how I would like it to work for a future email
once everyone has gotten on board.
Best Regards,

Leif


Thank you for your time and consideration.



--
Signed,
Ethin D. Probst

--
Signed,
Ethin D. Probst



--
Signed,
Ethin D. Probst


Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

rebecca@...
 

On 2019-09-17 15:08, Sean via Groups.Io wrote:
That process is to run the CI process locally.

If you want to run on the Azure servers the idea is you push to a branch that has a CI policy or create a PR to a branch that has this build as a "gate".

So for example if you
1. fork https://github.com/spbrogan/edk2-staging
2. Checkout the edk2-stuart-ci-latest branch from your fork.
3. Make a change and commit it.
4. Push to your fork.
5. Go to your fork on the github website or the PR page of my fork you can create a Pull Request from your branch to mine.
6. Create it. Once you do that it will spawn builds on the server. If I had branch policies that required those builds to pass the PR could not be completed/merged without passing.

Does that help.
Thanks, that does help.


--
Rebecca Cran


Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

Sean
 

Rebecca,

That process is to run the CI process locally.

If you want to run on the Azure servers the idea is you push to a branch that has a CI policy or create a PR to a branch that has this build as a "gate".

So for example if you
1. fork https://github.com/spbrogan/edk2-staging
2. Checkout the edk2-stuart-ci-latest branch from your fork.
3. Make a change and commit it.
4. Push to your fork.
5. Go to your fork on the github website or the PR page of my fork you can create a Pull Request from your branch to mine.
6. Create it. Once you do that it will spawn builds on the server. If I had branch policies that required those builds to pass the PR could not be completed/merged without passing.

Does that help.

Thanks
Sean


Re: [edk2-devel] [PATCH] q35: lpc: allow to lock down 128K RAM at default SMBASE address

Igor Mammedov <imammedo@...>
 

On Wed, 11 Sep 2019 19:30:46 +0200
"Laszlo Ersek" <lersek@redhat.com> wrote:

On 09/10/19 17:58, Igor Mammedov wrote:
On Mon, 9 Sep 2019 21:15:44 +0200
Laszlo Ersek <lersek@redhat.com> wrote:
[...]

It looks like fwcfg smi feature negotiation isn't reusable in this case.
I'd prefer not to add another device just for another SMI feature
negotiation/activation.
I thought it could be a register on the new CPU hotplug controller that
we're going to need anyway (if I understand correctly, at least).
If we don't have to 'park' hotplugged CPUs, then I don't see a need for
an extra controller.


But:

How about stealing reserved register from pci-host similar to your
extended TSEG commit (2f295167 q35/mch: implement extended TSEG sizes)?
(Looking into spec can (ab)use 0x58 or 0x59 register)
Yes, that should work.

In fact, I had considered 0x58 / 0x59 when looking for unused registers
for extended TSEG configuration:

http://mid.mail-archive.com/d8802612-0b11-776f-b209-53bbdaf67515@redhat.com

So I'm OK with this, thank you.
Thanks for the tip!
... patches with a stolen register are on the way to mail-list.


[PATCH 2/2] tests: q35: MCH: add default SMBASE SMRAM lock test

Igor Mammedov <imammedo@...>
 

test lockable SMRAM at default SMBASE feature introduced by
commit "q35: implement 128K SMRAM at default SMBASE address"

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
tests/q35-test.c | 105 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 105 insertions(+)

diff --git a/tests/q35-test.c b/tests/q35-test.c
index a68183d513..dd02660303 100644
--- a/tests/q35-test.c
+++ b/tests/q35-test.c
@@ -186,6 +186,109 @@ static void test_tseg_size(const void *data)
qtest_quit(qts);
}

+#define SMBASE 0x30000
+#define SMRAM_TEST_PATTERN 0x32
+#define SMRAM_TEST_RESET_PATTERN 0x23
+
+static void test_smram_smbase_lock(void)
+{
+ QPCIBus *pcibus;
+ QPCIDevice *pcidev;
+ QDict *response;
+ QTestState *qts;
+ int i;
+
+ qts = qtest_init("-M q35");
+
+ pcibus = qpci_new_pc(qts, NULL);
+ g_assert(pcibus != NULL);
+
+ pcidev = qpci_device_find(pcibus, 0);
+ g_assert(pcidev != NULL);
+
+ /* check that SMRAM is not enabled by default */
+ g_assert(qpci_config_readb(pcidev, MCH_HOST_BRIDGE_F_SMBASE) == 0);
+ qtest_writeb(qts, SMBASE, SMRAM_TEST_PATTERN);
+ g_assert_cmpint(qtest_readb(qts, SMBASE), ==, SMRAM_TEST_PATTERN);
+
+ /* check that writinng junk to 0x9c before before negotiating is ignorred */
+ for (i = 0; i < 0xff; i++) {
+ qpci_config_writeb(pcidev, MCH_HOST_BRIDGE_F_SMBASE, i);
+ g_assert(qpci_config_readb(pcidev, MCH_HOST_BRIDGE_F_SMBASE) == 0);
+ }
+
+ /* enable SMRAM at SMBASE */
+ qpci_config_writeb(pcidev, MCH_HOST_BRIDGE_F_SMBASE, 0xff);
+ g_assert(qpci_config_readb(pcidev, MCH_HOST_BRIDGE_F_SMBASE) == 0x01);
+ /* lock SMRAM at SMBASE */
+ qpci_config_writeb(pcidev, MCH_HOST_BRIDGE_F_SMBASE, 0x02);
+ g_assert(qpci_config_readb(pcidev, MCH_HOST_BRIDGE_F_SMBASE) == 0x02);
+
+ /* check that SMRAM at SMBASE is locked and can't be unlocked */
+ g_assert_cmpint(qtest_readb(qts, SMBASE), ==, 0xff);
+ for (i = 0; i <= 0xff; i++) {
+ /* make sure register is immutable */
+ qpci_config_writeb(pcidev, MCH_HOST_BRIDGE_F_SMBASE, i);
+ g_assert(qpci_config_readb(pcidev, MCH_HOST_BRIDGE_F_SMBASE) == 0x02);
+
+ /* RAM access should go inot black hole */
+ qtest_writeb(qts, SMBASE, SMRAM_TEST_PATTERN);
+ g_assert_cmpint(qtest_readb(qts, SMBASE), ==, 0xff);
+ }
+
+ /* reset */
+ response = qtest_qmp(qts, "{'execute': 'system_reset', 'arguments': {} }");
+ g_assert(response);
+ g_assert(!qdict_haskey(response, "error"));
+ qobject_unref(response);
+
+ /* check RAM at SMBASE is available after reset */
+ g_assert_cmpint(qtest_readb(qts, SMBASE), ==, SMRAM_TEST_PATTERN);
+ g_assert(qpci_config_readb(pcidev, MCH_HOST_BRIDGE_F_SMBASE) == 0);
+ qtest_writeb(qts, SMBASE, SMRAM_TEST_RESET_PATTERN);
+ g_assert_cmpint(qtest_readb(qts, SMBASE), ==, SMRAM_TEST_RESET_PATTERN);
+
+ g_free(pcidev);
+ qpci_free_pc(pcibus);
+
+ qtest_quit(qts);
+}
+
+static void test_without_smram_base(void)
+{
+ QPCIBus *pcibus;
+ QPCIDevice *pcidev;
+ QTestState *qts;
+ int i;
+
+ qts = qtest_init("-M pc-q35-4.1");
+
+ pcibus = qpci_new_pc(qts, NULL);
+ g_assert(pcibus != NULL);
+
+ pcidev = qpci_device_find(pcibus, 0);
+ g_assert(pcidev != NULL);
+
+ /* check that RAM accessible */
+ qtest_writeb(qts, SMBASE, SMRAM_TEST_PATTERN);
+ g_assert_cmpint(qtest_readb(qts, SMBASE), ==, SMRAM_TEST_PATTERN);
+
+ /* check that writing to 0x9c succeeds */
+ for (i = 0; i <= 0xff; i++) {
+ qpci_config_writeb(pcidev, MCH_HOST_BRIDGE_F_SMBASE, i);
+ g_assert(qpci_config_readb(pcidev, MCH_HOST_BRIDGE_F_SMBASE) == i);
+ }
+
+ /* check that RAM is still accessible */
+ qtest_writeb(qts, SMBASE, SMRAM_TEST_PATTERN + 1);
+ g_assert_cmpint(qtest_readb(qts, SMBASE), ==, (SMRAM_TEST_PATTERN + 1));
+
+ g_free(pcidev);
+ qpci_free_pc(pcibus);
+
+ qtest_quit(qts);
+}
+
int main(int argc, char **argv)
{
g_test_init(&argc, &argv, NULL);
@@ -197,5 +300,7 @@ int main(int argc, char **argv)
qtest_add_data_func("/q35/tseg-size/8mb", &tseg_8mb, test_tseg_size);
qtest_add_data_func("/q35/tseg-size/ext/16mb", &tseg_ext_16mb,
test_tseg_size);
+ qtest_add_func("/q35/smram/smbase_lock", test_smram_smbase_lock);
+ qtest_add_func("/q35/smram/legacy_smbase", test_without_smram_base);
return g_test_run();
}
--
2.18.1


[PATCH 1/2] q35: implement 128K SMRAM at default SMBASE address

Igor Mammedov <imammedo@...>
 

Use commit (2f295167e0 q35/mch: implement extended TSEG sizes) for
inspiration and (ab)use reserved register in config space at 0x9c
offset [*] to extend q35 pci-host with ability to use 128K at
0x30000 as SMRAM and hide it (like TSEG) from non-SMM context.

Usage:
1: write 0xff in the register
2: if the feature is supported, follow up read from the register
should return 0x01. At this point RAM at 0x30000 is still
available for SMI handler configuration from non-SMM context
3: writing 0x02 in the register, locks SMBASE area, making its contents
available only from SMM context. In non-SMM context, reads return
0xff and writes are ignored. Further writes into the register are
ignored until the system reset.

*) https://www.mail-archive.com/qemu-devel@nongnu.org/msg455991.html

Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
include/hw/pci-host/q35.h | 10 +++++
hw/i386/pc.c | 4 +-
hw/pci-host/q35.c | 80 +++++++++++++++++++++++++++++++++++----
3 files changed, 86 insertions(+), 8 deletions(-)

diff --git a/include/hw/pci-host/q35.h b/include/hw/pci-host/q35.h
index b3bcf2e632..976fbae599 100644
--- a/include/hw/pci-host/q35.h
+++ b/include/hw/pci-host/q35.h
@@ -32,6 +32,7 @@
#include "hw/acpi/ich9.h"
#include "hw/pci-host/pam.h"
#include "hw/i386/intel_iommu.h"
+#include "qemu/units.h"

#define TYPE_Q35_HOST_DEVICE "q35-pcihost"
#define Q35_HOST_DEVICE(obj) \
@@ -54,6 +55,8 @@ typedef struct MCHPCIState {
MemoryRegion smram_region, open_high_smram;
MemoryRegion smram, low_smram, high_smram;
MemoryRegion tseg_blackhole, tseg_window;
+ MemoryRegion smbase_blackhole, smbase_window;
+ bool has_smram_at_smbase;
Range pci_hole;
uint64_t below_4g_mem_size;
uint64_t above_4g_mem_size;
@@ -97,6 +100,13 @@ typedef struct Q35PCIHost {
#define MCH_HOST_BRIDGE_EXT_TSEG_MBYTES_QUERY 0xffff
#define MCH_HOST_BRIDGE_EXT_TSEG_MBYTES_MAX 0xfff

+#define MCH_HOST_BRIDGE_SMBASE_SIZE (128 * KiB)
+#define MCH_HOST_BRIDGE_SMBASE_ADDR 0x30000
+#define MCH_HOST_BRIDGE_F_SMBASE 0x9c
+#define MCH_HOST_BRIDGE_F_SMBASE_QUERY 0xff
+#define MCH_HOST_BRIDGE_F_SMBASE_IN_RAM 0x01
+#define MCH_HOST_BRIDGE_F_SMBASE_LCK 0x02
+
#define MCH_HOST_BRIDGE_PCIEXBAR 0x60 /* 64bit register */
#define MCH_HOST_BRIDGE_PCIEXBAR_SIZE 8 /* 64bit register */
#define MCH_HOST_BRIDGE_PCIEXBAR_DEFAULT 0xb0000000
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index bad866fe44..288d28358a 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -119,7 +119,9 @@ struct hpet_fw_config hpet_cfg = {.count = UINT8_MAX};
/* Physical Address of PVH entry point read from kernel ELF NOTE */
static size_t pvh_start_addr;

-GlobalProperty pc_compat_4_1[] = {};
+GlobalProperty pc_compat_4_1[] = {
+ { "mch", "smbase-smram", "off" },
+};
const size_t pc_compat_4_1_len = G_N_ELEMENTS(pc_compat_4_1);

GlobalProperty pc_compat_4_0[] = {};
diff --git a/hw/pci-host/q35.c b/hw/pci-host/q35.c
index 158d270b9f..c1bd9f78ae 100644
--- a/hw/pci-host/q35.c
+++ b/hw/pci-host/q35.c
@@ -275,20 +275,20 @@ static const TypeInfo q35_host_info = {
* MCH D0:F0
*/

-static uint64_t tseg_blackhole_read(void *ptr, hwaddr reg, unsigned size)
+static uint64_t blackhole_read(void *ptr, hwaddr reg, unsigned size)
{
return 0xffffffff;
}

-static void tseg_blackhole_write(void *opaque, hwaddr addr, uint64_t val,
- unsigned width)
+static void blackhole_write(void *opaque, hwaddr addr, uint64_t val,
+ unsigned width)
{
/* nothing */
}

-static const MemoryRegionOps tseg_blackhole_ops = {
- .read = tseg_blackhole_read,
- .write = tseg_blackhole_write,
+static const MemoryRegionOps blackhole_ops = {
+ .read = blackhole_read,
+ .write = blackhole_write,
.endianness = DEVICE_NATIVE_ENDIAN,
.valid.min_access_size = 1,
.valid.max_access_size = 4,
@@ -430,6 +430,46 @@ static void mch_update_ext_tseg_mbytes(MCHPCIState *mch)
}
}

+static void mch_update_smbase_smram(MCHPCIState *mch)
+{
+ PCIDevice *pd = PCI_DEVICE(mch);
+ uint8_t *reg = pd->config + MCH_HOST_BRIDGE_F_SMBASE;
+ bool lck;
+
+ if (!mch->has_smram_at_smbase) {
+ return;
+ }
+
+ if (*reg == MCH_HOST_BRIDGE_F_SMBASE_QUERY) {
+ pd->wmask[MCH_HOST_BRIDGE_F_SMBASE] =
+ MCH_HOST_BRIDGE_F_SMBASE_LCK;
+ *reg = MCH_HOST_BRIDGE_F_SMBASE_IN_RAM;
+ return;
+ }
+
+ /*
+ * default/reset state, discard written value
+ * which will disable SMRAM balackhole at SMBASE
+ */
+ if (pd->wmask[MCH_HOST_BRIDGE_F_SMBASE] == 0xff) {
+ *reg = 0x00;
+ }
+
+ memory_region_transaction_begin();
+ if (*reg & MCH_HOST_BRIDGE_F_SMBASE_LCK) {
+ /* disable all writes */
+ pd->wmask[MCH_HOST_BRIDGE_F_SMBASE] &=
+ ~MCH_HOST_BRIDGE_F_SMBASE_LCK;
+ *reg = MCH_HOST_BRIDGE_F_SMBASE_LCK;
+ lck = true;
+ } else {
+ lck = false;
+ }
+ memory_region_set_enabled(&mch->smbase_blackhole, lck);
+ memory_region_set_enabled(&mch->smbase_window, lck);
+ memory_region_transaction_commit();
+}
+
static void mch_write_config(PCIDevice *d,
uint32_t address, uint32_t val, int len)
{
@@ -456,6 +496,10 @@ static void mch_write_config(PCIDevice *d,
MCH_HOST_BRIDGE_EXT_TSEG_MBYTES_SIZE)) {
mch_update_ext_tseg_mbytes(mch);
}
+
+ if (ranges_overlap(address, len, MCH_HOST_BRIDGE_F_SMBASE, 1)) {
+ mch_update_smbase_smram(mch);
+ }
}

static void mch_update(MCHPCIState *mch)
@@ -464,6 +508,7 @@ static void mch_update(MCHPCIState *mch)
mch_update_pam(mch);
mch_update_smram(mch);
mch_update_ext_tseg_mbytes(mch);
+ mch_update_smbase_smram(mch);

/*
* pci hole goes from end-of-low-ram to io-apic.
@@ -514,6 +559,9 @@ static void mch_reset(DeviceState *qdev)
MCH_HOST_BRIDGE_EXT_TSEG_MBYTES_QUERY);
}

+ d->config[MCH_HOST_BRIDGE_F_SMBASE] = 0;
+ d->wmask[MCH_HOST_BRIDGE_F_SMBASE] = 0xff;
+
mch_update(mch);
}

@@ -563,7 +611,7 @@ static void mch_realize(PCIDevice *d, Error **errp)
memory_region_add_subregion(&mch->smram, 0xfeda0000, &mch->high_smram);

memory_region_init_io(&mch->tseg_blackhole, OBJECT(mch),
- &tseg_blackhole_ops, NULL,
+ &blackhole_ops, NULL,
"tseg-blackhole", 0);
memory_region_set_enabled(&mch->tseg_blackhole, false);
memory_region_add_subregion_overlap(mch->system_memory,
@@ -575,6 +623,23 @@ static void mch_realize(PCIDevice *d, Error **errp)
memory_region_set_enabled(&mch->tseg_window, false);
memory_region_add_subregion(&mch->smram, mch->below_4g_mem_size,
&mch->tseg_window);
+
+ memory_region_init_io(&mch->smbase_blackhole, OBJECT(mch), &blackhole_ops,
+ NULL, "smbase-blackhole",
+ MCH_HOST_BRIDGE_SMBASE_SIZE);
+ memory_region_set_enabled(&mch->smbase_blackhole, false);
+ memory_region_add_subregion_overlap(mch->system_memory,
+ MCH_HOST_BRIDGE_SMBASE_ADDR,
+ &mch->smbase_blackhole, 1);
+
+ memory_region_init_alias(&mch->smbase_window, OBJECT(mch),
+ "smbase-window", mch->ram_memory,
+ MCH_HOST_BRIDGE_SMBASE_ADDR,
+ MCH_HOST_BRIDGE_SMBASE_SIZE);
+ memory_region_set_enabled(&mch->smbase_window, false);
+ memory_region_add_subregion(&mch->smram, MCH_HOST_BRIDGE_SMBASE_ADDR,
+ &mch->smbase_window);
+
object_property_add_const_link(qdev_get_machine(), "smram",
OBJECT(&mch->smram), &error_abort);

@@ -601,6 +666,7 @@ uint64_t mch_mcfg_base(void)
static Property mch_props[] = {
DEFINE_PROP_UINT16("extended-tseg-mbytes", MCHPCIState, ext_tseg_mbytes,
16),
+ DEFINE_PROP_BOOL("smbase-smram", MCHPCIState, has_smram_at_smbase, true),
DEFINE_PROP_END_OF_LIST(),
};

--
2.18.1


[PATCH 0/2] q35: mch: allow to lock down 128K RAM at default SMBASE address

Igor Mammedov <imammedo@...>
 

Try #2 using PCI config space of MCH to negotiate/lock SMRAM
at 0x30000.

CC: yingwen.chen@intel.com
CC: devel@edk2.groups.io
CC: phillip.goerl@oracle.com
CC: alex.williamson@redhat.com
CC: jiewen.yao@intel.com
CC: jun.nakajima@intel.com
CC: michael.d.kinney@intel.com
CC: pbonzini@redhat.com
CC: boris.ostrovsky@oracle.com
CC: rfc@edk2.groups.io
CC: joao.m.martins@oracle.com
CC: lersek@redhat.com


Igor Mammedov (2):
q35: implement 128K SMRAM at default SMBASE address
tests: q35: MCH: add default SMBASE SMRAM lock test

include/hw/pci-host/q35.h | 10 ++++
hw/i386/pc.c | 4 +-
hw/pci-host/q35.c | 80 ++++++++++++++++++++++++++---
tests/q35-test.c | 105 ++++++++++++++++++++++++++++++++++++++
4 files changed, 191 insertions(+), 8 deletions(-)

--
2.18.1


Re: [edk2-devel] [RFC] EDK II Continuous Integration Phase 1

rebecca@...
 

On 8/29/19 2:22 PM, Michael D Kinney wrote:
The following link is a link to an EDK II WIKI page that
contains a summary of the work to date. Please provide
feedback in the EDK II mailing lists. The WIKI pages will
be updated with input from the entire EDK II community.

https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Continuous-Integration

In that wiki page, it says:

"To work with this branch and run tests immediately, all you need to do is:

|pip install --upgrade -r requirements.txt stuart_setup -c .\CISettings.py stuart_update -c .\CISettings.py stuart_ci_build -c .\CISettings.py --Tool_Chain VS2017" |


I tried doing that in my KUbuntu system, but it was wanting to run NuGet. Once I installed the needed libmono packages via apt, it was wanting to install mu-nasm and then mu-Basetools. I thought it would be pushing changes to Azure to test, not trying to build it locally?


--

Rebecca Cran


Re: UEFI accessibility mandate

Ethin Probst
 

You need to post about yourself in this topic to actually get
unrestricted (its a way of stopping bots):
https://forum.audiogames.net/post/461466/#p461466
Sorry I didn't mention that, I forgot about that. :(

On 9/13/19, Rafael Machado <rafaelrodrigues.machado@gmail.com> wrote:
Hi Ethin

I just get registered at the forum, but have no permission to post there.
The message "Sorry! no permission to post a reply" is presented to my
user.

Thanks
Rafael

Em sex, 13 de set de 2019 às 08:16, Rafael Machado <
rafaelrodrigues.machado@gmail.com> escreveu:

Hi Ethin
I will contact the community you mentioned to get feedback and align
expectations.

Hope to hear comments and feedback of other member of discussion (here at
EDK2, that is where the work will happen).

Thanks and Regards
Rafael

Em ter, 10 de set de 2019 às 20:50, Ethin Probst
<harlydavidsen@gmail.com>
escreveu:

I've posted this to the largest forum of blind visually impaired
people I know of to date and have gotten some excellent feedback from
them. I also provided in that post a link to this discussion. However,
I can't possibly forward all their questions onto this group (and I
doubt I can answer them all). If you wish to join the discussion over
there, its over at https://forum.audiogames.net/post/461041/#p461041.
(If we can find a way to merge the two discussions, that would make
everything easier. If we can't, I'm fine with that.)

On 9/10/19, Rafael Machado <rafaelrodrigues.machado@gmail.com> wrote:
Hi Ethin and all

As Leif mentioned, my mastering thesys was the implementation of a
reference code so a driver for audio at pre-OS would be possible,
eliminating the first barrier for the creation of a screen reader at
UEFI,
that is the lack of audio drivers.

My project was based on Intel High Definition Audio, so it is not a
generic
code that would fit at all scenarios, but it a first step.
The idea was to proceed this work using the Google Summer of Code
initiative,and I will try my best to make this happen in future
editions of
this initiative.
Unfortunately my thesys was written in portuguese, because there was
almost
nothing related to BIOS in Portuguese, so the decision was to write in
Portuguese to help the community to attract brasilian developers. As
soon
as I find some slot, I'll translate to english.
But you are not alone. Just in Brazil I discovered around 30 blind
developers that area waiting for accessibility at BIOS. Also had some
contacts from US and Ireland.
So it is a market to be explored. Everyone will win.
The companies, because they will attract more people and their
families,
and the ones who need accessibility.
By the way, most of the population will need accessible solutions if
they
live enough. This includes the uefi community members, that today do
not
have special needs, but that may require it in future.
Accessibility is not a wast of money.
This ia what I would like to show to the UEFI community.
Hope to have the opportunity to present this in some future event.

Since this discussion was started, who knows what are the next steps?
My code is available at github if someone gets interested (ps.: The
code
has failures. It was just a Prof of Concept)

https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Lets keep the discussion, because it is important.

Thanks and Regards
Rafael R. Machado


Em ter, 10 de set de 2019 17:00, Ethin Probst
<harlydavidsen@gmail.com>
escreveu:

Woops, I just realized while reading my message after I'd sent it
that
I had linked to the wrong document. The priority system is outlined
over here (https://freebsoft.org/doc/speechd/ssip.html#Top). This
document discusses SsIP -- the Speech Synthesis Interface Protocol.

On 9/10/19, Ethin Probst <harlydavidsen@gmail.com> wrote:
While conquering refreshable braille displays would be nice, I
think
we should take it in stages. (Some blind people would want this in
along with the accessibility subsystems, however I do not believe
that
would be the wisest approach.) Instead, I think we should do what
you
suggest: implement protocols for communicating with audio devices
(both input and output) and add audio hooks for the HII. (We could
even make those audio hooks just general hooks/events instead of
partitioning them for a particular purpose.)
You mentioned that not all devices 'have a keyboard'. This is most
definitely true, but not all devices have audio outputs and/or
inputs
either. Therefore, if we can't mandate it for *all* devices,
perhaps
we could only mandate it (or at least somehow infer it as a
requirement) for devices that both have an interaction mechanism
(any
form of interacting with the system directly via the HII and not,
say,
over the wire, because that would be superfluous) and have
mechanisms/devices/drivers/... for audio output at minimum. (Audio
input is not necessarily something that is required unless the
device
is controlled primarily by voice, but I think having voice controls
is
a bit beyond the UEFI specification and is not something UEFI
should
worry about. Voice control is not something you should have in the
preboot environment anyway, correct?)
So, I think the first stage we should work on should include:
- Implementing protocols for communicating with audio outputs
(don't
worry about inputs right now)
- Implementing protocols/hooks/events that can be intercepted in
the
HII by a foreground/background UEFI driver/application
- Implementing some kind of method to allow the accessibility
protocols to read what is drawn to the text when navigating around
The third item -- when implemented properly -- will prevent the
accessibility subsystem from reading everything on the screen every
time the "cursor" moves. Perhaps another good idea would be to
implement some kind of priority system for text-to-speech, like
speech
dispatcher does
(
https://docs.microsoft.com/en-us/cortana/skills/speech-synthesis-markup-language
).

On 9/10/19, Leif Lindholm <leif.lindholm@linaro.org> wrote:
Hi Ethin,

Apologies for the delay in responding - I personally don't tend to
read the RFC list (and I know that applies to some other
developers
who have now subscribed).

On Sun, Aug 11, 2019 at 06:15:18PM -0700, Ethin Probst wrote:
Hello all,

I'm new here, and was recommended to the TianoCore project by
someone over at the UEFI forum. I've run across TianoCore before,
and like the project.
Before anyone gets worried by the subject line, no, this is not
any
kind of legal thing. Its just something I believe needs to
happen. :)
Back in 2016-2017 I contacted the UEFI forum about two problems,
one
of which was the format of the spec, which I figured out on my
own. The other problem was not so easily dealt with. Te other
problem relates to accessibility of UEFI-compliant systems and
platform firmware to persons with disabilities. As it currently
stands, such a thing is nonexistent. To be fair, I completely
understand the difficulty that such a thing would require, and I
would fully agree if we still used the PC-AT BIOS systems -- yes,
indeed, I would never suggest this kind of thing on such a system
given that there was no actual standard of any kind for
BIOSes. However, now that UEFI is here, we have such a
possibility.
As it currently stands, people with low vision or blind people
have
access to their computers in the general sense (I am blind
myself). We can do pretty much anything anyone else could do. We
can
code, play games, all that. There are few things that we cannot
do. One of those things is managing our systems firmware in the
preboot environment.
As it stands now, I can only boot other OSes or disks via
memorization. While that worked on BIOS machines (I have, or had,
an
old Toshiba laptop that was BIOS-based), it no longer works
because
UEFI is mercurial. When I access the boot menu now, I play a game
of
chance. If the cards are in my favor, the OS I want to boot
boots,
and I can go on my way. But if the cards aren't in my favor, I
end
up making something happen that was unintended, and, worst of
all,
I
have no idea what I did.
However, the boot menu is only one portion of a platform firmware
UI. What about the setup utility or other utilities offered by
computer manufacturers? What about diagnostic utilities,
bootloaders, etc? What do I do with those? Well, I only have one
option -- sited assistance. If I go into my computers setup
utility,
I cannot trust myself and say to myself, "OK, I know what I'm
doing. All I need to do is change x and save and quit." No, I
can't
do that, because memorizing such a complex interface is extremely
difficult, and its something I wouldn't expect anyone to do.
My proposal is simple, and I'm posting it here because I'd like
comments and feedback before it actually gets implemented (it
will
take a lot of time, I'm sure): mandate, in the UEFI
specification,
that accessibility features for persons with disabilities must be
implemented and documented, and, if such features are not
implemented, then that vendor is not compliant with the
specification. Place strict minimum requirements for what the
accessibility features should do and how they should work.
So, first of all, I need to point out that what you suggest would
be
very difficult for the UEFI Forum to mandate. *But* I don't think
this
needs to be a problem.

UEFI primarily exists as a means through which interoperability
can
be
guaranteed. This means it does not tend to so much mandate what is
actually implemented in a platform, as it defines how it behaves
if
it
is. This is not 100% true, but suffice to say that mandating the
functionality you are asking for *without* having a reference
implementation available would likely simply be ignored.

If you want to turn thumbscrews, get Microsoft to add the
requirement
to their windows logo requirements :)

However, as much as it is difficult to force people to develop
this
support, if a functional reference implementation was published
under
a permissive open source license, it would take a pretty
spectacularly
incompetent product owner not to incorporate it in their products.

Now, I'm sure someone out there will ask me how this can be
done. Well, that's why I've joined the group -- though as I
familiarize myself with EDK2 development and all that I may
actually
be able to participate as more than just an accessibility expert,
of
sorts.
I have added Rafael Machado on cc. He did a PhD on firmware
accessibility, including a prototype implementing an audio driver
for
EDK2. The resulting thesis can be found at
https://arxiv.org/abs/1712.03186 .

As a side note, I have been blind all my life. I was born with
retinopathy of prematurity (ROP), which resulted because I was
born
at 26 weeks. My retina was detached, and, though the doctors
attempted to fix it, it would not remain attached, and there is
no
chance of it getting fixed now. I would neither want nor care for
such a cure, however. I have lived my entire life blind, and
while
the thought of gaining site back is appealing, I am unwilling to
go
through the years and years of rewiring and reconditioning of my
brain that would be required for me to survive with site. To me,
it
is simply not worth the cost.
But back to the discussion at hand: I would be happy to discuss
how
the accessibility features would work and what would be
required. Even standardizing, through the specification, a key
combination to toggle the accessibility features would be nice,
as
that would alleviate the major problem of a blind person buying a
new computer and not knowing how to enable the accessibility
features. The overarching goal would be to make the preboot
environment (including applications run within it) accessible and
usable by blind and visually impaired people as a boot service
only.
Yes, I agree, operating systems are already doing their things.

Again, mandating such a key combination is not necessarily
something
that the UEFI specification can do. Some devices won't necessarily
have a keyboard.

Some things the UEFI specification could do is:
- Add protocols for Audio output (and possibly input) and talking
to
audio codecs.
- Add audio hooks for the Human Interaction Interface subsystem
(HII).

EDK2 could then start adding device drivers producing these
protocols,
and enable the support for some reference platforms.

Would we need any specific support for things like refreshable
braille
displays?

It would be superfluous to make this a runtime service, as all
major OSes already have accessibility features. Plus, managing
such
a thing would be impossible to do.
Agreed.

This email has gotten quite long, so I will suspend the
discussion
of functionality and how I would like it to work for a future
email
once everyone has gotten on board.
Best Regards,

Leif


Thank you for your time and consideration.



--
Signed,
Ethin D. Probst

--
Signed,
Ethin D. Probst



--
Signed,
Ethin D. Probst
--
Signed,
Ethin D. Probst


Re: UEFI accessibility mandate

Rafael Machado <rafaelrodrigues.machado@...>
 

Hi Ethin

I just get registered at the forum, but have no permission to post there.
The message "Sorry! no permission to post a reply" is presented to my user.

Thanks
Rafael

Em sex, 13 de set de 2019 às 08:16, Rafael Machado <
rafaelrodrigues.machado@gmail.com> escreveu:

Hi Ethin
I will contact the community you mentioned to get feedback and align
expectations.

Hope to hear comments and feedback of other member of discussion (here at
EDK2, that is where the work will happen).

Thanks and Regards
Rafael

Em ter, 10 de set de 2019 às 20:50, Ethin Probst <harlydavidsen@gmail.com>
escreveu:

I've posted this to the largest forum of blind visually impaired
people I know of to date and have gotten some excellent feedback from
them. I also provided in that post a link to this discussion. However,
I can't possibly forward all their questions onto this group (and I
doubt I can answer them all). If you wish to join the discussion over
there, its over at https://forum.audiogames.net/post/461041/#p461041.
(If we can find a way to merge the two discussions, that would make
everything easier. If we can't, I'm fine with that.)

On 9/10/19, Rafael Machado <rafaelrodrigues.machado@gmail.com> wrote:
Hi Ethin and all

As Leif mentioned, my mastering thesys was the implementation of a
reference code so a driver for audio at pre-OS would be possible,
eliminating the first barrier for the creation of a screen reader at
UEFI,
that is the lack of audio drivers.

My project was based on Intel High Definition Audio, so it is not a
generic
code that would fit at all scenarios, but it a first step.
The idea was to proceed this work using the Google Summer of Code
initiative,and I will try my best to make this happen in future
editions of
this initiative.
Unfortunately my thesys was written in portuguese, because there was
almost
nothing related to BIOS in Portuguese, so the decision was to write in
Portuguese to help the community to attract brasilian developers. As
soon
as I find some slot, I'll translate to english.
But you are not alone. Just in Brazil I discovered around 30 blind
developers that area waiting for accessibility at BIOS. Also had some
contacts from US and Ireland.
So it is a market to be explored. Everyone will win.
The companies, because they will attract more people and their families,
and the ones who need accessibility.
By the way, most of the population will need accessible solutions if
they
live enough. This includes the uefi community members, that today do not
have special needs, but that may require it in future.
Accessibility is not a wast of money.
This ia what I would like to show to the UEFI community.
Hope to have the opportunity to present this in some future event.

Since this discussion was started, who knows what are the next steps?
My code is available at github if someone gets interested (ps.: The code
has failures. It was just a Prof of Concept)

https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Lets keep the discussion, because it is important.

Thanks and Regards
Rafael R. Machado


Em ter, 10 de set de 2019 17:00, Ethin Probst <harlydavidsen@gmail.com>
escreveu:

Woops, I just realized while reading my message after I'd sent it that
I had linked to the wrong document. The priority system is outlined
over here (https://freebsoft.org/doc/speechd/ssip.html#Top). This
document discusses SsIP -- the Speech Synthesis Interface Protocol.

On 9/10/19, Ethin Probst <harlydavidsen@gmail.com> wrote:
While conquering refreshable braille displays would be nice, I think
we should take it in stages. (Some blind people would want this in
along with the accessibility subsystems, however I do not believe
that
would be the wisest approach.) Instead, I think we should do what you
suggest: implement protocols for communicating with audio devices
(both input and output) and add audio hooks for the HII. (We could
even make those audio hooks just general hooks/events instead of
partitioning them for a particular purpose.)
You mentioned that not all devices 'have a keyboard'. This is most
definitely true, but not all devices have audio outputs and/or inputs
either. Therefore, if we can't mandate it for *all* devices, perhaps
we could only mandate it (or at least somehow infer it as a
requirement) for devices that both have an interaction mechanism (any
form of interacting with the system directly via the HII and not,
say,
over the wire, because that would be superfluous) and have
mechanisms/devices/drivers/... for audio output at minimum. (Audio
input is not necessarily something that is required unless the device
is controlled primarily by voice, but I think having voice controls
is
a bit beyond the UEFI specification and is not something UEFI should
worry about. Voice control is not something you should have in the
preboot environment anyway, correct?)
So, I think the first stage we should work on should include:
- Implementing protocols for communicating with audio outputs (don't
worry about inputs right now)
- Implementing protocols/hooks/events that can be intercepted in the
HII by a foreground/background UEFI driver/application
- Implementing some kind of method to allow the accessibility
protocols to read what is drawn to the text when navigating around
The third item -- when implemented properly -- will prevent the
accessibility subsystem from reading everything on the screen every
time the "cursor" moves. Perhaps another good idea would be to
implement some kind of priority system for text-to-speech, like
speech
dispatcher does
(
https://docs.microsoft.com/en-us/cortana/skills/speech-synthesis-markup-language
).

On 9/10/19, Leif Lindholm <leif.lindholm@linaro.org> wrote:
Hi Ethin,

Apologies for the delay in responding - I personally don't tend to
read the RFC list (and I know that applies to some other developers
who have now subscribed).

On Sun, Aug 11, 2019 at 06:15:18PM -0700, Ethin Probst wrote:
Hello all,

I'm new here, and was recommended to the TianoCore project by
someone over at the UEFI forum. I've run across TianoCore before,
and like the project.
Before anyone gets worried by the subject line, no, this is not any
kind of legal thing. Its just something I believe needs to
happen. :)
Back in 2016-2017 I contacted the UEFI forum about two problems,
one
of which was the format of the spec, which I figured out on my
own. The other problem was not so easily dealt with. Te other
problem relates to accessibility of UEFI-compliant systems and
platform firmware to persons with disabilities. As it currently
stands, such a thing is nonexistent. To be fair, I completely
understand the difficulty that such a thing would require, and I
would fully agree if we still used the PC-AT BIOS systems -- yes,
indeed, I would never suggest this kind of thing on such a system
given that there was no actual standard of any kind for
BIOSes. However, now that UEFI is here, we have such a possibility.
As it currently stands, people with low vision or blind people have
access to their computers in the general sense (I am blind
myself). We can do pretty much anything anyone else could do. We
can
code, play games, all that. There are few things that we cannot
do. One of those things is managing our systems firmware in the
preboot environment.
As it stands now, I can only boot other OSes or disks via
memorization. While that worked on BIOS machines (I have, or had,
an
old Toshiba laptop that was BIOS-based), it no longer works because
UEFI is mercurial. When I access the boot menu now, I play a game
of
chance. If the cards are in my favor, the OS I want to boot boots,
and I can go on my way. But if the cards aren't in my favor, I end
up making something happen that was unintended, and, worst of all,
I
have no idea what I did.
However, the boot menu is only one portion of a platform firmware
UI. What about the setup utility or other utilities offered by
computer manufacturers? What about diagnostic utilities,
bootloaders, etc? What do I do with those? Well, I only have one
option -- sited assistance. If I go into my computers setup
utility,
I cannot trust myself and say to myself, "OK, I know what I'm
doing. All I need to do is change x and save and quit." No, I can't
do that, because memorizing such a complex interface is extremely
difficult, and its something I wouldn't expect anyone to do.
My proposal is simple, and I'm posting it here because I'd like
comments and feedback before it actually gets implemented (it will
take a lot of time, I'm sure): mandate, in the UEFI specification,
that accessibility features for persons with disabilities must be
implemented and documented, and, if such features are not
implemented, then that vendor is not compliant with the
specification. Place strict minimum requirements for what the
accessibility features should do and how they should work.
So, first of all, I need to point out that what you suggest would be
very difficult for the UEFI Forum to mandate. *But* I don't think
this
needs to be a problem.

UEFI primarily exists as a means through which interoperability can
be
guaranteed. This means it does not tend to so much mandate what is
actually implemented in a platform, as it defines how it behaves if
it
is. This is not 100% true, but suffice to say that mandating the
functionality you are asking for *without* having a reference
implementation available would likely simply be ignored.

If you want to turn thumbscrews, get Microsoft to add the
requirement
to their windows logo requirements :)

However, as much as it is difficult to force people to develop this
support, if a functional reference implementation was published
under
a permissive open source license, it would take a pretty
spectacularly
incompetent product owner not to incorporate it in their products.

Now, I'm sure someone out there will ask me how this can be
done. Well, that's why I've joined the group -- though as I
familiarize myself with EDK2 development and all that I may
actually
be able to participate as more than just an accessibility expert,
of
sorts.
I have added Rafael Machado on cc. He did a PhD on firmware
accessibility, including a prototype implementing an audio driver
for
EDK2. The resulting thesis can be found at
https://arxiv.org/abs/1712.03186 .

As a side note, I have been blind all my life. I was born with
retinopathy of prematurity (ROP), which resulted because I was born
at 26 weeks. My retina was detached, and, though the doctors
attempted to fix it, it would not remain attached, and there is no
chance of it getting fixed now. I would neither want nor care for
such a cure, however. I have lived my entire life blind, and while
the thought of gaining site back is appealing, I am unwilling to go
through the years and years of rewiring and reconditioning of my
brain that would be required for me to survive with site. To me, it
is simply not worth the cost.
But back to the discussion at hand: I would be happy to discuss how
the accessibility features would work and what would be
required. Even standardizing, through the specification, a key
combination to toggle the accessibility features would be nice, as
that would alleviate the major problem of a blind person buying a
new computer and not knowing how to enable the accessibility
features. The overarching goal would be to make the preboot
environment (including applications run within it) accessible and
usable by blind and visually impaired people as a boot service
only.
Yes, I agree, operating systems are already doing their things.

Again, mandating such a key combination is not necessarily something
that the UEFI specification can do. Some devices won't necessarily
have a keyboard.

Some things the UEFI specification could do is:
- Add protocols for Audio output (and possibly input) and talking to
audio codecs.
- Add audio hooks for the Human Interaction Interface subsystem
(HII).

EDK2 could then start adding device drivers producing these
protocols,
and enable the support for some reference platforms.

Would we need any specific support for things like refreshable
braille
displays?

It would be superfluous to make this a runtime service, as all
major OSes already have accessibility features. Plus, managing such
a thing would be impossible to do.
Agreed.

This email has gotten quite long, so I will suspend the discussion
of functionality and how I would like it to work for a future email
once everyone has gotten on board.
Best Regards,

Leif


Thank you for your time and consideration.



--
Signed,
Ethin D. Probst

--
Signed,
Ethin D. Probst



--
Signed,
Ethin D. Probst


Re: UEFI accessibility mandate

Rafael Machado <rafaelrodrigues.machado@...>
 

Hi Ethin
I will contact the community you mentioned to get feedback and align
expectations.

Hope to hear comments and feedback of other member of discussion (here at
EDK2, that is where the work will happen).

Thanks and Regards
Rafael

Em ter, 10 de set de 2019 às 20:50, Ethin Probst <harlydavidsen@gmail.com>
escreveu:

I've posted this to the largest forum of blind visually impaired
people I know of to date and have gotten some excellent feedback from
them. I also provided in that post a link to this discussion. However,
I can't possibly forward all their questions onto this group (and I
doubt I can answer them all). If you wish to join the discussion over
there, its over at https://forum.audiogames.net/post/461041/#p461041.
(If we can find a way to merge the two discussions, that would make
everything easier. If we can't, I'm fine with that.)

On 9/10/19, Rafael Machado <rafaelrodrigues.machado@gmail.com> wrote:
Hi Ethin and all

As Leif mentioned, my mastering thesys was the implementation of a
reference code so a driver for audio at pre-OS would be possible,
eliminating the first barrier for the creation of a screen reader at
UEFI,
that is the lack of audio drivers.

My project was based on Intel High Definition Audio, so it is not a
generic
code that would fit at all scenarios, but it a first step.
The idea was to proceed this work using the Google Summer of Code
initiative,and I will try my best to make this happen in future editions
of
this initiative.
Unfortunately my thesys was written in portuguese, because there was
almost
nothing related to BIOS in Portuguese, so the decision was to write in
Portuguese to help the community to attract brasilian developers. As soon
as I find some slot, I'll translate to english.
But you are not alone. Just in Brazil I discovered around 30 blind
developers that area waiting for accessibility at BIOS. Also had some
contacts from US and Ireland.
So it is a market to be explored. Everyone will win.
The companies, because they will attract more people and their families,
and the ones who need accessibility.
By the way, most of the population will need accessible solutions if they
live enough. This includes the uefi community members, that today do not
have special needs, but that may require it in future.
Accessibility is not a wast of money.
This ia what I would like to show to the UEFI community.
Hope to have the opportunity to present this in some future event.

Since this discussion was started, who knows what are the next steps?
My code is available at github if someone gets interested (ps.: The code
has failures. It was just a Prof of Concept)

https://github.com/RafaelRMachado/Msc_UefiHda_PreOs_Accessibility

Lets keep the discussion, because it is important.

Thanks and Regards
Rafael R. Machado


Em ter, 10 de set de 2019 17:00, Ethin Probst <harlydavidsen@gmail.com>
escreveu:

Woops, I just realized while reading my message after I'd sent it that
I had linked to the wrong document. The priority system is outlined
over here (https://freebsoft.org/doc/speechd/ssip.html#Top). This
document discusses SsIP -- the Speech Synthesis Interface Protocol.

On 9/10/19, Ethin Probst <harlydavidsen@gmail.com> wrote:
While conquering refreshable braille displays would be nice, I think
we should take it in stages. (Some blind people would want this in
along with the accessibility subsystems, however I do not believe that
would be the wisest approach.) Instead, I think we should do what you
suggest: implement protocols for communicating with audio devices
(both input and output) and add audio hooks for the HII. (We could
even make those audio hooks just general hooks/events instead of
partitioning them for a particular purpose.)
You mentioned that not all devices 'have a keyboard'. This is most
definitely true, but not all devices have audio outputs and/or inputs
either. Therefore, if we can't mandate it for *all* devices, perhaps
we could only mandate it (or at least somehow infer it as a
requirement) for devices that both have an interaction mechanism (any
form of interacting with the system directly via the HII and not, say,
over the wire, because that would be superfluous) and have
mechanisms/devices/drivers/... for audio output at minimum. (Audio
input is not necessarily something that is required unless the device
is controlled primarily by voice, but I think having voice controls is
a bit beyond the UEFI specification and is not something UEFI should
worry about. Voice control is not something you should have in the
preboot environment anyway, correct?)
So, I think the first stage we should work on should include:
- Implementing protocols for communicating with audio outputs (don't
worry about inputs right now)
- Implementing protocols/hooks/events that can be intercepted in the
HII by a foreground/background UEFI driver/application
- Implementing some kind of method to allow the accessibility
protocols to read what is drawn to the text when navigating around
The third item -- when implemented properly -- will prevent the
accessibility subsystem from reading everything on the screen every
time the "cursor" moves. Perhaps another good idea would be to
implement some kind of priority system for text-to-speech, like speech
dispatcher does
(
https://docs.microsoft.com/en-us/cortana/skills/speech-synthesis-markup-language
).

On 9/10/19, Leif Lindholm <leif.lindholm@linaro.org> wrote:
Hi Ethin,

Apologies for the delay in responding - I personally don't tend to
read the RFC list (and I know that applies to some other developers
who have now subscribed).

On Sun, Aug 11, 2019 at 06:15:18PM -0700, Ethin Probst wrote:
Hello all,

I'm new here, and was recommended to the TianoCore project by
someone over at the UEFI forum. I've run across TianoCore before,
and like the project.
Before anyone gets worried by the subject line, no, this is not any
kind of legal thing. Its just something I believe needs to
happen. :)
Back in 2016-2017 I contacted the UEFI forum about two problems, one
of which was the format of the spec, which I figured out on my
own. The other problem was not so easily dealt with. Te other
problem relates to accessibility of UEFI-compliant systems and
platform firmware to persons with disabilities. As it currently
stands, such a thing is nonexistent. To be fair, I completely
understand the difficulty that such a thing would require, and I
would fully agree if we still used the PC-AT BIOS systems -- yes,
indeed, I would never suggest this kind of thing on such a system
given that there was no actual standard of any kind for
BIOSes. However, now that UEFI is here, we have such a possibility.
As it currently stands, people with low vision or blind people have
access to their computers in the general sense (I am blind
myself). We can do pretty much anything anyone else could do. We can
code, play games, all that. There are few things that we cannot
do. One of those things is managing our systems firmware in the
preboot environment.
As it stands now, I can only boot other OSes or disks via
memorization. While that worked on BIOS machines (I have, or had, an
old Toshiba laptop that was BIOS-based), it no longer works because
UEFI is mercurial. When I access the boot menu now, I play a game of
chance. If the cards are in my favor, the OS I want to boot boots,
and I can go on my way. But if the cards aren't in my favor, I end
up making something happen that was unintended, and, worst of all, I
have no idea what I did.
However, the boot menu is only one portion of a platform firmware
UI. What about the setup utility or other utilities offered by
computer manufacturers? What about diagnostic utilities,
bootloaders, etc? What do I do with those? Well, I only have one
option -- sited assistance. If I go into my computers setup utility,
I cannot trust myself and say to myself, "OK, I know what I'm
doing. All I need to do is change x and save and quit." No, I can't
do that, because memorizing such a complex interface is extremely
difficult, and its something I wouldn't expect anyone to do.
My proposal is simple, and I'm posting it here because I'd like
comments and feedback before it actually gets implemented (it will
take a lot of time, I'm sure): mandate, in the UEFI specification,
that accessibility features for persons with disabilities must be
implemented and documented, and, if such features are not
implemented, then that vendor is not compliant with the
specification. Place strict minimum requirements for what the
accessibility features should do and how they should work.
So, first of all, I need to point out that what you suggest would be
very difficult for the UEFI Forum to mandate. *But* I don't think
this
needs to be a problem.

UEFI primarily exists as a means through which interoperability can
be
guaranteed. This means it does not tend to so much mandate what is
actually implemented in a platform, as it defines how it behaves if
it
is. This is not 100% true, but suffice to say that mandating the
functionality you are asking for *without* having a reference
implementation available would likely simply be ignored.

If you want to turn thumbscrews, get Microsoft to add the requirement
to their windows logo requirements :)

However, as much as it is difficult to force people to develop this
support, if a functional reference implementation was published under
a permissive open source license, it would take a pretty
spectacularly
incompetent product owner not to incorporate it in their products.

Now, I'm sure someone out there will ask me how this can be
done. Well, that's why I've joined the group -- though as I
familiarize myself with EDK2 development and all that I may actually
be able to participate as more than just an accessibility expert, of
sorts.
I have added Rafael Machado on cc. He did a PhD on firmware
accessibility, including a prototype implementing an audio driver for
EDK2. The resulting thesis can be found at
https://arxiv.org/abs/1712.03186 .

As a side note, I have been blind all my life. I was born with
retinopathy of prematurity (ROP), which resulted because I was born
at 26 weeks. My retina was detached, and, though the doctors
attempted to fix it, it would not remain attached, and there is no
chance of it getting fixed now. I would neither want nor care for
such a cure, however. I have lived my entire life blind, and while
the thought of gaining site back is appealing, I am unwilling to go
through the years and years of rewiring and reconditioning of my
brain that would be required for me to survive with site. To me, it
is simply not worth the cost.
But back to the discussion at hand: I would be happy to discuss how
the accessibility features would work and what would be
required. Even standardizing, through the specification, a key
combination to toggle the accessibility features would be nice, as
that would alleviate the major problem of a blind person buying a
new computer and not knowing how to enable the accessibility
features. The overarching goal would be to make the preboot
environment (including applications run within it) accessible and
usable by blind and visually impaired people as a boot service
only.
Yes, I agree, operating systems are already doing their things.

Again, mandating such a key combination is not necessarily something
that the UEFI specification can do. Some devices won't necessarily
have a keyboard.

Some things the UEFI specification could do is:
- Add protocols for Audio output (and possibly input) and talking to
audio codecs.
- Add audio hooks for the Human Interaction Interface subsystem
(HII).

EDK2 could then start adding device drivers producing these
protocols,
and enable the support for some reference platforms.

Would we need any specific support for things like refreshable
braille
displays?

It would be superfluous to make this a runtime service, as all
major OSes already have accessibility features. Plus, managing such
a thing would be impossible to do.
Agreed.

This email has gotten quite long, so I will suspend the discussion
of functionality and how I would like it to work for a future email
once everyone has gotten on board.
Best Regards,

Leif


Thank you for your time and consideration.



--
Signed,
Ethin D. Probst

--
Signed,
Ethin D. Probst



--
Signed,
Ethin D. Probst


Re: [PATCH] q35: lpc: allow to lock down 128K RAM at default SMBASE address

Laszlo Ersek
 

On 09/10/19 17:58, Igor Mammedov wrote:
On Mon, 9 Sep 2019 21:15:44 +0200
Laszlo Ersek <lersek@redhat.com> wrote:
[...]

It looks like fwcfg smi feature negotiation isn't reusable in this case.
I'd prefer not to add another device just for another SMI feature
negotiation/activation.
I thought it could be a register on the new CPU hotplug controller that
we're going to need anyway (if I understand correctly, at least).

But:

How about stealing reserved register from pci-host similar to your
extended TSEG commit (2f295167 q35/mch: implement extended TSEG sizes)?
(Looking into spec can (ab)use 0x58 or 0x59 register)
Yes, that should work.

In fact, I had considered 0x58 / 0x59 when looking for unused registers
for extended TSEG configuration:

http://mid.mail-archive.com/d8802612-0b11-776f-b209-53bbdaf67515@redhat.com

So I'm OK with this, thank you.

More below:

... I've done some testing too. Applying the QEMU patch on top of
89ea03a7dc83, my plan was:

- do not change OVMF, just see if it continues booting with the QEMU
patch

- then negotiate bit#1 too, in step (1a) -- this is when I'd expect (3a)
to break.

Unfortunately, the result is worse than that; even without negotiating
bit#1 (i.e. in the baseline test), the firmware crashes (reboots) in
step (3a). I've checked "info mtree", and all occurences of
"smbase-blackhole" and "smbase-blackhole" are marked [disabled]. I'm not
sure what's wrong with the baseline test (i.e. without negotiating
bit#1). If I drop the patch (build QEMU at 89ea03a7dc83), then things
work fine.
that was a bug in my code, which always made lock down effective on
feature_ok selection, which breaks relocation for reasons you've
explained above.

diff --git a/hw/isa/lpc_ich9.c b/hw/isa/lpc_ich9.c
index 17a8cd1b51..32ddf54fc2 100644
--- a/hw/isa/lpc_ich9.c
+++ b/hw/isa/lpc_ich9.c
@@ -383,7 +383,7 @@ static const MemoryRegionOps smbase_blackhole_ops = {

static void ich9_lpc_smbase_locked_update(ICH9LPCState *lpc)
{
- bool en = lpc->smi_negotiated_features & ICH9_LPC_SMI_F_LOCKED_SMBASE_BIT;
+ bool en = lpc->smi_negotiated_features & (UINT64_C(1) << ICH9_LPC_SMI_F_LOCKED_SMBASE_BIT);

memory_region_transaction_begin();
memory_region_set_enabled(&lpc->smbase_blackhole, en);
I see.

ICH9_LPC_SMI_F_LOCKED_SMBASE_BIT is 1, with the intended value for
bitmask checkin) being 1LLU<<1 == 2LLU.

Due to the bug, the function would check value 1 in the bitmask -- which
in fact corresponds to bit#0. Bit#0 happens to be
ICH9_LPC_SMI_F_BROADCAST_BIT.

And because OVMF would negotiate that feature (= broadcast SMI) even in
the baseline test, it ended up enabling the "locked smbase" feature too.

So ultimately I think we can consider this a valid test (= with
meaningful result); the result is that we can't reuse these fw_cfg files
for "locked smbase", as discussed above.

Thanks!
Laszlo


Re: [RFC] Error reporting upon SecureBoot image validation failures?

Laszlo Ersek
 

On 09/11/19 01:41, aaron.young@oracle.com wrote:

 Hello, I have a question about user error reporting during SecureBoot
image validation.

 From my testing with OVMF/Secureboot, when an image to be loaded fails
validation (i.e. due to missing or incorrect signature), it silently
fails. i.e. there is no indication given to the user via the console.
All that seems to be done as a result is an entry added to the Image
Execution Information Table.

 I see there used to be code implemented to give an indication (e.g.
QUERY_USER_ON_SECURITY_VIOLATION), but it was deemed a violation of the
UEFI spec and removed.


 I didn't see any code in edk2 at present which appears to consume the
data in the Image Execution Information Table (though I may have missed
it).

 The UEFI spec (Version 2.8, Section 32.4.2) briefly discusses an "agent
that executes later" can consume the table data, but is vague about what
this agent can/should do.

 We'd like users to be able to see image validation failures and the
reason for the failure. There are currently some DEBUG output statements
in the code, but this is inadequate for general user applications.

 Questions:

 Is there any (planned) replacement for QUERY_USER_ON_SECURITY_VIOLATION
support? i.e. to give users an indication of the image validation failure?

 Is there any "agent" code (as I reference above) at present to process
the Image Execution Information Table - or any work planned in this area?


 What I envisioned is some HII code added to the Boot Manager to display
the contents of the Image Execution Information Table in a human
readable manner. Does this seem reasonable?

 Thanks in advance for any insight/comments/etc.
A special case of your question is covered, at the moment, as follows
("give users an indication of the image validation failure"):

- When a UEFI boot option is rejected due to secure boot verification,
the BDS driver reports a structured status code that carries the
LoadImage / StartImage failure code.

- ArmVirtQemu and OVMF are subscribed to the status code in question,
and report it to the UEFI console (which is multiplexed to graphics and
serial). If you have log capture enabled for the serial console, you can
see the error report later, even if it disappears quickly from the
console otherwise (e.g. because the next boot option succeeds and clears
the screen quickly).

Please see commit range 2df879827442..1797f32e0a19.

I have no other input on this.

Thanks
Laszlo


[RFC] Error reporting upon SecureBoot image validation failures?

Aaron Young
 

Hello, I have a question about user error reporting during SecureBoot image validation.

From my testing with OVMF/Secureboot, when an image to be loaded fails validation (i.e. due to missing or incorrect signature), it silently fails. i.e. there is no indication given to the user via the console. All that seems to be done as a result is an entry added to the Image Execution Information Table.

I see there used to be code implemented to give an indication (e.g. QUERY_USER_ON_SECURITY_VIOLATION), but it was deemed a violation of the UEFI spec and removed.


I didn't see any code in edk2 at present which appears to consume the data in the Image Execution Information Table (though I may have missed it).

The UEFI spec (Version 2.8, Section 32.4.2) briefly discusses an "agent that executes later" can consume the table data, but is vague about what this agent can/should do.

We'd like users to be able to see image validation failures and the reason for the failure. There are currently some DEBUG output statements in the code, but this is inadequate for general user applications.

Questions:

Is there any (planned) replacement for QUERY_USER_ON_SECURITY_VIOLATION support? i.e. to give users an indication of the image validation failure?

Is there any "agent" code (as I reference above) at present to process the Image Execution Information Table - or any work planned in this area?


What I envisioned is some HII code added to the Boot Manager to display the contents of the Image Execution Information Table in a human readable manner. Does this seem reasonable?

Thanks in advance for any insight/comments/etc.

-Aaron Young

601 - 620 of 753