Cc: Michael D Kinney <michael.d.kinney@...>=0D
Cc: Bret Barkelew <bret.barkelew@...>=0D
Signed-off-by: Bret Barkelew <bret.barkelew@...>=0D
---=0D
=0D
Notes:=0D
v2:=0D
- Update the SpellCheck YAML to incude GitHub handles in the dictionary=
=0D
=0D
.pytool/Readme.md | 125 +++++++++++++++---=
--=0D
UnitTestFrameworkPkg/ReadMe.md | 82 ++++++++++++-=0D
UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml | 7 +-=0D
3 files changed, 177 insertions(+), 37 deletions(-)=0D
=0D
diff --git a/.pytool/Readme.md b/.pytool/Readme.md=0D
index c7dce3b64ca0..c401dba18fbf 100644=0D
--- a/.pytool/Readme.md=0D
+++ b/.pytool/Readme.md=0D
@@ -2,31 +2,32 @@=0D
=0D
## Basic Status=0D
=0D
-| Package | Windows VS2019 (IA32/X64)| Ubuntu GCC (IA32/X64/AR=
M/AARCH64) | Known Issues |=0D
-| :---- | :----- | :---- =
| :--- |=0D
-| ArmPkg |=0D
-| ArmPlatformPkg |=0D
-| ArmVirtPkg | SEE PACKAGE README | SEE PACKAGE README |=0D
-| CryptoPkg | :heavy_check_mark: | :heavy_check_mark: | Spell ch=
ecking in audit mode=0D
-| DynamicTablesPkg |=0D
-| EmbeddedPkg |=0D
-| EmulatorPkg | SEE PACKAGE README | SEE PACKAGE README | Spell ch=
ecking in audit mode=0D
-| FatPkg | :heavy_check_mark: | :heavy_check_mark: |=0D
-| FmpDevicePkg | :heavy_check_mark: | :heavy_check_mark: |=0D
-| IntelFsp2Pkg |=0D
-| IntelFsp2WrapperPkg |=0D
-| MdeModulePkg | :heavy_check_mark: | :heavy_check_mark: | DxeIpl d=
ependency on ArmPkg, Depends on StandaloneMmPkg, Spell checking in audit mo=
de=0D
-| MdePkg | :heavy_check_mark: | :heavy_check_mark: | Spell ch=
ecking in audit mode=0D
-| NetworkPkg | :heavy_check_mark: | :heavy_check_mark: | Spell ch=
ecking in audit mode=0D
-| OvmfPkg | SEE PACKAGE README | SEE PACKAGE README | Spell ch=
ecking in audit mode=0D
-| PcAtChipsetPkg | :heavy_check_mark: | :heavy_check_mark: |=0D
-| SecurityPkg | :heavy_check_mark: | :heavy_check_mark: | Spell ch=
ecking in audit mode=0D
-| ShellPkg | :heavy_check_mark: | :heavy_check_mark: | Spell ch=
ecking in audit mode, 3 modules are not being built by DSC=0D
-| SignedCapsulePkg |=0D
-| SourceLevelDebugPkg |=0D
-| StandaloneMmPkg |=0D
-| UefiCpuPkg | :heavy_check_mark: | :heavy_check_mark: | Spell ch=
ecking in audit mode, 2 binary modules not being built by DSC=0D
-| UefiPayloadPkg |=0D
+| Package | Windows VS2019 (IA32/X64)| Ubuntu GCC (IA32/X64/A=
RM/AARCH64) | Known Issues |=0D
+| :---- | :----- | :---- =
| :--- |=0D
+| ArmPkg |=0D
+| ArmPlatformPkg |=0D
+| ArmVirtPkg | SEE PACKAGE README | SEE PACKAGE README |=0D
+| CryptoPkg | :heavy_check_mark: | :heavy_check_mark: | Spell c=
hecking in audit mode=0D
+| DynamicTablesPkg |=0D
+| EmbeddedPkg |=0D
+| EmulatorPkg | SEE PACKAGE README | SEE PACKAGE README | Spell c=
hecking in audit mode=0D
+| FatPkg | :heavy_check_mark: | :heavy_check_mark: |=0D
+| FmpDevicePkg | :heavy_check_mark: | :heavy_check_mark: |=0D
+| IntelFsp2Pkg |=0D
+| IntelFsp2WrapperPkg |=0D
+| MdeModulePkg | :heavy_check_mark: | :heavy_check_mark: | DxeIpl =
dependency on ArmPkg, Depends on StandaloneMmPkg, Spell checking in audit m=
ode=0D
+| MdePkg | :heavy_check_mark: | :heavy_check_mark: | Spell c=
hecking in audit mode=0D
+| NetworkPkg | :heavy_check_mark: | :heavy_check_mark: | Spell c=
hecking in audit mode=0D
+| OvmfPkg | SEE PACKAGE README | SEE PACKAGE README | Spell c=
hecking in audit mode=0D
+| PcAtChipsetPkg | :heavy_check_mark: | :heavy_check_mark: |=0D
+| SecurityPkg | :heavy_check_mark: | :heavy_check_mark: | Spell c=
hecking in audit mode=0D
+| ShellPkg | :heavy_check_mark: | :heavy_check_mark: | Spell c=
hecking in audit mode, 3 modules are not being built by DSC=0D
+| SignedCapsulePkg |=0D
+| SourceLevelDebugPkg |=0D
+| StandaloneMmPkg |=0D
+| UefiCpuPkg | :heavy_check_mark: | :heavy_check_mark: | Spell c=
hecking in audit mode, 2 binary modules not being built by DSC=0D
+| UefiPayloadPkg |=0D
+| UnitTestFrameworkPkg | :heavy_check_mark: | :heavy_check_mark: |=0D
=0D
For more detailed status look at the test results of the latest CI run on =
the=0D
repo readme.=0D
@@ -88,7 +89,7 @@ that a few steps should be followed. Details of EDKII To=
ols can be found in the=0D
* VS 2017 or VS 2019=0D
* Windows SDK (for rc)=0D
* Windows WDK (for capsules)=0D
- * Ubuntu 16.04=0D
+ * Ubuntu 18.04 or Fedora=0D
* GCC5=0D
* Easy to add more but this is the current state=0D
2. Python 3.7.x or newer on path=0D
@@ -137,11 +138,31 @@ location makes more sense for the community.=0D
=0D
### Module Inclusion Test - DscCompleteCheck=0D
=0D
-This test scans all available modules (via INF files) and compares them to=
the=0D
-package-level DSC file for the package each module is contained within. Th=
e test=0D
-considers it an error if any module does not appear in the `Components` se=
ction=0D
-of at least one package-level DSC (indicating that it would not be built i=
f the=0D
-package were built).=0D
+This scans all INF files from a package and confirms they are=0D
+listed in the package level DSC file. The test considers it an error if an=
y INF=0D
+does not appear in the `Components` section of the package-level DSC (indi=
cating=0D
+that it would not be built if the package were built). This is critical be=
cause=0D
+much of the CI infrastructure assumes that all modules will be listed in t=
he DSC=0D
+and compiled.=0D
+=0D
+This test will ignore INFs in the following cases:=0D
+=0D
+1. When `MODULE_TYPE` =3D `HOST_APPLICATION`=0D
+2. When a Library instance **only** supports the `HOST_APPLICATION` enviro=
nment=0D
+=0D
+### Host Module Inclusion Test - HostUnitTestDscCompleteCheck=0D
+=0D
+This test scans all INF files from a package for those related to host=0D
+based unit tests and confirms they are listed in the unit test DSC file fo=
r the package.=0D
+The test considers it an error if any INF meeting the requirements does no=
t appear=0D
+in the `Components` section of the unit test DSC. This is critical because=
=0D
+much of the CI infrastructure assumes that modules will be listed in the =
DSC=0D
+and compiled.=0D
+=0D
+This test will only require INFs in the following cases:=0D
+=0D
+1. When `MODULE_TYPE` =3D `HOST_APPLICATION`=0D
+2. When a Library instance explicitly supports the `HOST_APPLICATION` envi=
ronment=0D
=0D
### Code Compilation Test - CompilerPlugin=0D
=0D
@@ -150,6 +171,46 @@ all package-level DSCs were built, the Code Compilatio=
n Test simply runs through=0D
and builds every package-level DSC on every toolchain and for every archit=
ecture=0D
that is supported. Any module that fails to build is considered an error.=
=0D
=0D
+### Host Unit Test Compilation and Run Test - HostUnitTestCompilerPlugin=0D
+=0D
+A test that compiles the dsc for host based unit test apps.=0D
+On Windows this will also enable a build plugin to execute that will run t=
he unit tests and verify the results.=0D
+=0D
+These tools will be invoked on any CI=0D
+pass that includes the NOOPT target. In order for these tools to do their =
job,=0D
+the package and tests must be configured in a particular way...=0D
+=0D
+#### Including Host-Based Tests in the Package YAML=0D
+=0D
+For example, looking at the `MdeModulePkg.ci.yaml` config file, there are =
two=0D
+config options that control HostBased test behavior:=0D
+=0D
+```json=0D
+ ## options defined .pytool/Plugin/HostUnitTestCompilerPlugin=0D
+ "HostUnitTestCompilerPlugin": {=0D
+ "DscPath": "Test/MdeModulePkgHostTest.dsc"=0D
+ },=0D
+```=0D
+=0D
+This option tell the test builder to run. The test builder needs to know w=
hich=0D
+modules in this package are host-based tests, so that DSC path is provided=
.=0D
+=0D
+#### Configuring the HostBased DSC=0D
+=0D
+The HostBased DSC for `MdeModulePkg` is located at=0D
+`MdeModulePkg/Test/MdeModulePkgHostTest.dsc`.=0D
+=0D
+To add automated host-based unit test building to a new package, create a=
=0D
+similar DSC. The new DSC should make sure to have the `NOOPT` BUILD_TARGET=
=0D
+and should include the line:=0D
+=0D
+```=0D
+!include UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc=0D
+```=0D
+=0D
+All of the modules that are included in the `Components` section of this=0D
+DSC should be of type HOST_APPLICATION.=0D
+=0D
### GUID Uniqueness Test - GuidCheck=0D
=0D
This test works on the collection of all packages rather than an individua=
l=0D
@@ -207,6 +268,8 @@ few standard scopes.=0D
| global-nix | edk2_invocable++ | Runn=
ing on Linux based OS |=0D
| edk2-build | | This=
indicates that an invocable is building EDK2 based UEFI code |=0D
| cibuild | set in .pytool/CISettings.py | Sugg=
ested target for edk2 continuous integration builds. Tools used for CiBuil=
ds can use this scope. Example: asl compiler |=0D
+| host-based-test | set in .pytool/CISettings.py | Turn=
s on the host based tests and plugin |=0D
+| host-test-win | set in .pytool/CISettings.py | Enab=
les the host based test runner for Windows |=0D
=0D
## Future investments=0D
=0D
diff --git a/UnitTestFrameworkPkg/ReadMe.md b/UnitTestFrameworkPkg/ReadMe.m=
d=0D
index 7296f0a45c5f..64386941cb4b 100644=0D
--- a/UnitTestFrameworkPkg/ReadMe.md=0D
+++ b/UnitTestFrameworkPkg/ReadMe.md=0D
@@ -58,7 +58,7 @@ header for the `UnitTestLib` is located in `MdePkg`, so y=
ou shouldn't need to de=0D
packages. As long as your DSC file knows where to find the lib implementat=
ion that you want to use,=0D
you should be good to go.=0D
=0D
-See this example in 'SampleUnitTestApp.inf'...=0D
+See this example in 'SampleUnitTestUefiShell.inf'...=0D
=0D
```=0D
[Packages]=0D
@@ -72,6 +72,14 @@ See this example in 'SampleUnitTestApp.inf'...=0D
PrintLib=0D
```=0D
=0D
+Also, if you want you test to automatically be picked up by the Test Runne=
r plugin, you will need=0D
+to make sure that the module `BASE_NAME` contains the word `Test`...=0D
+=0D
+```=0D
+[Defines]=0D
+ BASE_NAME =3D SampleUnitTestUefiShell=0D
+```=0D
+=0D
### Requirements - Code=0D
=0D
Not to state the obvious, but let's make sure we have the following includ=
e before getting too far along...=0D
@@ -221,9 +229,11 @@ https://api.cmocka.org/=0D
=0D
## Development=0D
=0D
-When using the EDK2 Pytools for CI testing, the host-based unit tests will=
be built and run on any build that includes the `NOOPT` build target.=0D
+When using the EDK2 Pytools for CI testing, the host-based unit tests will=
be built and run on any build that includes=0D
+the `NOOPT` build target.=0D
=0D
-If you are trying to iterate on a single test, a convenient pattern is to =
build only that test module. For example, the following command will build =
only the SafeIntLib host-based test from the MdePkg...=0D
+If you are trying to iterate on a single test, a convenient pattern is to =
build only that test module. For example,=0D
+the following command will build only the SafeIntLib host-based test from =
the MdePkg...=0D
=0D
```bash=0D
stuart_ci_build -c .pytool/CISettings.py TOOL_CHAIN_TAG=3DVS2017 -p MdePkg=
-t NOOPT BUILDMODULE=3DMdePkg/Test/UnitTest/Library/BaseSafeIntLib/TestBas=
eSafeIntLib.inf=0D
@@ -250,8 +260,72 @@ reporting lib. This isn't currently possible with host=
-based. Only the assertion=0D
=0D
We will continue trying to make these as similar as possible.=0D
=0D
+## Unit Test Location/Layout Rules=0D
+=0D
+Code/Test | Location=0D
+--------- | --------=0D
+Host-Based Unit Tests for a Library/Protocol/PPI/GUID Interface | If wha=
t's being tested is an interface (e.g. a library with a public header file,=
like DebugLib), the test should be scoped to the parent package.<br/>Examp=
le: `MdePkg/Test/UnitTest/[Library/Protocol/Ppi/Guid]/`<br/><br/>A real-wor=
ld example of this is the BaseSafeIntLib test in MdePkg.<br/>`MdePkg/Test/U=
nitTest/Library/BaseSafeIntLib/TestBaseSafeIntLibHost.inf`=0D
+Host-Based Unit Tests for a Library/Driver (PEI/DXE/SMM) implementation =
| If what's being tested is a specific implementation (e.g. BaseDebugLibSer=
ialPort for DebugLib), the test should be scoped to the implementation dire=
ctory itself, in a UnitTest subdirectory.<br/><br/>Module Example: `MdeModu=
lePkg/Universal/EsrtFmpDxe/UnitTest/`<br/>Library Example: `MdePkg/Library/=
BaseMemoryLib/UnitTest/`=0D
+Host-Based Tests for a Functionality or Feature | If you're writing a fu=
nctional test that operates at the module level (i.e. if it's more than a s=
ingle file or library), the test should be located in the package-level Tes=
ts directory under the HostFuncTest subdirectory.<br/>For example, if you w=
ere writing a test for the entire FMP Device Framework, you might put your =
test in:<br/>`FmpDevicePkg/Test/HostFuncTest/FmpDeviceFramework`<br/><br/>I=
f the feature spans multiple packages, it's location should be determined b=
y the package owners related to the feature.=0D
+Non-Host-Based (PEI/DXE/SMM/Shell) Tests for a Functionality or Feature =
| Similar to Host-Based, if the feature is in one package, should be locate=
d in the `*Pkg/Test/[Shell/Dxe/Smm/Pei]Test` directory.<br/><br/>If the fea=
ture spans multiple packages, it's location should be determined by the pac=
kage owners related to the feature.<br/><br/>USAGE EXAMPLES<br/>PEI Example=
: MP_SERVICE_PPI. Or check MTRR configuration in a notification function.<b=
r/> SMM Example: a test in a protocol callback function. (It is different w=
ith the solution that SmmAgent+ShellApp)<br/>DXE Example: a test in a UEFI =
event call back to check SPI/SMRAM status. <br/> Shell Example: the SMM han=
dler audit test has a shell-based app that interacts with an SMM handler to=
get information. The SMM paging audit test gathers information about both =
DXE and SMM. And the SMM paging functional test actually forces errors into=
SMM via a DXE driver.=0D
+=0D
+### Example Directory Tree=0D
+=0D
+```text=0D
+<PackageName>Pkg/=0D
+ ComponentY/=0D
+ ComponentY.inf=0D
+ ComponentY.c=0D
+ UnitTest/=0D
+ ComponentYHostUnitTest.inf # Host-Based Test for Driver Module=
=0D
+ ComponentYUnitTest.c=0D
+=0D
+ Library/=0D
+ GeneralPurposeLibBase/=0D
+ ...=0D
+=0D
+ GeneralPurposeLibSerial/=0D
+ ...=0D
+=0D
+ SpecificLibDxe/=0D
+ SpecificLibDxe.c=0D
+ SpecificLibDxe.inf=0D
+ UnitTest/ # Host-Based Test for Specific Librar=
y Implementation=0D
+ SpecificLibDxeHostUnitTest.c=0D
+ SpecificLibDxeHostUnitTest.inf=0D
+ Test/=0D
+ <Package>HostTest.dsc # Host-Based Test Apps=0D
+ UnitTest/=0D
+ InterfaceX=0D
+ InterfaceXHostUnitTest.inf # Host-Based App (should be in Test/=
<Package>HostTest.dsc)=0D
+ InterfaceXPeiUnitTest.inf # PEIM Target-Based Test (if applica=
ble)=0D
+ InterfaceXDxeUnitTest.inf # DXE Target-Based Test (if applicab=
le)=0D
+ InterfaceXSmmUnitTest.inf # SMM Target-Based Test (if applicab=
le)=0D
+ InterfaceXShellUnitTest.inf # Shell App Target-Based Test (if ap=
plicable)=0D
+ InterfaceXUnitTest.c # Test Logic=0D
+=0D
+ GeneralPurposeLib/ # Host-Based Test for any implementa=
tion of GeneralPurposeLib=0D
+ GeneralPurposeLibTest.c=0D
+ GeneralPurposeLibHostUnitTest.inf=0D
+=0D
+ <Package>Pkg.dsc # Standard Modules and any Target-Based Test A=
pps (including in Test/)=0D
+=0D
+```=0D
+=0D
+### Future Locations in Consideration=0D
+=0D
+We don't know if these types will exist or be applicable yet, but if you w=
rite a support library or module that matches the following, please make su=
re they live in the correct place.=0D
+=0D
+Code/Test | Location=0D
+--------- | --------=0D
+Host-Based Library Implementations | Host-Based Implementa=
tions of common libraries (eg. MemoryAllocationLibHost) should live in the =
same package that declares the library interface in its .DEC file in the `*=
Pkg/HostLibrary` directory. Should have 'Host' in the name.=0D
+Host-Based Mocks and Stubs | Mock and Stub libraries should live in the `=
UefiTestFrameworkPkg/StubLibrary` with either 'Mock' or 'Stub' in the libra=
ry name.=0D
+=0D
+### If still in doubt...=0D
+=0D
+Hop on GitHub and ask @corthon, @mdkinney, or @spbrogan. ;)=0D
+=0D
## Copyright=0D
=0D
Copyright (c) Microsoft Corporation.=0D
SPDX-License-Identifier: BSD-2-Clause-Patent=0D
-=0D
diff --git a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml b/UnitTestFr=
ameworkPkg/UnitTestFrameworkPkg.ci.yaml=0D
index 01648595055e..51e172537f8a 100644=0D
--- a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml=0D
+++ b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml=0D
@@ -59,7 +59,7 @@=0D
"SpellCheck": {=0D
"AuditOnly": False, # Fails test but run in AuditOnly mo=
de to collect log=0D
"IgnoreFiles": [ # use gitignore syntax to ignore erro=
rs in matching files=0D
- "/Library/CmockaLib/cmocka/**/*.*" # not going to spell check=
a submodule=0D
+ "Library/CmockaLib/cmocka/**/*.*" # not going to spell check =
a submodule=0D
],=0D
"ExtendWords": [ # words to extend to the dictionary f=
or this package=0D
"cmocka",=0D
@@ -68,7 +68,10 @@=0D
"pytool",=0D
"pytools",=0D
"NOFAILURE",=0D
- "DHAVE" # build flag for cmocka in the INF=0D
+ "DHAVE", # build flag for cmocka in the INF=0D
+ "corthon", # Contact GitHub account in Readme=0D
+ "mdkinney", # Contact GitHub account in Readme=0D
+ "spbrogan" # Contact GitHub account in Readme=0D
],=0D
"IgnoreStandardPaths": [], # Standard Plugin defined paths that =
should be ignore=0D
"AdditionalIncludePaths": [] # Additional paths to spell check (wi=
ldcards supported)=0D
-- =0D
2.26.2.windows.1.8.g01c50adf56.20200515075929=0D
=0D

Reviewed-by: Michael D Kinney <michael.d.kinney@...>

Mike