Re: [PATCH v2 1/1] UnitTestFrameworkPkg: Add info to readme about working with UnitTests


Michael D Kinney
 

Reviewed-by: Michael D Kinney <michael.d.kinney@...>

Mike

-----Original Message-----
From: Bret Barkelew <bret@...>
Sent: Friday, May 29, 2020 10:34 AM
To: devel@edk2.groups.io
Cc: Kinney, Michael D <michael.d.kinney@...>;
Bret Barkelew <bret.barkelew@...>
Subject: [PATCH v2 1/1] UnitTestFrameworkPkg: Add info
to readme about working with UnitTests

Cc: Michael D Kinney <michael.d.kinney@...>

Cc: Bret Barkelew <bret.barkelew@...>

Signed-off-by: Bret Barkelew
<bret.barkelew@...>

---



Notes:

v2:

- Update the SpellCheck YAML to incude GitHub
handles in the dictionary



.pytool/Readme.md |
125 +++++++++++++++-----

UnitTestFrameworkPkg/ReadMe.md |
82 ++++++++++++-

UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml |
7 +-

3 files changed, 177 insertions(+), 37 deletions(-)



diff --git a/.pytool/Readme.md b/.pytool/Readme.md

index c7dce3b64ca0..c401dba18fbf 100644

--- a/.pytool/Readme.md

+++ b/.pytool/Readme.md

@@ -2,31 +2,32 @@



## Basic Status



-| Package | Windows VS2019 (IA32/X64)|
Ubuntu GCC (IA32/X64/ARM/AARCH64) | Known Issues |

-| :---- | :----- | :--
-- | :--- |

-| ArmPkg |

-| ArmPlatformPkg |

-| ArmVirtPkg | SEE PACKAGE README | SEE
PACKAGE README |

-| CryptoPkg | :heavy_check_mark: |
:heavy_check_mark: | Spell checking in audit mode

-| DynamicTablesPkg |

-| EmbeddedPkg |

-| EmulatorPkg | SEE PACKAGE README | SEE
PACKAGE README | Spell checking in audit mode

-| FatPkg | :heavy_check_mark: |
:heavy_check_mark: |

-| FmpDevicePkg | :heavy_check_mark: |
:heavy_check_mark: |

-| IntelFsp2Pkg |

-| IntelFsp2WrapperPkg |

-| MdeModulePkg | :heavy_check_mark: |
:heavy_check_mark: | DxeIpl dependency on ArmPkg,
Depends on StandaloneMmPkg, Spell checking in audit
mode

-| MdePkg | :heavy_check_mark: |
:heavy_check_mark: | Spell checking in audit mode

-| NetworkPkg | :heavy_check_mark: |
:heavy_check_mark: | Spell checking in audit mode

-| OvmfPkg | SEE PACKAGE README | SEE
PACKAGE README | Spell checking in audit mode

-| PcAtChipsetPkg | :heavy_check_mark: |
:heavy_check_mark: |

-| SecurityPkg | :heavy_check_mark: |
:heavy_check_mark: | Spell checking in audit mode

-| ShellPkg | :heavy_check_mark: |
:heavy_check_mark: | Spell checking in audit mode, 3
modules are not being built by DSC

-| SignedCapsulePkg |

-| SourceLevelDebugPkg |

-| StandaloneMmPkg |

-| UefiCpuPkg | :heavy_check_mark: |
:heavy_check_mark: | Spell checking in audit mode, 2
binary modules not being built by DSC

-| UefiPayloadPkg |

+| Package | Windows VS2019 (IA32/X64)|
Ubuntu GCC (IA32/X64/ARM/AARCH64) | Known Issues |

+| :---- | :----- | :-
--- | :--- |

+| ArmPkg |

+| ArmPlatformPkg |

+| ArmVirtPkg | SEE PACKAGE README | SEE
PACKAGE README |

+| CryptoPkg | :heavy_check_mark: |
:heavy_check_mark: | Spell checking in audit mode

+| DynamicTablesPkg |

+| EmbeddedPkg |

+| EmulatorPkg | SEE PACKAGE README | SEE
PACKAGE README | Spell checking in audit mode

+| FatPkg | :heavy_check_mark: |
:heavy_check_mark: |

+| FmpDevicePkg | :heavy_check_mark: |
:heavy_check_mark: |

+| IntelFsp2Pkg |

+| IntelFsp2WrapperPkg |

+| MdeModulePkg | :heavy_check_mark: |
:heavy_check_mark: | DxeIpl dependency on ArmPkg,
Depends on StandaloneMmPkg, Spell checking in audit
mode

+| MdePkg | :heavy_check_mark: |
:heavy_check_mark: | Spell checking in audit mode

+| NetworkPkg | :heavy_check_mark: |
:heavy_check_mark: | Spell checking in audit mode

+| OvmfPkg | SEE PACKAGE README | SEE
PACKAGE README | Spell checking in audit mode

+| PcAtChipsetPkg | :heavy_check_mark: |
:heavy_check_mark: |

+| SecurityPkg | :heavy_check_mark: |
:heavy_check_mark: | Spell checking in audit mode

+| ShellPkg | :heavy_check_mark: |
:heavy_check_mark: | Spell checking in audit mode, 3
modules are not being built by DSC

+| SignedCapsulePkg |

+| SourceLevelDebugPkg |

+| StandaloneMmPkg |

+| UefiCpuPkg | :heavy_check_mark: |
:heavy_check_mark: | Spell checking in audit mode, 2
binary modules not being built by DSC

+| UefiPayloadPkg |

+| UnitTestFrameworkPkg | :heavy_check_mark: |
:heavy_check_mark: |



For more detailed status look at the test results of
the latest CI run on the

repo readme.

@@ -88,7 +89,7 @@ that a few steps should be followed.
Details of EDKII Tools can be found in the

* VS 2017 or VS 2019

* Windows SDK (for rc)

* Windows WDK (for capsules)

- * Ubuntu 16.04

+ * Ubuntu 18.04 or Fedora

* GCC5

* Easy to add more but this is the current state

2. Python 3.7.x or newer on path

@@ -137,11 +138,31 @@ location makes more sense for the
community.



### Module Inclusion Test - DscCompleteCheck



-This test scans all available modules (via INF files)
and compares them to the

-package-level DSC file for the package each module is
contained within. The test

-considers it an error if any module does not appear in
the `Components` section

-of at least one package-level DSC (indicating that it
would not be built if the

-package were built).

+This scans all INF files from a package and confirms
they are

+listed in the package level DSC file. The test
considers it an error if any INF

+does not appear in the `Components` section of the
package-level DSC (indicating

+that it would not be built if the package were built).
This is critical because

+much of the CI infrastructure assumes that all modules
will be listed in the DSC

+and compiled.

+

+This test will ignore INFs in the following cases:

+

+1. When `MODULE_TYPE` = `HOST_APPLICATION`

+2. When a Library instance **only** supports the
`HOST_APPLICATION` environment

+

+### Host Module Inclusion Test -
HostUnitTestDscCompleteCheck

+

+This test scans all INF files from a package for those
related to host

+based unit tests and confirms they are listed in the
unit test DSC file for the package.

+The test considers it an error if any INF meeting the
requirements does not appear

+in the `Components` section of the unit test DSC. This
is critical because

+much of the CI infrastructure assumes that modules
will be listed in the DSC

+and compiled.

+

+This test will only require INFs in the following
cases:

+

+1. When `MODULE_TYPE` = `HOST_APPLICATION`

+2. When a Library instance explicitly supports the
`HOST_APPLICATION` environment



### Code Compilation Test - CompilerPlugin



@@ -150,6 +171,46 @@ all package-level DSCs were built,
the Code Compilation Test simply runs through

and builds every package-level DSC on every toolchain
and for every architecture

that is supported. Any module that fails to build is
considered an error.



+### Host Unit Test Compilation and Run Test -
HostUnitTestCompilerPlugin

+

+A test that compiles the dsc for host based unit test
apps.

+On Windows this will also enable a build plugin to
execute that will run the unit tests and verify the
results.

+

+These tools will be invoked on any CI

+pass that includes the NOOPT target. In order for
these tools to do their job,

+the package and tests must be configured in a
particular way...

+

+#### Including Host-Based Tests in the Package YAML

+

+For example, looking at the `MdeModulePkg.ci.yaml`
config file, there are two

+config options that control HostBased test behavior:

+

+```json

+ ## options defined
.pytool/Plugin/HostUnitTestCompilerPlugin

+ "HostUnitTestCompilerPlugin": {

+ "DscPath": "Test/MdeModulePkgHostTest.dsc"

+ },

+```

+

+This option tell the test builder to run. The test
builder needs to know which

+modules in this package are host-based tests, so that
DSC path is provided.

+

+#### Configuring the HostBased DSC

+

+The HostBased DSC for `MdeModulePkg` is located at

+`MdeModulePkg/Test/MdeModulePkgHostTest.dsc`.

+

+To add automated host-based unit test building to a
new package, create a

+similar DSC. The new DSC should make sure to have the
`NOOPT` BUILD_TARGET

+and should include the line:

+

+```

+!include
UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc

+```

+

+All of the modules that are included in the
`Components` section of this

+DSC should be of type HOST_APPLICATION.

+

### GUID Uniqueness Test - GuidCheck



This test works on the collection of all packages
rather than an individual

@@ -207,6 +268,8 @@ few standard scopes.

| global-nix | edk2_invocable++
| Running on Linux based OS |

| edk2-build |
| This indicates that an invocable is building EDK2
based UEFI code |

| cibuild | set in .pytool/CISettings.py
| Suggested target for edk2 continuous integration
builds. Tools used for CiBuilds can use this scope.
Example: asl compiler |

+| host-based-test | set in .pytool/CISettings.py
| Turns on the host based tests and plugin |

+| host-test-win | set in .pytool/CISettings.py
| Enables the host based test runner for Windows |



## Future investments



diff --git a/UnitTestFrameworkPkg/ReadMe.md
b/UnitTestFrameworkPkg/ReadMe.md

index 7296f0a45c5f..64386941cb4b 100644

--- a/UnitTestFrameworkPkg/ReadMe.md

+++ b/UnitTestFrameworkPkg/ReadMe.md

@@ -58,7 +58,7 @@ header for the `UnitTestLib` is
located in `MdePkg`, so you shouldn't need to de

packages. As long as your DSC file knows where to find
the lib implementation that you want to use,

you should be good to go.



-See this example in 'SampleUnitTestApp.inf'...

+See this example in 'SampleUnitTestUefiShell.inf'...



```

[Packages]

@@ -72,6 +72,14 @@ See this example in
'SampleUnitTestApp.inf'...

PrintLib

```



+Also, if you want you test to automatically be picked
up by the Test Runner plugin, you will need

+to make sure that the module `BASE_NAME` contains the
word `Test`...

+

+```

+[Defines]

+ BASE_NAME = SampleUnitTestUefiShell

+```

+

### Requirements - Code



Not to state the obvious, but let's make sure we have
the following include before getting too far along...

@@ -221,9 +229,11 @@ https://api.cmocka.org/



## Development



-When using the EDK2 Pytools for CI testing, the host-
based unit tests will be built and run on any build
that includes the `NOOPT` build target.

+When using the EDK2 Pytools for CI testing, the host-
based unit tests will be built and run on any build
that includes

+the `NOOPT` build target.



-If you are trying to iterate on a single test, a
convenient pattern is to build only that test module.
For example, the following command will build only the
SafeIntLib host-based test from the MdePkg...

+If you are trying to iterate on a single test, a
convenient pattern is to build only that test module.
For example,

+the following command will build only the SafeIntLib
host-based test from the MdePkg...



```bash

stuart_ci_build -c .pytool/CISettings.py
TOOL_CHAIN_TAG=VS2017 -p MdePkg -t NOOPT
BUILDMODULE=MdePkg/Test/UnitTest/Library/BaseSafeIntLib
/TestBaseSafeIntLib.inf

@@ -250,8 +260,72 @@ reporting lib. This isn't
currently possible with host-based. Only the assertion



We will continue trying to make these as similar as
possible.



+## Unit Test Location/Layout Rules

+

+Code/Test | Location

+--------- | --------

+Host-Based Unit Tests for a Library/Protocol/PPI/GUID
Interface | If what's being tested is an interface
(e.g. a library with a public header file, like
DebugLib), the test should be scoped to the parent
package.<br/>Example:
`MdePkg/Test/UnitTest/[Library/Protocol/Ppi/Guid]/`<br/
<br/>A real-world example of this is the
BaseSafeIntLib test in
MdePkg.<br/>`MdePkg/Test/UnitTest/Library/BaseSafeIntLi
b/TestBaseSafeIntLibHost.inf`

+Host-Based Unit Tests for a Library/Driver
(PEI/DXE/SMM) implementation | If what's being tested
is a specific implementation (e.g.
BaseDebugLibSerialPort for DebugLib), the test should
be scoped to the implementation directory itself, in a
UnitTest subdirectory.<br/><br/>Module Example:
`MdeModulePkg/Universal/EsrtFmpDxe/UnitTest/`<br/>Libra
ry Example: `MdePkg/Library/BaseMemoryLib/UnitTest/`

+Host-Based Tests for a Functionality or Feature | If
you're writing a functional test that operates at the
module level (i.e. if it's more than a single file or
library), the test should be located in the package-
level Tests directory under the HostFuncTest
subdirectory.<br/>For example, if you were writing a
test for the entire FMP Device Framework, you might put
your test
in:<br/>`FmpDevicePkg/Test/HostFuncTest/FmpDeviceFramew
ork`<br/><br/>If the feature spans multiple packages,
it's location should be determined by the package
owners related to the feature.

+Non-Host-Based (PEI/DXE/SMM/Shell) Tests for a
Functionality or Feature | Similar to Host-Based, if
the feature is in one package, should be located in the
`*Pkg/Test/[Shell/Dxe/Smm/Pei]Test`
directory.<br/><br/>If the feature spans multiple
packages, it's location should be determined by the
package owners related to the feature.<br/><br/>USAGE
EXAMPLES<br/>PEI Example: MP_SERVICE_PPI. Or check MTRR
configuration in a notification function.<br/> SMM
Example: a test in a protocol callback function. (It is
different with the solution that
SmmAgent+ShellApp)<br/>DXE Example: a test in a UEFI
event call back to check SPI/SMRAM status. <br/> Shell
Example: the SMM handler audit test has a shell-based
app that interacts with an SMM handler to get
information. The SMM paging audit test gathers
information about both DXE and SMM. And the SMM paging
functional test actually forces errors into SMM via a
DXE driver.

+

+### Example Directory Tree

+

+```text

+<PackageName>Pkg/

+ ComponentY/

+ ComponentY.inf

+ ComponentY.c

+ UnitTest/

+ ComponentYHostUnitTest.inf # Host-Based
Test for Driver Module

+ ComponentYUnitTest.c

+

+ Library/

+ GeneralPurposeLibBase/

+ ...

+

+ GeneralPurposeLibSerial/

+ ...

+

+ SpecificLibDxe/

+ SpecificLibDxe.c

+ SpecificLibDxe.inf

+ UnitTest/ # Host-Based Test
for Specific Library Implementation

+ SpecificLibDxeHostUnitTest.c

+ SpecificLibDxeHostUnitTest.inf

+ Test/

+ <Package>HostTest.dsc # Host-Based
Test Apps

+ UnitTest/

+ InterfaceX

+ InterfaceXHostUnitTest.inf # Host-Based App
(should be in Test/<Package>HostTest.dsc)

+ InterfaceXPeiUnitTest.inf # PEIM Target-
Based Test (if applicable)

+ InterfaceXDxeUnitTest.inf # DXE Target-
Based Test (if applicable)

+ InterfaceXSmmUnitTest.inf # SMM Target-
Based Test (if applicable)

+ InterfaceXShellUnitTest.inf # Shell App
Target-Based Test (if applicable)

+ InterfaceXUnitTest.c # Test Logic

+

+ GeneralPurposeLib/ # Host-Based
Test for any implementation of GeneralPurposeLib

+ GeneralPurposeLibTest.c

+ GeneralPurposeLibHostUnitTest.inf

+

+ <Package>Pkg.dsc # Standard Modules and any
Target-Based Test Apps (including in Test/)

+

+```

+

+### Future Locations in Consideration

+

+We don't know if these types will exist or be
applicable yet, but if you write a support library or
module that matches the following, please make sure
they live in the correct place.

+

+Code/Test | Location

+--------- | --------

+Host-Based Library Implementations |
Host-Based Implementations of common libraries (eg.
MemoryAllocationLibHost) should live in the same
package that declares the library interface in its .DEC
file in the `*Pkg/HostLibrary` directory. Should have
'Host' in the name.

+Host-Based Mocks and Stubs | Mock and Stub libraries
should live in the `UefiTestFrameworkPkg/StubLibrary`
with either 'Mock' or 'Stub' in the library name.

+

+### If still in doubt...

+

+Hop on GitHub and ask @corthon, @mdkinney, or
@spbrogan. ;)

+

## Copyright



Copyright (c) Microsoft Corporation.

SPDX-License-Identifier: BSD-2-Clause-Patent

-

diff --git
a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml

index 01648595055e..51e172537f8a 100644

--- a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml

+++ b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml

@@ -59,7 +59,7 @@

"SpellCheck": {

"AuditOnly": False, # Fails test but
run in AuditOnly mode to collect log

"IgnoreFiles": [ # use gitignore
syntax to ignore errors in matching files

- "/Library/CmockaLib/cmocka/**/*.*" # not
going to spell check a submodule

+ "Library/CmockaLib/cmocka/**/*.*" # not
going to spell check a submodule

],

"ExtendWords": [ # words to extend
to the dictionary for this package

"cmocka",

@@ -68,7 +68,10 @@

"pytool",

"pytools",

"NOFAILURE",

- "DHAVE" # build flag for cmocka in the INF

+ "DHAVE", # build flag for cmocka in the
INF

+ "corthon", # Contact GitHub account
in Readme

+ "mdkinney", # Contact GitHub account
in Readme

+ "spbrogan" # Contact GitHub account
in Readme

],

"IgnoreStandardPaths": [], # Standard Plugin
defined paths that should be ignore

"AdditionalIncludePaths": [] # Additional
paths to spell check (wildcards supported)

--

2.26.2.windows.1.8.g01c50adf56.20200515075929

Join devel@edk2.groups.io to automatically receive all group messages.