Uncrustify Conversion Detailed Plan and Extended Hard Freeze Update #4


Michael D Kinney
 

Hello,

Thank you for your patience during this extended hard freeze.

Just one more step to go. There has been a delay in the review of
the patch series with the uncrustify source changes. PR(6). This
patch series was not sent out as patch review email because of its
very large size. It only contains source style changes and the
CompareBuild tool and GitHub action has shown there are no binary
differences introduced with these source style changes.

If you are a package maintainer, then please review the following
branch/PR for your package contents and review the EDK II CI results
and BuildCompare results. I do not expect a line by line review
because we already had time to provide feedback on the source style
performed by uncrustify. Instead, a Reviewed-by for your package
indicates that you have reviewed the EDK II CI results and CompareBuild
tool functionality and results and you accept the source style
changes to your package.

* https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChanges_V5
* https://github.com/tianocore/edk2/pull/2229
* https://github.com/mdkinney/edk2/actions/runs/1521618836

Additional details on this update below.

Thank you,

Mike


Changes from Update #3
----------------------------------------------------------------------------
* Pushed PR (5)
* Added link to PR(6). EDK II CI Status is PASS. Build Compare PASS.
* Waiting for review of PR (6)
* Review of PR (7) completed and waiting for review of PR (6)
----------------------------------------------------------------------------

Changes from Update #2
----------------------------------------------------------------------------
* Changed order of PRs swapping (4) and (5). The PR that activates
increases the max CI agent job time is independent of all the other
PRs and its review is complete, so it can be committed now.
* Pushed PRs (1), (2), (3), (4).
* Waiting for review to complete for PRs (5) and (6)
* Reviews complete for PR (7)
* Identifies steps using git filter-branch to apply uncrustify changes to a
code review patch series that was generated before the uncrustify changes
avoiding manual merge.
* Identified steps using git filter-repo to generate an alternate history of
the edk2 repo with uncrustify changes applied on every commit. This may
be useful when evaluating changes to files using tools like git blame
without the large uncrustify patch series.
---------------------------------------------------------------------------

Changes from Update #1
----------------------------------------------------------------------------
* Changed order of PRs swapping (6) and (7). The PR that activates
EDK II CI check UncrustifyCheck has to be last because it unconditionally
checks all C/H files in all packages. Not just files that have been
modified like some of the other checkers.
* Updated link to the branch with the UncrustifyCheck plugin that has been
updated with a one line change and Reviewed-by and Tested-by tags.
https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plugin_v6
* Reviews complete for (1), (2), (3), (5), and (7)
---------------------------------------------------------------------------

Michael Kubacki and I have prepared the patches required to apply the
uncrustify changes and enable EDK II CI to check all submitted
patches have been run through uncrustify.

We have verified through the CompareBuild GitHub Action that the
format changes performed by uncrustify have no functional changes.
All of the OBJ, LIB, DLL, EFI, FFS, FV, and FD files match 100%
across 70 VS2019/GCC5 builds of all package/platform DSC files in
the edk2 repo.

The hard freeze will be extended after the edk2-stable202111 tag until
all uncrustify related changes are committed. We do not expect this
to take more than a few days. Do not push any PRs until the hard
freeze is lifted.

The changes are broken up into 7 patch series/PRs. The PRs are ordered
so they can be submitted using the normal submission process and EDK II
CI will pass for each one. Details are listed below.

Uncrustify 73.0.3 for EDK II
=============================
* Sources: https://dev.azure.com/projectmu/_git/Uncrustify
* Documentation: https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme
* Download: https://dev.azure.com/projectmu/Uncrustify/_packaging?_a=package&feed=mu_uncrustify&package=mu-uncrustify-release&protocolType=NuGet&version=73.0.3

Installing Uncrustify
======================
The Uncrustify tool is installed automatically when the Pytools
environment is used and the stuart* commands are run to complete the
environment setup. Please see:

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-locally

Uncrustify can also be installed from the download page listed above
or built from sources from the source link above.

The Documentation link provides instruction on how to run uncrustify from
the command line or install as a Visual Studio Code plugin. The main
uncrustify documentation also describes how to integrate with a few other
editors.

We have also discussed a client side githook. That effort has not started.
Let us know if that is a feature you would find useful.

Developer impact for new code reviews
======================================
Once the uncrustify checker is active in EDK II CI, developers must
make sure their patches are run through the uncrustify tool before
sending the patches for review.

Developers must install and run uncrustify against changes files before
sending patch review emails or submitting PR for EDK II CI. If EDK II CI
detects and differences in source formatting, then EDK II CI will fail
and the developer must run uncrustify and resubmit the patches.

Developer impact to patch series/PRs reviewed during edk2-stable201121 soft/hard freeze
=======================================================================================
Developers must rebase their changes after the uncrustify source changes are
committed. The branch with a preview of the uncrustify changes can be used
to start this rebase work.

https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChanges_V5

The following steps can be used to update an existing branch with the
required uncrustify format. This is the Windows version. I will add
the Linux version soon.

1) Fetch and checkout and rebase to latest edk2/master

git fetch origin
git checkout master
git rebase origin/master

2) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

3) Check out the patch series branch (e.g. MyBranch)

git checkout MyBranch

4) Rebase patch series against edk2-stable202111

git rebase edk2-stable202111

5) Create new branch for the uncrustifed version (e.g. MyBranch_Uncrustified)

git checkout -b MyBranch_Uncrustified

6) Use git filter-branch to uncrustify all the commits in the series
between the rebase target from (2) and HEAD of the branch. A filter
can be used to scope the uncrustify operations to only the C/H files
in the specific package the patch series is against. (e.g. DynamicTablesPkg).
BaseTools should always be excluded. If the package scoped filter is
not used, it will still work, but will take longer to run because
uncrustify will rescan every C/H files in the whole repo.

git filter-branch --tree-filter "git ls-files DynamicTablesPkg*.c DynamicTablesPkg*.h :!BaseTools/* | c:\\Temp\\UncrustifyCheck\\mu-uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c c:\\Temp\\UncrustifyCheck\\uncrustify.cfg -F - --replace --no-backup --if-changed" edk2-stable202111..HEAD

7) Now that all the individual patches in the branch are uncrustified,
rebase against latest edk2/master that is already uncrustified.

git rebase master

8) Verify the patches in this new branch.

Impacts to tracing history across the uncrusity changes
=======================================================
Tools the view file and line history do work with the large uncrustify
patch series. One impact is that the operations can be very slow due
to the large uncrustify patches.

One option to provide a faster experience is to provide an alternate
version of the edk2 repository as "documentation" that has the
entire history re-written with uncrustify run on every commit.
The tool called git-filter-repo can be used to perform this
transformation and runs in a reasonable period of time (a few hours)

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-repo/blob/main/contrib/filter-repo-demos/lint-history

The following steps can be used to perform this transformation.
This is the Windows version. I will add the Linux version soon.

** WARNING ** This operation modifies(rewrites) all the commits
in the local copy of the repo. Do not perform
these steps on a local repo you are using for
active development.

1) Clone edk2 into a new directory (see **WARNING**)

git clone https://github.com/tianocore/edk2.git edk2-uncrustified
cd edk2-uncrustified

2) Setup python virtual env, install pytools, and run stuart commands
to setup build environment which includes installing uncrustify tools.

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-locally

3) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

4) Use lint-history.py from git-filter-repo examples

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-repo/blob/main/contrib/filter-repo-demos/lint-history

Line #127 - Add try except around subprocess.check_call() with except
being pass. This is required because there are a few commits of C
files in the edk2 repo that have incorrect C syntax and do not
build with a C compiler and break the uncrustify parser. Skip reformat
of C files that can not be parsed by uncrustify. These rare instances
are addressed in the commit that fixes the C syntax error.

Run this slightly modified version of lint-history. Include only
C/H files and exclude directories that start with 'Tools' or 'BaseTools'.
This step took about 2.2 hours on a laptop.

lint-history.py
--relevant "return (not filename.startswith(b'Tools') and not filename.startswith(b'BaseTools') and (filename.endswith(b'.c') or filename.endswith(b'.h')))"
c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\mu-uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\uncrustify.cfg --replace --no-backup --if-changed

Order of PRs to apply during extended hard freeze
==================================================
1) Update EmulatorPkg Win Host [BuildOptions] MSFT CC_FLAGS to not force debug information
* https://bugzilla.tianocore.org/show_bug.cgi?id=3747
* https://github.com/mdkinney/edk2/tree/Bug_3747_EmulatorPkg_WinHost_ReproducibleBuild
* https://github.com/tianocore/edk2/pull/2215
* Required for EmulatorPkg to pass CompareBuild for VS2019 IA32/X64 builds.
* Status: Review complete. PR pushed.

2) EccCheck should not revert staged and local changes
* https://bugzilla.tianocore.org/show_bug.cgi?id=2986
* https://github.com/mdkinney/edk2/tree/Bug_2986_EccCheckRemoveGitRevert_V2
* https://github.com/tianocore/edk2/pull/2216
* Required for EDK II CI to complete in a reasonable period of time when
processing the 4000+ source file style changes made by uncrustify.
* Also fixes critical bugs that can potentially corrupt git state when
EccCheck is run locally.
* Status: Review complete. PR pushed.

3) Update pytool LicenseCheck plugin to use temp directory for diff output file
* https://bugzilla.tianocore.org/show_bug.cgi?id=3746
* https://github.com/mdkinney/edk2/tree/Bug_3746_LicenseCheckUseDiffOutputFile_V2
* https://github.com/tianocore/edk2/pull/2217
* Required to reduce EDK II CI build times.
* Status: Review complete. PR pushed.

4) Update max job time from 60 min to 120 minutes in .azurepipelines/templates
* https://bugzilla.tianocore.org/show_bug.cgi?id=3750
* https://github.com/mdkinney/edk2/tree/Bug_3750_IncreaseAzurePipelinesTimeout
* https://github.com/tianocore/edk2/pull/2219
* Required to allow EccCheck of uncrustify changes to complete on Azure
Pipelines CI agents without timing out.
* Status: Review complete. PR pushed.

5) Update Package YAML to ignore specific ECC files/errors
* https://bugzilla.tianocore.org/show_bug.cgi?id=3749
* https://github.com/mdkinney/edk2/tree/Bug_3749_EccCheckIgnoreFilesErrors
* https://github.com/tianocore/edk2/pull/2218
* Required to pass EccCheck
* Status: Review complete. PR pushed

6) Uncrustify Source Changes
* https://bugzilla.tianocore.org/show_bug.cgi?id=3737
* https://bugzilla.tianocore.org/show_bug.cgi?id=3739
* https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChanges_V5
* https://github.com/tianocore/edk2/pull/2229
* Build comparison result PASS: https://github.com/mdkinney/edk2/actions/runs/1521618836
* EFI_D_ -> DEBUG changes required to pass PatchCheck
* Uncrustify format changes required to pass UncrustifyCheck
* Status:
Waiting for review

7) UncrustifyCheck EDK II CI Plugin
* https://bugzilla.tianocore.org/show_bug.cgi?id=3748
* https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plugin_v6
* Required to enforce all PRs submitted to EDK II CI match uncrustify format.
* Unconditionally checks all packages. Can not be committed until all C/H
source files have been updated.
* Status: Review complete

Combined Branch/PR for Review/Test
==================================
* Build Comparison results must pass 100% across the full set of PRs before
the individual PRs can be pushed in the order listed above.
* Branch: https://github.com/mdkinney/edk2/tree/TestOnly_Uncrustify_PR_Series
* PR: https://github.com/tianocore/edk2/pull/2229
Status = PASS
* CompareBuild:
Branch: https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChanges_V5
--ref1: ef9a059cdb15844fe52a49af2bf7d86b9dd3e9bf
--ref2: Bug_3737_3739_ApplyUncrustifyChanges_V5
Extra Options: -n 4 --quiet
Results: https://github.com/mdkinney/edk2/actions/runs/1521618836
30 VS2019 build comparisons PASS
40 GCC5 build comparisons PASS
100% PASS

The following git log shows the set of patches from --ref1 to --ref 2across
which there are no differences in any of the OBJ/LIB/DLL/EFI/FFS/FV/FD files.

--ref2
b7d4bf0675b7 (HEAD -> Bug_3737_3739_ApplyUncrustifyChanges_V5) UnitTestFrameworkPkg: Apply uncrusitify changes
7f03d25f60e7 UefiPayloadPkg: Apply uncrusitify changes
0bfd8d9b5ac9 UefiCpuPkg: Apply uncrusitify changes
e1cd9bfb9dea StandaloneMmPkg: Apply uncrusitify changes
5da2f65be378 SourceLevelDebugPkg: Apply uncrusitify changes
95b86de07e5d SignedCapsulePkg: Apply uncrusitify changes
fe71d97246c4 ShellPkg: Apply uncrusitify changes
54c21c952992 SecurityPkg: Apply uncrusitify changes
187a3785f12b RedfishPkg: Apply uncrusitify changes
810100002a46 PcAtChipsetPkg: Apply uncrusitify changes
276a695c0cf2 OvmfPkg: Apply uncrusitify changes
303c0a91ab07 NetworkPkg: Apply uncrusitify changes
bc80792cd1b1 MdePkg: Apply uncrusitify changes
3ea86be17a2a MdeModulePkg: Apply uncrusitify changes
c70ef11ed0cd IntelFsp2WrapperPkg: Apply uncrusitify changes
c0291221f252 IntelFsp2Pkg: Apply uncrusitify changes
6a479952a690 FmpDevicePkg: Apply uncrusitify changes
3a7c05b7070d FatPkg: Apply uncrusitify changes
b789f98c8959 EmulatorPkg: Apply uncrusitify changes
952d7a1c9220 EmbeddedPkg: Apply uncrusitify changes
a1cc9881bab6 DynamicTablesPkg: Apply uncrusitify changes
50654dfe5785 CryptoPkg: Apply uncrusitify changes
ed965a02dfa1 ArmVirtPkg: Apply uncrusitify changes
9744023fbc46 ArmPlatformPkg: Apply uncrusitify changes
7a1cde5f5bba ArmPkg: Apply uncrusitify changes
19d17e0913e8 UefiCpuPkg: Change use of EFI_D_* to DEBUG_*
ffa718b4f994 SourceLevelDebugPkg: Change use of EFI_D_* to DEBUG_*
b86cb3c5e5b4 ShellPkg: Change use of EFI_D_* to DEBUG_*
c7c42204dc07 SecurityPkg: Change use of EFI_D_* to DEBUG_*
16b8e6f958e4 PcAtChipsetPkg: Change use of EFI_D_* to DEBUG_*
0ac3f8b2dac5 OvmfPkg: Change use of EFI_D_* to DEBUG_*
bc5004b8d294 NetworkPkg: Change use of EFI_D_* to DEBUG_*
6f671a8e2377 MdePkg: Change use of EFI_D_* to DEBUG_*
a10c610ff9a3 MdeModulePkg: Change use of EFI_D_* to DEBUG_*
09a3bddba390 FatPkg: Change use of EFI_D_* to DEBUG_*
59c61318246a EmulatorPkg: Change use of EFI_D_* to DEBUG_*
3a80367dda3b EmbeddedPkg: Change use of EFI_D_* to DEBUG_*
23eb1aaf80ca ArmVirtPkg: Change use of EFI_D_* to DEBUG_*
875914b45c54 ArmPlatformPkg: Change use of EFI_D_* to DEBUG_*
eb2eca82b451 ArmPkg: Change use of EFI_D_* to DEBUG_*
f0f3f5aae7c4 (origin/master, origin/HEAD, master) UnitTestFrameworkPkg: Update YAML to ignore specific ECC files/errors
c05734797790 UefiPayloadPkg: Update YAML to ignore specific ECC files/errors
c30c40d6c63d StandaloneMmPkg: Update YAML to ignore specific ECC files/errors
9944508e85f1 ShellPkg: Update YAML to ignore specific ECC files/errors
60fa40be458d SecurityPkg: Update YAML to ignore specific ECC files/errors
df790cd6b37e MdePkg: Update YAML to ignore specific ECC files/errors
9deb9370766e MdeModulePkg: Update YAML to ignore specific ECC files/errors
d7d30e8f219f EmulatorPkg: Update YAML to ignore specific ECC files/errors
d5744ecba813 CryptoPkg: Update YAML to ignore specific ECC files/errors
c97fee87f0f9 ArmVirtPkg: Update YAML to ignore specific ECC files/errors
1939fc9569f2 ArmPlatformPkg: Update YAML to ignore specific ECC files/errors
365dced2c37a ArmPkg: Update YAML to ignore specific ECC files/errors
76a1ce4d5fec .azurepipelines/templates: Update max pipeline job time to 2 hours
99f84ff47390 .pytools/Plugin/LicenseCheck: Use temp directory for git diff output
3019f1bbabf1 .pytool/Plugin/EccCheck: Add performance optimizations
854462bd3479 .pytool/Plugin/EccCheck: Remove temp directory on exception
69877614fdee .pytool/Plugin/EccCheck: Remove RevertCode()
--ref1
ef9a059cdb15 EmulatorPkg/Win/Host: Update CC_FLAGS
bb1bba3d7767 (tag: edk2-stable202111) NetworkPkg: Fix invalid pointer for DNS response token on error

Best regards,

Mike


Wu, Hao A
 

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Michael
D Kinney
Sent: Wednesday, December 1, 2021 6:34 AM
To: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>;
Michael Kubacki <michael.kubacki@microsoft.com>; Andrew Fish
(afish@apple.com) <afish@apple.com>; Leif Lindholm <leif@nuviainc.com>
Subject: [edk2-devel] Uncrustify Conversion Detailed Plan and Extended
Hard Freeze Update #4

Hello,

Thank you for your patience during this extended hard freeze.

Just one more step to go. There has been a delay in the review of the patch
series with the uncrustify source changes. PR(6). This patch series was not
sent out as patch review email because of its very large size. It only contains
source style changes and the CompareBuild tool and GitHub action has
shown there are no binary differences introduced with these source style
changes.

If you are a package maintainer, then please review the following branch/PR
for your package contents and review the EDK II CI results and BuildCompare
results. I do not expect a line by line review because we already had time to
provide feedback on the source style performed by uncrustify. Instead, a
Reviewed-by for your package indicates that you have reviewed the EDK II CI
results and CompareBuild tool functionality and results and you accept the
source style changes to your package.

For SourceLevelDebugPkg:
Reviewed-by: Hao A Wu <hao.a.wu@intel.com>

Best Regards,
Hao Wu



*
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
hanges_V5
* https://github.com/tianocore/edk2/pull/2229
* https://github.com/mdkinney/edk2/actions/runs/1521618836

Additional details on this update below.

Thank you,

Mike


Changes from Update #3
----------------------------------------------------------------------------
* Pushed PR (5)
* Added link to PR(6). EDK II CI Status is PASS. Build Compare PASS.
* Waiting for review of PR (6)
* Review of PR (7) completed and waiting for review of PR (6)
----------------------------------------------------------------------------

Changes from Update #2
----------------------------------------------------------------------------
* Changed order of PRs swapping (4) and (5). The PR that activates
increases the max CI agent job time is independent of all the other
PRs and its review is complete, so it can be committed now.
* Pushed PRs (1), (2), (3), (4).
* Waiting for review to complete for PRs (5) and (6)
* Reviews complete for PR (7)
* Identifies steps using git filter-branch to apply uncrustify changes to a
code review patch series that was generated before the uncrustify changes
avoiding manual merge.
* Identified steps using git filter-repo to generate an alternate history of
the edk2 repo with uncrustify changes applied on every commit. This may
be useful when evaluating changes to files using tools like git blame
without the large uncrustify patch series.
---------------------------------------------------------------------------

Changes from Update #1
----------------------------------------------------------------------------
* Changed order of PRs swapping (6) and (7). The PR that activates
EDK II CI check UncrustifyCheck has to be last because it unconditionally
checks all C/H files in all packages. Not just files that have been
modified like some of the other checkers.
* Updated link to the branch with the UncrustifyCheck plugin that has been
updated with a one line change and Reviewed-by and Tested-by tags.

https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plug
in_v6
* Reviews complete for (1), (2), (3), (5), and (7)
---------------------------------------------------------------------------

Michael Kubacki and I have prepared the patches required to apply the
uncrustify changes and enable EDK II CI to check all submitted patches have
been run through uncrustify.

We have verified through the CompareBuild GitHub Action that the format
changes performed by uncrustify have no functional changes.
All of the OBJ, LIB, DLL, EFI, FFS, FV, and FD files match 100% across 70
VS2019/GCC5 builds of all package/platform DSC files in the edk2 repo.

The hard freeze will be extended after the edk2-stable202111 tag until all
uncrustify related changes are committed. We do not expect this to take
more than a few days. Do not push any PRs until the hard freeze is lifted.

The changes are broken up into 7 patch series/PRs. The PRs are ordered so
they can be submitted using the normal submission process and EDK II CI will
pass for each one. Details are listed below.

Uncrustify 73.0.3 for EDK II
=============================
* Sources: https://dev.azure.com/projectmu/_git/Uncrustify
* Documentation:
https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/
Project-Mu-(EDK-II)-Fork-Readme
* Download:
https://dev.azure.com/projectmu/Uncrustify/_packaging?_a=package&fee
d=mu_uncrustify&package=mu-uncrustify-
release&protocolType=NuGet&version=73.0.3

Installing Uncrustify
======================
The Uncrustify tool is installed automatically when the Pytools environment
is used and the stuart* commands are run to complete the environment
setup. Please see:

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-locally

Uncrustify can also be installed from the download page listed above or built
from sources from the source link above.

The Documentation link provides instruction on how to run uncrustify from
the command line or install as a Visual Studio Code plugin. The main
uncrustify documentation also describes how to integrate with a few other
editors.

We have also discussed a client side githook. That effort has not started.
Let us know if that is a feature you would find useful.

Developer impact for new code reviews
======================================
Once the uncrustify checker is active in EDK II CI, developers must make sure
their patches are run through the uncrustify tool before sending the patches
for review.

Developers must install and run uncrustify against changes files before
sending patch review emails or submitting PR for EDK II CI. If EDK II CI detects
and differences in source formatting, then EDK II CI will fail and the
developer must run uncrustify and resubmit the patches.

Developer impact to patch series/PRs reviewed during edk2-stable201121
soft/hard freeze
==========================================================
=============================
Developers must rebase their changes after the uncrustify source changes
are committed. The branch with a preview of the uncrustify changes can be
used to start this rebase work.


https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
hanges_V5

The following steps can be used to update an existing branch with the
required uncrustify format. This is the Windows version. I will add the Linux
version soon.

1) Fetch and checkout and rebase to latest edk2/master

git fetch origin
git checkout master
git rebase origin/master

2) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

3) Check out the patch series branch (e.g. MyBranch)

git checkout MyBranch

4) Rebase patch series against edk2-stable202111

git rebase edk2-stable202111

5) Create new branch for the uncrustifed version (e.g.
MyBranch_Uncrustified)

git checkout -b MyBranch_Uncrustified

6) Use git filter-branch to uncrustify all the commits in the series
between the rebase target from (2) and HEAD of the branch. A filter
can be used to scope the uncrustify operations to only the C/H files
in the specific package the patch series is against. (e.g.
DynamicTablesPkg).
BaseTools should always be excluded. If the package scoped filter is
not used, it will still work, but will take longer to run because
uncrustify will rescan every C/H files in the whole repo.

git filter-branch --tree-filter "git ls-files DynamicTablesPkg*.c
DynamicTablesPkg*.h :!BaseTools/* | c:\\Temp\\UncrustifyCheck\\mu-
uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c
c:\\Temp\\UncrustifyCheck\\uncrustify.cfg -F - --replace --no-backup --if-
changed" edk2-stable202111..HEAD

7) Now that all the individual patches in the branch are uncrustified,
rebase against latest edk2/master that is already uncrustified.

git rebase master

8) Verify the patches in this new branch.

Impacts to tracing history across the uncrusity changes
=======================================================
Tools the view file and line history do work with the large uncrustify patch
series. One impact is that the operations can be very slow due to the large
uncrustify patches.

One option to provide a faster experience is to provide an alternate version
of the edk2 repository as "documentation" that has the entire history re-
written with uncrustify run on every commit.
The tool called git-filter-repo can be used to perform this transformation and
runs in a reasonable period of time (a few hours)

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-repo/blob/main/contrib/filter-repo-
demos/lint-history

The following steps can be used to perform this transformation.
This is the Windows version. I will add the Linux version soon.

** WARNING ** This operation modifies(rewrites) all the commits
in the local copy of the repo. Do not perform
these steps on a local repo you are using for
active development.

1) Clone edk2 into a new directory (see **WARNING**)

git clone https://github.com/tianocore/edk2.git edk2-uncrustified
cd edk2-uncrustified

2) Setup python virtual env, install pytools, and run stuart commands
to setup build environment which includes installing uncrustify tools.

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-
locally

3) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

4) Use lint-history.py from git-filter-repo examples

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-repo/blob/main/contrib/filter-
repo-demos/lint-history

Line #127 - Add try except around subprocess.check_call() with except
being pass. This is required because there are a few commits of C
files in the edk2 repo that have incorrect C syntax and do not
build with a C compiler and break the uncrustify parser. Skip reformat
of C files that can not be parsed by uncrustify. These rare instances
are addressed in the commit that fixes the C syntax error.

Run this slightly modified version of lint-history. Include only
C/H files and exclude directories that start with 'Tools' or 'BaseTools'.
This step took about 2.2 hours on a laptop.

lint-history.py
--relevant "return (not filename.startswith(b'Tools') and not
filename.startswith(b'BaseTools') and (filename.endswith(b'.c') or
filename.endswith(b'.h')))"
c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\mu-
uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c
c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\uncrustify.cfg --
replace --no-backup --if-changed

Order of PRs to apply during extended hard freeze
==================================================
1) Update EmulatorPkg Win Host [BuildOptions] MSFT CC_FLAGS to not force
debug information
* https://bugzilla.tianocore.org/show_bug.cgi?id=3747
*
https://github.com/mdkinney/edk2/tree/Bug_3747_EmulatorPkg_WinHost_
ReproducibleBuild
* https://github.com/tianocore/edk2/pull/2215
* Required for EmulatorPkg to pass CompareBuild for VS2019 IA32/X64
builds.
* Status: Review complete. PR pushed.

2) EccCheck should not revert staged and local changes
* https://bugzilla.tianocore.org/show_bug.cgi?id=2986
*
https://github.com/mdkinney/edk2/tree/Bug_2986_EccCheckRemoveGitRe
vert_V2
* https://github.com/tianocore/edk2/pull/2216
* Required for EDK II CI to complete in a reasonable period of time when
processing the 4000+ source file style changes made by uncrustify.
* Also fixes critical bugs that can potentially corrupt git state when
EccCheck is run locally.
* Status: Review complete. PR pushed.

3) Update pytool LicenseCheck plugin to use temp directory for diff output
file
* https://bugzilla.tianocore.org/show_bug.cgi?id=3746
*
https://github.com/mdkinney/edk2/tree/Bug_3746_LicenseCheckUseDiffO
utputFile_V2
* https://github.com/tianocore/edk2/pull/2217
* Required to reduce EDK II CI build times.
* Status: Review complete. PR pushed.

4) Update max job time from 60 min to 120 minutes
in .azurepipelines/templates
* https://bugzilla.tianocore.org/show_bug.cgi?id=3750
*
https://github.com/mdkinney/edk2/tree/Bug_3750_IncreaseAzurePipelines
Timeout
* https://github.com/tianocore/edk2/pull/2219
* Required to allow EccCheck of uncrustify changes to complete on Azure
Pipelines CI agents without timing out.
* Status: Review complete. PR pushed.

5) Update Package YAML to ignore specific ECC files/errors
* https://bugzilla.tianocore.org/show_bug.cgi?id=3749
*
https://github.com/mdkinney/edk2/tree/Bug_3749_EccCheckIgnoreFilesErr
ors
* https://github.com/tianocore/edk2/pull/2218
* Required to pass EccCheck
* Status: Review complete. PR pushed

6) Uncrustify Source Changes
* https://bugzilla.tianocore.org/show_bug.cgi?id=3737
* https://bugzilla.tianocore.org/show_bug.cgi?id=3739
*
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
hanges_V5
* https://github.com/tianocore/edk2/pull/2229
* Build comparison result PASS:
https://github.com/mdkinney/edk2/actions/runs/1521618836
* EFI_D_ -> DEBUG changes required to pass PatchCheck
* Uncrustify format changes required to pass UncrustifyCheck
* Status:
Waiting for review

7) UncrustifyCheck EDK II CI Plugin
* https://bugzilla.tianocore.org/show_bug.cgi?id=3748
*
https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plug
in_v6
* Required to enforce all PRs submitted to EDK II CI match uncrustify format.
* Unconditionally checks all packages. Can not be committed until all C/H
source files have been updated.
* Status: Review complete

Combined Branch/PR for Review/Test
==================================
* Build Comparison results must pass 100% across the full set of PRs before
the individual PRs can be pushed in the order listed above.
* Branch:
https://github.com/mdkinney/edk2/tree/TestOnly_Uncrustify_PR_Series
* PR: https://github.com/tianocore/edk2/pull/2229
Status = PASS
* CompareBuild:
Branch:
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
hanges_V5
--ref1: ef9a059cdb15844fe52a49af2bf7d86b9dd3e9bf
--ref2: Bug_3737_3739_ApplyUncrustifyChanges_V5
Extra Options: -n 4 --quiet
Results: https://github.com/mdkinney/edk2/actions/runs/1521618836
30 VS2019 build comparisons PASS
40 GCC5 build comparisons PASS
100% PASS

The following git log shows the set of patches from --ref1 to --ref 2across
which there are no differences in any of the OBJ/LIB/DLL/EFI/FFS/FV/FD files.

--ref2
b7d4bf0675b7 (HEAD -> Bug_3737_3739_ApplyUncrustifyChanges_V5)
UnitTestFrameworkPkg: Apply uncrusitify changes
7f03d25f60e7 UefiPayloadPkg: Apply uncrusitify changes
0bfd8d9b5ac9 UefiCpuPkg: Apply uncrusitify changes
e1cd9bfb9dea StandaloneMmPkg: Apply uncrusitify changes
5da2f65be378 SourceLevelDebugPkg: Apply uncrusitify changes
95b86de07e5d SignedCapsulePkg: Apply uncrusitify changes
fe71d97246c4 ShellPkg: Apply uncrusitify changes
54c21c952992 SecurityPkg: Apply uncrusitify changes
187a3785f12b RedfishPkg: Apply uncrusitify changes
810100002a46 PcAtChipsetPkg: Apply uncrusitify changes
276a695c0cf2 OvmfPkg: Apply uncrusitify changes
303c0a91ab07 NetworkPkg: Apply uncrusitify changes
bc80792cd1b1 MdePkg: Apply uncrusitify changes
3ea86be17a2a MdeModulePkg: Apply uncrusitify changes
c70ef11ed0cd IntelFsp2WrapperPkg: Apply uncrusitify changes
c0291221f252 IntelFsp2Pkg: Apply uncrusitify changes
6a479952a690 FmpDevicePkg: Apply uncrusitify changes
3a7c05b7070d FatPkg: Apply uncrusitify changes
b789f98c8959 EmulatorPkg: Apply uncrusitify changes
952d7a1c9220 EmbeddedPkg: Apply uncrusitify changes
a1cc9881bab6 DynamicTablesPkg: Apply uncrusitify changes
50654dfe5785 CryptoPkg: Apply uncrusitify changes
ed965a02dfa1 ArmVirtPkg: Apply uncrusitify changes
9744023fbc46 ArmPlatformPkg: Apply uncrusitify changes
7a1cde5f5bba ArmPkg: Apply uncrusitify changes
19d17e0913e8 UefiCpuPkg: Change use of EFI_D_* to DEBUG_*
ffa718b4f994 SourceLevelDebugPkg: Change use of EFI_D_* to DEBUG_*
b86cb3c5e5b4 ShellPkg: Change use of EFI_D_* to DEBUG_*
c7c42204dc07 SecurityPkg: Change use of EFI_D_* to DEBUG_*
16b8e6f958e4 PcAtChipsetPkg: Change use of EFI_D_* to DEBUG_*
0ac3f8b2dac5 OvmfPkg: Change use of EFI_D_* to DEBUG_*
bc5004b8d294 NetworkPkg: Change use of EFI_D_* to DEBUG_*
6f671a8e2377 MdePkg: Change use of EFI_D_* to DEBUG_*
a10c610ff9a3 MdeModulePkg: Change use of EFI_D_* to DEBUG_*
09a3bddba390 FatPkg: Change use of EFI_D_* to DEBUG_*
59c61318246a EmulatorPkg: Change use of EFI_D_* to DEBUG_*
3a80367dda3b EmbeddedPkg: Change use of EFI_D_* to DEBUG_*
23eb1aaf80ca ArmVirtPkg: Change use of EFI_D_* to DEBUG_*
875914b45c54 ArmPlatformPkg: Change use of EFI_D_* to DEBUG_*
eb2eca82b451 ArmPkg: Change use of EFI_D_* to DEBUG_*
f0f3f5aae7c4 (origin/master, origin/HEAD, master) UnitTestFrameworkPkg:
Update YAML to ignore specific ECC files/errors
c05734797790 UefiPayloadPkg: Update YAML to ignore specific ECC
files/errors
c30c40d6c63d StandaloneMmPkg: Update YAML to ignore specific ECC
files/errors
9944508e85f1 ShellPkg: Update YAML to ignore specific ECC files/errors
60fa40be458d SecurityPkg: Update YAML to ignore specific ECC files/errors
df790cd6b37e MdePkg: Update YAML to ignore specific ECC files/errors
9deb9370766e MdeModulePkg: Update YAML to ignore specific ECC
files/errors
d7d30e8f219f EmulatorPkg: Update YAML to ignore specific ECC files/errors
d5744ecba813 CryptoPkg: Update YAML to ignore specific ECC files/errors
c97fee87f0f9 ArmVirtPkg: Update YAML to ignore specific ECC files/errors
1939fc9569f2 ArmPlatformPkg: Update YAML to ignore specific ECC
files/errors
365dced2c37a ArmPkg: Update YAML to ignore specific ECC files/errors
76a1ce4d5fec .azurepipelines/templates: Update max pipeline job time to 2
hours
99f84ff47390 .pytools/Plugin/LicenseCheck: Use temp directory for git diff
output
3019f1bbabf1 .pytool/Plugin/EccCheck: Add performance optimizations
854462bd3479 .pytool/Plugin/EccCheck: Remove temp directory on
exception
69877614fdee .pytool/Plugin/EccCheck: Remove RevertCode()
--ref1
ef9a059cdb15 EmulatorPkg/Win/Host: Update CC_FLAGS
bb1bba3d7767 (tag: edk2-stable202111) NetworkPkg: Fix invalid pointer for
DNS response token on error

Best regards,

Mike









Ni, Ray
 

For below packages, Reviewed-by: Ray Ni <ray.ni@intel.com>
* EmulatorPkg
* FatPkg
* PcAtChipsetPkg
* ShellPkg
* UefiCpuPkg
* UefiPayloadPkg

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Michael D Kinney
Sent: Wednesday, December 1, 2021 6:34 AM
To: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>; Michael Kubacki <michael.kubacki@microsoft.com>; Andrew Fish (afish@apple.com) <afish@apple.com>; Leif Lindholm <leif@nuviainc.com>
Subject: [edk2-devel] Uncrustify Conversion Detailed Plan and Extended Hard Freeze Update #4

Hello,

Thank you for your patience during this extended hard freeze.

Just one more step to go. There has been a delay in the review of the patch series with the uncrustify source changes. PR(6). This patch series was not sent out as patch review email because of its very large size. It only contains source style changes and the CompareBuild tool and GitHub action has shown there are no binary differences introduced with these source style changes.

If you are a package maintainer, then please review the following branch/PR for your package contents and review the EDK II CI results and BuildCompare results. I do not expect a line by line review because we already had time to provide feedback on the source style performed by uncrustify. Instead, a Reviewed-by for your package indicates that you have reviewed the EDK II CI results and CompareBuild tool functionality and results and you accept the source style changes to your package.

* https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChanges_V5
* https://github.com/tianocore/edk2/pull/2229
* https://github.com/mdkinney/edk2/actions/runs/1521618836

Additional details on this update below.

Thank you,

Mike


Changes from Update #3
----------------------------------------------------------------------------
* Pushed PR (5)
* Added link to PR(6). EDK II CI Status is PASS. Build Compare PASS.
* Waiting for review of PR (6)
* Review of PR (7) completed and waiting for review of PR (6)
----------------------------------------------------------------------------

Changes from Update #2
----------------------------------------------------------------------------
* Changed order of PRs swapping (4) and (5). The PR that activates
increases the max CI agent job time is independent of all the other
PRs and its review is complete, so it can be committed now.
* Pushed PRs (1), (2), (3), (4).
* Waiting for review to complete for PRs (5) and (6)
* Reviews complete for PR (7)
* Identifies steps using git filter-branch to apply uncrustify changes to a
code review patch series that was generated before the uncrustify changes
avoiding manual merge.
* Identified steps using git filter-repo to generate an alternate history of
the edk2 repo with uncrustify changes applied on every commit. This may
be useful when evaluating changes to files using tools like git blame
without the large uncrustify patch series.
---------------------------------------------------------------------------

Changes from Update #1
----------------------------------------------------------------------------
* Changed order of PRs swapping (6) and (7). The PR that activates
EDK II CI check UncrustifyCheck has to be last because it unconditionally
checks all C/H files in all packages. Not just files that have been
modified like some of the other checkers.
* Updated link to the branch with the UncrustifyCheck plugin that has been
updated with a one line change and Reviewed-by and Tested-by tags.
https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plugin_v6
* Reviews complete for (1), (2), (3), (5), and (7)
---------------------------------------------------------------------------

Michael Kubacki and I have prepared the patches required to apply the uncrustify changes and enable EDK II CI to check all submitted patches have been run through uncrustify.

We have verified through the CompareBuild GitHub Action that the format changes performed by uncrustify have no functional changes.
All of the OBJ, LIB, DLL, EFI, FFS, FV, and FD files match 100% across 70 VS2019/GCC5 builds of all package/platform DSC files in the edk2 repo.

The hard freeze will be extended after the edk2-stable202111 tag until all uncrustify related changes are committed. We do not expect this to take more than a few days. Do not push any PRs until the hard freeze is lifted.

The changes are broken up into 7 patch series/PRs. The PRs are ordered so they can be submitted using the normal submission process and EDK II CI will pass for each one. Details are listed below.

Uncrustify 73.0.3 for EDK II
=============================
* Sources: https://dev.azure.com/projectmu/_git/Uncrustify
* Documentation: https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme
* Download: https://dev.azure.com/projectmu/Uncrustify/_packaging?_a=package&feed=mu_uncrustify&package=mu-uncrustify-release&protocolType=NuGet&version=73.0.3

Installing Uncrustify
======================
The Uncrustify tool is installed automatically when the Pytools environment is used and the stuart* commands are run to complete the environment setup. Please see:

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-locally

Uncrustify can also be installed from the download page listed above or built from sources from the source link above.

The Documentation link provides instruction on how to run uncrustify from the command line or install as a Visual Studio Code plugin. The main uncrustify documentation also describes how to integrate with a few other editors.

We have also discussed a client side githook. That effort has not started.
Let us know if that is a feature you would find useful.

Developer impact for new code reviews
======================================
Once the uncrustify checker is active in EDK II CI, developers must make sure their patches are run through the uncrustify tool before sending the patches for review.

Developers must install and run uncrustify against changes files before sending patch review emails or submitting PR for EDK II CI. If EDK II CI detects and differences in source formatting, then EDK II CI will fail and the developer must run uncrustify and resubmit the patches.

Developer impact to patch series/PRs reviewed during edk2-stable201121 soft/hard freeze =======================================================================================
Developers must rebase their changes after the uncrustify source changes are committed. The branch with a preview of the uncrustify changes can be used to start this rebase work.

https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChanges_V5

The following steps can be used to update an existing branch with the required uncrustify format. This is the Windows version. I will add the Linux version soon.

1) Fetch and checkout and rebase to latest edk2/master

git fetch origin
git checkout master
git rebase origin/master

2) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

3) Check out the patch series branch (e.g. MyBranch)

git checkout MyBranch

4) Rebase patch series against edk2-stable202111

git rebase edk2-stable202111

5) Create new branch for the uncrustifed version (e.g. MyBranch_Uncrustified)

git checkout -b MyBranch_Uncrustified

6) Use git filter-branch to uncrustify all the commits in the series
between the rebase target from (2) and HEAD of the branch. A filter
can be used to scope the uncrustify operations to only the C/H files
in the specific package the patch series is against. (e.g. DynamicTablesPkg).
BaseTools should always be excluded. If the package scoped filter is
not used, it will still work, but will take longer to run because
uncrustify will rescan every C/H files in the whole repo.

git filter-branch --tree-filter "git ls-files DynamicTablesPkg*.c DynamicTablesPkg*.h :!BaseTools/* | c:\\Temp\\UncrustifyCheck\\mu-uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c c:\\Temp\\UncrustifyCheck\\uncrustify.cfg -F - --replace --no-backup --if-changed" edk2-stable202111..HEAD

7) Now that all the individual patches in the branch are uncrustified,
rebase against latest edk2/master that is already uncrustified.

git rebase master

8) Verify the patches in this new branch.

Impacts to tracing history across the uncrusity changes =======================================================
Tools the view file and line history do work with the large uncrustify patch series. One impact is that the operations can be very slow due to the large uncrustify patches.

One option to provide a faster experience is to provide an alternate version of the edk2 repository as "documentation" that has the entire history re-written with uncrustify run on every commit.
The tool called git-filter-repo can be used to perform this transformation and runs in a reasonable period of time (a few hours)

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-repo/blob/main/contrib/filter-repo-demos/lint-history

The following steps can be used to perform this transformation.
This is the Windows version. I will add the Linux version soon.

** WARNING ** This operation modifies(rewrites) all the commits
in the local copy of the repo. Do not perform
these steps on a local repo you are using for
active development.

1) Clone edk2 into a new directory (see **WARNING**)

git clone https://github.com/tianocore/edk2.git edk2-uncrustified
cd edk2-uncrustified

2) Setup python virtual env, install pytools, and run stuart commands
to setup build environment which includes installing uncrustify tools.

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-locally

3) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

4) Use lint-history.py from git-filter-repo examples

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-repo/blob/main/contrib/filter-repo-demos/lint-history

Line #127 - Add try except around subprocess.check_call() with except
being pass. This is required because there are a few commits of C
files in the edk2 repo that have incorrect C syntax and do not
build with a C compiler and break the uncrustify parser. Skip reformat
of C files that can not be parsed by uncrustify. These rare instances
are addressed in the commit that fixes the C syntax error.

Run this slightly modified version of lint-history. Include only
C/H files and exclude directories that start with 'Tools' or 'BaseTools'.
This step took about 2.2 hours on a laptop.

lint-history.py
--relevant "return (not filename.startswith(b'Tools') and not filename.startswith(b'BaseTools') and (filename.endswith(b'.c') or filename.endswith(b'.h')))"
c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\mu-uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\uncrustify.cfg --replace --no-backup --if-changed

Order of PRs to apply during extended hard freeze ==================================================
1) Update EmulatorPkg Win Host [BuildOptions] MSFT CC_FLAGS to not force debug information
* https://bugzilla.tianocore.org/show_bug.cgi?id=3747
* https://github.com/mdkinney/edk2/tree/Bug_3747_EmulatorPkg_WinHost_ReproducibleBuild
* https://github.com/tianocore/edk2/pull/2215
* Required for EmulatorPkg to pass CompareBuild for VS2019 IA32/X64 builds.
* Status: Review complete. PR pushed.

2) EccCheck should not revert staged and local changes
* https://bugzilla.tianocore.org/show_bug.cgi?id=2986
* https://github.com/mdkinney/edk2/tree/Bug_2986_EccCheckRemoveGitRevert_V2
* https://github.com/tianocore/edk2/pull/2216
* Required for EDK II CI to complete in a reasonable period of time when
processing the 4000+ source file style changes made by uncrustify.
* Also fixes critical bugs that can potentially corrupt git state when
EccCheck is run locally.
* Status: Review complete. PR pushed.

3) Update pytool LicenseCheck plugin to use temp directory for diff output file
* https://bugzilla.tianocore.org/show_bug.cgi?id=3746
* https://github.com/mdkinney/edk2/tree/Bug_3746_LicenseCheckUseDiffOutputFile_V2
* https://github.com/tianocore/edk2/pull/2217
* Required to reduce EDK II CI build times.
* Status: Review complete. PR pushed.

4) Update max job time from 60 min to 120 minutes in .azurepipelines/templates
* https://bugzilla.tianocore.org/show_bug.cgi?id=3750
* https://github.com/mdkinney/edk2/tree/Bug_3750_IncreaseAzurePipelinesTimeout
* https://github.com/tianocore/edk2/pull/2219
* Required to allow EccCheck of uncrustify changes to complete on Azure
Pipelines CI agents without timing out.
* Status: Review complete. PR pushed.

5) Update Package YAML to ignore specific ECC files/errors
* https://bugzilla.tianocore.org/show_bug.cgi?id=3749
* https://github.com/mdkinney/edk2/tree/Bug_3749_EccCheckIgnoreFilesErrors
* https://github.com/tianocore/edk2/pull/2218
* Required to pass EccCheck
* Status: Review complete. PR pushed

6) Uncrustify Source Changes
* https://bugzilla.tianocore.org/show_bug.cgi?id=3737
* https://bugzilla.tianocore.org/show_bug.cgi?id=3739
* https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChanges_V5
* https://github.com/tianocore/edk2/pull/2229
* Build comparison result PASS: https://github.com/mdkinney/edk2/actions/runs/1521618836
* EFI_D_ -> DEBUG changes required to pass PatchCheck
* Uncrustify format changes required to pass UncrustifyCheck
* Status:
Waiting for review

7) UncrustifyCheck EDK II CI Plugin
* https://bugzilla.tianocore.org/show_bug.cgi?id=3748
* https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plugin_v6
* Required to enforce all PRs submitted to EDK II CI match uncrustify format.
* Unconditionally checks all packages. Can not be committed until all C/H
source files have been updated.
* Status: Review complete

Combined Branch/PR for Review/Test
==================================
* Build Comparison results must pass 100% across the full set of PRs before
the individual PRs can be pushed in the order listed above.
* Branch: https://github.com/mdkinney/edk2/tree/TestOnly_Uncrustify_PR_Series
* PR: https://github.com/tianocore/edk2/pull/2229
Status = PASS
* CompareBuild:
Branch: https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChanges_V5
--ref1: ef9a059cdb15844fe52a49af2bf7d86b9dd3e9bf
--ref2: Bug_3737_3739_ApplyUncrustifyChanges_V5
Extra Options: -n 4 --quiet
Results: https://github.com/mdkinney/edk2/actions/runs/1521618836
30 VS2019 build comparisons PASS
40 GCC5 build comparisons PASS
100% PASS

The following git log shows the set of patches from --ref1 to --ref 2across which there are no differences in any of the OBJ/LIB/DLL/EFI/FFS/FV/FD files.

--ref2
b7d4bf0675b7 (HEAD -> Bug_3737_3739_ApplyUncrustifyChanges_V5) UnitTestFrameworkPkg: Apply uncrusitify changes
7f03d25f60e7 UefiPayloadPkg: Apply uncrusitify changes
0bfd8d9b5ac9 UefiCpuPkg: Apply uncrusitify changes
e1cd9bfb9dea StandaloneMmPkg: Apply uncrusitify changes
5da2f65be378 SourceLevelDebugPkg: Apply uncrusitify changes
95b86de07e5d SignedCapsulePkg: Apply uncrusitify changes
fe71d97246c4 ShellPkg: Apply uncrusitify changes
54c21c952992 SecurityPkg: Apply uncrusitify changes
187a3785f12b RedfishPkg: Apply uncrusitify changes
810100002a46 PcAtChipsetPkg: Apply uncrusitify changes
276a695c0cf2 OvmfPkg: Apply uncrusitify changes
303c0a91ab07 NetworkPkg: Apply uncrusitify changes
bc80792cd1b1 MdePkg: Apply uncrusitify changes
3ea86be17a2a MdeModulePkg: Apply uncrusitify changes
c70ef11ed0cd IntelFsp2WrapperPkg: Apply uncrusitify changes
c0291221f252 IntelFsp2Pkg: Apply uncrusitify changes
6a479952a690 FmpDevicePkg: Apply uncrusitify changes
3a7c05b7070d FatPkg: Apply uncrusitify changes
b789f98c8959 EmulatorPkg: Apply uncrusitify changes
952d7a1c9220 EmbeddedPkg: Apply uncrusitify changes
a1cc9881bab6 DynamicTablesPkg: Apply uncrusitify changes
50654dfe5785 CryptoPkg: Apply uncrusitify changes
ed965a02dfa1 ArmVirtPkg: Apply uncrusitify changes
9744023fbc46 ArmPlatformPkg: Apply uncrusitify changes
7a1cde5f5bba ArmPkg: Apply uncrusitify changes
19d17e0913e8 UefiCpuPkg: Change use of EFI_D_* to DEBUG_*
ffa718b4f994 SourceLevelDebugPkg: Change use of EFI_D_* to DEBUG_*
b86cb3c5e5b4 ShellPkg: Change use of EFI_D_* to DEBUG_*
c7c42204dc07 SecurityPkg: Change use of EFI_D_* to DEBUG_*
16b8e6f958e4 PcAtChipsetPkg: Change use of EFI_D_* to DEBUG_*
0ac3f8b2dac5 OvmfPkg: Change use of EFI_D_* to DEBUG_*
bc5004b8d294 NetworkPkg: Change use of EFI_D_* to DEBUG_*
6f671a8e2377 MdePkg: Change use of EFI_D_* to DEBUG_*
a10c610ff9a3 MdeModulePkg: Change use of EFI_D_* to DEBUG_*
09a3bddba390 FatPkg: Change use of EFI_D_* to DEBUG_*
59c61318246a EmulatorPkg: Change use of EFI_D_* to DEBUG_*
3a80367dda3b EmbeddedPkg: Change use of EFI_D_* to DEBUG_*
23eb1aaf80ca ArmVirtPkg: Change use of EFI_D_* to DEBUG_*
875914b45c54 ArmPlatformPkg: Change use of EFI_D_* to DEBUG_*
eb2eca82b451 ArmPkg: Change use of EFI_D_* to DEBUG_*
f0f3f5aae7c4 (origin/master, origin/HEAD, master) UnitTestFrameworkPkg: Update YAML to ignore specific ECC files/errors
c05734797790 UefiPayloadPkg: Update YAML to ignore specific ECC files/errors
c30c40d6c63d StandaloneMmPkg: Update YAML to ignore specific ECC files/errors
9944508e85f1 ShellPkg: Update YAML to ignore specific ECC files/errors
60fa40be458d SecurityPkg: Update YAML to ignore specific ECC files/errors
df790cd6b37e MdePkg: Update YAML to ignore specific ECC files/errors
9deb9370766e MdeModulePkg: Update YAML to ignore specific ECC files/errors
d7d30e8f219f EmulatorPkg: Update YAML to ignore specific ECC files/errors
d5744ecba813 CryptoPkg: Update YAML to ignore specific ECC files/errors
c97fee87f0f9 ArmVirtPkg: Update YAML to ignore specific ECC files/errors
1939fc9569f2 ArmPlatformPkg: Update YAML to ignore specific ECC files/errors
365dced2c37a ArmPkg: Update YAML to ignore specific ECC files/errors
76a1ce4d5fec .azurepipelines/templates: Update max pipeline job time to 2 hours
99f84ff47390 .pytools/Plugin/LicenseCheck: Use temp directory for git diff output
3019f1bbabf1 .pytool/Plugin/EccCheck: Add performance optimizations
854462bd3479 .pytool/Plugin/EccCheck: Remove temp directory on exception
69877614fdee .pytool/Plugin/EccCheck: Remove RevertCode()
--ref1
ef9a059cdb15 EmulatorPkg/Win/Host: Update CC_FLAGS
bb1bba3d7767 (tag: edk2-stable202111) NetworkPkg: Fix invalid pointer for DNS response token on error

Best regards,

Mike


Chiu, Chasel
 

For below patches, Reviewed-by: Chasel Chiu <chasel.chiu@intel.com>

Thanks,
Chasel

IntelFsp2Pkg: Apply uncrusitify changes
IntelFsp2WrapperPkg: Apply uncrusitify changes

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Michael D
Kinney
Sent: Wednesday, December 1, 2021 6:34 AM
To: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>;
Michael Kubacki <michael.kubacki@microsoft.com>; Andrew Fish
(afish@apple.com) <afish@apple.com>; Leif Lindholm <leif@nuviainc.com>
Subject: [edk2-devel] Uncrustify Conversion Detailed Plan and Extended Hard
Freeze Update #4

Hello,

Thank you for your patience during this extended hard freeze.

Just one more step to go. There has been a delay in the review of the patch
series with the uncrustify source changes. PR(6). This patch series was not sent
out as patch review email because of its very large size. It only contains source
style changes and the CompareBuild tool and GitHub action has shown there are
no binary differences introduced with these source style changes.

If you are a package maintainer, then please review the following branch/PR for
your package contents and review the EDK II CI results and BuildCompare results.
I do not expect a line by line review because we already had time to provide
feedback on the source style performed by uncrustify. Instead, a Reviewed-by
for your package indicates that you have reviewed the EDK II CI results and
CompareBuild tool functionality and results and you accept the source style
changes to your package.

*
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChan
ges_V5
* https://github.com/tianocore/edk2/pull/2229
* https://github.com/mdkinney/edk2/actions/runs/1521618836

Additional details on this update below.

Thank you,

Mike


Changes from Update #3
----------------------------------------------------------------------------
* Pushed PR (5)
* Added link to PR(6). EDK II CI Status is PASS. Build Compare PASS.
* Waiting for review of PR (6)
* Review of PR (7) completed and waiting for review of PR (6)
----------------------------------------------------------------------------

Changes from Update #2
----------------------------------------------------------------------------
* Changed order of PRs swapping (4) and (5). The PR that activates
increases the max CI agent job time is independent of all the other
PRs and its review is complete, so it can be committed now.
* Pushed PRs (1), (2), (3), (4).
* Waiting for review to complete for PRs (5) and (6)
* Reviews complete for PR (7)
* Identifies steps using git filter-branch to apply uncrustify changes to a
code review patch series that was generated before the uncrustify changes
avoiding manual merge.
* Identified steps using git filter-repo to generate an alternate history of
the edk2 repo with uncrustify changes applied on every commit. This may
be useful when evaluating changes to files using tools like git blame
without the large uncrustify patch series.
---------------------------------------------------------------------------

Changes from Update #1
----------------------------------------------------------------------------
* Changed order of PRs swapping (6) and (7). The PR that activates
EDK II CI check UncrustifyCheck has to be last because it unconditionally
checks all C/H files in all packages. Not just files that have been
modified like some of the other checkers.
* Updated link to the branch with the UncrustifyCheck plugin that has been
updated with a one line change and Reviewed-by and Tested-by tags.

https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plugin_v
6
* Reviews complete for (1), (2), (3), (5), and (7)
---------------------------------------------------------------------------

Michael Kubacki and I have prepared the patches required to apply the uncrustify
changes and enable EDK II CI to check all submitted patches have been run
through uncrustify.

We have verified through the CompareBuild GitHub Action that the format
changes performed by uncrustify have no functional changes.
All of the OBJ, LIB, DLL, EFI, FFS, FV, and FD files match 100% across 70
VS2019/GCC5 builds of all package/platform DSC files in the edk2 repo.

The hard freeze will be extended after the edk2-stable202111 tag until all
uncrustify related changes are committed. We do not expect this to take more
than a few days. Do not push any PRs until the hard freeze is lifted.

The changes are broken up into 7 patch series/PRs. The PRs are ordered so they
can be submitted using the normal submission process and EDK II CI will pass for
each one. Details are listed below.

Uncrustify 73.0.3 for EDK II
=============================
* Sources: https://dev.azure.com/projectmu/_git/Uncrustify
* Documentation:
https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Proj
ect-Mu-(EDK-II)-Fork-Readme
* Download:
https://dev.azure.com/projectmu/Uncrustify/_packaging?_a=package&feed=m
u_uncrustify&package=mu-uncrustify-
release&protocolType=NuGet&version=73.0.3

Installing Uncrustify
======================
The Uncrustify tool is installed automatically when the Pytools environment is
used and the stuart* commands are run to complete the environment setup.
Please see:

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-locally

Uncrustify can also be installed from the download page listed above or built
from sources from the source link above.

The Documentation link provides instruction on how to run uncrustify from the
command line or install as a Visual Studio Code plugin. The main uncrustify
documentation also describes how to integrate with a few other editors.

We have also discussed a client side githook. That effort has not started.
Let us know if that is a feature you would find useful.

Developer impact for new code reviews
======================================
Once the uncrustify checker is active in EDK II CI, developers must make sure
their patches are run through the uncrustify tool before sending the patches for
review.

Developers must install and run uncrustify against changes files before sending
patch review emails or submitting PR for EDK II CI. If EDK II CI detects and
differences in source formatting, then EDK II CI will fail and the developer must
run uncrustify and resubmit the patches.

Developer impact to patch series/PRs reviewed during edk2-stable201121
soft/hard freeze
=================================================================
======================
Developers must rebase their changes after the uncrustify source changes are
committed. The branch with a preview of the uncrustify changes can be used to
start this rebase work.


https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChan
ges_V5

The following steps can be used to update an existing branch with the required
uncrustify format. This is the Windows version. I will add the Linux version soon.

1) Fetch and checkout and rebase to latest edk2/master

git fetch origin
git checkout master
git rebase origin/master

2) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

3) Check out the patch series branch (e.g. MyBranch)

git checkout MyBranch

4) Rebase patch series against edk2-stable202111

git rebase edk2-stable202111

5) Create new branch for the uncrustifed version (e.g. MyBranch_Uncrustified)

git checkout -b MyBranch_Uncrustified

6) Use git filter-branch to uncrustify all the commits in the series
between the rebase target from (2) and HEAD of the branch. A filter
can be used to scope the uncrustify operations to only the C/H files
in the specific package the patch series is against. (e.g. DynamicTablesPkg).
BaseTools should always be excluded. If the package scoped filter is
not used, it will still work, but will take longer to run because
uncrustify will rescan every C/H files in the whole repo.

git filter-branch --tree-filter "git ls-files DynamicTablesPkg*.c
DynamicTablesPkg*.h :!BaseTools/* | c:\\Temp\\UncrustifyCheck\\mu-
uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c
c:\\Temp\\UncrustifyCheck\\uncrustify.cfg -F - --replace --no-backup --if-
changed" edk2-stable202111..HEAD

7) Now that all the individual patches in the branch are uncrustified,
rebase against latest edk2/master that is already uncrustified.

git rebase master

8) Verify the patches in this new branch.

Impacts to tracing history across the uncrusity changes
=======================================================
Tools the view file and line history do work with the large uncrustify patch series.
One impact is that the operations can be very slow due to the large uncrustify
patches.

One option to provide a faster experience is to provide an alternate version of
the edk2 repository as "documentation" that has the entire history re-written
with uncrustify run on every commit.
The tool called git-filter-repo can be used to perform this transformation and
runs in a reasonable period of time (a few hours)

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-repo/blob/main/contrib/filter-repo-
demos/lint-history

The following steps can be used to perform this transformation.
This is the Windows version. I will add the Linux version soon.

** WARNING ** This operation modifies(rewrites) all the commits
in the local copy of the repo. Do not perform
these steps on a local repo you are using for
active development.

1) Clone edk2 into a new directory (see **WARNING**)

git clone https://github.com/tianocore/edk2.git edk2-uncrustified
cd edk2-uncrustified

2) Setup python virtual env, install pytools, and run stuart commands
to setup build environment which includes installing uncrustify tools.

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-
locally

3) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

4) Use lint-history.py from git-filter-repo examples

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-repo/blob/main/contrib/filter-repo-
demos/lint-history

Line #127 - Add try except around subprocess.check_call() with except
being pass. This is required because there are a few commits of C
files in the edk2 repo that have incorrect C syntax and do not
build with a C compiler and break the uncrustify parser. Skip reformat
of C files that can not be parsed by uncrustify. These rare instances
are addressed in the commit that fixes the C syntax error.

Run this slightly modified version of lint-history. Include only
C/H files and exclude directories that start with 'Tools' or 'BaseTools'.
This step took about 2.2 hours on a laptop.

lint-history.py
--relevant "return (not filename.startswith(b'Tools') and not
filename.startswith(b'BaseTools') and (filename.endswith(b'.c') or
filename.endswith(b'.h')))"
c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\mu-uncrustify-
release_extdep\\Windows-x86\\uncrustify.exe -c
c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\uncrustify.cfg --replace --
no-backup --if-changed

Order of PRs to apply during extended hard freeze
==================================================
1) Update EmulatorPkg Win Host [BuildOptions] MSFT CC_FLAGS to not force
debug information
* https://bugzilla.tianocore.org/show_bug.cgi?id=3747
*
https://github.com/mdkinney/edk2/tree/Bug_3747_EmulatorPkg_WinHost_Rep
roducibleBuild
* https://github.com/tianocore/edk2/pull/2215
* Required for EmulatorPkg to pass CompareBuild for VS2019 IA32/X64 builds.
* Status: Review complete. PR pushed.

2) EccCheck should not revert staged and local changes
* https://bugzilla.tianocore.org/show_bug.cgi?id=2986
*
https://github.com/mdkinney/edk2/tree/Bug_2986_EccCheckRemoveGitRevert
_V2
* https://github.com/tianocore/edk2/pull/2216
* Required for EDK II CI to complete in a reasonable period of time when
processing the 4000+ source file style changes made by uncrustify.
* Also fixes critical bugs that can potentially corrupt git state when
EccCheck is run locally.
* Status: Review complete. PR pushed.

3) Update pytool LicenseCheck plugin to use temp directory for diff output file
* https://bugzilla.tianocore.org/show_bug.cgi?id=3746
*
https://github.com/mdkinney/edk2/tree/Bug_3746_LicenseCheckUseDiffOutput
File_V2
* https://github.com/tianocore/edk2/pull/2217
* Required to reduce EDK II CI build times.
* Status: Review complete. PR pushed.

4) Update max job time from 60 min to 120 minutes in .azurepipelines/templates
* https://bugzilla.tianocore.org/show_bug.cgi?id=3750
*
https://github.com/mdkinney/edk2/tree/Bug_3750_IncreaseAzurePipelinesTime
out
* https://github.com/tianocore/edk2/pull/2219
* Required to allow EccCheck of uncrustify changes to complete on Azure
Pipelines CI agents without timing out.
* Status: Review complete. PR pushed.

5) Update Package YAML to ignore specific ECC files/errors
* https://bugzilla.tianocore.org/show_bug.cgi?id=3749
*
https://github.com/mdkinney/edk2/tree/Bug_3749_EccCheckIgnoreFilesErrors
* https://github.com/tianocore/edk2/pull/2218
* Required to pass EccCheck
* Status: Review complete. PR pushed

6) Uncrustify Source Changes
* https://bugzilla.tianocore.org/show_bug.cgi?id=3737
* https://bugzilla.tianocore.org/show_bug.cgi?id=3739
*
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChan
ges_V5
* https://github.com/tianocore/edk2/pull/2229
* Build comparison result PASS:
https://github.com/mdkinney/edk2/actions/runs/1521618836
* EFI_D_ -> DEBUG changes required to pass PatchCheck
* Uncrustify format changes required to pass UncrustifyCheck
* Status:
Waiting for review

7) UncrustifyCheck EDK II CI Plugin
* https://bugzilla.tianocore.org/show_bug.cgi?id=3748
*
https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plugin_v
6
* Required to enforce all PRs submitted to EDK II CI match uncrustify format.
* Unconditionally checks all packages. Can not be committed until all C/H
source files have been updated.
* Status: Review complete

Combined Branch/PR for Review/Test
==================================
* Build Comparison results must pass 100% across the full set of PRs before
the individual PRs can be pushed in the order listed above.
* Branch:
https://github.com/mdkinney/edk2/tree/TestOnly_Uncrustify_PR_Series
* PR: https://github.com/tianocore/edk2/pull/2229
Status = PASS
* CompareBuild:
Branch:
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChan
ges_V5
--ref1: ef9a059cdb15844fe52a49af2bf7d86b9dd3e9bf
--ref2: Bug_3737_3739_ApplyUncrustifyChanges_V5
Extra Options: -n 4 --quiet
Results: https://github.com/mdkinney/edk2/actions/runs/1521618836
30 VS2019 build comparisons PASS
40 GCC5 build comparisons PASS
100% PASS

The following git log shows the set of patches from --ref1 to --ref 2across which
there are no differences in any of the OBJ/LIB/DLL/EFI/FFS/FV/FD files.

--ref2
b7d4bf0675b7 (HEAD -> Bug_3737_3739_ApplyUncrustifyChanges_V5)
UnitTestFrameworkPkg: Apply uncrusitify changes
7f03d25f60e7 UefiPayloadPkg: Apply uncrusitify changes
0bfd8d9b5ac9 UefiCpuPkg: Apply uncrusitify changes
e1cd9bfb9dea StandaloneMmPkg: Apply uncrusitify changes
5da2f65be378 SourceLevelDebugPkg: Apply uncrusitify changes
95b86de07e5d SignedCapsulePkg: Apply uncrusitify changes
fe71d97246c4 ShellPkg: Apply uncrusitify changes
54c21c952992 SecurityPkg: Apply uncrusitify changes
187a3785f12b RedfishPkg: Apply uncrusitify changes
810100002a46 PcAtChipsetPkg: Apply uncrusitify changes
276a695c0cf2 OvmfPkg: Apply uncrusitify changes
303c0a91ab07 NetworkPkg: Apply uncrusitify changes
bc80792cd1b1 MdePkg: Apply uncrusitify changes
3ea86be17a2a MdeModulePkg: Apply uncrusitify changes
c70ef11ed0cd IntelFsp2WrapperPkg: Apply uncrusitify changes
c0291221f252 IntelFsp2Pkg: Apply uncrusitify changes
6a479952a690 FmpDevicePkg: Apply uncrusitify changes
3a7c05b7070d FatPkg: Apply uncrusitify changes
b789f98c8959 EmulatorPkg: Apply uncrusitify changes
952d7a1c9220 EmbeddedPkg: Apply uncrusitify changes
a1cc9881bab6 DynamicTablesPkg: Apply uncrusitify changes
50654dfe5785 CryptoPkg: Apply uncrusitify changes
ed965a02dfa1 ArmVirtPkg: Apply uncrusitify changes
9744023fbc46 ArmPlatformPkg: Apply uncrusitify changes
7a1cde5f5bba ArmPkg: Apply uncrusitify changes
19d17e0913e8 UefiCpuPkg: Change use of EFI_D_* to DEBUG_*
ffa718b4f994 SourceLevelDebugPkg: Change use of EFI_D_* to DEBUG_*
b86cb3c5e5b4 ShellPkg: Change use of EFI_D_* to DEBUG_*
c7c42204dc07 SecurityPkg: Change use of EFI_D_* to DEBUG_*
16b8e6f958e4 PcAtChipsetPkg: Change use of EFI_D_* to DEBUG_*
0ac3f8b2dac5 OvmfPkg: Change use of EFI_D_* to DEBUG_*
bc5004b8d294 NetworkPkg: Change use of EFI_D_* to DEBUG_*
6f671a8e2377 MdePkg: Change use of EFI_D_* to DEBUG_*
a10c610ff9a3 MdeModulePkg: Change use of EFI_D_* to DEBUG_*
09a3bddba390 FatPkg: Change use of EFI_D_* to DEBUG_*
59c61318246a EmulatorPkg: Change use of EFI_D_* to DEBUG_*
3a80367dda3b EmbeddedPkg: Change use of EFI_D_* to DEBUG_*
23eb1aaf80ca ArmVirtPkg: Change use of EFI_D_* to DEBUG_*
875914b45c54 ArmPlatformPkg: Change use of EFI_D_* to DEBUG_*
eb2eca82b451 ArmPkg: Change use of EFI_D_* to DEBUG_*
f0f3f5aae7c4 (origin/master, origin/HEAD, master) UnitTestFrameworkPkg:
Update YAML to ignore specific ECC files/errors
c05734797790 UefiPayloadPkg: Update YAML to ignore specific ECC files/errors
c30c40d6c63d StandaloneMmPkg: Update YAML to ignore specific ECC
files/errors
9944508e85f1 ShellPkg: Update YAML to ignore specific ECC files/errors
60fa40be458d SecurityPkg: Update YAML to ignore specific ECC files/errors
df790cd6b37e MdePkg: Update YAML to ignore specific ECC files/errors
9deb9370766e MdeModulePkg: Update YAML to ignore specific ECC files/errors
d7d30e8f219f EmulatorPkg: Update YAML to ignore specific ECC files/errors
d5744ecba813 CryptoPkg: Update YAML to ignore specific ECC files/errors
c97fee87f0f9 ArmVirtPkg: Update YAML to ignore specific ECC files/errors
1939fc9569f2 ArmPlatformPkg: Update YAML to ignore specific ECC files/errors
365dced2c37a ArmPkg: Update YAML to ignore specific ECC files/errors
76a1ce4d5fec .azurepipelines/templates: Update max pipeline job time to 2
hours
99f84ff47390 .pytools/Plugin/LicenseCheck: Use temp directory for git diff
output
3019f1bbabf1 .pytool/Plugin/EccCheck: Add performance optimizations
854462bd3479 .pytool/Plugin/EccCheck: Remove temp directory on exception
69877614fdee .pytool/Plugin/EccCheck: Remove RevertCode()
--ref1
ef9a059cdb15 EmulatorPkg/Win/Host: Update CC_FLAGS
bb1bba3d7767 (tag: edk2-stable202111) NetworkPkg: Fix invalid pointer for DNS
response token on error

Best regards,

Mike









Wang, Jian J
 

For SecurityPkg and CryptoPkg,

Reviewed-by: Jian J Wang <jian.j.wang@intel.com>

Regards,
Jian

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Michael D
Kinney
Sent: Wednesday, December 01, 2021 6:34 AM
To: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>;
Michael Kubacki <michael.kubacki@microsoft.com>; Andrew Fish
(afish@apple.com) <afish@apple.com>; Leif Lindholm <leif@nuviainc.com>
Subject: [edk2-devel] Uncrustify Conversion Detailed Plan and Extended Hard
Freeze Update #4

Hello,

Thank you for your patience during this extended hard freeze.

Just one more step to go. There has been a delay in the review of
the patch series with the uncrustify source changes. PR(6). This
patch series was not sent out as patch review email because of its
very large size. It only contains source style changes and the
CompareBuild tool and GitHub action has shown there are no binary
differences introduced with these source style changes.

If you are a package maintainer, then please review the following
branch/PR for your package contents and review the EDK II CI results
and BuildCompare results. I do not expect a line by line review
because we already had time to provide feedback on the source style
performed by uncrustify. Instead, a Reviewed-by for your package
indicates that you have reviewed the EDK II CI results and CompareBuild
tool functionality and results and you accept the source style
changes to your package.

*
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChan
ges_V5
* https://github.com/tianocore/edk2/pull/2229
* https://github.com/mdkinney/edk2/actions/runs/1521618836

Additional details on this update below.

Thank you,

Mike


Changes from Update #3
----------------------------------------------------------------------------
* Pushed PR (5)
* Added link to PR(6). EDK II CI Status is PASS. Build Compare PASS.
* Waiting for review of PR (6)
* Review of PR (7) completed and waiting for review of PR (6)
----------------------------------------------------------------------------

Changes from Update #2
----------------------------------------------------------------------------
* Changed order of PRs swapping (4) and (5). The PR that activates
increases the max CI agent job time is independent of all the other
PRs and its review is complete, so it can be committed now.
* Pushed PRs (1), (2), (3), (4).
* Waiting for review to complete for PRs (5) and (6)
* Reviews complete for PR (7)
* Identifies steps using git filter-branch to apply uncrustify changes to a
code review patch series that was generated before the uncrustify changes
avoiding manual merge.
* Identified steps using git filter-repo to generate an alternate history of
the edk2 repo with uncrustify changes applied on every commit. This may
be useful when evaluating changes to files using tools like git blame
without the large uncrustify patch series.
---------------------------------------------------------------------------

Changes from Update #1
----------------------------------------------------------------------------
* Changed order of PRs swapping (6) and (7). The PR that activates
EDK II CI check UncrustifyCheck has to be last because it unconditionally
checks all C/H files in all packages. Not just files that have been
modified like some of the other checkers.
* Updated link to the branch with the UncrustifyCheck plugin that has been
updated with a one line change and Reviewed-by and Tested-by tags.

https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plugin_v
6
* Reviews complete for (1), (2), (3), (5), and (7)
---------------------------------------------------------------------------

Michael Kubacki and I have prepared the patches required to apply the
uncrustify changes and enable EDK II CI to check all submitted
patches have been run through uncrustify.

We have verified through the CompareBuild GitHub Action that the
format changes performed by uncrustify have no functional changes.
All of the OBJ, LIB, DLL, EFI, FFS, FV, and FD files match 100%
across 70 VS2019/GCC5 builds of all package/platform DSC files in
the edk2 repo.

The hard freeze will be extended after the edk2-stable202111 tag until
all uncrustify related changes are committed. We do not expect this
to take more than a few days. Do not push any PRs until the hard
freeze is lifted.

The changes are broken up into 7 patch series/PRs. The PRs are ordered
so they can be submitted using the normal submission process and EDK II
CI will pass for each one. Details are listed below.

Uncrustify 73.0.3 for EDK II
=============================
* Sources: https://dev.azure.com/projectmu/_git/Uncrustify
* Documentation:
https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Proj
ect-Mu-(EDK-II)-Fork-Readme
* Download:
https://dev.azure.com/projectmu/Uncrustify/_packaging?_a=package&feed=m
u_uncrustify&package=mu-uncrustify-
release&protocolType=NuGet&version=73.0.3

Installing Uncrustify
======================
The Uncrustify tool is installed automatically when the Pytools
environment is used and the stuart* commands are run to complete the
environment setup. Please see:

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-locally

Uncrustify can also be installed from the download page listed above
or built from sources from the source link above.

The Documentation link provides instruction on how to run uncrustify from
the command line or install as a Visual Studio Code plugin. The main
uncrustify documentation also describes how to integrate with a few other
editors.

We have also discussed a client side githook. That effort has not started.
Let us know if that is a feature you would find useful.

Developer impact for new code reviews
======================================
Once the uncrustify checker is active in EDK II CI, developers must
make sure their patches are run through the uncrustify tool before
sending the patches for review.

Developers must install and run uncrustify against changes files before
sending patch review emails or submitting PR for EDK II CI. If EDK II CI
detects and differences in source formatting, then EDK II CI will fail
and the developer must run uncrustify and resubmit the patches.

Developer impact to patch series/PRs reviewed during edk2-stable201121
soft/hard freeze
=================================================================
======================
Developers must rebase their changes after the uncrustify source changes are
committed. The branch with a preview of the uncrustify changes can be used
to start this rebase work.


https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChan
ges_V5

The following steps can be used to update an existing branch with the
required uncrustify format. This is the Windows version. I will add
the Linux version soon.

1) Fetch and checkout and rebase to latest edk2/master

git fetch origin
git checkout master
git rebase origin/master

2) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

3) Check out the patch series branch (e.g. MyBranch)

git checkout MyBranch

4) Rebase patch series against edk2-stable202111

git rebase edk2-stable202111

5) Create new branch for the uncrustifed version (e.g. MyBranch_Uncrustified)

git checkout -b MyBranch_Uncrustified

6) Use git filter-branch to uncrustify all the commits in the series
between the rebase target from (2) and HEAD of the branch. A filter
can be used to scope the uncrustify operations to only the C/H files
in the specific package the patch series is against. (e.g. DynamicTablesPkg).
BaseTools should always be excluded. If the package scoped filter is
not used, it will still work, but will take longer to run because
uncrustify will rescan every C/H files in the whole repo.

git filter-branch --tree-filter "git ls-files DynamicTablesPkg*.c
DynamicTablesPkg*.h :!BaseTools/* | c:\\Temp\\UncrustifyCheck\\mu-
uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c
c:\\Temp\\UncrustifyCheck\\uncrustify.cfg -F - --replace --no-backup --if-
changed" edk2-stable202111..HEAD

7) Now that all the individual patches in the branch are uncrustified,
rebase against latest edk2/master that is already uncrustified.

git rebase master

8) Verify the patches in this new branch.

Impacts to tracing history across the uncrusity changes
=======================================================
Tools the view file and line history do work with the large uncrustify
patch series. One impact is that the operations can be very slow due
to the large uncrustify patches.

One option to provide a faster experience is to provide an alternate
version of the edk2 repository as "documentation" that has the
entire history re-written with uncrustify run on every commit.
The tool called git-filter-repo can be used to perform this
transformation and runs in a reasonable period of time (a few hours)

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-repo/blob/main/contrib/filter-repo-
demos/lint-history

The following steps can be used to perform this transformation.
This is the Windows version. I will add the Linux version soon.

** WARNING ** This operation modifies(rewrites) all the commits
in the local copy of the repo. Do not perform
these steps on a local repo you are using for
active development.

1) Clone edk2 into a new directory (see **WARNING**)

git clone https://github.com/tianocore/edk2.git edk2-uncrustified
cd edk2-uncrustified

2) Setup python virtual env, install pytools, and run stuart commands
to setup build environment which includes installing uncrustify tools.

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-
locally

3) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

4) Use lint-history.py from git-filter-repo examples

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-repo/blob/main/contrib/filter-repo-
demos/lint-history

Line #127 - Add try except around subprocess.check_call() with except
being pass. This is required because there are a few commits of C
files in the edk2 repo that have incorrect C syntax and do not
build with a C compiler and break the uncrustify parser. Skip reformat
of C files that can not be parsed by uncrustify. These rare instances
are addressed in the commit that fixes the C syntax error.

Run this slightly modified version of lint-history. Include only
C/H files and exclude directories that start with 'Tools' or 'BaseTools'.
This step took about 2.2 hours on a laptop.

lint-history.py
--relevant "return (not filename.startswith(b'Tools') and not
filename.startswith(b'BaseTools') and (filename.endswith(b'.c') or
filename.endswith(b'.h')))"
c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\mu-uncrustify-
release_extdep\\Windows-x86\\uncrustify.exe -c
c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\uncrustify.cfg --replace --
no-backup --if-changed

Order of PRs to apply during extended hard freeze
==================================================
1) Update EmulatorPkg Win Host [BuildOptions] MSFT CC_FLAGS to not force
debug information
* https://bugzilla.tianocore.org/show_bug.cgi?id=3747
*
https://github.com/mdkinney/edk2/tree/Bug_3747_EmulatorPkg_WinHost_Rep
roducibleBuild
* https://github.com/tianocore/edk2/pull/2215
* Required for EmulatorPkg to pass CompareBuild for VS2019 IA32/X64 builds.
* Status: Review complete. PR pushed.

2) EccCheck should not revert staged and local changes
* https://bugzilla.tianocore.org/show_bug.cgi?id=2986
*
https://github.com/mdkinney/edk2/tree/Bug_2986_EccCheckRemoveGitRevert
_V2
* https://github.com/tianocore/edk2/pull/2216
* Required for EDK II CI to complete in a reasonable period of time when
processing the 4000+ source file style changes made by uncrustify.
* Also fixes critical bugs that can potentially corrupt git state when
EccCheck is run locally.
* Status: Review complete. PR pushed.

3) Update pytool LicenseCheck plugin to use temp directory for diff output file
* https://bugzilla.tianocore.org/show_bug.cgi?id=3746
*
https://github.com/mdkinney/edk2/tree/Bug_3746_LicenseCheckUseDiffOutput
File_V2
* https://github.com/tianocore/edk2/pull/2217
* Required to reduce EDK II CI build times.
* Status: Review complete. PR pushed.

4) Update max job time from 60 min to 120 minutes in .azurepipelines/templates
* https://bugzilla.tianocore.org/show_bug.cgi?id=3750
*
https://github.com/mdkinney/edk2/tree/Bug_3750_IncreaseAzurePipelinesTime
out
* https://github.com/tianocore/edk2/pull/2219
* Required to allow EccCheck of uncrustify changes to complete on Azure
Pipelines CI agents without timing out.
* Status: Review complete. PR pushed.

5) Update Package YAML to ignore specific ECC files/errors
* https://bugzilla.tianocore.org/show_bug.cgi?id=3749
*
https://github.com/mdkinney/edk2/tree/Bug_3749_EccCheckIgnoreFilesErrors
* https://github.com/tianocore/edk2/pull/2218
* Required to pass EccCheck
* Status: Review complete. PR pushed

6) Uncrustify Source Changes
* https://bugzilla.tianocore.org/show_bug.cgi?id=3737
* https://bugzilla.tianocore.org/show_bug.cgi?id=3739
*
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChan
ges_V5
* https://github.com/tianocore/edk2/pull/2229
* Build comparison result PASS:
https://github.com/mdkinney/edk2/actions/runs/1521618836
* EFI_D_ -> DEBUG changes required to pass PatchCheck
* Uncrustify format changes required to pass UncrustifyCheck
* Status:
Waiting for review

7) UncrustifyCheck EDK II CI Plugin
* https://bugzilla.tianocore.org/show_bug.cgi?id=3748
*
https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plugin_v
6
* Required to enforce all PRs submitted to EDK II CI match uncrustify format.
* Unconditionally checks all packages. Can not be committed until all C/H
source files have been updated.
* Status: Review complete

Combined Branch/PR for Review/Test
==================================
* Build Comparison results must pass 100% across the full set of PRs before
the individual PRs can be pushed in the order listed above.
* Branch:
https://github.com/mdkinney/edk2/tree/TestOnly_Uncrustify_PR_Series
* PR: https://github.com/tianocore/edk2/pull/2229
Status = PASS
* CompareBuild:
Branch:
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChan
ges_V5
--ref1: ef9a059cdb15844fe52a49af2bf7d86b9dd3e9bf
--ref2: Bug_3737_3739_ApplyUncrustifyChanges_V5
Extra Options: -n 4 --quiet
Results: https://github.com/mdkinney/edk2/actions/runs/1521618836
30 VS2019 build comparisons PASS
40 GCC5 build comparisons PASS
100% PASS

The following git log shows the set of patches from --ref1 to --ref 2across
which there are no differences in any of the OBJ/LIB/DLL/EFI/FFS/FV/FD files.

--ref2
b7d4bf0675b7 (HEAD -> Bug_3737_3739_ApplyUncrustifyChanges_V5)
UnitTestFrameworkPkg: Apply uncrusitify changes
7f03d25f60e7 UefiPayloadPkg: Apply uncrusitify changes
0bfd8d9b5ac9 UefiCpuPkg: Apply uncrusitify changes
e1cd9bfb9dea StandaloneMmPkg: Apply uncrusitify changes
5da2f65be378 SourceLevelDebugPkg: Apply uncrusitify changes
95b86de07e5d SignedCapsulePkg: Apply uncrusitify changes
fe71d97246c4 ShellPkg: Apply uncrusitify changes
54c21c952992 SecurityPkg: Apply uncrusitify changes
187a3785f12b RedfishPkg: Apply uncrusitify changes
810100002a46 PcAtChipsetPkg: Apply uncrusitify changes
276a695c0cf2 OvmfPkg: Apply uncrusitify changes
303c0a91ab07 NetworkPkg: Apply uncrusitify changes
bc80792cd1b1 MdePkg: Apply uncrusitify changes
3ea86be17a2a MdeModulePkg: Apply uncrusitify changes
c70ef11ed0cd IntelFsp2WrapperPkg: Apply uncrusitify changes
c0291221f252 IntelFsp2Pkg: Apply uncrusitify changes
6a479952a690 FmpDevicePkg: Apply uncrusitify changes
3a7c05b7070d FatPkg: Apply uncrusitify changes
b789f98c8959 EmulatorPkg: Apply uncrusitify changes
952d7a1c9220 EmbeddedPkg: Apply uncrusitify changes
a1cc9881bab6 DynamicTablesPkg: Apply uncrusitify changes
50654dfe5785 CryptoPkg: Apply uncrusitify changes
ed965a02dfa1 ArmVirtPkg: Apply uncrusitify changes
9744023fbc46 ArmPlatformPkg: Apply uncrusitify changes
7a1cde5f5bba ArmPkg: Apply uncrusitify changes
19d17e0913e8 UefiCpuPkg: Change use of EFI_D_* to DEBUG_*
ffa718b4f994 SourceLevelDebugPkg: Change use of EFI_D_* to DEBUG_*
b86cb3c5e5b4 ShellPkg: Change use of EFI_D_* to DEBUG_*
c7c42204dc07 SecurityPkg: Change use of EFI_D_* to DEBUG_*
16b8e6f958e4 PcAtChipsetPkg: Change use of EFI_D_* to DEBUG_*
0ac3f8b2dac5 OvmfPkg: Change use of EFI_D_* to DEBUG_*
bc5004b8d294 NetworkPkg: Change use of EFI_D_* to DEBUG_*
6f671a8e2377 MdePkg: Change use of EFI_D_* to DEBUG_*
a10c610ff9a3 MdeModulePkg: Change use of EFI_D_* to DEBUG_*
09a3bddba390 FatPkg: Change use of EFI_D_* to DEBUG_*
59c61318246a EmulatorPkg: Change use of EFI_D_* to DEBUG_*
3a80367dda3b EmbeddedPkg: Change use of EFI_D_* to DEBUG_*
23eb1aaf80ca ArmVirtPkg: Change use of EFI_D_* to DEBUG_*
875914b45c54 ArmPlatformPkg: Change use of EFI_D_* to DEBUG_*
eb2eca82b451 ArmPkg: Change use of EFI_D_* to DEBUG_*
f0f3f5aae7c4 (origin/master, origin/HEAD, master) UnitTestFrameworkPkg:
Update YAML to ignore specific ECC files/errors
c05734797790 UefiPayloadPkg: Update YAML to ignore specific ECC files/errors
c30c40d6c63d StandaloneMmPkg: Update YAML to ignore specific ECC
files/errors
9944508e85f1 ShellPkg: Update YAML to ignore specific ECC files/errors
60fa40be458d SecurityPkg: Update YAML to ignore specific ECC files/errors
df790cd6b37e MdePkg: Update YAML to ignore specific ECC files/errors
9deb9370766e MdeModulePkg: Update YAML to ignore specific ECC files/errors
d7d30e8f219f EmulatorPkg: Update YAML to ignore specific ECC files/errors
d5744ecba813 CryptoPkg: Update YAML to ignore specific ECC files/errors
c97fee87f0f9 ArmVirtPkg: Update YAML to ignore specific ECC files/errors
1939fc9569f2 ArmPlatformPkg: Update YAML to ignore specific ECC files/errors
365dced2c37a ArmPkg: Update YAML to ignore specific ECC files/errors
76a1ce4d5fec .azurepipelines/templates: Update max pipeline job time to 2
hours
99f84ff47390 .pytools/Plugin/LicenseCheck: Use temp directory for git diff
output
3019f1bbabf1 .pytool/Plugin/EccCheck: Add performance optimizations
854462bd3479 .pytool/Plugin/EccCheck: Remove temp directory on exception
69877614fdee .pytool/Plugin/EccCheck: Remove RevertCode()
--ref1
ef9a059cdb15 EmulatorPkg/Win/Host: Update CC_FLAGS
bb1bba3d7767 (tag: edk2-stable202111) NetworkPkg: Fix invalid pointer for DNS
response token on error

Best regards,

Mike









Wang, Jian J
 

For SignedCapulePkg,

Reviewed-by: Jian J Wang <jian.j.wang@intel.com>

Regards,
Jian

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Michael D
Kinney
Sent: Wednesday, December 01, 2021 6:34 AM
To: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>;
Michael Kubacki <michael.kubacki@microsoft.com>; Andrew Fish
(afish@apple.com) <afish@apple.com>; Leif Lindholm <leif@nuviainc.com>
Subject: [edk2-devel] Uncrustify Conversion Detailed Plan and Extended Hard
Freeze Update #4

Hello,

Thank you for your patience during this extended hard freeze.

Just one more step to go. There has been a delay in the review of
the patch series with the uncrustify source changes. PR(6). This
patch series was not sent out as patch review email because of its
very large size. It only contains source style changes and the
CompareBuild tool and GitHub action has shown there are no binary
differences introduced with these source style changes.

If you are a package maintainer, then please review the following
branch/PR for your package contents and review the EDK II CI results
and BuildCompare results. I do not expect a line by line review
because we already had time to provide feedback on the source style
performed by uncrustify. Instead, a Reviewed-by for your package
indicates that you have reviewed the EDK II CI results and CompareBuild
tool functionality and results and you accept the source style
changes to your package.

*
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChan
ges_V5
* https://github.com/tianocore/edk2/pull/2229
* https://github.com/mdkinney/edk2/actions/runs/1521618836

Additional details on this update below.

Thank you,

Mike


Changes from Update #3
----------------------------------------------------------------------------
* Pushed PR (5)
* Added link to PR(6). EDK II CI Status is PASS. Build Compare PASS.
* Waiting for review of PR (6)
* Review of PR (7) completed and waiting for review of PR (6)
----------------------------------------------------------------------------

Changes from Update #2
----------------------------------------------------------------------------
* Changed order of PRs swapping (4) and (5). The PR that activates
increases the max CI agent job time is independent of all the other
PRs and its review is complete, so it can be committed now.
* Pushed PRs (1), (2), (3), (4).
* Waiting for review to complete for PRs (5) and (6)
* Reviews complete for PR (7)
* Identifies steps using git filter-branch to apply uncrustify changes to a
code review patch series that was generated before the uncrustify changes
avoiding manual merge.
* Identified steps using git filter-repo to generate an alternate history of
the edk2 repo with uncrustify changes applied on every commit. This may
be useful when evaluating changes to files using tools like git blame
without the large uncrustify patch series.
---------------------------------------------------------------------------

Changes from Update #1
----------------------------------------------------------------------------
* Changed order of PRs swapping (6) and (7). The PR that activates
EDK II CI check UncrustifyCheck has to be last because it unconditionally
checks all C/H files in all packages. Not just files that have been
modified like some of the other checkers.
* Updated link to the branch with the UncrustifyCheck plugin that has been
updated with a one line change and Reviewed-by and Tested-by tags.

https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plugin_v
6
* Reviews complete for (1), (2), (3), (5), and (7)
---------------------------------------------------------------------------

Michael Kubacki and I have prepared the patches required to apply the
uncrustify changes and enable EDK II CI to check all submitted
patches have been run through uncrustify.

We have verified through the CompareBuild GitHub Action that the
format changes performed by uncrustify have no functional changes.
All of the OBJ, LIB, DLL, EFI, FFS, FV, and FD files match 100%
across 70 VS2019/GCC5 builds of all package/platform DSC files in
the edk2 repo.

The hard freeze will be extended after the edk2-stable202111 tag until
all uncrustify related changes are committed. We do not expect this
to take more than a few days. Do not push any PRs until the hard
freeze is lifted.

The changes are broken up into 7 patch series/PRs. The PRs are ordered
so they can be submitted using the normal submission process and EDK II
CI will pass for each one. Details are listed below.

Uncrustify 73.0.3 for EDK II
=============================
* Sources: https://dev.azure.com/projectmu/_git/Uncrustify
* Documentation:
https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Proj
ect-Mu-(EDK-II)-Fork-Readme
* Download:
https://dev.azure.com/projectmu/Uncrustify/_packaging?_a=package&feed=m
u_uncrustify&package=mu-uncrustify-
release&protocolType=NuGet&version=73.0.3

Installing Uncrustify
======================
The Uncrustify tool is installed automatically when the Pytools
environment is used and the stuart* commands are run to complete the
environment setup. Please see:

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-locally

Uncrustify can also be installed from the download page listed above
or built from sources from the source link above.

The Documentation link provides instruction on how to run uncrustify from
the command line or install as a Visual Studio Code plugin. The main
uncrustify documentation also describes how to integrate with a few other
editors.

We have also discussed a client side githook. That effort has not started.
Let us know if that is a feature you would find useful.

Developer impact for new code reviews
======================================
Once the uncrustify checker is active in EDK II CI, developers must
make sure their patches are run through the uncrustify tool before
sending the patches for review.

Developers must install and run uncrustify against changes files before
sending patch review emails or submitting PR for EDK II CI. If EDK II CI
detects and differences in source formatting, then EDK II CI will fail
and the developer must run uncrustify and resubmit the patches.

Developer impact to patch series/PRs reviewed during edk2-stable201121
soft/hard freeze
=================================================================
======================
Developers must rebase their changes after the uncrustify source changes are
committed. The branch with a preview of the uncrustify changes can be used
to start this rebase work.


https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChan
ges_V5

The following steps can be used to update an existing branch with the
required uncrustify format. This is the Windows version. I will add
the Linux version soon.

1) Fetch and checkout and rebase to latest edk2/master

git fetch origin
git checkout master
git rebase origin/master

2) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

3) Check out the patch series branch (e.g. MyBranch)

git checkout MyBranch

4) Rebase patch series against edk2-stable202111

git rebase edk2-stable202111

5) Create new branch for the uncrustifed version (e.g. MyBranch_Uncrustified)

git checkout -b MyBranch_Uncrustified

6) Use git filter-branch to uncrustify all the commits in the series
between the rebase target from (2) and HEAD of the branch. A filter
can be used to scope the uncrustify operations to only the C/H files
in the specific package the patch series is against. (e.g. DynamicTablesPkg).
BaseTools should always be excluded. If the package scoped filter is
not used, it will still work, but will take longer to run because
uncrustify will rescan every C/H files in the whole repo.

git filter-branch --tree-filter "git ls-files DynamicTablesPkg*.c
DynamicTablesPkg*.h :!BaseTools/* | c:\\Temp\\UncrustifyCheck\\mu-
uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c
c:\\Temp\\UncrustifyCheck\\uncrustify.cfg -F - --replace --no-backup --if-
changed" edk2-stable202111..HEAD

7) Now that all the individual patches in the branch are uncrustified,
rebase against latest edk2/master that is already uncrustified.

git rebase master

8) Verify the patches in this new branch.

Impacts to tracing history across the uncrusity changes
=======================================================
Tools the view file and line history do work with the large uncrustify
patch series. One impact is that the operations can be very slow due
to the large uncrustify patches.

One option to provide a faster experience is to provide an alternate
version of the edk2 repository as "documentation" that has the
entire history re-written with uncrustify run on every commit.
The tool called git-filter-repo can be used to perform this
transformation and runs in a reasonable period of time (a few hours)

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-repo/blob/main/contrib/filter-repo-
demos/lint-history

The following steps can be used to perform this transformation.
This is the Windows version. I will add the Linux version soon.

** WARNING ** This operation modifies(rewrites) all the commits
in the local copy of the repo. Do not perform
these steps on a local repo you are using for
active development.

1) Clone edk2 into a new directory (see **WARNING**)

git clone https://github.com/tianocore/edk2.git edk2-uncrustified
cd edk2-uncrustified

2) Setup python virtual env, install pytools, and run stuart commands
to setup build environment which includes installing uncrustify tools.

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-
locally

3) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

4) Use lint-history.py from git-filter-repo examples

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-repo/blob/main/contrib/filter-repo-
demos/lint-history

Line #127 - Add try except around subprocess.check_call() with except
being pass. This is required because there are a few commits of C
files in the edk2 repo that have incorrect C syntax and do not
build with a C compiler and break the uncrustify parser. Skip reformat
of C files that can not be parsed by uncrustify. These rare instances
are addressed in the commit that fixes the C syntax error.

Run this slightly modified version of lint-history. Include only
C/H files and exclude directories that start with 'Tools' or 'BaseTools'.
This step took about 2.2 hours on a laptop.

lint-history.py
--relevant "return (not filename.startswith(b'Tools') and not
filename.startswith(b'BaseTools') and (filename.endswith(b'.c') or
filename.endswith(b'.h')))"
c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\mu-uncrustify-
release_extdep\\Windows-x86\\uncrustify.exe -c
c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\uncrustify.cfg --replace --
no-backup --if-changed

Order of PRs to apply during extended hard freeze
==================================================
1) Update EmulatorPkg Win Host [BuildOptions] MSFT CC_FLAGS to not force
debug information
* https://bugzilla.tianocore.org/show_bug.cgi?id=3747
*
https://github.com/mdkinney/edk2/tree/Bug_3747_EmulatorPkg_WinHost_Rep
roducibleBuild
* https://github.com/tianocore/edk2/pull/2215
* Required for EmulatorPkg to pass CompareBuild for VS2019 IA32/X64 builds.
* Status: Review complete. PR pushed.

2) EccCheck should not revert staged and local changes
* https://bugzilla.tianocore.org/show_bug.cgi?id=2986
*
https://github.com/mdkinney/edk2/tree/Bug_2986_EccCheckRemoveGitRevert
_V2
* https://github.com/tianocore/edk2/pull/2216
* Required for EDK II CI to complete in a reasonable period of time when
processing the 4000+ source file style changes made by uncrustify.
* Also fixes critical bugs that can potentially corrupt git state when
EccCheck is run locally.
* Status: Review complete. PR pushed.

3) Update pytool LicenseCheck plugin to use temp directory for diff output file
* https://bugzilla.tianocore.org/show_bug.cgi?id=3746
*
https://github.com/mdkinney/edk2/tree/Bug_3746_LicenseCheckUseDiffOutput
File_V2
* https://github.com/tianocore/edk2/pull/2217
* Required to reduce EDK II CI build times.
* Status: Review complete. PR pushed.

4) Update max job time from 60 min to 120 minutes in .azurepipelines/templates
* https://bugzilla.tianocore.org/show_bug.cgi?id=3750
*
https://github.com/mdkinney/edk2/tree/Bug_3750_IncreaseAzurePipelinesTime
out
* https://github.com/tianocore/edk2/pull/2219
* Required to allow EccCheck of uncrustify changes to complete on Azure
Pipelines CI agents without timing out.
* Status: Review complete. PR pushed.

5) Update Package YAML to ignore specific ECC files/errors
* https://bugzilla.tianocore.org/show_bug.cgi?id=3749
*
https://github.com/mdkinney/edk2/tree/Bug_3749_EccCheckIgnoreFilesErrors
* https://github.com/tianocore/edk2/pull/2218
* Required to pass EccCheck
* Status: Review complete. PR pushed

6) Uncrustify Source Changes
* https://bugzilla.tianocore.org/show_bug.cgi?id=3737
* https://bugzilla.tianocore.org/show_bug.cgi?id=3739
*
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChan
ges_V5
* https://github.com/tianocore/edk2/pull/2229
* Build comparison result PASS:
https://github.com/mdkinney/edk2/actions/runs/1521618836
* EFI_D_ -> DEBUG changes required to pass PatchCheck
* Uncrustify format changes required to pass UncrustifyCheck
* Status:
Waiting for review

7) UncrustifyCheck EDK II CI Plugin
* https://bugzilla.tianocore.org/show_bug.cgi?id=3748
*
https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plugin_v
6
* Required to enforce all PRs submitted to EDK II CI match uncrustify format.
* Unconditionally checks all packages. Can not be committed until all C/H
source files have been updated.
* Status: Review complete

Combined Branch/PR for Review/Test
==================================
* Build Comparison results must pass 100% across the full set of PRs before
the individual PRs can be pushed in the order listed above.
* Branch:
https://github.com/mdkinney/edk2/tree/TestOnly_Uncrustify_PR_Series
* PR: https://github.com/tianocore/edk2/pull/2229
Status = PASS
* CompareBuild:
Branch:
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChan
ges_V5
--ref1: ef9a059cdb15844fe52a49af2bf7d86b9dd3e9bf
--ref2: Bug_3737_3739_ApplyUncrustifyChanges_V5
Extra Options: -n 4 --quiet
Results: https://github.com/mdkinney/edk2/actions/runs/1521618836
30 VS2019 build comparisons PASS
40 GCC5 build comparisons PASS
100% PASS

The following git log shows the set of patches from --ref1 to --ref 2across
which there are no differences in any of the OBJ/LIB/DLL/EFI/FFS/FV/FD files.

--ref2
b7d4bf0675b7 (HEAD -> Bug_3737_3739_ApplyUncrustifyChanges_V5)
UnitTestFrameworkPkg: Apply uncrusitify changes
7f03d25f60e7 UefiPayloadPkg: Apply uncrusitify changes
0bfd8d9b5ac9 UefiCpuPkg: Apply uncrusitify changes
e1cd9bfb9dea StandaloneMmPkg: Apply uncrusitify changes
5da2f65be378 SourceLevelDebugPkg: Apply uncrusitify changes
95b86de07e5d SignedCapsulePkg: Apply uncrusitify changes
fe71d97246c4 ShellPkg: Apply uncrusitify changes
54c21c952992 SecurityPkg: Apply uncrusitify changes
187a3785f12b RedfishPkg: Apply uncrusitify changes
810100002a46 PcAtChipsetPkg: Apply uncrusitify changes
276a695c0cf2 OvmfPkg: Apply uncrusitify changes
303c0a91ab07 NetworkPkg: Apply uncrusitify changes
bc80792cd1b1 MdePkg: Apply uncrusitify changes
3ea86be17a2a MdeModulePkg: Apply uncrusitify changes
c70ef11ed0cd IntelFsp2WrapperPkg: Apply uncrusitify changes
c0291221f252 IntelFsp2Pkg: Apply uncrusitify changes
6a479952a690 FmpDevicePkg: Apply uncrusitify changes
3a7c05b7070d FatPkg: Apply uncrusitify changes
b789f98c8959 EmulatorPkg: Apply uncrusitify changes
952d7a1c9220 EmbeddedPkg: Apply uncrusitify changes
a1cc9881bab6 DynamicTablesPkg: Apply uncrusitify changes
50654dfe5785 CryptoPkg: Apply uncrusitify changes
ed965a02dfa1 ArmVirtPkg: Apply uncrusitify changes
9744023fbc46 ArmPlatformPkg: Apply uncrusitify changes
7a1cde5f5bba ArmPkg: Apply uncrusitify changes
19d17e0913e8 UefiCpuPkg: Change use of EFI_D_* to DEBUG_*
ffa718b4f994 SourceLevelDebugPkg: Change use of EFI_D_* to DEBUG_*
b86cb3c5e5b4 ShellPkg: Change use of EFI_D_* to DEBUG_*
c7c42204dc07 SecurityPkg: Change use of EFI_D_* to DEBUG_*
16b8e6f958e4 PcAtChipsetPkg: Change use of EFI_D_* to DEBUG_*
0ac3f8b2dac5 OvmfPkg: Change use of EFI_D_* to DEBUG_*
bc5004b8d294 NetworkPkg: Change use of EFI_D_* to DEBUG_*
6f671a8e2377 MdePkg: Change use of EFI_D_* to DEBUG_*
a10c610ff9a3 MdeModulePkg: Change use of EFI_D_* to DEBUG_*
09a3bddba390 FatPkg: Change use of EFI_D_* to DEBUG_*
59c61318246a EmulatorPkg: Change use of EFI_D_* to DEBUG_*
3a80367dda3b EmbeddedPkg: Change use of EFI_D_* to DEBUG_*
23eb1aaf80ca ArmVirtPkg: Change use of EFI_D_* to DEBUG_*
875914b45c54 ArmPlatformPkg: Change use of EFI_D_* to DEBUG_*
eb2eca82b451 ArmPkg: Change use of EFI_D_* to DEBUG_*
f0f3f5aae7c4 (origin/master, origin/HEAD, master) UnitTestFrameworkPkg:
Update YAML to ignore specific ECC files/errors
c05734797790 UefiPayloadPkg: Update YAML to ignore specific ECC files/errors
c30c40d6c63d StandaloneMmPkg: Update YAML to ignore specific ECC
files/errors
9944508e85f1 ShellPkg: Update YAML to ignore specific ECC files/errors
60fa40be458d SecurityPkg: Update YAML to ignore specific ECC files/errors
df790cd6b37e MdePkg: Update YAML to ignore specific ECC files/errors
9deb9370766e MdeModulePkg: Update YAML to ignore specific ECC files/errors
d7d30e8f219f EmulatorPkg: Update YAML to ignore specific ECC files/errors
d5744ecba813 CryptoPkg: Update YAML to ignore specific ECC files/errors
c97fee87f0f9 ArmVirtPkg: Update YAML to ignore specific ECC files/errors
1939fc9569f2 ArmPlatformPkg: Update YAML to ignore specific ECC files/errors
365dced2c37a ArmPkg: Update YAML to ignore specific ECC files/errors
76a1ce4d5fec .azurepipelines/templates: Update max pipeline job time to 2
hours
99f84ff47390 .pytools/Plugin/LicenseCheck: Use temp directory for git diff
output
3019f1bbabf1 .pytool/Plugin/EccCheck: Add performance optimizations
854462bd3479 .pytool/Plugin/EccCheck: Remove temp directory on exception
69877614fdee .pytool/Plugin/EccCheck: Remove RevertCode()
--ref1
ef9a059cdb15 EmulatorPkg/Win/Host: Update CC_FLAGS
bb1bba3d7767 (tag: edk2-stable202111) NetworkPkg: Fix invalid pointer for DNS
response token on error

Best regards,

Mike









Abner Chang
 

Hi Mike,
Is that possible to not applying uncrusitify on the source files that cloned from other open source?
e.g. RedfishPkg/PrivateLibrary/RedfishLib/edk2libredfish/*.*

Thanks
Abner

-----Original Message-----
From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of
Wang, Jian J
Sent: Wednesday, December 1, 2021 2:39 PM
To: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>;
Michael Kubacki <michael.kubacki@microsoft.com>; Andrew Fish
(afish@apple.com) <afish@apple.com>; Leif Lindholm <leif@nuviainc.com>
Subject: Re: [edk2-devel] Uncrustify Conversion Detailed Plan and Extended
Hard Freeze Update #4

For SignedCapulePkg,

Reviewed-by: Jian J Wang <jian.j.wang@intel.com>

Regards,
Jian

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
Michael D
Kinney
Sent: Wednesday, December 01, 2021 6:34 AM
To: devel@edk2.groups.io; Kinney, Michael D
<michael.d.kinney@intel.com>;
Michael Kubacki <michael.kubacki@microsoft.com>; Andrew Fish
(afish@apple.com) <afish@apple.com>; Leif Lindholm
<leif@nuviainc.com>
Subject: [edk2-devel] Uncrustify Conversion Detailed Plan and Extended
Hard
Freeze Update #4

Hello,

Thank you for your patience during this extended hard freeze.

Just one more step to go. There has been a delay in the review of
the patch series with the uncrustify source changes. PR(6). This
patch series was not sent out as patch review email because of its
very large size. It only contains source style changes and the
CompareBuild tool and GitHub action has shown there are no binary
differences introduced with these source style changes.

If you are a package maintainer, then please review the following
branch/PR for your package contents and review the EDK II CI results
and BuildCompare results. I do not expect a line by line review
because we already had time to provide feedback on the source style
performed by uncrustify. Instead, a Reviewed-by for your package
indicates that you have reviewed the EDK II CI results and CompareBuild
tool functionality and results and you accept the source style
changes to your package.

*
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
han
ges_V5
* https://github.com/tianocore/edk2/pull/2229
* https://github.com/mdkinney/edk2/actions/runs/1521618836

Additional details on this update below.

Thank you,

Mike


Changes from Update #3
----------------------------------------------------------------------------
* Pushed PR (5)
* Added link to PR(6). EDK II CI Status is PASS. Build Compare PASS.
* Waiting for review of PR (6)
* Review of PR (7) completed and waiting for review of PR (6)
----------------------------------------------------------------------------

Changes from Update #2
----------------------------------------------------------------------------
* Changed order of PRs swapping (4) and (5). The PR that activates
increases the max CI agent job time is independent of all the other
PRs and its review is complete, so it can be committed now.
* Pushed PRs (1), (2), (3), (4).
* Waiting for review to complete for PRs (5) and (6)
* Reviews complete for PR (7)
* Identifies steps using git filter-branch to apply uncrustify changes to a
code review patch series that was generated before the uncrustify
changes
avoiding manual merge.
* Identified steps using git filter-repo to generate an alternate history of
the edk2 repo with uncrustify changes applied on every commit. This may
be useful when evaluating changes to files using tools like git blame
without the large uncrustify patch series.
---------------------------------------------------------------------------

Changes from Update #1
----------------------------------------------------------------------------
* Changed order of PRs swapping (6) and (7). The PR that activates
EDK II CI check UncrustifyCheck has to be last because it unconditionally
checks all C/H files in all packages. Not just files that have been
modified like some of the other checkers.
* Updated link to the branch with the UncrustifyCheck plugin that has been
updated with a one line change and Reviewed-by and Tested-by tags.

https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plug
in_v
6
* Reviews complete for (1), (2), (3), (5), and (7)
---------------------------------------------------------------------------

Michael Kubacki and I have prepared the patches required to apply the
uncrustify changes and enable EDK II CI to check all submitted
patches have been run through uncrustify.

We have verified through the CompareBuild GitHub Action that the
format changes performed by uncrustify have no functional changes.
All of the OBJ, LIB, DLL, EFI, FFS, FV, and FD files match 100%
across 70 VS2019/GCC5 builds of all package/platform DSC files in
the edk2 repo.

The hard freeze will be extended after the edk2-stable202111 tag until
all uncrustify related changes are committed. We do not expect this
to take more than a few days. Do not push any PRs until the hard
freeze is lifted.

The changes are broken up into 7 patch series/PRs. The PRs are ordered
so they can be submitted using the normal submission process and EDK II
CI will pass for each one. Details are listed below.

Uncrustify 73.0.3 for EDK II
=============================
* Sources:
INVALID URI REMOVED
stify__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJOYp
cSVHoHwMD2t8$
* Documentation:
INVALID URI REMOVED
/_wiki/wikis/Uncrustify.wiki/1/Proj__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKk
oslR8D0LLPZTWXL0OKtMZJOYpcSVHoP8OtlxM$
ect-Mu-(EDK-II)-Fork-Readme
* Download:
INVALID URI REMOVED
/_packaging?_a=package&feed=m__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKk
oslR8D0LLPZTWXL0OKtMZJOYpcSVHo4yTWeAM$
u_uncrustify&package=mu-uncrustify-
release&protocolType=NuGet&version=73.0.3

Installing Uncrustify
======================
The Uncrustify tool is installed automatically when the Pytools
environment is used and the stuart* commands are run to complete the
environment setup. Please see:

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-
locally

Uncrustify can also be installed from the download page listed above
or built from sources from the source link above.

The Documentation link provides instruction on how to run uncrustify from
the command line or install as a Visual Studio Code plugin. The main
uncrustify documentation also describes how to integrate with a few other
editors.

We have also discussed a client side githook. That effort has not started.
Let us know if that is a feature you would find useful.

Developer impact for new code reviews
======================================
Once the uncrustify checker is active in EDK II CI, developers must
make sure their patches are run through the uncrustify tool before
sending the patches for review.

Developers must install and run uncrustify against changes files before
sending patch review emails or submitting PR for EDK II CI. If EDK II CI
detects and differences in source formatting, then EDK II CI will fail
and the developer must run uncrustify and resubmit the patches.

Developer impact to patch series/PRs reviewed during edk2-stable201121
soft/hard freeze
==========================================================
=======
======================
Developers must rebase their changes after the uncrustify source changes
are
committed. The branch with a preview of the uncrustify changes can be
used
to start this rebase work.


https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
han
ges_V5

The following steps can be used to update an existing branch with the
required uncrustify format. This is the Windows version. I will add
the Linux version soon.

1) Fetch and checkout and rebase to latest edk2/master

git fetch origin
git checkout master
git rebase origin/master

2) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

3) Check out the patch series branch (e.g. MyBranch)

git checkout MyBranch

4) Rebase patch series against edk2-stable202111

git rebase edk2-stable202111

5) Create new branch for the uncrustifed version (e.g.
MyBranch_Uncrustified)

git checkout -b MyBranch_Uncrustified

6) Use git filter-branch to uncrustify all the commits in the series
between the rebase target from (2) and HEAD of the branch. A filter
can be used to scope the uncrustify operations to only the C/H files
in the specific package the patch series is against. (e.g.
DynamicTablesPkg).
BaseTools should always be excluded. If the package scoped filter is
not used, it will still work, but will take longer to run because
uncrustify will rescan every C/H files in the whole repo.

git filter-branch --tree-filter "git ls-files DynamicTablesPkg*.c
DynamicTablesPkg*.h :!BaseTools/* | c:\\Temp\\UncrustifyCheck\\mu-
uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c
c:\\Temp\\UncrustifyCheck\\uncrustify.cfg -F - --replace --no-backup --if-
changed" edk2-stable202111..HEAD

7) Now that all the individual patches in the branch are uncrustified,
rebase against latest edk2/master that is already uncrustified.

git rebase master

8) Verify the patches in this new branch.

Impacts to tracing history across the uncrusity changes
=======================================================
Tools the view file and line history do work with the large uncrustify
patch series. One impact is that the operations can be very slow due
to the large uncrustify patches.

One option to provide a faster experience is to provide an alternate
version of the edk2 repository as "documentation" that has the
entire history re-written with uncrustify run on every commit.
The tool called git-filter-repo can be used to perform this
transformation and runs in a reasonable period of time (a few hours)

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-repo/blob/main/contrib/filter-
repo-
demos/lint-history

The following steps can be used to perform this transformation.
This is the Windows version. I will add the Linux version soon.

** WARNING ** This operation modifies(rewrites) all the commits
in the local copy of the repo. Do not perform
these steps on a local repo you are using for
active development.

1) Clone edk2 into a new directory (see **WARNING**)

git clone https://github.com/tianocore/edk2.git edk2-uncrustified
cd edk2-uncrustified

2) Setup python virtual env, install pytools, and run stuart commands
to setup build environment which includes installing uncrustify tools.

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-
locally

3) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

4) Use lint-history.py from git-filter-repo examples

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-repo/blob/main/contrib/filter-
repo-
demos/lint-history

Line #127 - Add try except around subprocess.check_call() with except
being pass. This is required because there are a few commits of C
files in the edk2 repo that have incorrect C syntax and do not
build with a C compiler and break the uncrustify parser. Skip reformat
of C files that can not be parsed by uncrustify. These rare instances
are addressed in the commit that fixes the C syntax error.

Run this slightly modified version of lint-history. Include only
C/H files and exclude directories that start with 'Tools' or 'BaseTools'.
This step took about 2.2 hours on a laptop.

lint-history.py
--relevant "return (not filename.startswith(b'Tools') and not
filename.startswith(b'BaseTools') and (filename.endswith(b'.c') or
filename.endswith(b'.h')))"
c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\mu-
uncrustify-
release_extdep\\Windows-x86\\uncrustify.exe -c
c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\uncrustify.cfg --
replace --
no-backup --if-changed

Order of PRs to apply during extended hard freeze
==================================================
1) Update EmulatorPkg Win Host [BuildOptions] MSFT CC_FLAGS to not
force
debug information
*
INVALID URI REMOVED
d=3747__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHo2BEG33c$
*
https://github.com/mdkinney/edk2/tree/Bug_3747_EmulatorPkg_WinHost_
Rep
roducibleBuild
* https://github.com/tianocore/edk2/pull/2215
* Required for EmulatorPkg to pass CompareBuild for VS2019 IA32/X64
builds.
* Status: Review complete. PR pushed.

2) EccCheck should not revert staged and local changes
*
INVALID URI REMOVED
d=2986__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHotvR0Unk$
*
https://github.com/mdkinney/edk2/tree/Bug_2986_EccCheckRemoveGitRe
vert
_V2
* https://github.com/tianocore/edk2/pull/2216
* Required for EDK II CI to complete in a reasonable period of time when
processing the 4000+ source file style changes made by uncrustify.
* Also fixes critical bugs that can potentially corrupt git state when
EccCheck is run locally.
* Status: Review complete. PR pushed.

3) Update pytool LicenseCheck plugin to use temp directory for diff output
file
*
INVALID URI REMOVED
d=3746__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHoWRuHaBw$
*
https://github.com/mdkinney/edk2/tree/Bug_3746_LicenseCheckUseDiffO
utput
File_V2
* https://github.com/tianocore/edk2/pull/2217
* Required to reduce EDK II CI build times.
* Status: Review complete. PR pushed.

4) Update max job time from 60 min to 120 minutes
in .azurepipelines/templates
*
INVALID URI REMOVED
d=3750__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHozaVshgc$
*
https://github.com/mdkinney/edk2/tree/Bug_3750_IncreaseAzurePipelines
Time
out
* https://github.com/tianocore/edk2/pull/2219
* Required to allow EccCheck of uncrustify changes to complete on Azure
Pipelines CI agents without timing out.
* Status: Review complete. PR pushed.

5) Update Package YAML to ignore specific ECC files/errors
*
INVALID URI REMOVED
d=3749__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHo-fY3dMo$
*
https://github.com/mdkinney/edk2/tree/Bug_3749_EccCheckIgnoreFilesErr
ors
* https://github.com/tianocore/edk2/pull/2218
* Required to pass EccCheck
* Status: Review complete. PR pushed

6) Uncrustify Source Changes
*
INVALID URI REMOVED
d=3737__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHoMRa47YU$
*
INVALID URI REMOVED
d=3739__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHoPAlckP8$
*
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
han
ges_V5
* https://github.com/tianocore/edk2/pull/2229
* Build comparison result PASS:
https://github.com/mdkinney/edk2/actions/runs/1521618836
* EFI_D_ -> DEBUG changes required to pass PatchCheck
* Uncrustify format changes required to pass UncrustifyCheck
* Status:
Waiting for review

7) UncrustifyCheck EDK II CI Plugin
*
INVALID URI REMOVED
d=3748__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHoCzEWymQ$
*
https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plug
in_v
6
* Required to enforce all PRs submitted to EDK II CI match uncrustify
format.
* Unconditionally checks all packages. Can not be committed until all C/H
source files have been updated.
* Status: Review complete

Combined Branch/PR for Review/Test
==================================
* Build Comparison results must pass 100% across the full set of PRs before
the individual PRs can be pushed in the order listed above.
* Branch:
https://github.com/mdkinney/edk2/tree/TestOnly_Uncrustify_PR_Series
* PR: https://github.com/tianocore/edk2/pull/2229
Status = PASS
* CompareBuild:
Branch:
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
han
ges_V5
--ref1: ef9a059cdb15844fe52a49af2bf7d86b9dd3e9bf
--ref2: Bug_3737_3739_ApplyUncrustifyChanges_V5
Extra Options: -n 4 --quiet
Results: https://github.com/mdkinney/edk2/actions/runs/1521618836
30 VS2019 build comparisons PASS
40 GCC5 build comparisons PASS
100% PASS

The following git log shows the set of patches from --ref1 to --ref 2across
which there are no differences in any of the OBJ/LIB/DLL/EFI/FFS/FV/FD
files.

--ref2
b7d4bf0675b7 (HEAD -> Bug_3737_3739_ApplyUncrustifyChanges_V5)
UnitTestFrameworkPkg: Apply uncrusitify changes
7f03d25f60e7 UefiPayloadPkg: Apply uncrusitify changes
0bfd8d9b5ac9 UefiCpuPkg: Apply uncrusitify changes
e1cd9bfb9dea StandaloneMmPkg: Apply uncrusitify changes
5da2f65be378 SourceLevelDebugPkg: Apply uncrusitify changes
95b86de07e5d SignedCapsulePkg: Apply uncrusitify changes
fe71d97246c4 ShellPkg: Apply uncrusitify changes
54c21c952992 SecurityPkg: Apply uncrusitify changes
187a3785f12b RedfishPkg: Apply uncrusitify changes
810100002a46 PcAtChipsetPkg: Apply uncrusitify changes
276a695c0cf2 OvmfPkg: Apply uncrusitify changes
303c0a91ab07 NetworkPkg: Apply uncrusitify changes
bc80792cd1b1 MdePkg: Apply uncrusitify changes
3ea86be17a2a MdeModulePkg: Apply uncrusitify changes
c70ef11ed0cd IntelFsp2WrapperPkg: Apply uncrusitify changes
c0291221f252 IntelFsp2Pkg: Apply uncrusitify changes
6a479952a690 FmpDevicePkg: Apply uncrusitify changes
3a7c05b7070d FatPkg: Apply uncrusitify changes
b789f98c8959 EmulatorPkg: Apply uncrusitify changes
952d7a1c9220 EmbeddedPkg: Apply uncrusitify changes
a1cc9881bab6 DynamicTablesPkg: Apply uncrusitify changes
50654dfe5785 CryptoPkg: Apply uncrusitify changes
ed965a02dfa1 ArmVirtPkg: Apply uncrusitify changes
9744023fbc46 ArmPlatformPkg: Apply uncrusitify changes
7a1cde5f5bba ArmPkg: Apply uncrusitify changes
19d17e0913e8 UefiCpuPkg: Change use of EFI_D_* to DEBUG_*
ffa718b4f994 SourceLevelDebugPkg: Change use of EFI_D_* to DEBUG_*
b86cb3c5e5b4 ShellPkg: Change use of EFI_D_* to DEBUG_*
c7c42204dc07 SecurityPkg: Change use of EFI_D_* to DEBUG_*
16b8e6f958e4 PcAtChipsetPkg: Change use of EFI_D_* to DEBUG_*
0ac3f8b2dac5 OvmfPkg: Change use of EFI_D_* to DEBUG_*
bc5004b8d294 NetworkPkg: Change use of EFI_D_* to DEBUG_*
6f671a8e2377 MdePkg: Change use of EFI_D_* to DEBUG_*
a10c610ff9a3 MdeModulePkg: Change use of EFI_D_* to DEBUG_*
09a3bddba390 FatPkg: Change use of EFI_D_* to DEBUG_*
59c61318246a EmulatorPkg: Change use of EFI_D_* to DEBUG_*
3a80367dda3b EmbeddedPkg: Change use of EFI_D_* to DEBUG_*
23eb1aaf80ca ArmVirtPkg: Change use of EFI_D_* to DEBUG_*
875914b45c54 ArmPlatformPkg: Change use of EFI_D_* to DEBUG_*
eb2eca82b451 ArmPkg: Change use of EFI_D_* to DEBUG_*
f0f3f5aae7c4 (origin/master, origin/HEAD, master) UnitTestFrameworkPkg:
Update YAML to ignore specific ECC files/errors
c05734797790 UefiPayloadPkg: Update YAML to ignore specific ECC
files/errors
c30c40d6c63d StandaloneMmPkg: Update YAML to ignore specific ECC
files/errors
9944508e85f1 ShellPkg: Update YAML to ignore specific ECC files/errors
60fa40be458d SecurityPkg: Update YAML to ignore specific ECC files/errors
df790cd6b37e MdePkg: Update YAML to ignore specific ECC files/errors
9deb9370766e MdeModulePkg: Update YAML to ignore specific ECC
files/errors
d7d30e8f219f EmulatorPkg: Update YAML to ignore specific ECC files/errors
d5744ecba813 CryptoPkg: Update YAML to ignore specific ECC files/errors
c97fee87f0f9 ArmVirtPkg: Update YAML to ignore specific ECC files/errors
1939fc9569f2 ArmPlatformPkg: Update YAML to ignore specific ECC
files/errors
365dced2c37a ArmPkg: Update YAML to ignore specific ECC files/errors
76a1ce4d5fec .azurepipelines/templates: Update max pipeline job time to
2
hours
99f84ff47390 .pytools/Plugin/LicenseCheck: Use temp directory for git diff
output
3019f1bbabf1 .pytool/Plugin/EccCheck: Add performance optimizations
854462bd3479 .pytool/Plugin/EccCheck: Remove temp directory on
exception
69877614fdee .pytool/Plugin/EccCheck: Remove RevertCode()
--ref1
ef9a059cdb15 EmulatorPkg/Win/Host: Update CC_FLAGS
bb1bba3d7767 (tag: edk2-stable202111) NetworkPkg: Fix invalid pointer for
DNS
response token on error

Best regards,

Mike












Sami Mujawar
 

Hi Mike,

For DynamicTablesPkg, ArmPkg, ArmPlatformPkg, ArmVirtPkg & StandaloneMmPkg

Reviewed-by: Sami Mujawar <sami.mujawar@...>

Regards,

Sami Mujawar


Michael D Kinney
 

Hi Abner,

Yes. That is possible. We are already excluding BaseTools.

In order to get the file lists that apply to uncrustify, we would have to change from:

git ls-files *.c *.h :!BaseTools/*

To:

git ls-files *.c *.h :!BaseTools/* :!RedfishPkg/PrivateLibrary/RedfishLib/edk2libredfish/*

However, I do not see a feature in the UncrustifyCheck to check all c/h files in RedFishPkg and exclude one of its directories.
If we can specify an exclusion like this in the RedFishPkg YAML file, then I think that would work well.

Michael Kubacki would have to comment on how we support this in EDK II CI checks.

Best regards,

Mike

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Abner Chang
Sent: Tuesday, November 30, 2021 11:34 PM
To: devel@edk2.groups.io; Wang, Jian J <jian.j.wang@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com>; Michael
Kubacki <michael.kubacki@microsoft.com>; Andrew Fish (afish@apple.com) <afish@apple.com>; Leif Lindholm
<leif@nuviainc.com>
Subject: Re: [edk2-devel] Uncrustify Conversion Detailed Plan and Extended Hard Freeze Update #4

Hi Mike,
Is that possible to not applying uncrusitify on the source files that cloned from other open source?
e.g. RedfishPkg/PrivateLibrary/RedfishLib/edk2libredfish/*.*

Thanks
Abner

-----Original Message-----
From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of
Wang, Jian J
Sent: Wednesday, December 1, 2021 2:39 PM
To: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>;
Michael Kubacki <michael.kubacki@microsoft.com>; Andrew Fish
(afish@apple.com) <afish@apple.com>; Leif Lindholm <leif@nuviainc.com>
Subject: Re: [edk2-devel] Uncrustify Conversion Detailed Plan and Extended
Hard Freeze Update #4

For SignedCapulePkg,

Reviewed-by: Jian J Wang <jian.j.wang@intel.com>

Regards,
Jian

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
Michael D
Kinney
Sent: Wednesday, December 01, 2021 6:34 AM
To: devel@edk2.groups.io; Kinney, Michael D
<michael.d.kinney@intel.com>;
Michael Kubacki <michael.kubacki@microsoft.com>; Andrew Fish
(afish@apple.com) <afish@apple.com>; Leif Lindholm
<leif@nuviainc.com>
Subject: [edk2-devel] Uncrustify Conversion Detailed Plan and Extended
Hard
Freeze Update #4

Hello,

Thank you for your patience during this extended hard freeze.

Just one more step to go. There has been a delay in the review of
the patch series with the uncrustify source changes. PR(6). This
patch series was not sent out as patch review email because of its
very large size. It only contains source style changes and the
CompareBuild tool and GitHub action has shown there are no binary
differences introduced with these source style changes.

If you are a package maintainer, then please review the following
branch/PR for your package contents and review the EDK II CI results
and BuildCompare results. I do not expect a line by line review
because we already had time to provide feedback on the source style
performed by uncrustify. Instead, a Reviewed-by for your package
indicates that you have reviewed the EDK II CI results and CompareBuild
tool functionality and results and you accept the source style
changes to your package.

*
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
han
ges_V5
* https://github.com/tianocore/edk2/pull/2229
* https://github.com/mdkinney/edk2/actions/runs/1521618836

Additional details on this update below.

Thank you,

Mike


Changes from Update #3
----------------------------------------------------------------------------
* Pushed PR (5)
* Added link to PR(6). EDK II CI Status is PASS. Build Compare PASS.
* Waiting for review of PR (6)
* Review of PR (7) completed and waiting for review of PR (6)
----------------------------------------------------------------------------

Changes from Update #2
----------------------------------------------------------------------------
* Changed order of PRs swapping (4) and (5). The PR that activates
increases the max CI agent job time is independent of all the other
PRs and its review is complete, so it can be committed now.
* Pushed PRs (1), (2), (3), (4).
* Waiting for review to complete for PRs (5) and (6)
* Reviews complete for PR (7)
* Identifies steps using git filter-branch to apply uncrustify changes to a
code review patch series that was generated before the uncrustify
changes
avoiding manual merge.
* Identified steps using git filter-repo to generate an alternate history of
the edk2 repo with uncrustify changes applied on every commit. This may
be useful when evaluating changes to files using tools like git blame
without the large uncrustify patch series.
---------------------------------------------------------------------------

Changes from Update #1
----------------------------------------------------------------------------
* Changed order of PRs swapping (6) and (7). The PR that activates
EDK II CI check UncrustifyCheck has to be last because it unconditionally
checks all C/H files in all packages. Not just files that have been
modified like some of the other checkers.
* Updated link to the branch with the UncrustifyCheck plugin that has been
updated with a one line change and Reviewed-by and Tested-by tags.

https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plug
in_v
6
* Reviews complete for (1), (2), (3), (5), and (7)
---------------------------------------------------------------------------

Michael Kubacki and I have prepared the patches required to apply the
uncrustify changes and enable EDK II CI to check all submitted
patches have been run through uncrustify.

We have verified through the CompareBuild GitHub Action that the
format changes performed by uncrustify have no functional changes.
All of the OBJ, LIB, DLL, EFI, FFS, FV, and FD files match 100%
across 70 VS2019/GCC5 builds of all package/platform DSC files in
the edk2 repo.

The hard freeze will be extended after the edk2-stable202111 tag until
all uncrustify related changes are committed. We do not expect this
to take more than a few days. Do not push any PRs until the hard
freeze is lifted.

The changes are broken up into 7 patch series/PRs. The PRs are ordered
so they can be submitted using the normal submission process and EDK II
CI will pass for each one. Details are listed below.

Uncrustify 73.0.3 for EDK II
=============================
* Sources:
INVALID URI REMOVED
stify__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJOYp
cSVHoHwMD2t8$
* Documentation:
INVALID URI REMOVED
/_wiki/wikis/Uncrustify.wiki/1/Proj__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKk
oslR8D0LLPZTWXL0OKtMZJOYpcSVHoP8OtlxM$
ect-Mu-(EDK-II)-Fork-Readme
* Download:
INVALID URI REMOVED
/_packaging?_a=package&feed=m__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKk
oslR8D0LLPZTWXL0OKtMZJOYpcSVHo4yTWeAM$
u_uncrustify&package=mu-uncrustify-
release&protocolType=NuGet&version=73.0.3

Installing Uncrustify
======================
The Uncrustify tool is installed automatically when the Pytools
environment is used and the stuart* commands are run to complete the
environment setup. Please see:

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-
locally

Uncrustify can also be installed from the download page listed above
or built from sources from the source link above.

The Documentation link provides instruction on how to run uncrustify from
the command line or install as a Visual Studio Code plugin. The main
uncrustify documentation also describes how to integrate with a few other
editors.

We have also discussed a client side githook. That effort has not started.
Let us know if that is a feature you would find useful.

Developer impact for new code reviews
======================================
Once the uncrustify checker is active in EDK II CI, developers must
make sure their patches are run through the uncrustify tool before
sending the patches for review.

Developers must install and run uncrustify against changes files before
sending patch review emails or submitting PR for EDK II CI. If EDK II CI
detects and differences in source formatting, then EDK II CI will fail
and the developer must run uncrustify and resubmit the patches.

Developer impact to patch series/PRs reviewed during edk2-stable201121
soft/hard freeze
==========================================================
=======
======================
Developers must rebase their changes after the uncrustify source changes
are
committed. The branch with a preview of the uncrustify changes can be
used
to start this rebase work.


https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
han
ges_V5

The following steps can be used to update an existing branch with the
required uncrustify format. This is the Windows version. I will add
the Linux version soon.

1) Fetch and checkout and rebase to latest edk2/master

git fetch origin
git checkout master
git rebase origin/master

2) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

3) Check out the patch series branch (e.g. MyBranch)

git checkout MyBranch

4) Rebase patch series against edk2-stable202111

git rebase edk2-stable202111

5) Create new branch for the uncrustifed version (e.g.
MyBranch_Uncrustified)

git checkout -b MyBranch_Uncrustified

6) Use git filter-branch to uncrustify all the commits in the series
between the rebase target from (2) and HEAD of the branch. A filter
can be used to scope the uncrustify operations to only the C/H files
in the specific package the patch series is against. (e.g.
DynamicTablesPkg).
BaseTools should always be excluded. If the package scoped filter is
not used, it will still work, but will take longer to run because
uncrustify will rescan every C/H files in the whole repo.

git filter-branch --tree-filter "git ls-files DynamicTablesPkg*.c
DynamicTablesPkg*.h :!BaseTools/* | c:\\Temp\\UncrustifyCheck\\mu-
uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c
c:\\Temp\\UncrustifyCheck\\uncrustify.cfg -F - --replace --no-backup --if-
changed" edk2-stable202111..HEAD

7) Now that all the individual patches in the branch are uncrustified,
rebase against latest edk2/master that is already uncrustified.

git rebase master

8) Verify the patches in this new branch.

Impacts to tracing history across the uncrusity changes
=======================================================
Tools the view file and line history do work with the large uncrustify
patch series. One impact is that the operations can be very slow due
to the large uncrustify patches.

One option to provide a faster experience is to provide an alternate
version of the edk2 repository as "documentation" that has the
entire history re-written with uncrustify run on every commit.
The tool called git-filter-repo can be used to perform this
transformation and runs in a reasonable period of time (a few hours)

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-repo/blob/main/contrib/filter-
repo-
demos/lint-history

The following steps can be used to perform this transformation.
This is the Windows version. I will add the Linux version soon.

** WARNING ** This operation modifies(rewrites) all the commits
in the local copy of the repo. Do not perform
these steps on a local repo you are using for
active development.

1) Clone edk2 into a new directory (see **WARNING**)

git clone https://github.com/tianocore/edk2.git edk2-uncrustified
cd edk2-uncrustified

2) Setup python virtual env, install pytools, and run stuart commands
to setup build environment which includes installing uncrustify tools.

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-
locally

3) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

4) Use lint-history.py from git-filter-repo examples

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-repo/blob/main/contrib/filter-
repo-
demos/lint-history

Line #127 - Add try except around subprocess.check_call() with except
being pass. This is required because there are a few commits of C
files in the edk2 repo that have incorrect C syntax and do not
build with a C compiler and break the uncrustify parser. Skip reformat
of C files that can not be parsed by uncrustify. These rare instances
are addressed in the commit that fixes the C syntax error.

Run this slightly modified version of lint-history. Include only
C/H files and exclude directories that start with 'Tools' or 'BaseTools'.
This step took about 2.2 hours on a laptop.

lint-history.py
--relevant "return (not filename.startswith(b'Tools') and not
filename.startswith(b'BaseTools') and (filename.endswith(b'.c') or
filename.endswith(b'.h')))"
c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\mu-
uncrustify-
release_extdep\\Windows-x86\\uncrustify.exe -c
c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\uncrustify.cfg --
replace --
no-backup --if-changed

Order of PRs to apply during extended hard freeze
==================================================
1) Update EmulatorPkg Win Host [BuildOptions] MSFT CC_FLAGS to not
force
debug information
*
INVALID URI REMOVED
d=3747__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHo2BEG33c$
*
https://github.com/mdkinney/edk2/tree/Bug_3747_EmulatorPkg_WinHost_
Rep
roducibleBuild
* https://github.com/tianocore/edk2/pull/2215
* Required for EmulatorPkg to pass CompareBuild for VS2019 IA32/X64
builds.
* Status: Review complete. PR pushed.

2) EccCheck should not revert staged and local changes
*
INVALID URI REMOVED
d=2986__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHotvR0Unk$
*
https://github.com/mdkinney/edk2/tree/Bug_2986_EccCheckRemoveGitRe
vert
_V2
* https://github.com/tianocore/edk2/pull/2216
* Required for EDK II CI to complete in a reasonable period of time when
processing the 4000+ source file style changes made by uncrustify.
* Also fixes critical bugs that can potentially corrupt git state when
EccCheck is run locally.
* Status: Review complete. PR pushed.

3) Update pytool LicenseCheck plugin to use temp directory for diff output
file
*
INVALID URI REMOVED
d=3746__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHoWRuHaBw$
*
https://github.com/mdkinney/edk2/tree/Bug_3746_LicenseCheckUseDiffO
utput
File_V2
* https://github.com/tianocore/edk2/pull/2217
* Required to reduce EDK II CI build times.
* Status: Review complete. PR pushed.

4) Update max job time from 60 min to 120 minutes
in .azurepipelines/templates
*
INVALID URI REMOVED
d=3750__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHozaVshgc$
*
https://github.com/mdkinney/edk2/tree/Bug_3750_IncreaseAzurePipelines
Time
out
* https://github.com/tianocore/edk2/pull/2219
* Required to allow EccCheck of uncrustify changes to complete on Azure
Pipelines CI agents without timing out.
* Status: Review complete. PR pushed.

5) Update Package YAML to ignore specific ECC files/errors
*
INVALID URI REMOVED
d=3749__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHo-fY3dMo$
*
https://github.com/mdkinney/edk2/tree/Bug_3749_EccCheckIgnoreFilesErr
ors
* https://github.com/tianocore/edk2/pull/2218
* Required to pass EccCheck
* Status: Review complete. PR pushed

6) Uncrustify Source Changes
*
INVALID URI REMOVED
d=3737__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHoMRa47YU$
*
INVALID URI REMOVED
d=3739__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHoPAlckP8$
*
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
han
ges_V5
* https://github.com/tianocore/edk2/pull/2229
* Build comparison result PASS:
https://github.com/mdkinney/edk2/actions/runs/1521618836
* EFI_D_ -> DEBUG changes required to pass PatchCheck
* Uncrustify format changes required to pass UncrustifyCheck
* Status:
Waiting for review

7) UncrustifyCheck EDK II CI Plugin
*
INVALID URI REMOVED
d=3748__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHoCzEWymQ$
*
https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plug
in_v
6
* Required to enforce all PRs submitted to EDK II CI match uncrustify
format.
* Unconditionally checks all packages. Can not be committed until all C/H
source files have been updated.
* Status: Review complete

Combined Branch/PR for Review/Test
==================================
* Build Comparison results must pass 100% across the full set of PRs before
the individual PRs can be pushed in the order listed above.
* Branch:
https://github.com/mdkinney/edk2/tree/TestOnly_Uncrustify_PR_Series
* PR: https://github.com/tianocore/edk2/pull/2229
Status = PASS
* CompareBuild:
Branch:
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
han
ges_V5
--ref1: ef9a059cdb15844fe52a49af2bf7d86b9dd3e9bf
--ref2: Bug_3737_3739_ApplyUncrustifyChanges_V5
Extra Options: -n 4 --quiet
Results: https://github.com/mdkinney/edk2/actions/runs/1521618836
30 VS2019 build comparisons PASS
40 GCC5 build comparisons PASS
100% PASS

The following git log shows the set of patches from --ref1 to --ref 2across
which there are no differences in any of the OBJ/LIB/DLL/EFI/FFS/FV/FD
files.

--ref2
b7d4bf0675b7 (HEAD -> Bug_3737_3739_ApplyUncrustifyChanges_V5)
UnitTestFrameworkPkg: Apply uncrusitify changes
7f03d25f60e7 UefiPayloadPkg: Apply uncrusitify changes
0bfd8d9b5ac9 UefiCpuPkg: Apply uncrusitify changes
e1cd9bfb9dea StandaloneMmPkg: Apply uncrusitify changes
5da2f65be378 SourceLevelDebugPkg: Apply uncrusitify changes
95b86de07e5d SignedCapsulePkg: Apply uncrusitify changes
fe71d97246c4 ShellPkg: Apply uncrusitify changes
54c21c952992 SecurityPkg: Apply uncrusitify changes
187a3785f12b RedfishPkg: Apply uncrusitify changes
810100002a46 PcAtChipsetPkg: Apply uncrusitify changes
276a695c0cf2 OvmfPkg: Apply uncrusitify changes
303c0a91ab07 NetworkPkg: Apply uncrusitify changes
bc80792cd1b1 MdePkg: Apply uncrusitify changes
3ea86be17a2a MdeModulePkg: Apply uncrusitify changes
c70ef11ed0cd IntelFsp2WrapperPkg: Apply uncrusitify changes
c0291221f252 IntelFsp2Pkg: Apply uncrusitify changes
6a479952a690 FmpDevicePkg: Apply uncrusitify changes
3a7c05b7070d FatPkg: Apply uncrusitify changes
b789f98c8959 EmulatorPkg: Apply uncrusitify changes
952d7a1c9220 EmbeddedPkg: Apply uncrusitify changes
a1cc9881bab6 DynamicTablesPkg: Apply uncrusitify changes
50654dfe5785 CryptoPkg: Apply uncrusitify changes
ed965a02dfa1 ArmVirtPkg: Apply uncrusitify changes
9744023fbc46 ArmPlatformPkg: Apply uncrusitify changes
7a1cde5f5bba ArmPkg: Apply uncrusitify changes
19d17e0913e8 UefiCpuPkg: Change use of EFI_D_* to DEBUG_*
ffa718b4f994 SourceLevelDebugPkg: Change use of EFI_D_* to DEBUG_*
b86cb3c5e5b4 ShellPkg: Change use of EFI_D_* to DEBUG_*
c7c42204dc07 SecurityPkg: Change use of EFI_D_* to DEBUG_*
16b8e6f958e4 PcAtChipsetPkg: Change use of EFI_D_* to DEBUG_*
0ac3f8b2dac5 OvmfPkg: Change use of EFI_D_* to DEBUG_*
bc5004b8d294 NetworkPkg: Change use of EFI_D_* to DEBUG_*
6f671a8e2377 MdePkg: Change use of EFI_D_* to DEBUG_*
a10c610ff9a3 MdeModulePkg: Change use of EFI_D_* to DEBUG_*
09a3bddba390 FatPkg: Change use of EFI_D_* to DEBUG_*
59c61318246a EmulatorPkg: Change use of EFI_D_* to DEBUG_*
3a80367dda3b EmbeddedPkg: Change use of EFI_D_* to DEBUG_*
23eb1aaf80ca ArmVirtPkg: Change use of EFI_D_* to DEBUG_*
875914b45c54 ArmPlatformPkg: Change use of EFI_D_* to DEBUG_*
eb2eca82b451 ArmPkg: Change use of EFI_D_* to DEBUG_*
f0f3f5aae7c4 (origin/master, origin/HEAD, master) UnitTestFrameworkPkg:
Update YAML to ignore specific ECC files/errors
c05734797790 UefiPayloadPkg: Update YAML to ignore specific ECC
files/errors
c30c40d6c63d StandaloneMmPkg: Update YAML to ignore specific ECC
files/errors
9944508e85f1 ShellPkg: Update YAML to ignore specific ECC files/errors
60fa40be458d SecurityPkg: Update YAML to ignore specific ECC files/errors
df790cd6b37e MdePkg: Update YAML to ignore specific ECC files/errors
9deb9370766e MdeModulePkg: Update YAML to ignore specific ECC
files/errors
d7d30e8f219f EmulatorPkg: Update YAML to ignore specific ECC files/errors
d5744ecba813 CryptoPkg: Update YAML to ignore specific ECC files/errors
c97fee87f0f9 ArmVirtPkg: Update YAML to ignore specific ECC files/errors
1939fc9569f2 ArmPlatformPkg: Update YAML to ignore specific ECC
files/errors
365dced2c37a ArmPkg: Update YAML to ignore specific ECC files/errors
76a1ce4d5fec .azurepipelines/templates: Update max pipeline job time to
2
hours
99f84ff47390 .pytools/Plugin/LicenseCheck: Use temp directory for git diff
output
3019f1bbabf1 .pytool/Plugin/EccCheck: Add performance optimizations
854462bd3479 .pytool/Plugin/EccCheck: Remove temp directory on
exception
69877614fdee .pytool/Plugin/EccCheck: Remove RevertCode()
--ref1
ef9a059cdb15 EmulatorPkg/Win/Host: Update CC_FLAGS
bb1bba3d7767 (tag: edk2-stable202111) NetworkPkg: Fix invalid pointer for
DNS
response token on error

Best regards,

Mike















Michael Kubacki
 

This can of course be done at a technical level but it smells of a code cohesion problem.

The code is in the edk2 project so it would reason that it should be uniform in style with other code and at least close to the EDK II C Coding Standard Specification.

Currently, git submodules and ignored path are already excluded in addition to the ability to mark packages in "audit mode" so they do not fail the results. I would personally prefer not to allow arbitrary subdirectories to diverge from the rest of the codebase / package but if there's consensus this is a good idea, I can look into adding it.

Regards,
Michael

On 12/1/2021 11:43 AM, Michael D Kinney wrote:
Hi Abner,
Yes. That is possible. We are already excluding BaseTools.
In order to get the file lists that apply to uncrustify, we would have to change from:
git ls-files *.c *.h :!BaseTools/*
To:
git ls-files *.c *.h :!BaseTools/* :!RedfishPkg/PrivateLibrary/RedfishLib/edk2libredfish/*
However, I do not see a feature in the UncrustifyCheck to check all c/h files in RedFishPkg and exclude one of its directories.
If we can specify an exclusion like this in the RedFishPkg YAML file, then I think that would work well.
Michael Kubacki would have to comment on how we support this in EDK II CI checks.
Best regards,
Mike

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Abner Chang
Sent: Tuesday, November 30, 2021 11:34 PM
To: devel@edk2.groups.io; Wang, Jian J <jian.j.wang@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com>; Michael
Kubacki <michael.kubacki@microsoft.com>; Andrew Fish (afish@apple.com) <afish@apple.com>; Leif Lindholm
<leif@nuviainc.com>
Subject: Re: [edk2-devel] Uncrustify Conversion Detailed Plan and Extended Hard Freeze Update #4

Hi Mike,
Is that possible to not applying uncrusitify on the source files that cloned from other open source?
e.g. RedfishPkg/PrivateLibrary/RedfishLib/edk2libredfish/*.*

Thanks
Abner

-----Original Message-----
From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of
Wang, Jian J
Sent: Wednesday, December 1, 2021 2:39 PM
To: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>;
Michael Kubacki <michael.kubacki@microsoft.com>; Andrew Fish
(afish@apple.com) <afish@apple.com>; Leif Lindholm <leif@nuviainc.com>
Subject: Re: [edk2-devel] Uncrustify Conversion Detailed Plan and Extended
Hard Freeze Update #4

For SignedCapulePkg,

Reviewed-by: Jian J Wang <jian.j.wang@intel.com>

Regards,
Jian

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
Michael D
Kinney
Sent: Wednesday, December 01, 2021 6:34 AM
To: devel@edk2.groups.io; Kinney, Michael D
<michael.d.kinney@intel.com>;
Michael Kubacki <michael.kubacki@microsoft.com>; Andrew Fish
(afish@apple.com) <afish@apple.com>; Leif Lindholm
<leif@nuviainc.com>
Subject: [edk2-devel] Uncrustify Conversion Detailed Plan and Extended
Hard
Freeze Update #4

Hello,

Thank you for your patience during this extended hard freeze.

Just one more step to go. There has been a delay in the review of
the patch series with the uncrustify source changes. PR(6). This
patch series was not sent out as patch review email because of its
very large size. It only contains source style changes and the
CompareBuild tool and GitHub action has shown there are no binary
differences introduced with these source style changes.

If you are a package maintainer, then please review the following
branch/PR for your package contents and review the EDK II CI results
and BuildCompare results. I do not expect a line by line review
because we already had time to provide feedback on the source style
performed by uncrustify. Instead, a Reviewed-by for your package
indicates that you have reviewed the EDK II CI results and CompareBuild
tool functionality and results and you accept the source style
changes to your package.

*
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
han
ges_V5
* https://github.com/tianocore/edk2/pull/2229
* https://github.com/mdkinney/edk2/actions/runs/1521618836

Additional details on this update below.

Thank you,

Mike


Changes from Update #3
----------------------------------------------------------------------------
* Pushed PR (5)
* Added link to PR(6). EDK II CI Status is PASS. Build Compare PASS.
* Waiting for review of PR (6)
* Review of PR (7) completed and waiting for review of PR (6)
----------------------------------------------------------------------------

Changes from Update #2
----------------------------------------------------------------------------
* Changed order of PRs swapping (4) and (5). The PR that activates
increases the max CI agent job time is independent of all the other
PRs and its review is complete, so it can be committed now.
* Pushed PRs (1), (2), (3), (4).
* Waiting for review to complete for PRs (5) and (6)
* Reviews complete for PR (7)
* Identifies steps using git filter-branch to apply uncrustify changes to a
code review patch series that was generated before the uncrustify
changes
avoiding manual merge.
* Identified steps using git filter-repo to generate an alternate history of
the edk2 repo with uncrustify changes applied on every commit. This may
be useful when evaluating changes to files using tools like git blame
without the large uncrustify patch series.
---------------------------------------------------------------------------

Changes from Update #1
----------------------------------------------------------------------------
* Changed order of PRs swapping (6) and (7). The PR that activates
EDK II CI check UncrustifyCheck has to be last because it unconditionally
checks all C/H files in all packages. Not just files that have been
modified like some of the other checkers.
* Updated link to the branch with the UncrustifyCheck plugin that has been
updated with a one line change and Reviewed-by and Tested-by tags.

https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plug
in_v
6
* Reviews complete for (1), (2), (3), (5), and (7)
---------------------------------------------------------------------------

Michael Kubacki and I have prepared the patches required to apply the
uncrustify changes and enable EDK II CI to check all submitted
patches have been run through uncrustify.

We have verified through the CompareBuild GitHub Action that the
format changes performed by uncrustify have no functional changes.
All of the OBJ, LIB, DLL, EFI, FFS, FV, and FD files match 100%
across 70 VS2019/GCC5 builds of all package/platform DSC files in
the edk2 repo.

The hard freeze will be extended after the edk2-stable202111 tag until
all uncrustify related changes are committed. We do not expect this
to take more than a few days. Do not push any PRs until the hard
freeze is lifted.

The changes are broken up into 7 patch series/PRs. The PRs are ordered
so they can be submitted using the normal submission process and EDK II
CI will pass for each one. Details are listed below.

Uncrustify 73.0.3 for EDK II
=============================
* Sources:
INVALID URI REMOVED
stify__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJOYp
cSVHoHwMD2t8$
* Documentation:
INVALID URI REMOVED
/_wiki/wikis/Uncrustify.wiki/1/Proj__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKk
oslR8D0LLPZTWXL0OKtMZJOYpcSVHoP8OtlxM$
ect-Mu-(EDK-II)-Fork-Readme
* Download:
INVALID URI REMOVED
/_packaging?_a=package&feed=m__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKk
oslR8D0LLPZTWXL0OKtMZJOYpcSVHo4yTWeAM$
u_uncrustify&package=mu-uncrustify-
release&protocolType=NuGet&version=73.0.3

Installing Uncrustify
======================
The Uncrustify tool is installed automatically when the Pytools
environment is used and the stuart* commands are run to complete the
environment setup. Please see:

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-
locally

Uncrustify can also be installed from the download page listed above
or built from sources from the source link above.

The Documentation link provides instruction on how to run uncrustify from
the command line or install as a Visual Studio Code plugin. The main
uncrustify documentation also describes how to integrate with a few other
editors.

We have also discussed a client side githook. That effort has not started.
Let us know if that is a feature you would find useful.

Developer impact for new code reviews
======================================
Once the uncrustify checker is active in EDK II CI, developers must
make sure their patches are run through the uncrustify tool before
sending the patches for review.

Developers must install and run uncrustify against changes files before
sending patch review emails or submitting PR for EDK II CI. If EDK II CI
detects and differences in source formatting, then EDK II CI will fail
and the developer must run uncrustify and resubmit the patches.

Developer impact to patch series/PRs reviewed during edk2-stable201121
soft/hard freeze
==========================================================
=======
======================
Developers must rebase their changes after the uncrustify source changes
are
committed. The branch with a preview of the uncrustify changes can be
used
to start this rebase work.


https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
han
ges_V5

The following steps can be used to update an existing branch with the
required uncrustify format. This is the Windows version. I will add
the Linux version soon.

1) Fetch and checkout and rebase to latest edk2/master

git fetch origin
git checkout master
git rebase origin/master

2) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

3) Check out the patch series branch (e.g. MyBranch)

git checkout MyBranch

4) Rebase patch series against edk2-stable202111

git rebase edk2-stable202111

5) Create new branch for the uncrustifed version (e.g.
MyBranch_Uncrustified)

git checkout -b MyBranch_Uncrustified

6) Use git filter-branch to uncrustify all the commits in the series
between the rebase target from (2) and HEAD of the branch. A filter
can be used to scope the uncrustify operations to only the C/H files
in the specific package the patch series is against. (e.g.
DynamicTablesPkg).
BaseTools should always be excluded. If the package scoped filter is
not used, it will still work, but will take longer to run because
uncrustify will rescan every C/H files in the whole repo.

git filter-branch --tree-filter "git ls-files DynamicTablesPkg*.c
DynamicTablesPkg*.h :!BaseTools/* | c:\\Temp\\UncrustifyCheck\\mu-
uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c
c:\\Temp\\UncrustifyCheck\\uncrustify.cfg -F - --replace --no-backup --if-
changed" edk2-stable202111..HEAD

7) Now that all the individual patches in the branch are uncrustified,
rebase against latest edk2/master that is already uncrustified.

git rebase master

8) Verify the patches in this new branch.

Impacts to tracing history across the uncrusity changes
=======================================================
Tools the view file and line history do work with the large uncrustify
patch series. One impact is that the operations can be very slow due
to the large uncrustify patches.

One option to provide a faster experience is to provide an alternate
version of the edk2 repository as "documentation" that has the
entire history re-written with uncrustify run on every commit.
The tool called git-filter-repo can be used to perform this
transformation and runs in a reasonable period of time (a few hours)

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-repo/blob/main/contrib/filter-
repo-
demos/lint-history

The following steps can be used to perform this transformation.
This is the Windows version. I will add the Linux version soon.

** WARNING ** This operation modifies(rewrites) all the commits
in the local copy of the repo. Do not perform
these steps on a local repo you are using for
active development.

1) Clone edk2 into a new directory (see **WARNING**)

git clone https://github.com/tianocore/edk2.git edk2-uncrustified
cd edk2-uncrustified

2) Setup python virtual env, install pytools, and run stuart commands
to setup build environment which includes installing uncrustify tools.

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-
locally

3) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

4) Use lint-history.py from git-filter-repo examples

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-repo/blob/main/contrib/filter-
repo-
demos/lint-history

Line #127 - Add try except around subprocess.check_call() with except
being pass. This is required because there are a few commits of C
files in the edk2 repo that have incorrect C syntax and do not
build with a C compiler and break the uncrustify parser. Skip reformat
of C files that can not be parsed by uncrustify. These rare instances
are addressed in the commit that fixes the C syntax error.

Run this slightly modified version of lint-history. Include only
C/H files and exclude directories that start with 'Tools' or 'BaseTools'.
This step took about 2.2 hours on a laptop.

lint-history.py
--relevant "return (not filename.startswith(b'Tools') and not
filename.startswith(b'BaseTools') and (filename.endswith(b'.c') or
filename.endswith(b'.h')))"
c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\mu-
uncrustify-
release_extdep\\Windows-x86\\uncrustify.exe -c
c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\uncrustify.cfg --
replace --
no-backup --if-changed

Order of PRs to apply during extended hard freeze
==================================================
1) Update EmulatorPkg Win Host [BuildOptions] MSFT CC_FLAGS to not
force
debug information
*
INVALID URI REMOVED
d=3747__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHo2BEG33c$
*
https://github.com/mdkinney/edk2/tree/Bug_3747_EmulatorPkg_WinHost_
Rep
roducibleBuild
* https://github.com/tianocore/edk2/pull/2215
* Required for EmulatorPkg to pass CompareBuild for VS2019 IA32/X64
builds.
* Status: Review complete. PR pushed.

2) EccCheck should not revert staged and local changes
*
INVALID URI REMOVED
d=2986__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHotvR0Unk$
*
https://github.com/mdkinney/edk2/tree/Bug_2986_EccCheckRemoveGitRe
vert
_V2
* https://github.com/tianocore/edk2/pull/2216
* Required for EDK II CI to complete in a reasonable period of time when
processing the 4000+ source file style changes made by uncrustify.
* Also fixes critical bugs that can potentially corrupt git state when
EccCheck is run locally.
* Status: Review complete. PR pushed.

3) Update pytool LicenseCheck plugin to use temp directory for diff output
file
*
INVALID URI REMOVED
d=3746__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHoWRuHaBw$
*
https://github.com/mdkinney/edk2/tree/Bug_3746_LicenseCheckUseDiffO
utput
File_V2
* https://github.com/tianocore/edk2/pull/2217
* Required to reduce EDK II CI build times.
* Status: Review complete. PR pushed.

4) Update max job time from 60 min to 120 minutes
in .azurepipelines/templates
*
INVALID URI REMOVED
d=3750__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHozaVshgc$
*
https://github.com/mdkinney/edk2/tree/Bug_3750_IncreaseAzurePipelines
Time
out
* https://github.com/tianocore/edk2/pull/2219
* Required to allow EccCheck of uncrustify changes to complete on Azure
Pipelines CI agents without timing out.
* Status: Review complete. PR pushed.

5) Update Package YAML to ignore specific ECC files/errors
*
INVALID URI REMOVED
d=3749__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHo-fY3dMo$
*
https://github.com/mdkinney/edk2/tree/Bug_3749_EccCheckIgnoreFilesErr
ors
* https://github.com/tianocore/edk2/pull/2218
* Required to pass EccCheck
* Status: Review complete. PR pushed

6) Uncrustify Source Changes
*
INVALID URI REMOVED
d=3737__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHoMRa47YU$
*
INVALID URI REMOVED
d=3739__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHoPAlckP8$
*
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
han
ges_V5
* https://github.com/tianocore/edk2/pull/2229
* Build comparison result PASS:
https://github.com/mdkinney/edk2/actions/runs/1521618836
* EFI_D_ -> DEBUG changes required to pass PatchCheck
* Uncrustify format changes required to pass UncrustifyCheck
* Status:
Waiting for review

7) UncrustifyCheck EDK II CI Plugin
*
INVALID URI REMOVED
d=3748__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHoCzEWymQ$
*
https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plug
in_v
6
* Required to enforce all PRs submitted to EDK II CI match uncrustify
format.
* Unconditionally checks all packages. Can not be committed until all C/H
source files have been updated.
* Status: Review complete

Combined Branch/PR for Review/Test
==================================
* Build Comparison results must pass 100% across the full set of PRs before
the individual PRs can be pushed in the order listed above.
* Branch:
https://github.com/mdkinney/edk2/tree/TestOnly_Uncrustify_PR_Series
* PR: https://github.com/tianocore/edk2/pull/2229
Status = PASS
* CompareBuild:
Branch:
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
han
ges_V5
--ref1: ef9a059cdb15844fe52a49af2bf7d86b9dd3e9bf
--ref2: Bug_3737_3739_ApplyUncrustifyChanges_V5
Extra Options: -n 4 --quiet
Results: https://github.com/mdkinney/edk2/actions/runs/1521618836
30 VS2019 build comparisons PASS
40 GCC5 build comparisons PASS
100% PASS

The following git log shows the set of patches from --ref1 to --ref 2across
which there are no differences in any of the OBJ/LIB/DLL/EFI/FFS/FV/FD
files.

--ref2
b7d4bf0675b7 (HEAD -> Bug_3737_3739_ApplyUncrustifyChanges_V5)
UnitTestFrameworkPkg: Apply uncrusitify changes
7f03d25f60e7 UefiPayloadPkg: Apply uncrusitify changes
0bfd8d9b5ac9 UefiCpuPkg: Apply uncrusitify changes
e1cd9bfb9dea StandaloneMmPkg: Apply uncrusitify changes
5da2f65be378 SourceLevelDebugPkg: Apply uncrusitify changes
95b86de07e5d SignedCapsulePkg: Apply uncrusitify changes
fe71d97246c4 ShellPkg: Apply uncrusitify changes
54c21c952992 SecurityPkg: Apply uncrusitify changes
187a3785f12b RedfishPkg: Apply uncrusitify changes
810100002a46 PcAtChipsetPkg: Apply uncrusitify changes
276a695c0cf2 OvmfPkg: Apply uncrusitify changes
303c0a91ab07 NetworkPkg: Apply uncrusitify changes
bc80792cd1b1 MdePkg: Apply uncrusitify changes
3ea86be17a2a MdeModulePkg: Apply uncrusitify changes
c70ef11ed0cd IntelFsp2WrapperPkg: Apply uncrusitify changes
c0291221f252 IntelFsp2Pkg: Apply uncrusitify changes
6a479952a690 FmpDevicePkg: Apply uncrusitify changes
3a7c05b7070d FatPkg: Apply uncrusitify changes
b789f98c8959 EmulatorPkg: Apply uncrusitify changes
952d7a1c9220 EmbeddedPkg: Apply uncrusitify changes
a1cc9881bab6 DynamicTablesPkg: Apply uncrusitify changes
50654dfe5785 CryptoPkg: Apply uncrusitify changes
ed965a02dfa1 ArmVirtPkg: Apply uncrusitify changes
9744023fbc46 ArmPlatformPkg: Apply uncrusitify changes
7a1cde5f5bba ArmPkg: Apply uncrusitify changes
19d17e0913e8 UefiCpuPkg: Change use of EFI_D_* to DEBUG_*
ffa718b4f994 SourceLevelDebugPkg: Change use of EFI_D_* to DEBUG_*
b86cb3c5e5b4 ShellPkg: Change use of EFI_D_* to DEBUG_*
c7c42204dc07 SecurityPkg: Change use of EFI_D_* to DEBUG_*
16b8e6f958e4 PcAtChipsetPkg: Change use of EFI_D_* to DEBUG_*
0ac3f8b2dac5 OvmfPkg: Change use of EFI_D_* to DEBUG_*
bc5004b8d294 NetworkPkg: Change use of EFI_D_* to DEBUG_*
6f671a8e2377 MdePkg: Change use of EFI_D_* to DEBUG_*
a10c610ff9a3 MdeModulePkg: Change use of EFI_D_* to DEBUG_*
09a3bddba390 FatPkg: Change use of EFI_D_* to DEBUG_*
59c61318246a EmulatorPkg: Change use of EFI_D_* to DEBUG_*
3a80367dda3b EmbeddedPkg: Change use of EFI_D_* to DEBUG_*
23eb1aaf80ca ArmVirtPkg: Change use of EFI_D_* to DEBUG_*
875914b45c54 ArmPlatformPkg: Change use of EFI_D_* to DEBUG_*
eb2eca82b451 ArmPkg: Change use of EFI_D_* to DEBUG_*
f0f3f5aae7c4 (origin/master, origin/HEAD, master) UnitTestFrameworkPkg:
Update YAML to ignore specific ECC files/errors
c05734797790 UefiPayloadPkg: Update YAML to ignore specific ECC
files/errors
c30c40d6c63d StandaloneMmPkg: Update YAML to ignore specific ECC
files/errors
9944508e85f1 ShellPkg: Update YAML to ignore specific ECC files/errors
60fa40be458d SecurityPkg: Update YAML to ignore specific ECC files/errors
df790cd6b37e MdePkg: Update YAML to ignore specific ECC files/errors
9deb9370766e MdeModulePkg: Update YAML to ignore specific ECC
files/errors
d7d30e8f219f EmulatorPkg: Update YAML to ignore specific ECC files/errors
d5744ecba813 CryptoPkg: Update YAML to ignore specific ECC files/errors
c97fee87f0f9 ArmVirtPkg: Update YAML to ignore specific ECC files/errors
1939fc9569f2 ArmPlatformPkg: Update YAML to ignore specific ECC
files/errors
365dced2c37a ArmPkg: Update YAML to ignore specific ECC files/errors
76a1ce4d5fec .azurepipelines/templates: Update max pipeline job time to
2
hours
99f84ff47390 .pytools/Plugin/LicenseCheck: Use temp directory for git diff
output
3019f1bbabf1 .pytool/Plugin/EccCheck: Add performance optimizations
854462bd3479 .pytool/Plugin/EccCheck: Remove temp directory on
exception
69877614fdee .pytool/Plugin/EccCheck: Remove RevertCode()
--ref1
ef9a059cdb15 EmulatorPkg/Win/Host: Update CC_FLAGS
bb1bba3d7767 (tag: edk2-stable202111) NetworkPkg: Fix invalid pointer for
DNS
response token on error

Best regards,

Mike















Michael D Kinney
 

Hi Michael,

This commit provides the background on why this content was copied into edk2
instead of used as a submodule.

https://github.com/tianocore/edk2/commit/4751a48aeb2ab828b0a5cbdc585fd3642967cda1#diff-9f77cfa3b537eee28489f8713c17d8702330f5cef1795f154201dd463aabed27

I imagine the main reason for this request is to minimize the effort required
to sync these copies with new versions. The scope right now is only 7 files.
One alternative to help with the sync is for Abner to run uncrustify on the
files from the upstream source on these 7 files and then do the diff to see if
there is anything significant that needs to be merged.

Abner, are you ok with doing this 2 step process to sync the files that were copied?

Best regards,

Mike

-----Original Message-----
From: Michael Kubacki <mikuback@linux.microsoft.com>
Sent: Wednesday, December 1, 2021 9:05 AM
To: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>; Chang, Abner <abner.chang@hpe.com>; Wang, Jian J
<jian.j.wang@intel.com>; Michael Kubacki <michael.kubacki@microsoft.com>; Andrew Fish (afish@apple.com) <afish@apple.com>;
Leif Lindholm <leif@nuviainc.com>
Subject: Re: [edk2-devel] Uncrustify Conversion Detailed Plan and Extended Hard Freeze Update #4

This can of course be done at a technical level but it smells of a code
cohesion problem.

The code is in the edk2 project so it would reason that it should be
uniform in style with other code and at least close to the EDK II C
Coding Standard Specification.

Currently, git submodules and ignored path are already excluded in
addition to the ability to mark packages in "audit mode" so they do not
fail the results. I would personally prefer not to allow arbitrary
subdirectories to diverge from the rest of the codebase / package but if
there's consensus this is a good idea, I can look into adding it.

Regards,
Michael

On 12/1/2021 11:43 AM, Michael D Kinney wrote:
Hi Abner,

Yes. That is possible. We are already excluding BaseTools.

In order to get the file lists that apply to uncrustify, we would have to change from:

git ls-files *.c *.h :!BaseTools/*

To:

git ls-files *.c *.h :!BaseTools/* :!RedfishPkg/PrivateLibrary/RedfishLib/edk2libredfish/*

However, I do not see a feature in the UncrustifyCheck to check all c/h files in RedFishPkg and exclude one of its
directories.
If we can specify an exclusion like this in the RedFishPkg YAML file, then I think that would work well.

Michael Kubacki would have to comment on how we support this in EDK II CI checks.

Best regards,

Mike


-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Abner Chang
Sent: Tuesday, November 30, 2021 11:34 PM
To: devel@edk2.groups.io; Wang, Jian J <jian.j.wang@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com>; Michael
Kubacki <michael.kubacki@microsoft.com>; Andrew Fish (afish@apple.com) <afish@apple.com>; Leif Lindholm
<leif@nuviainc.com>
Subject: Re: [edk2-devel] Uncrustify Conversion Detailed Plan and Extended Hard Freeze Update #4

Hi Mike,
Is that possible to not applying uncrusitify on the source files that cloned from other open source?
e.g. RedfishPkg/PrivateLibrary/RedfishLib/edk2libredfish/*.*

Thanks
Abner

-----Original Message-----
From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of
Wang, Jian J
Sent: Wednesday, December 1, 2021 2:39 PM
To: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>;
Michael Kubacki <michael.kubacki@microsoft.com>; Andrew Fish
(afish@apple.com) <afish@apple.com>; Leif Lindholm <leif@nuviainc.com>
Subject: Re: [edk2-devel] Uncrustify Conversion Detailed Plan and Extended
Hard Freeze Update #4

For SignedCapulePkg,

Reviewed-by: Jian J Wang <jian.j.wang@intel.com>

Regards,
Jian

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
Michael D
Kinney
Sent: Wednesday, December 01, 2021 6:34 AM
To: devel@edk2.groups.io; Kinney, Michael D
<michael.d.kinney@intel.com>;
Michael Kubacki <michael.kubacki@microsoft.com>; Andrew Fish
(afish@apple.com) <afish@apple.com>; Leif Lindholm
<leif@nuviainc.com>
Subject: [edk2-devel] Uncrustify Conversion Detailed Plan and Extended
Hard
Freeze Update #4

Hello,

Thank you for your patience during this extended hard freeze.

Just one more step to go. There has been a delay in the review of
the patch series with the uncrustify source changes. PR(6). This
patch series was not sent out as patch review email because of its
very large size. It only contains source style changes and the
CompareBuild tool and GitHub action has shown there are no binary
differences introduced with these source style changes.

If you are a package maintainer, then please review the following
branch/PR for your package contents and review the EDK II CI results
and BuildCompare results. I do not expect a line by line review
because we already had time to provide feedback on the source style
performed by uncrustify. Instead, a Reviewed-by for your package
indicates that you have reviewed the EDK II CI results and CompareBuild
tool functionality and results and you accept the source style
changes to your package.

*
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
han
ges_V5
* https://github.com/tianocore/edk2/pull/2229
* https://github.com/mdkinney/edk2/actions/runs/1521618836

Additional details on this update below.

Thank you,

Mike


Changes from Update #3
----------------------------------------------------------------------------
* Pushed PR (5)
* Added link to PR(6). EDK II CI Status is PASS. Build Compare PASS.
* Waiting for review of PR (6)
* Review of PR (7) completed and waiting for review of PR (6)
----------------------------------------------------------------------------

Changes from Update #2
----------------------------------------------------------------------------
* Changed order of PRs swapping (4) and (5). The PR that activates
increases the max CI agent job time is independent of all the other
PRs and its review is complete, so it can be committed now.
* Pushed PRs (1), (2), (3), (4).
* Waiting for review to complete for PRs (5) and (6)
* Reviews complete for PR (7)
* Identifies steps using git filter-branch to apply uncrustify changes to a
code review patch series that was generated before the uncrustify
changes
avoiding manual merge.
* Identified steps using git filter-repo to generate an alternate history of
the edk2 repo with uncrustify changes applied on every commit. This may
be useful when evaluating changes to files using tools like git blame
without the large uncrustify patch series.
---------------------------------------------------------------------------

Changes from Update #1
----------------------------------------------------------------------------
* Changed order of PRs swapping (6) and (7). The PR that activates
EDK II CI check UncrustifyCheck has to be last because it unconditionally
checks all C/H files in all packages. Not just files that have been
modified like some of the other checkers.
* Updated link to the branch with the UncrustifyCheck plugin that has been
updated with a one line change and Reviewed-by and Tested-by tags.

https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plug
in_v
6
* Reviews complete for (1), (2), (3), (5), and (7)
---------------------------------------------------------------------------

Michael Kubacki and I have prepared the patches required to apply the
uncrustify changes and enable EDK II CI to check all submitted
patches have been run through uncrustify.

We have verified through the CompareBuild GitHub Action that the
format changes performed by uncrustify have no functional changes.
All of the OBJ, LIB, DLL, EFI, FFS, FV, and FD files match 100%
across 70 VS2019/GCC5 builds of all package/platform DSC files in
the edk2 repo.

The hard freeze will be extended after the edk2-stable202111 tag until
all uncrustify related changes are committed. We do not expect this
to take more than a few days. Do not push any PRs until the hard
freeze is lifted.

The changes are broken up into 7 patch series/PRs. The PRs are ordered
so they can be submitted using the normal submission process and EDK II
CI will pass for each one. Details are listed below.

Uncrustify 73.0.3 for EDK II
=============================
* Sources:
INVALID URI REMOVED
stify__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJOYp
cSVHoHwMD2t8$
* Documentation:
INVALID URI REMOVED
/_wiki/wikis/Uncrustify.wiki/1/Proj__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKk
oslR8D0LLPZTWXL0OKtMZJOYpcSVHoP8OtlxM$
ect-Mu-(EDK-II)-Fork-Readme
* Download:
INVALID URI REMOVED
/_packaging?_a=package&feed=m__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKk
oslR8D0LLPZTWXL0OKtMZJOYpcSVHo4yTWeAM$
u_uncrustify&package=mu-uncrustify-
release&protocolType=NuGet&version=73.0.3

Installing Uncrustify
======================
The Uncrustify tool is installed automatically when the Pytools
environment is used and the stuart* commands are run to complete the
environment setup. Please see:

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-
locally

Uncrustify can also be installed from the download page listed above
or built from sources from the source link above.

The Documentation link provides instruction on how to run uncrustify from
the command line or install as a Visual Studio Code plugin. The main
uncrustify documentation also describes how to integrate with a few other
editors.

We have also discussed a client side githook. That effort has not started.
Let us know if that is a feature you would find useful.

Developer impact for new code reviews
======================================
Once the uncrustify checker is active in EDK II CI, developers must
make sure their patches are run through the uncrustify tool before
sending the patches for review.

Developers must install and run uncrustify against changes files before
sending patch review emails or submitting PR for EDK II CI. If EDK II CI
detects and differences in source formatting, then EDK II CI will fail
and the developer must run uncrustify and resubmit the patches.

Developer impact to patch series/PRs reviewed during edk2-stable201121
soft/hard freeze
==========================================================
=======
======================
Developers must rebase their changes after the uncrustify source changes
are
committed. The branch with a preview of the uncrustify changes can be
used
to start this rebase work.


https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
han
ges_V5

The following steps can be used to update an existing branch with the
required uncrustify format. This is the Windows version. I will add
the Linux version soon.

1) Fetch and checkout and rebase to latest edk2/master

git fetch origin
git checkout master
git rebase origin/master

2) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

3) Check out the patch series branch (e.g. MyBranch)

git checkout MyBranch

4) Rebase patch series against edk2-stable202111

git rebase edk2-stable202111

5) Create new branch for the uncrustifed version (e.g.
MyBranch_Uncrustified)

git checkout -b MyBranch_Uncrustified

6) Use git filter-branch to uncrustify all the commits in the series
between the rebase target from (2) and HEAD of the branch. A filter
can be used to scope the uncrustify operations to only the C/H files
in the specific package the patch series is against. (e.g.
DynamicTablesPkg).
BaseTools should always be excluded. If the package scoped filter is
not used, it will still work, but will take longer to run because
uncrustify will rescan every C/H files in the whole repo.

git filter-branch --tree-filter "git ls-files DynamicTablesPkg*.c
DynamicTablesPkg*.h :!BaseTools/* | c:\\Temp\\UncrustifyCheck\\mu-
uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c
c:\\Temp\\UncrustifyCheck\\uncrustify.cfg -F - --replace --no-backup --if-
changed" edk2-stable202111..HEAD

7) Now that all the individual patches in the branch are uncrustified,
rebase against latest edk2/master that is already uncrustified.

git rebase master

8) Verify the patches in this new branch.

Impacts to tracing history across the uncrusity changes
=======================================================
Tools the view file and line history do work with the large uncrustify
patch series. One impact is that the operations can be very slow due
to the large uncrustify patches.

One option to provide a faster experience is to provide an alternate
version of the edk2 repository as "documentation" that has the
entire history re-written with uncrustify run on every commit.
The tool called git-filter-repo can be used to perform this
transformation and runs in a reasonable period of time (a few hours)

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-repo/blob/main/contrib/filter-
repo-
demos/lint-history

The following steps can be used to perform this transformation.
This is the Windows version. I will add the Linux version soon.

** WARNING ** This operation modifies(rewrites) all the commits
in the local copy of the repo. Do not perform
these steps on a local repo you are using for
active development.

1) Clone edk2 into a new directory (see **WARNING**)

git clone https://github.com/tianocore/edk2.git edk2-uncrustified
cd edk2-uncrustified

2) Setup python virtual env, install pytools, and run stuart commands
to setup build environment which includes installing uncrustify tools.

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-
locally

3) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

4) Use lint-history.py from git-filter-repo examples

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-repo/blob/main/contrib/filter-
repo-
demos/lint-history

Line #127 - Add try except around subprocess.check_call() with except
being pass. This is required because there are a few commits of C
files in the edk2 repo that have incorrect C syntax and do not
build with a C compiler and break the uncrustify parser. Skip reformat
of C files that can not be parsed by uncrustify. These rare instances
are addressed in the commit that fixes the C syntax error.

Run this slightly modified version of lint-history. Include only
C/H files and exclude directories that start with 'Tools' or 'BaseTools'.
This step took about 2.2 hours on a laptop.

lint-history.py
--relevant "return (not filename.startswith(b'Tools') and not
filename.startswith(b'BaseTools') and (filename.endswith(b'.c') or
filename.endswith(b'.h')))"
c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\mu-
uncrustify-
release_extdep\\Windows-x86\\uncrustify.exe -c
c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\uncrustify.cfg --
replace --
no-backup --if-changed

Order of PRs to apply during extended hard freeze
==================================================
1) Update EmulatorPkg Win Host [BuildOptions] MSFT CC_FLAGS to not
force
debug information
*
INVALID URI REMOVED
d=3747__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHo2BEG33c$
*
https://github.com/mdkinney/edk2/tree/Bug_3747_EmulatorPkg_WinHost_
Rep
roducibleBuild
* https://github.com/tianocore/edk2/pull/2215
* Required for EmulatorPkg to pass CompareBuild for VS2019 IA32/X64
builds.
* Status: Review complete. PR pushed.

2) EccCheck should not revert staged and local changes
*
INVALID URI REMOVED
d=2986__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHotvR0Unk$
*
https://github.com/mdkinney/edk2/tree/Bug_2986_EccCheckRemoveGitRe
vert
_V2
* https://github.com/tianocore/edk2/pull/2216
* Required for EDK II CI to complete in a reasonable period of time when
processing the 4000+ source file style changes made by uncrustify.
* Also fixes critical bugs that can potentially corrupt git state when
EccCheck is run locally.
* Status: Review complete. PR pushed.

3) Update pytool LicenseCheck plugin to use temp directory for diff output
file
*
INVALID URI REMOVED
d=3746__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHoWRuHaBw$
*
https://github.com/mdkinney/edk2/tree/Bug_3746_LicenseCheckUseDiffO
utput
File_V2
* https://github.com/tianocore/edk2/pull/2217
* Required to reduce EDK II CI build times.
* Status: Review complete. PR pushed.

4) Update max job time from 60 min to 120 minutes
in .azurepipelines/templates
*
INVALID URI REMOVED
d=3750__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHozaVshgc$
*
https://github.com/mdkinney/edk2/tree/Bug_3750_IncreaseAzurePipelines
Time
out
* https://github.com/tianocore/edk2/pull/2219
* Required to allow EccCheck of uncrustify changes to complete on Azure
Pipelines CI agents without timing out.
* Status: Review complete. PR pushed.

5) Update Package YAML to ignore specific ECC files/errors
*
INVALID URI REMOVED
d=3749__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHo-fY3dMo$
*
https://github.com/mdkinney/edk2/tree/Bug_3749_EccCheckIgnoreFilesErr
ors
* https://github.com/tianocore/edk2/pull/2218
* Required to pass EccCheck
* Status: Review complete. PR pushed

6) Uncrustify Source Changes
*
INVALID URI REMOVED
d=3737__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHoMRa47YU$
*
INVALID URI REMOVED
d=3739__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHoPAlckP8$
*
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
han
ges_V5
* https://github.com/tianocore/edk2/pull/2229
* Build comparison result PASS:
https://github.com/mdkinney/edk2/actions/runs/1521618836
* EFI_D_ -> DEBUG changes required to pass PatchCheck
* Uncrustify format changes required to pass UncrustifyCheck
* Status:
Waiting for review

7) UncrustifyCheck EDK II CI Plugin
*
INVALID URI REMOVED
d=3748__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHoCzEWymQ$
*
https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plug
in_v
6
* Required to enforce all PRs submitted to EDK II CI match uncrustify
format.
* Unconditionally checks all packages. Can not be committed until all C/H
source files have been updated.
* Status: Review complete

Combined Branch/PR for Review/Test
==================================
* Build Comparison results must pass 100% across the full set of PRs before
the individual PRs can be pushed in the order listed above.
* Branch:
https://github.com/mdkinney/edk2/tree/TestOnly_Uncrustify_PR_Series
* PR: https://github.com/tianocore/edk2/pull/2229
Status = PASS
* CompareBuild:
Branch:
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
han
ges_V5
--ref1: ef9a059cdb15844fe52a49af2bf7d86b9dd3e9bf
--ref2: Bug_3737_3739_ApplyUncrustifyChanges_V5
Extra Options: -n 4 --quiet
Results: https://github.com/mdkinney/edk2/actions/runs/1521618836
30 VS2019 build comparisons PASS
40 GCC5 build comparisons PASS
100% PASS

The following git log shows the set of patches from --ref1 to --ref 2across
which there are no differences in any of the OBJ/LIB/DLL/EFI/FFS/FV/FD
files.

--ref2
b7d4bf0675b7 (HEAD -> Bug_3737_3739_ApplyUncrustifyChanges_V5)
UnitTestFrameworkPkg: Apply uncrusitify changes
7f03d25f60e7 UefiPayloadPkg: Apply uncrusitify changes
0bfd8d9b5ac9 UefiCpuPkg: Apply uncrusitify changes
e1cd9bfb9dea StandaloneMmPkg: Apply uncrusitify changes
5da2f65be378 SourceLevelDebugPkg: Apply uncrusitify changes
95b86de07e5d SignedCapsulePkg: Apply uncrusitify changes
fe71d97246c4 ShellPkg: Apply uncrusitify changes
54c21c952992 SecurityPkg: Apply uncrusitify changes
187a3785f12b RedfishPkg: Apply uncrusitify changes
810100002a46 PcAtChipsetPkg: Apply uncrusitify changes
276a695c0cf2 OvmfPkg: Apply uncrusitify changes
303c0a91ab07 NetworkPkg: Apply uncrusitify changes
bc80792cd1b1 MdePkg: Apply uncrusitify changes
3ea86be17a2a MdeModulePkg: Apply uncrusitify changes
c70ef11ed0cd IntelFsp2WrapperPkg: Apply uncrusitify changes
c0291221f252 IntelFsp2Pkg: Apply uncrusitify changes
6a479952a690 FmpDevicePkg: Apply uncrusitify changes
3a7c05b7070d FatPkg: Apply uncrusitify changes
b789f98c8959 EmulatorPkg: Apply uncrusitify changes
952d7a1c9220 EmbeddedPkg: Apply uncrusitify changes
a1cc9881bab6 DynamicTablesPkg: Apply uncrusitify changes
50654dfe5785 CryptoPkg: Apply uncrusitify changes
ed965a02dfa1 ArmVirtPkg: Apply uncrusitify changes
9744023fbc46 ArmPlatformPkg: Apply uncrusitify changes
7a1cde5f5bba ArmPkg: Apply uncrusitify changes
19d17e0913e8 UefiCpuPkg: Change use of EFI_D_* to DEBUG_*
ffa718b4f994 SourceLevelDebugPkg: Change use of EFI_D_* to DEBUG_*
b86cb3c5e5b4 ShellPkg: Change use of EFI_D_* to DEBUG_*
c7c42204dc07 SecurityPkg: Change use of EFI_D_* to DEBUG_*
16b8e6f958e4 PcAtChipsetPkg: Change use of EFI_D_* to DEBUG_*
0ac3f8b2dac5 OvmfPkg: Change use of EFI_D_* to DEBUG_*
bc5004b8d294 NetworkPkg: Change use of EFI_D_* to DEBUG_*
6f671a8e2377 MdePkg: Change use of EFI_D_* to DEBUG_*
a10c610ff9a3 MdeModulePkg: Change use of EFI_D_* to DEBUG_*
09a3bddba390 FatPkg: Change use of EFI_D_* to DEBUG_*
59c61318246a EmulatorPkg: Change use of EFI_D_* to DEBUG_*
3a80367dda3b EmbeddedPkg: Change use of EFI_D_* to DEBUG_*
23eb1aaf80ca ArmVirtPkg: Change use of EFI_D_* to DEBUG_*
875914b45c54 ArmPlatformPkg: Change use of EFI_D_* to DEBUG_*
eb2eca82b451 ArmPkg: Change use of EFI_D_* to DEBUG_*
f0f3f5aae7c4 (origin/master, origin/HEAD, master) UnitTestFrameworkPkg:
Update YAML to ignore specific ECC files/errors
c05734797790 UefiPayloadPkg: Update YAML to ignore specific ECC
files/errors
c30c40d6c63d StandaloneMmPkg: Update YAML to ignore specific ECC
files/errors
9944508e85f1 ShellPkg: Update YAML to ignore specific ECC files/errors
60fa40be458d SecurityPkg: Update YAML to ignore specific ECC files/errors
df790cd6b37e MdePkg: Update YAML to ignore specific ECC files/errors
9deb9370766e MdeModulePkg: Update YAML to ignore specific ECC
files/errors
d7d30e8f219f EmulatorPkg: Update YAML to ignore specific ECC files/errors
d5744ecba813 CryptoPkg: Update YAML to ignore specific ECC files/errors
c97fee87f0f9 ArmVirtPkg: Update YAML to ignore specific ECC files/errors
1939fc9569f2 ArmPlatformPkg: Update YAML to ignore specific ECC
files/errors
365dced2c37a ArmPkg: Update YAML to ignore specific ECC files/errors
76a1ce4d5fec .azurepipelines/templates: Update max pipeline job time to
2
hours
99f84ff47390 .pytools/Plugin/LicenseCheck: Use temp directory for git diff
output
3019f1bbabf1 .pytool/Plugin/EccCheck: Add performance optimizations
854462bd3479 .pytool/Plugin/EccCheck: Remove temp directory on
exception
69877614fdee .pytool/Plugin/EccCheck: Remove RevertCode()
--ref1
ef9a059cdb15 EmulatorPkg/Win/Host: Update CC_FLAGS
bb1bba3d7767 (tag: edk2-stable202111) NetworkPkg: Fix invalid pointer for
DNS
response token on error

Best regards,

Mike



















Bret Barkelew
 

So, so, so refreshing to see automated consistency.

 

For UnitTestFrameworkPkg…

Reviewed-by: Bret Barkelew <bret.barkelew@...>

 

- Bret

 

From: Michael D Kinney via groups.io
Sent: Tuesday, November 30, 2021 2:34 PM
To: devel@edk2.groups.io; Kinney, Michael D; Michael Kubacki; Andrew Fish (afish@...); Lindholm, Leif
Subject: [EXTERNAL] [edk2-devel] Uncrustify Conversion Detailed Plan and Extended Hard Freeze Update #4

 

Hello,

Thank you for your patience during this extended hard freeze.

Just one more step to go.  There has been a delay in the review of
the patch series with the uncrustify source changes.  PR(6).  This
patch series was not sent out as patch review email because of its
very large size.  It only contains source style changes and the
CompareBuild tool and GitHub action has shown there are no binary
differences introduced with these source style changes.

If you are a package maintainer, then please review the following
branch/PR for your package contents and review the EDK II CI results
and BuildCompare results.  I do not expect a line by line review
because we already had time to provide feedback on the source style
performed by uncrustify.  Instead, a Reviewed-by for your package
indicates that you have reviewed the EDK II CI results and CompareBuild
tool functionality and results and you accept the source style
changes to your package.

   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2%2Ftree%2FBug_3737_3739_ApplyUncrustifyChanges_V5&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=6OFgnyfsGhclcj8GTVKchzBmkbOQJtcm4o2iz4RzPU4%3D&amp;reserved=0
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Fpull%2F2229&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=BA8pmxk0T4Uom8%2F4dzyjJ2lvEl3mALyq3wOlVCKMPD0%3D&amp;reserved=0
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2%2Factions%2Fruns%2F1521618836&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=1EcBi3Mjb9LM%2BhrUDfufOJFmvEEVsb6IW7chJvs3dl0%3D&amp;reserved=0

Additional details on this update below.

Thank you,

Mike


Changes from Update #3
----------------------------------------------------------------------------
* Pushed PR (5)
* Added link to PR(6). EDK II CI Status is PASS. Build Compare PASS.
* Waiting for review of PR (6)
* Review of PR (7) completed and waiting for review of PR (6)
----------------------------------------------------------------------------

Changes from Update #2
----------------------------------------------------------------------------
* Changed order of PRs swapping (4) and (5).  The PR that activates
  increases the max CI agent job time is independent of all the other
  PRs and its review is complete, so it can be committed now.
* Pushed PRs (1), (2), (3), (4).
* Waiting for review to complete for PRs (5) and (6)
* Reviews complete for PR (7)
* Identifies steps using git filter-branch to apply uncrustify changes to a
  code review patch series that was generated before the uncrustify changes
  avoiding manual merge.
* Identified steps using git filter-repo to generate an alternate history of
  the edk2 repo with uncrustify changes applied on every commit.  This may
  be useful when evaluating changes to files using tools like git blame
  without the large uncrustify patch series.
---------------------------------------------------------------------------

Changes from Update #1
----------------------------------------------------------------------------
* Changed order of PRs swapping (6) and (7).  The PR that activates
  EDK II CI check UncrustifyCheck has to be last because it unconditionally
  checks all C/H files in all packages.  Not just files that have been
  modified like some of the other checkers.
* Updated link to the branch with the UncrustifyCheck plugin that has been
  updated with a one line change and Reviewed-by and Tested-by tags.
  https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2%2Ftree%2FBug_3748_add_uncrustify_ci_plugin_v6&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=j0Xjp7ss0vyQ2oNTRF4VC%2B5AwQrKQuxxGHH14q%2Fw1p8%3D&amp;reserved=0
* Reviews complete for (1), (2), (3), (5), and (7)
---------------------------------------------------------------------------

Michael Kubacki and I have prepared the patches required to apply the
uncrustify changes and enable EDK II CI to check all submitted
patches have been run through uncrustify.

We have verified through the CompareBuild GitHub Action that the
format changes performed by uncrustify have no functional changes.
All of the OBJ, LIB, DLL, EFI, FFS, FV, and FD files match 100%
across 70 VS2019/GCC5 builds of all package/platform DSC files in
the edk2 repo.

The hard freeze will be extended after the edk2-stable202111 tag until
all uncrustify related changes are committed.  We do not expect this
to take more than a few days.  Do not push any PRs until the hard
freeze is lifted.

The changes are broken up into 7 patch series/PRs.  The PRs are ordered
so they can be submitted using the normal submission process and EDK II
CI will pass for each one.  Details are listed below.

Uncrustify 73.0.3 for EDK II
=============================
* Sources: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdev.azure.com%2Fprojectmu%2F_git%2FUncrustify&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=oxs8hpNZPKyuizJMa1Q938NvF07fpc5XN5PzQKvO5B4%3D&amp;reserved=0
* Documentation: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdev.azure.com%2Fprojectmu%2FUncrustify%2F_wiki%2Fwikis%2FUncrustify.wiki%2F1%2FProject-Mu-&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=FPPTmTUhhXL%2FGybBIdaMcTumBnfibpjNnorESzob8jA%3D&amp;reserved=0(EDK-II)-Fork-Readme
* Download: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdev.azure.com%2Fprojectmu%2FUncrustify%2F_packaging%3F_a%3Dpackage%26feed%3Dmu_uncrustify%26package%3Dmu-uncrustify-release%26protocolType%3DNuGet%26version%3D73.0.3&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=BUOogU1P194ZKUDhSBoO7QcxPIDjZyM%2Bqvg2kwNaSKw%3D&amp;reserved=0

Installing Uncrustify
======================
The Uncrustify tool is installed automatically when the Pytools
environment is used and the stuart* commands are run to complete the
environment setup.  Please see:

    https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2F.pytool%23running-ci-locally&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=Gl8zfFjVrUvpnTO0UXb3bXR2lsIBz5PwLazkKyBvGIE%3D&amp;reserved=0

Uncrustify can also be installed from the download page listed above
or built from sources from the source link above.

The Documentation link provides instruction on how to run uncrustify from
the command line or install as a Visual Studio Code plugin.  The main
uncrustify documentation also describes how to integrate with a few other
editors.

We have also discussed a client side githook.  That effort has not started.
Let us know if that is a feature you would find useful.

Developer impact for new code reviews
======================================
Once the uncrustify checker is active in EDK II CI, developers must
make sure their patches are run through the uncrustify tool before
sending the patches for review.

Developers must install and run uncrustify against changes files before
sending patch review emails or submitting PR for EDK II CI.  If EDK II CI
detects and differences in source formatting, then EDK II CI will fail
and the developer must run uncrustify and resubmit the patches.

Developer impact to patch series/PRs reviewed during edk2-stable201121 soft/hard freeze
=======================================================================================
Developers must rebase their changes after the uncrustify source changes are
committed.  The branch with a preview of the uncrustify changes can be used
to start this rebase work.

    https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2%2Ftree%2FBug_3737_3739_ApplyUncrustifyChanges_V5&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=6OFgnyfsGhclcj8GTVKchzBmkbOQJtcm4o2iz4RzPU4%3D&amp;reserved=0

The following steps can be used to update an existing branch with the
required uncrustify format.  This is the Windows version.  I will add
the Linux version soon.

    1) Fetch and checkout and rebase to latest edk2/master

           git fetch origin
           git checkout master
           git rebase origin/master

    2) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
       (e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
       EDK II specific uncrustify configuration file available when working
       with a branch that does not have those tools in its scope.

           xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

    3) Check out the patch series branch (e.g. MyBranch)

           git checkout MyBranch

    4) Rebase patch series against edk2-stable202111
 
           git rebase edk2-stable202111

    5) Create new branch for the uncrustifed version (e.g. MyBranch_Uncrustified)

           git checkout -b MyBranch_Uncrustified
   
    6) Use git filter-branch to uncrustify all the commits in the series
       between the rebase target from (2) and HEAD of the branch.  A filter
       can be used to scope the uncrustify operations to only the C/H files
       in the specific package the patch series is against. (e.g. DynamicTablesPkg).
       BaseTools should always be excluded.  If the package scoped filter is
       not used, it will still work, but will take longer to run because
       uncrustify will rescan every C/H files in the whole repo.

           git filter-branch --tree-filter "git ls-files DynamicTablesPkg*.c DynamicTablesPkg*.h :!BaseTools/* | c:\\Temp\\UncrustifyCheck\\mu-uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c c:\\Temp\\UncrustifyCheck\\uncrustify.cfg -F - --replace --no-backup --if-changed" edk2-stable202111..HEAD

    7) Now that all the individual patches in the branch are uncrustified,
       rebase against latest edk2/master that is already uncrustified.

           git rebase master

    8) Verify the patches in this new branch.

Impacts to tracing history across the uncrusity changes
=======================================================
Tools the view file and line history do work with the large uncrustify
patch series.  One impact is that the operations can be very slow due
to the large uncrustify patches.

One option to provide a faster experience is to provide an alternate
version of the edk2 repository as "documentation" that has the
entire history re-written with uncrustify run on every commit.
The tool called git-filter-repo can be used to perform this
transformation and runs in a reasonable period of time (a few hours)

   https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnewren%2Fgit-filter-repo&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=ww5Wr752N7sQUbISyV8uXnv4EDRveeU6LXktsw7afIs%3D&amp;reserved=0
   https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnewren%2Fgit-filter-repo%2Fblob%2Fmain%2Fcontrib%2Ffilter-repo-demos%2Flint-history&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=VeoGuXB8K%2F1vp%2BwDdSRM6HvUhqtK4wkFfJLjdorspWs%3D&amp;reserved=0

The following steps can be used to perform this transformation.
This is the Windows version. I will add the Linux version soon.

    ** WARNING **  This operation modifies(rewrites) all the commits
                   in the local copy of the repo.  Do not perform
                   these steps on a local repo you are using for
                   active development.

    1) Clone edk2 into a new directory (see **WARNING**)

           git clone https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2.git&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=djea29CcnZTVc2MohA2WzGaLiKMp66sD9qdum7wcaFI%3D&amp;reserved=0 edk2-uncrustified
           cd edk2-uncrustified

    2) Setup python virtual env, install pytools, and run stuart commands
       to setup build environment which includes installing uncrustify tools.

           https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2F.pytool%23running-ci-locally&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=Gl8zfFjVrUvpnTO0UXb3bXR2lsIBz5PwLazkKyBvGIE%3D&amp;reserved=0

    3) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE.
       (e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
       EDK II specific uncrustify configuration file available when working
       with a branch that does not have those tools in its scope.

           xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

    4) Use lint-history.py from git-filter-repo examples

           https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnewren%2Fgit-filter-repo&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=ww5Wr752N7sQUbISyV8uXnv4EDRveeU6LXktsw7afIs%3D&amp;reserved=0
           https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnewren%2Fgit-filter-repo%2Fblob%2Fmain%2Fcontrib%2Ffilter-repo-demos%2Flint-history&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=VeoGuXB8K%2F1vp%2BwDdSRM6HvUhqtK4wkFfJLjdorspWs%3D&amp;reserved=0

       Line #127 - Add try except around subprocess.check_call() with except
       being pass.  This is required because there are a few commits of C
       files in the edk2 repo that have incorrect C syntax and do not
       build with a C compiler and break the uncrustify parser.  Skip reformat
       of C files that can not be parsed by uncrustify.  These rare instances
       are addressed in the commit that fixes the C syntax error.

       Run this slightly modified version of lint-history.  Include only
       C/H files and exclude directories that start with 'Tools' or 'BaseTools'.
       This step took about 2.2 hours on a laptop.

           lint-history.py
               --relevant "return (not filename.startswith(b'Tools') and not filename.startswith(b'BaseTools') and (filename.endswith(b'.c') or filename.endswith(b'.h')))"
               c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\mu-uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\uncrustify.cfg --replace --no-backup --if-changed

Order of PRs to apply during extended hard freeze
==================================================
1) Update EmulatorPkg Win Host [BuildOptions] MSFT CC_FLAGS to not force debug information
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D3747&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=vRu5LQSjbmwafbvaA6Iv0S%2FzUIbAE%2Bn2Ax9xbckTYLg%3D&amp;reserved=0
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2%2Ftree%2FBug_3747_EmulatorPkg_WinHost_ReproducibleBuild&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=oFCZQy6C9zf7kD4QugF9jORY%2Feh5%2B9T9odYGvMND%2FE0%3D&amp;reserved=0
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Fpull%2F2215&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=mfZMcsGnEof2DvpnVCz8NV9HTo%2FyqadoGfEquXwLjFI%3D&amp;reserved=0
   * Required for EmulatorPkg to pass CompareBuild for VS2019 IA32/X64 builds.
   * Status: Review complete.  PR pushed.

2) EccCheck should not revert staged and local changes
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D2986&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=SQVOuOT3Fg%2FVlH2471q0UAG9%2FGRgux3pvkRjrsM%2FkRU%3D&amp;reserved=0
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2%2Ftree%2FBug_2986_EccCheckRemoveGitRevert_V2&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=st1WwSlbkgoJ1nPpZqb9XnvCgLqPlpRAos5LtAM%2BcPA%3D&amp;reserved=0
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Fpull%2F2216&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=6ZQJvN7KLQl5pYsVVIosFbwIdTYtfBVU3rQgvq4cKxs%3D&amp;reserved=0
   * Required for EDK II CI to complete in a reasonable period of time when
     processing the 4000+ source file style changes made by uncrustify.
   * Also fixes critical bugs that can potentially corrupt git state when
     EccCheck is run locally.
   * Status: Review complete.  PR pushed.

3) Update pytool LicenseCheck plugin to use temp directory for diff output file
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D3746&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=cukX7Hc09DRj1v%2BfC9epfr0DDeP0NKunwwbweXjK2hI%3D&amp;reserved=0
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2%2Ftree%2FBug_3746_LicenseCheckUseDiffOutputFile_V2&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=i7uICY1zgaq3ukuWCvWTq211b2qhHM6lYRfxTg4pJRE%3D&amp;reserved=0
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Fpull%2F2217&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=kNvYpT8TrH%2FxcykLRVAg6Z7JQ7RkBtqhFAWyR3x6%2Faw%3D&amp;reserved=0
   * Required to reduce EDK II CI build times.
   * Status: Review complete.  PR pushed.

4) Update max job time from 60 min to 120 minutes in .azurepipelines/templates
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D3750&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=qlXfBBwzPrz4iLyVunIEF8N5mondsfLjMmYk7hlF5Qg%3D&amp;reserved=0
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2%2Ftree%2FBug_3750_IncreaseAzurePipelinesTimeout&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=3QhsR365UOEUjyOsrPXvH3oXXSx1F3g6BmpQemEjIxA%3D&amp;reserved=0
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Fpull%2F2219&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=yNZhoLZXW7hYqPNtKqq4X6rNWyblAhFPpEbkalXotVU%3D&amp;reserved=0
   * Required to allow EccCheck of uncrustify changes to complete on Azure
     Pipelines CI agents without timing out.
   * Status: Review complete.  PR pushed.

5) Update Package YAML to ignore specific ECC files/errors
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D3749&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=HE2PDTC15WKVaQMPU9HZDhfJ4cx71yYzSFnnwqfVk5U%3D&amp;reserved=0
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2%2Ftree%2FBug_3749_EccCheckIgnoreFilesErrors&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=d6PNQhqOzohY3OQcX5WVgxB5aWp4qylzFuIzRPsAbZk%3D&amp;reserved=0
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Fpull%2F2218&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=Vvf2btUzSOzwILvps3h7cRHRfH%2B5cLV%2FcC0Zdo64QaY%3D&amp;reserved=0
   * Required to pass EccCheck
   * Status: Review complete. PR pushed

6) Uncrustify Source Changes
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D3737&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=4iMSTuiCXmhNq9v1z65UYEQ4G9eS0TTkKT0gAC%2Fj%2Bjo%3D&amp;reserved=0
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D3739&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=vbr%2BU6nvHHVcfaMrmfH1tMPC3kNDJyv%2BvDhOoEgUyj4%3D&amp;reserved=0
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2%2Ftree%2FBug_3737_3739_ApplyUncrustifyChanges_V5&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=6OFgnyfsGhclcj8GTVKchzBmkbOQJtcm4o2iz4RzPU4%3D&amp;reserved=0
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Fpull%2F2229&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=BA8pmxk0T4Uom8%2F4dzyjJ2lvEl3mALyq3wOlVCKMPD0%3D&amp;reserved=0
   * Build comparison result PASS: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2%2Factions%2Fruns%2F1521618836&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=1EcBi3Mjb9LM%2BhrUDfufOJFmvEEVsb6IW7chJvs3dl0%3D&amp;reserved=0
   * EFI_D_ -> DEBUG changes required to pass PatchCheck
   * Uncrustify format changes required to pass UncrustifyCheck
   * Status:
       Waiting for review

7) UncrustifyCheck EDK II CI Plugin
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D3748&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=peNoGQGid33y42AXi3qugg4oRJycrj%2FoPZkFofszVVk%3D&amp;reserved=0
   * https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2%2Ftree%2FBug_3748_add_uncrustify_ci_plugin_v6&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=j0Xjp7ss0vyQ2oNTRF4VC%2B5AwQrKQuxxGHH14q%2Fw1p8%3D&amp;reserved=0
   * Required to enforce all PRs submitted to EDK II CI match uncrustify format.
   * Unconditionally checks all packages.  Can not be committed until all C/H
     source files have been updated.
   * Status: Review complete

Combined Branch/PR for Review/Test
==================================
* Build Comparison results must pass 100% across the full set of PRs before
  the individual PRs can be pushed in the order listed above.
* Branch: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2%2Ftree%2FTestOnly_Uncrustify_PR_Series&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=%2Fgacf3RVe6uGU%2BQ2gVdQYOfKLn2zwS1DXwuqRo8k%2Fco%3D&amp;reserved=0
* PR: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Fpull%2F2229&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=BA8pmxk0T4Uom8%2F4dzyjJ2lvEl3mALyq3wOlVCKMPD0%3D&amp;reserved=0
    Status = PASS
* CompareBuild:
    Branch: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2%2Ftree%2FBug_3737_3739_ApplyUncrustifyChanges_V5&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=6OFgnyfsGhclcj8GTVKchzBmkbOQJtcm4o2iz4RzPU4%3D&amp;reserved=0
    --ref1:        ef9a059cdb15844fe52a49af2bf7d86b9dd3e9bf
    --ref2:        Bug_3737_3739_ApplyUncrustifyChanges_V5
    Extra Options: -n 4 --quiet
    Results: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fedk2%2Factions%2Fruns%2F1521618836&amp;data=04%7C01%7CBret.Barkelew%40microsoft.com%7Cafe73bb0029e491612b908d9b4519560%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637739084803059341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=1EcBi3Mjb9LM%2BhrUDfufOJFmvEEVsb6IW7chJvs3dl0%3D&amp;reserved=0
        30 VS2019 build comparisons PASS
        40 GCC5 build comparisons PASS
        100% PASS

The following git log shows the set of patches from --ref1 to --ref 2across
which there are no differences in any of the OBJ/LIB/DLL/EFI/FFS/FV/FD files.

--ref2
b7d4bf0675b7 (HEAD -> Bug_3737_3739_ApplyUncrustifyChanges_V5) UnitTestFrameworkPkg: Apply uncrusitify changes
7f03d25f60e7 UefiPayloadPkg: Apply uncrusitify changes   
0bfd8d9b5ac9 UefiCpuPkg: Apply uncrusitify changes                                                                                                                                                                       
e1cd9bfb9dea StandaloneMmPkg: Apply uncrusitify changes                                                                                                                                                                  
5da2f65be378 SourceLevelDebugPkg: Apply uncrusitify changes                                                                                                                                                              
95b86de07e5d SignedCapsulePkg: Apply uncrusitify changes                                                                                                                                                                 
fe71d97246c4 ShellPkg: Apply uncrusitify changes                                                                                                                                                                         
54c21c952992 SecurityPkg: Apply uncrusitify changes                                                                                                                                                                      
187a3785f12b RedfishPkg: Apply uncrusitify changes                                                                                                                                                                       
810100002a46 PcAtChipsetPkg: Apply uncrusitify changes                                                                                                                                                                   
276a695c0cf2 OvmfPkg: Apply uncrusitify changes                                                                                                                                                                          
303c0a91ab07 NetworkPkg: Apply uncrusitify changes                                                                                                                                                                       
bc80792cd1b1 MdePkg: Apply uncrusitify changes                                                                                                                                                                           
3ea86be17a2a MdeModulePkg: Apply uncrusitify changes                                                                                                                                                                     
c70ef11ed0cd IntelFsp2WrapperPkg: Apply uncrusitify changes                                                                                                                                                              
c0291221f252 IntelFsp2Pkg: Apply uncrusitify changes                                                                                                                                                                     
6a479952a690 FmpDevicePkg: Apply uncrusitify changes                                                                                                                                                                     
3a7c05b7070d FatPkg: Apply uncrusitify changes                                                                                                                                                                           
b789f98c8959 EmulatorPkg: Apply uncrusitify changes                                                                                                                                                                      
952d7a1c9220 EmbeddedPkg: Apply uncrusitify changes                                                                                                                                                                      
a1cc9881bab6 DynamicTablesPkg: Apply uncrusitify changes                                                                                                                                                                 
50654dfe5785 CryptoPkg: Apply uncrusitify changes                                                                                                                                                                        
ed965a02dfa1 ArmVirtPkg: Apply uncrusitify changes                                                                                                                                                                       
9744023fbc46 ArmPlatformPkg: Apply uncrusitify changes                                                                                                                                                                   
7a1cde5f5bba ArmPkg: Apply uncrusitify changes                                                                                                                                                                           
19d17e0913e8 UefiCpuPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                
ffa718b4f994 SourceLevelDebugPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                       
b86cb3c5e5b4 ShellPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                  
c7c42204dc07 SecurityPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                               
16b8e6f958e4 PcAtChipsetPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                            
0ac3f8b2dac5 OvmfPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                   
bc5004b8d294 NetworkPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                
6f671a8e2377 MdePkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                    
a10c610ff9a3 MdeModulePkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                              
09a3bddba390 FatPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                    
59c61318246a EmulatorPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                               
3a80367dda3b EmbeddedPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                               
23eb1aaf80ca ArmVirtPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                
875914b45c54 ArmPlatformPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                            
eb2eca82b451 ArmPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                    
f0f3f5aae7c4 (origin/master, origin/HEAD, master) UnitTestFrameworkPkg: Update YAML to ignore specific ECC files/errors                                                                                                  
c05734797790 UefiPayloadPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                             
c30c40d6c63d StandaloneMmPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                            
9944508e85f1 ShellPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                   
60fa40be458d SecurityPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                
df790cd6b37e MdePkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                     
9deb9370766e MdeModulePkg: Update YAML to ignore specific ECC files/errors                                                                                                                                               
d7d30e8f219f EmulatorPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                
d5744ecba813 CryptoPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                  
c97fee87f0f9 ArmVirtPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                 
1939fc9569f2 ArmPlatformPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                             
365dced2c37a ArmPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                     
76a1ce4d5fec .azurepipelines/templates: Update max pipeline job time to 2 hours                                                                                                                                          
99f84ff47390 .pytools/Plugin/LicenseCheck: Use temp directory for git diff output                                                                                                                                        
3019f1bbabf1 .pytool/Plugin/EccCheck: Add performance optimizations                                                                                                                                                      
854462bd3479 .pytool/Plugin/EccCheck: Remove temp directory on exception                                                                                                                                                 
69877614fdee .pytool/Plugin/EccCheck: Remove RevertCode()
--ref1                                                                                                                                                               
ef9a059cdb15 EmulatorPkg/Win/Host: Update CC_FLAGS                                                                                                                                                                       
bb1bba3d7767 (tag: edk2-stable202111) NetworkPkg: Fix invalid pointer for DNS response token on error                                                                                                                    

Best regards,

Mike









 


Abner Chang
 

-----Original Message-----
From: Kinney, Michael D [mailto:michael.d.kinney@intel.com]
Sent: Thursday, December 2, 2021 1:39 AM
To: Michael Kubacki <mikuback@linux.microsoft.com>;
devel@edk2.groups.io; Chang, Abner (HPS SW/FW Technologist)
<abner.chang@hpe.com>; Wang, Jian J <jian.j.wang@intel.com>; Michael
Kubacki <michael.kubacki@microsoft.com>; Andrew Fish (afish@apple.com)
<afish@apple.com>; Leif Lindholm <leif@nuviainc.com>; Kinney, Michael D
<michael.d.kinney@intel.com>
Subject: RE: [edk2-devel] Uncrustify Conversion Detailed Plan and Extended
Hard Freeze Update #4

Hi Michael,

This commit provides the background on why this content was copied into
edk2
instead of used as a submodule.

https://github.com/tianocore/edk2/commit/4751a48aeb2ab828b0a5cbdc585
fd3642967cda1#diff-
9f77cfa3b537eee28489f8713c17d8702330f5cef1795f154201dd463aabed27

I imagine the main reason for this request is to minimize the effort required
to sync these copies with new versions.
Yes, this is what I worry about applying uncrustify on these files.

The scope right now is only 7 files.
One alternative to help with the sync is for Abner to run uncrustify on the
files from the upstream source on these 7 files and then do the diff to see if
there is anything significant that needs to be merged.

Abner, are you ok with doing this 2 step process to sync the files that were
copied?
I also consider to just uncrustify those files, maybe just compare two open source version and add the diff back to edk2 one. Then we can have this work moves forward.
Thanks

For the RedfishPkg/,
Acked-by: Abner Chang


Best regards,

Mike


-----Original Message-----
From: Michael Kubacki <mikuback@linux.microsoft.com>
Sent: Wednesday, December 1, 2021 9:05 AM
To: devel@edk2.groups.io; Kinney, Michael D
<michael.d.kinney@intel.com>; Chang, Abner <abner.chang@hpe.com>;
Wang, Jian J
<jian.j.wang@intel.com>; Michael Kubacki
<michael.kubacki@microsoft.com>; Andrew Fish (afish@apple.com)
<afish@apple.com>;
Leif Lindholm <leif@nuviainc.com>
Subject: Re: [edk2-devel] Uncrustify Conversion Detailed Plan and
Extended Hard Freeze Update #4

This can of course be done at a technical level but it smells of a code
cohesion problem.

The code is in the edk2 project so it would reason that it should be
uniform in style with other code and at least close to the EDK II C
Coding Standard Specification.

Currently, git submodules and ignored path are already excluded in
addition to the ability to mark packages in "audit mode" so they do not
fail the results. I would personally prefer not to allow arbitrary
subdirectories to diverge from the rest of the codebase / package but if
there's consensus this is a good idea, I can look into adding it.

Regards,
Michael

On 12/1/2021 11:43 AM, Michael D Kinney wrote:
Hi Abner,

Yes. That is possible. We are already excluding BaseTools.

In order to get the file lists that apply to uncrustify, we would have to
change from:

git ls-files *.c *.h :!BaseTools/*

To:

git ls-files *.c
*.h :!BaseTools/* :!RedfishPkg/PrivateLibrary/RedfishLib/edk2libredfish/*

However, I do not see a feature in the UncrustifyCheck to check all c/h
files in RedFishPkg and exclude one of its
directories.
If we can specify an exclusion like this in the RedFishPkg YAML file, then I
think that would work well.

Michael Kubacki would have to comment on how we support this in EDK
II CI checks.

Best regards,

Mike


-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
Abner Chang
Sent: Tuesday, November 30, 2021 11:34 PM
To: devel@edk2.groups.io; Wang, Jian J <jian.j.wang@intel.com>;
Kinney, Michael D <michael.d.kinney@intel.com>; Michael
Kubacki <michael.kubacki@microsoft.com>; Andrew Fish
(afish@apple.com) <afish@apple.com>; Leif Lindholm
<leif@nuviainc.com>
Subject: Re: [edk2-devel] Uncrustify Conversion Detailed Plan and
Extended Hard Freeze Update #4

Hi Mike,
Is that possible to not applying uncrusitify on the source files that cloned
from other open source?
e.g. RedfishPkg/PrivateLibrary/RedfishLib/edk2libredfish/*.*

Thanks
Abner

-----Original Message-----
From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On
Behalf Of
Wang, Jian J
Sent: Wednesday, December 1, 2021 2:39 PM
To: devel@edk2.groups.io; Kinney, Michael D
<michael.d.kinney@intel.com>;
Michael Kubacki <michael.kubacki@microsoft.com>; Andrew Fish
(afish@apple.com) <afish@apple.com>; Leif Lindholm
<leif@nuviainc.com>
Subject: Re: [edk2-devel] Uncrustify Conversion Detailed Plan and
Extended
Hard Freeze Update #4

For SignedCapulePkg,

Reviewed-by: Jian J Wang <jian.j.wang@intel.com>

Regards,
Jian

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
Michael D
Kinney
Sent: Wednesday, December 01, 2021 6:34 AM
To: devel@edk2.groups.io; Kinney, Michael D
<michael.d.kinney@intel.com>;
Michael Kubacki <michael.kubacki@microsoft.com>; Andrew Fish
(afish@apple.com) <afish@apple.com>; Leif Lindholm
<leif@nuviainc.com>
Subject: [edk2-devel] Uncrustify Conversion Detailed Plan and
Extended
Hard
Freeze Update #4

Hello,

Thank you for your patience during this extended hard freeze.

Just one more step to go. There has been a delay in the review of
the patch series with the uncrustify source changes. PR(6). This
patch series was not sent out as patch review email because of its
very large size. It only contains source style changes and the
CompareBuild tool and GitHub action has shown there are no binary
differences introduced with these source style changes.

If you are a package maintainer, then please review the following
branch/PR for your package contents and review the EDK II CI results
and BuildCompare results. I do not expect a line by line review
because we already had time to provide feedback on the source style
performed by uncrustify. Instead, a Reviewed-by for your package
indicates that you have reviewed the EDK II CI results and
CompareBuild
tool functionality and results and you accept the source style
changes to your package.

*
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
han
ges_V5
* https://github.com/tianocore/edk2/pull/2229
* https://github.com/mdkinney/edk2/actions/runs/1521618836

Additional details on this update below.

Thank you,

Mike


Changes from Update #3
----------------------------------------------------------------------------
* Pushed PR (5)
* Added link to PR(6). EDK II CI Status is PASS. Build Compare PASS.
* Waiting for review of PR (6)
* Review of PR (7) completed and waiting for review of PR (6)
----------------------------------------------------------------------------

Changes from Update #2
----------------------------------------------------------------------------
* Changed order of PRs swapping (4) and (5). The PR that activates
increases the max CI agent job time is independent of all the other
PRs and its review is complete, so it can be committed now.
* Pushed PRs (1), (2), (3), (4).
* Waiting for review to complete for PRs (5) and (6)
* Reviews complete for PR (7)
* Identifies steps using git filter-branch to apply uncrustify changes to
a
code review patch series that was generated before the uncrustify
changes
avoiding manual merge.
* Identified steps using git filter-repo to generate an alternate history
of
the edk2 repo with uncrustify changes applied on every commit.
This may
be useful when evaluating changes to files using tools like git blame
without the large uncrustify patch series.
---------------------------------------------------------------------------

Changes from Update #1
----------------------------------------------------------------------------
* Changed order of PRs swapping (6) and (7). The PR that activates
EDK II CI check UncrustifyCheck has to be last because it
unconditionally
checks all C/H files in all packages. Not just files that have been
modified like some of the other checkers.
* Updated link to the branch with the UncrustifyCheck plugin that has
been
updated with a one line change and Reviewed-by and Tested-by
tags.

https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plug
in_v
6
* Reviews complete for (1), (2), (3), (5), and (7)
---------------------------------------------------------------------------

Michael Kubacki and I have prepared the patches required to apply
the
uncrustify changes and enable EDK II CI to check all submitted
patches have been run through uncrustify.

We have verified through the CompareBuild GitHub Action that the
format changes performed by uncrustify have no functional changes.
All of the OBJ, LIB, DLL, EFI, FFS, FV, and FD files match 100%
across 70 VS2019/GCC5 builds of all package/platform DSC files in
the edk2 repo.

The hard freeze will be extended after the edk2-stable202111 tag
until
all uncrustify related changes are committed. We do not expect this
to take more than a few days. Do not push any PRs until the hard
freeze is lifted.

The changes are broken up into 7 patch series/PRs. The PRs are
ordered
so they can be submitted using the normal submission process and
EDK II
CI will pass for each one. Details are listed below.

Uncrustify 73.0.3 for EDK II
=============================
* Sources:
INVALID URI REMOVED
stify__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJOYp
cSVHoHwMD2t8$
* Documentation:
INVALID URI REMOVED
/_wiki/wikis/Uncrustify.wiki/1/Proj__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKk
oslR8D0LLPZTWXL0OKtMZJOYpcSVHoP8OtlxM$
ect-Mu-(EDK-II)-Fork-Readme
* Download:
INVALID URI REMOVED
/_packaging?_a=package&feed=m__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKk
oslR8D0LLPZTWXL0OKtMZJOYpcSVHo4yTWeAM$
u_uncrustify&package=mu-uncrustify-
release&protocolType=NuGet&version=73.0.3

Installing Uncrustify
======================
The Uncrustify tool is installed automatically when the Pytools
environment is used and the stuart* commands are run to complete
the
environment setup. Please see:

https://github.com/tianocore/edk2/tree/master/.pytool#running-
ci-
locally

Uncrustify can also be installed from the download page listed above
or built from sources from the source link above.

The Documentation link provides instruction on how to run uncrustify
from
the command line or install as a Visual Studio Code plugin. The main
uncrustify documentation also describes how to integrate with a few
other
editors.

We have also discussed a client side githook. That effort has not
started.
Let us know if that is a feature you would find useful.

Developer impact for new code reviews
======================================
Once the uncrustify checker is active in EDK II CI, developers must
make sure their patches are run through the uncrustify tool before
sending the patches for review.

Developers must install and run uncrustify against changes files
before
sending patch review emails or submitting PR for EDK II CI. If EDK II CI
detects and differences in source formatting, then EDK II CI will fail
and the developer must run uncrustify and resubmit the patches.

Developer impact to patch series/PRs reviewed during edk2-
stable201121
soft/hard freeze
==========================================================
=======
======================
Developers must rebase their changes after the uncrustify source
changes
are
committed. The branch with a preview of the uncrustify changes can
be
used
to start this rebase work.


https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
han
ges_V5

The following steps can be used to update an existing branch with the
required uncrustify format. This is the Windows version. I will add
the Linux version soon.

1) Fetch and checkout and rebase to latest edk2/master

git fetch origin
git checkout master
git rebase origin/master

2) Make a backup copy of plugin UncrustifyCheck outside
WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable
and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck
C:\Temp\UncrustifyCheck

3) Check out the patch series branch (e.g. MyBranch)

git checkout MyBranch

4) Rebase patch series against edk2-stable202111

git rebase edk2-stable202111

5) Create new branch for the uncrustifed version (e.g.
MyBranch_Uncrustified)

git checkout -b MyBranch_Uncrustified

6) Use git filter-branch to uncrustify all the commits in the series
between the rebase target from (2) and HEAD of the branch. A
filter
can be used to scope the uncrustify operations to only the C/H
files
in the specific package the patch series is against. (e.g.
DynamicTablesPkg).
BaseTools should always be excluded. If the package scoped
filter is
not used, it will still work, but will take longer to run because
uncrustify will rescan every C/H files in the whole repo.

git filter-branch --tree-filter "git ls-files DynamicTablesPkg*.c
DynamicTablesPkg*.h :!BaseTools/* |
c:\\Temp\\UncrustifyCheck\\mu-
uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c
c:\\Temp\\UncrustifyCheck\\uncrustify.cfg -F - --replace --no-backup
--if-
changed" edk2-stable202111..HEAD

7) Now that all the individual patches in the branch are uncrustified,
rebase against latest edk2/master that is already uncrustified.

git rebase master

8) Verify the patches in this new branch.

Impacts to tracing history across the uncrusity changes
=======================================================
Tools the view file and line history do work with the large uncrustify
patch series. One impact is that the operations can be very slow due
to the large uncrustify patches.

One option to provide a faster experience is to provide an alternate
version of the edk2 repository as "documentation" that has the
entire history re-written with uncrustify run on every commit.
The tool called git-filter-repo can be used to perform this
transformation and runs in a reasonable period of time (a few hours)

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-
repo/blob/main/contrib/filter-
repo-
demos/lint-history

The following steps can be used to perform this transformation.
This is the Windows version. I will add the Linux version soon.

** WARNING ** This operation modifies(rewrites) all the commits
in the local copy of the repo. Do not perform
these steps on a local repo you are using for
active development.

1) Clone edk2 into a new directory (see **WARNING**)

git clone https://github.com/tianocore/edk2.git edk2-
uncrustified
cd edk2-uncrustified

2) Setup python virtual env, install pytools, and run stuart
commands
to setup build environment which includes installing uncrustify
tools.

https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-
locally

3) Make a backup copy of plugin UncrustifyCheck outside
WORKSPACE.
(e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable
and
EDK II specific uncrustify configuration file available when working
with a branch that does not have those tools in its scope.

xcopy .pytool\Plugin\UncrustifyCheck
C:\Temp\UncrustifyCheck

4) Use lint-history.py from git-filter-repo examples

https://github.com/newren/git-filter-repo
https://github.com/newren/git-filter-
repo/blob/main/contrib/filter-
repo-
demos/lint-history

Line #127 - Add try except around subprocess.check_call() with
except
being pass. This is required because there are a few commits of C
files in the edk2 repo that have incorrect C syntax and do not
build with a C compiler and break the uncrustify parser. Skip
reformat
of C files that can not be parsed by uncrustify. These rare
instances
are addressed in the commit that fixes the C syntax error.

Run this slightly modified version of lint-history. Include only
C/H files and exclude directories that start with 'Tools' or
'BaseTools'.
This step took about 2.2 hours on a laptop.

lint-history.py
--relevant "return (not filename.startswith(b'Tools') and not
filename.startswith(b'BaseTools') and (filename.endswith(b'.c') or
filename.endswith(b'.h')))"
c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\mu-
uncrustify-
release_extdep\\Windows-x86\\uncrustify.exe -c
c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\uncrustify.cfg
--
replace --
no-backup --if-changed

Order of PRs to apply during extended hard freeze
==================================================
1) Update EmulatorPkg Win Host [BuildOptions] MSFT CC_FLAGS to
not
force
debug information
*
INVALID URI REMOVED
d=3747__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHo2BEG33c$
*
https://github.com/mdkinney/edk2/tree/Bug_3747_EmulatorPkg_WinHost_
Rep
roducibleBuild
* https://github.com/tianocore/edk2/pull/2215
* Required for EmulatorPkg to pass CompareBuild for VS2019
IA32/X64
builds.
* Status: Review complete. PR pushed.

2) EccCheck should not revert staged and local changes
*
INVALID URI REMOVED
d=2986__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHotvR0Unk$
*
https://github.com/mdkinney/edk2/tree/Bug_2986_EccCheckRemoveGitRe
vert
_V2
* https://github.com/tianocore/edk2/pull/2216
* Required for EDK II CI to complete in a reasonable period of time
when
processing the 4000+ source file style changes made by uncrustify.
* Also fixes critical bugs that can potentially corrupt git state when
EccCheck is run locally.
* Status: Review complete. PR pushed.

3) Update pytool LicenseCheck plugin to use temp directory for diff
output
file
*
INVALID URI REMOVED
d=3746__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHoWRuHaBw$
*
https://github.com/mdkinney/edk2/tree/Bug_3746_LicenseCheckUseDiffO
utput
File_V2
* https://github.com/tianocore/edk2/pull/2217
* Required to reduce EDK II CI build times.
* Status: Review complete. PR pushed.

4) Update max job time from 60 min to 120 minutes
in .azurepipelines/templates
*
INVALID URI REMOVED
d=3750__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHozaVshgc$
*
https://github.com/mdkinney/edk2/tree/Bug_3750_IncreaseAzurePipelines
Time
out
* https://github.com/tianocore/edk2/pull/2219
* Required to allow EccCheck of uncrustify changes to complete on
Azure
Pipelines CI agents without timing out.
* Status: Review complete. PR pushed.

5) Update Package YAML to ignore specific ECC files/errors
*
INVALID URI REMOVED
d=3749__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHo-fY3dMo$
*
https://github.com/mdkinney/edk2/tree/Bug_3749_EccCheckIgnoreFilesErr
ors
* https://github.com/tianocore/edk2/pull/2218
* Required to pass EccCheck
* Status: Review complete. PR pushed

6) Uncrustify Source Changes
*
INVALID URI REMOVED
d=3737__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHoMRa47YU$
*
INVALID URI REMOVED
d=3739__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHoPAlckP8$
*
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
han
ges_V5
* https://github.com/tianocore/edk2/pull/2229
* Build comparison result PASS:
https://github.com/mdkinney/edk2/actions/runs/1521618836
* EFI_D_ -> DEBUG changes required to pass PatchCheck
* Uncrustify format changes required to pass UncrustifyCheck
* Status:
Waiting for review

7) UncrustifyCheck EDK II CI Plugin
*
INVALID URI REMOVED
d=3748__;!!NpxR!3rTN8lYPg_5bczi9Qai9VWvHKkoslR8D0LLPZTWXL0OKtMZJ
OYpcSVHoCzEWymQ$
*
https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plug
in_v
6
* Required to enforce all PRs submitted to EDK II CI match
uncrustify
format.
* Unconditionally checks all packages. Can not be committed until
all C/H
source files have been updated.
* Status: Review complete

Combined Branch/PR for Review/Test
==================================
* Build Comparison results must pass 100% across the full set of PRs
before
the individual PRs can be pushed in the order listed above.
* Branch:
https://github.com/mdkinney/edk2/tree/TestOnly_Uncrustify_PR_Series
* PR: https://github.com/tianocore/edk2/pull/2229
Status = PASS
* CompareBuild:
Branch:
https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyC
han
ges_V5
--ref1: ef9a059cdb15844fe52a49af2bf7d86b9dd3e9bf
--ref2: Bug_3737_3739_ApplyUncrustifyChanges_V5
Extra Options: -n 4 --quiet
Results:
https://github.com/mdkinney/edk2/actions/runs/1521618836
30 VS2019 build comparisons PASS
40 GCC5 build comparisons PASS
100% PASS

The following git log shows the set of patches from --ref1 to --ref
2across
which there are no differences in any of the
OBJ/LIB/DLL/EFI/FFS/FV/FD
files.

--ref2
b7d4bf0675b7 (HEAD -> Bug_3737_3739_ApplyUncrustifyChanges_V5)
UnitTestFrameworkPkg: Apply uncrusitify changes
7f03d25f60e7 UefiPayloadPkg: Apply uncrusitify changes
0bfd8d9b5ac9 UefiCpuPkg: Apply uncrusitify changes
e1cd9bfb9dea StandaloneMmPkg: Apply uncrusitify changes
5da2f65be378 SourceLevelDebugPkg: Apply uncrusitify changes
95b86de07e5d SignedCapsulePkg: Apply uncrusitify changes
fe71d97246c4 ShellPkg: Apply uncrusitify changes
54c21c952992 SecurityPkg: Apply uncrusitify changes
187a3785f12b RedfishPkg: Apply uncrusitify changes
810100002a46 PcAtChipsetPkg: Apply uncrusitify changes
276a695c0cf2 OvmfPkg: Apply uncrusitify changes
303c0a91ab07 NetworkPkg: Apply uncrusitify changes
bc80792cd1b1 MdePkg: Apply uncrusitify changes
3ea86be17a2a MdeModulePkg: Apply uncrusitify changes
c70ef11ed0cd IntelFsp2WrapperPkg: Apply uncrusitify changes
c0291221f252 IntelFsp2Pkg: Apply uncrusitify changes
6a479952a690 FmpDevicePkg: Apply uncrusitify changes
3a7c05b7070d FatPkg: Apply uncrusitify changes
b789f98c8959 EmulatorPkg: Apply uncrusitify changes
952d7a1c9220 EmbeddedPkg: Apply uncrusitify changes
a1cc9881bab6 DynamicTablesPkg: Apply uncrusitify changes
50654dfe5785 CryptoPkg: Apply uncrusitify changes
ed965a02dfa1 ArmVirtPkg: Apply uncrusitify changes
9744023fbc46 ArmPlatformPkg: Apply uncrusitify changes
7a1cde5f5bba ArmPkg: Apply uncrusitify changes
19d17e0913e8 UefiCpuPkg: Change use of EFI_D_* to DEBUG_*
ffa718b4f994 SourceLevelDebugPkg: Change use of EFI_D_* to
DEBUG_*
b86cb3c5e5b4 ShellPkg: Change use of EFI_D_* to DEBUG_*
c7c42204dc07 SecurityPkg: Change use of EFI_D_* to DEBUG_*
16b8e6f958e4 PcAtChipsetPkg: Change use of EFI_D_* to DEBUG_*
0ac3f8b2dac5 OvmfPkg: Change use of EFI_D_* to DEBUG_*
bc5004b8d294 NetworkPkg: Change use of EFI_D_* to DEBUG_*
6f671a8e2377 MdePkg: Change use of EFI_D_* to DEBUG_*
a10c610ff9a3 MdeModulePkg: Change use of EFI_D_* to DEBUG_*
09a3bddba390 FatPkg: Change use of EFI_D_* to DEBUG_*
59c61318246a EmulatorPkg: Change use of EFI_D_* to DEBUG_*
3a80367dda3b EmbeddedPkg: Change use of EFI_D_* to DEBUG_*
23eb1aaf80ca ArmVirtPkg: Change use of EFI_D_* to DEBUG_*
875914b45c54 ArmPlatformPkg: Change use of EFI_D_* to DEBUG_*
eb2eca82b451 ArmPkg: Change use of EFI_D_* to DEBUG_*
f0f3f5aae7c4 (origin/master, origin/HEAD, master)
UnitTestFrameworkPkg:
Update YAML to ignore specific ECC files/errors
c05734797790 UefiPayloadPkg: Update YAML to ignore specific ECC
files/errors
c30c40d6c63d StandaloneMmPkg: Update YAML to ignore specific
ECC
files/errors
9944508e85f1 ShellPkg: Update YAML to ignore specific ECC
files/errors
60fa40be458d SecurityPkg: Update YAML to ignore specific ECC
files/errors
df790cd6b37e MdePkg: Update YAML to ignore specific ECC
files/errors
9deb9370766e MdeModulePkg: Update YAML to ignore specific ECC
files/errors
d7d30e8f219f EmulatorPkg: Update YAML to ignore specific ECC
files/errors
d5744ecba813 CryptoPkg: Update YAML to ignore specific ECC
files/errors
c97fee87f0f9 ArmVirtPkg: Update YAML to ignore specific ECC
files/errors
1939fc9569f2 ArmPlatformPkg: Update YAML to ignore specific ECC
files/errors
365dced2c37a ArmPkg: Update YAML to ignore specific ECC
files/errors
76a1ce4d5fec .azurepipelines/templates: Update max pipeline job
time to
2
hours
99f84ff47390 .pytools/Plugin/LicenseCheck: Use temp directory for
git diff
output
3019f1bbabf1 .pytool/Plugin/EccCheck: Add performance
optimizations
854462bd3479 .pytool/Plugin/EccCheck: Remove temp directory on
exception
69877614fdee .pytool/Plugin/EccCheck: Remove RevertCode()
--ref1
ef9a059cdb15 EmulatorPkg/Win/Host: Update CC_FLAGS
bb1bba3d7767 (tag: edk2-stable202111) NetworkPkg: Fix invalid
pointer for
DNS
response token on error

Best regards,

Mike



















Gerd Hoffmann
 

On Wed, Dec 01, 2021 at 12:05:24PM -0500, Michael Kubacki wrote:
This can of course be done at a technical level but it smells of a code
cohesion problem.

The code is in the edk2 project so it would reason that it should be uniform
in style with other code and at least close to the EDK II C Coding Standard
Specification.

Currently, git submodules and ignored path are already excluded in addition
to the ability to mark packages in "audit mode" so they do not fail the
results. I would personally prefer not to allow arbitrary subdirectories to
diverge from the rest of the codebase / package but if there's consensus
this is a good idea, I can look into adding it.
What is the plan for generated include files, for example the ones in
CryptoPkg/Library/Include/openssl?

take care,
Gerd


Michael Kubacki
 

Hi Gerd,

To help me understand which files you're specifically referring to, can you please point them out from this commit? Or provide additional details?

https://github.com/tianocore/edk2/pull/2229/commits/50654dfe5785964c9ae72961d13a50b26af77794

CryptoPkg/Library/Include/openssl/opensslconf.h is currently formatted.

Regards,
Michael

On 12/2/2021 6:00 AM, Gerd Hoffmann wrote:
On Wed, Dec 01, 2021 at 12:05:24PM -0500, Michael Kubacki wrote:
This can of course be done at a technical level but it smells of a code
cohesion problem.

The code is in the edk2 project so it would reason that it should be uniform
in style with other code and at least close to the EDK II C Coding Standard
Specification.

Currently, git submodules and ignored path are already excluded in addition
to the ability to mark packages in "audit mode" so they do not fail the
results. I would personally prefer not to allow arbitrary subdirectories to
diverge from the rest of the codebase / package but if there's consensus
this is a good idea, I can look into adding it.
What is the plan for generated include files, for example the ones in
CryptoPkg/Library/Include/openssl?
take care,
Gerd


Gerd Hoffmann
 

On Thu, Dec 02, 2021 at 10:33:14AM -0500, Michael Kubacki wrote:
Hi Gerd,

To help me understand which files you're specifically referring to, can you
please point them out from this commit? Or provide additional details?

https://github.com/tianocore/edk2/pull/2229/commits/50654dfe5785964c9ae72961d13a50b26af77794

CryptoPkg/Library/Include/openssl/opensslconf.h is currently formatted.
Yes, that one, and there is another in CryptoPkg/Library/Include/crypto/
The switch to openssl 3.0 will add more of those files[1]

They are generated from openssl source code,
CryptoPkg/Library/OpensslLib/process_files.pl handles that.

I think the two reasonable options to deal with that are:

(1) exclude those files from formating, or
(2) call uncrustify in process_files.pl to reformat them
each time they are generated.

Given that these files will never be edited manually my personal
preference would be (1).

take care,
Gerd

[1] wip branch @ https://github.com/kraxel/edk2/commits/openssl3


Michael Kubacki
 

Thank you for clarifying. This case is more reasonable to me since the files are auto generated and not ported into edk2.

There is an option to bring files like this in as a new submodule but I understand why that might not be desirable and would be disruptive to the current process.

If there's no objection, I'll add an option to exclude paths within a package in a v6 patch of the plugin and leave this to maintainer discretion. I just ask that the spirit of consistent formatting be considered when making these decisions.

Regards,
Michael

On 12/2/2021 11:23 AM, Gerd Hoffmann wrote:
On Thu, Dec 02, 2021 at 10:33:14AM -0500, Michael Kubacki wrote:
Hi Gerd,

To help me understand which files you're specifically referring to, can you
please point them out from this commit? Or provide additional details?

https://github.com/tianocore/edk2/pull/2229/commits/50654dfe5785964c9ae72961d13a50b26af77794

CryptoPkg/Library/Include/openssl/opensslconf.h is currently formatted.
Yes, that one, and there is another in CryptoPkg/Library/Include/crypto/
The switch to openssl 3.0 will add more of those files[1]
They are generated from openssl source code,
CryptoPkg/Library/OpensslLib/process_files.pl handles that.
I think the two reasonable options to deal with that are:
(1) exclude those files from formating, or
(2) call uncrustify in process_files.pl to reformat them
each time they are generated.
Given that these files will never be edited manually my personal
preference would be (1).
take care,
Gerd
[1] wip branch @ https://github.com/kraxel/edk2/commits/openssl3


Michael D Kinney
 

Michael,

Given that there are now 2 requests for this feature, I think it is a
good idea to add the feature.

Package YAML files are part of the review content, so we can make sure
we all look at these YAML changes and provide feedback when this
uncrustify exclusion feature is applied in areas that do not make
sense.

One additional challenge with the exclusion feature is how developers
can run uncrustify locally to prepare their patch reviews. They can
not run it on all c/h files in a package. Now they have to know which
directories to exclude. This applies to running uncrustify from command
line and usages such as Visual Studio Code plugin. In order to make
it easy for developers to always get the right format, you would need
a helper tool that can look at the YAML file to know if there are
areas that are excluded and exclude files from the uncrustify processing.
I am concerned that someone will miss the exclusion list in patch
generation and review and the uncrustified versions will get checked
in by accident at some point. EDK II CI will not care because it will
skip the UncrustifyCheck on excluded files/dirs.

Thanks,

Mike

-----Original Message-----
From: Michael Kubacki <mikuback@linux.microsoft.com>
Sent: Thursday, December 2, 2021 9:15 AM
To: Gerd Hoffmann <kraxel@redhat.com>
Cc: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>; Chang, Abner <abner.chang@hpe.com>; Wang, Jian J
<jian.j.wang@intel.com>; Michael Kubacki <michael.kubacki@microsoft.com>; Andrew Fish (afish@apple.com) <afish@apple.com>;
Leif Lindholm <leif@nuviainc.com>
Subject: Re: [edk2-devel] Uncrustify Conversion Detailed Plan and Extended Hard Freeze Update #4

Thank you for clarifying. This case is more reasonable to me since the
files are auto generated and not ported into edk2.

There is an option to bring files like this in as a new submodule but I
understand why that might not be desirable and would be disruptive to
the current process.

If there's no objection, I'll add an option to exclude paths within a
package in a v6 patch of the plugin and leave this to maintainer
discretion. I just ask that the spirit of consistent formatting be
considered when making these decisions.

Regards,
Michael

On 12/2/2021 11:23 AM, Gerd Hoffmann wrote:
On Thu, Dec 02, 2021 at 10:33:14AM -0500, Michael Kubacki wrote:
Hi Gerd,

To help me understand which files you're specifically referring to, can you
please point them out from this commit? Or provide additional details?

https://github.com/tianocore/edk2/pull/2229/commits/50654dfe5785964c9ae72961d13a50b26af77794

CryptoPkg/Library/Include/openssl/opensslconf.h is currently formatted.
Yes, that one, and there is another in CryptoPkg/Library/Include/crypto/
The switch to openssl 3.0 will add more of those files[1]

They are generated from openssl source code,
CryptoPkg/Library/OpensslLib/process_files.pl handles that.

I think the two reasonable options to deal with that are:

(1) exclude those files from formating, or
(2) call uncrustify in process_files.pl to reformat them
each time they are generated.

Given that these files will never be edited manually my personal
preference would be (1).

take care,
Gerd

[1] wip branch @ https://github.com/kraxel/edk2/commits/openssl3


Maciej Rabeda
 

Hey Mike,

While most of the changes related to fixing coding style violations, there are a couple of changes to NetworkPkg in that PR that make the code less readable. Examples below.

Example 1:



Example 2:


Example 3:


Example 4:


On 30-Nov-21 23:34, Michael D Kinney wrote:
Hello,

Thank you for your patience during this extended hard freeze.

Just one more step to go.  There has been a delay in the review of
the patch series with the uncrustify source changes.  PR(6).  This
patch series was not sent out as patch review email because of its 
very large size.  It only contains source style changes and the 
CompareBuild tool and GitHub action has shown there are no binary 
differences introduced with these source style changes.

If you are a package maintainer, then please review the following
branch/PR for your package contents and review the EDK II CI results
and BuildCompare results.  I do not expect a line by line review
because we already had time to provide feedback on the source style
performed by uncrustify.  Instead, a Reviewed-by for your package
indicates that you have reviewed the EDK II CI results and CompareBuild
tool functionality and results and you accept the source style 
changes to your package.

   * https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChanges_V5
   * https://github.com/tianocore/edk2/pull/2229
   * https://github.com/mdkinney/edk2/actions/runs/1521618836

Additional details on this update below.

Thank you,

Mike


Changes from Update #3
----------------------------------------------------------------------------
* Pushed PR (5)
* Added link to PR(6). EDK II CI Status is PASS. Build Compare PASS.
* Waiting for review of PR (6)
* Review of PR (7) completed and waiting for review of PR (6)
----------------------------------------------------------------------------

Changes from Update #2
----------------------------------------------------------------------------
* Changed order of PRs swapping (4) and (5).  The PR that activates
  increases the max CI agent job time is independent of all the other
  PRs and its review is complete, so it can be committed now.
* Pushed PRs (1), (2), (3), (4).
* Waiting for review to complete for PRs (5) and (6)
* Reviews complete for PR (7)
* Identifies steps using git filter-branch to apply uncrustify changes to a
  code review patch series that was generated before the uncrustify changes
  avoiding manual merge.
* Identified steps using git filter-repo to generate an alternate history of
  the edk2 repo with uncrustify changes applied on every commit.  This may
  be useful when evaluating changes to files using tools like git blame
  without the large uncrustify patch series.
---------------------------------------------------------------------------

Changes from Update #1
----------------------------------------------------------------------------
* Changed order of PRs swapping (6) and (7).  The PR that activates
  EDK II CI check UncrustifyCheck has to be last because it unconditionally
  checks all C/H files in all packages.  Not just files that have been
  modified like some of the other checkers.
* Updated link to the branch with the UncrustifyCheck plugin that has been
  updated with a one line change and Reviewed-by and Tested-by tags.
  https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plugin_v6 
* Reviews complete for (1), (2), (3), (5), and (7)
---------------------------------------------------------------------------

Michael Kubacki and I have prepared the patches required to apply the 
uncrustify changes and enable EDK II CI to check all submitted
patches have been run through uncrustify.

We have verified through the CompareBuild GitHub Action that the
format changes performed by uncrustify have no functional changes.
All of the OBJ, LIB, DLL, EFI, FFS, FV, and FD files match 100%
across 70 VS2019/GCC5 builds of all package/platform DSC files in
the edk2 repo.

The hard freeze will be extended after the edk2-stable202111 tag until
all uncrustify related changes are committed.  We do not expect this 
to take more than a few days.  Do not push any PRs until the hard 
freeze is lifted.

The changes are broken up into 7 patch series/PRs.  The PRs are ordered
so they can be submitted using the normal submission process and EDK II
CI will pass for each one.  Details are listed below.

Uncrustify 73.0.3 for EDK II
=============================
* Sources: https://dev.azure.com/projectmu/_git/Uncrustify
* Documentation: https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme
* Download: https://dev.azure.com/projectmu/Uncrustify/_packaging?_a=package&feed=mu_uncrustify&package=mu-uncrustify-release&protocolType=NuGet&version=73.0.3

Installing Uncrustify
======================
The Uncrustify tool is installed automatically when the Pytools
environment is used and the stuart* commands are run to complete the
environment setup.  Please see:

    https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-locally

Uncrustify can also be installed from the download page listed above 
or built from sources from the source link above.

The Documentation link provides instruction on how to run uncrustify from 
the command line or install as a Visual Studio Code plugin.  The main
uncrustify documentation also describes how to integrate with a few other
editors.

We have also discussed a client side githook.  That effort has not started.
Let us know if that is a feature you would find useful.

Developer impact for new code reviews
======================================
Once the uncrustify checker is active in EDK II CI, developers must 
make sure their patches are run through the uncrustify tool before 
sending the patches for review.

Developers must install and run uncrustify against changes files before 
sending patch review emails or submitting PR for EDK II CI.  If EDK II CI
detects and differences in source formatting, then EDK II CI will fail
and the developer must run uncrustify and resubmit the patches.

Developer impact to patch series/PRs reviewed during edk2-stable201121 soft/hard freeze
=======================================================================================
Developers must rebase their changes after the uncrustify source changes are
committed.  The branch with a preview of the uncrustify changes can be used
to start this rebase work.

    https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChanges_V5

The following steps can be used to update an existing branch with the 
required uncrustify format.  This is the Windows version.  I will add
the Linux version soon.

    1) Fetch and checkout and rebase to latest edk2/master

           git fetch origin
           git checkout master
           git rebase origin/master

    2) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE. 
       (e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
       EDK II specific uncrustify configuration file available when working
       with a branch that does not have those tools in its scope.

           xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

    3) Check out the patch series branch (e.g. MyBranch)

           git checkout MyBranch

    4) Rebase patch series against edk2-stable202111
 
           git rebase edk2-stable202111

    5) Create new branch for the uncrustifed version (e.g. MyBranch_Uncrustified)

           git checkout -b MyBranch_Uncrustified
    
    6) Use git filter-branch to uncrustify all the commits in the series
       between the rebase target from (2) and HEAD of the branch.  A filter
       can be used to scope the uncrustify operations to only the C/H files
       in the specific package the patch series is against. (e.g. DynamicTablesPkg).
       BaseTools should always be excluded.  If the package scoped filter is 
       not used, it will still work, but will take longer to run because 
       uncrustify will rescan every C/H files in the whole repo.

           git filter-branch --tree-filter "git ls-files DynamicTablesPkg*.c DynamicTablesPkg*.h :!BaseTools/* | c:\\Temp\\UncrustifyCheck\\mu-uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c c:\\Temp\\UncrustifyCheck\\uncrustify.cfg -F - --replace --no-backup --if-changed" edk2-stable202111..HEAD

    7) Now that all the individual patches in the branch are uncrustified, 
       rebase against latest edk2/master that is already uncrustified.

           git rebase master

    8) Verify the patches in this new branch.

Impacts to tracing history across the uncrusity changes
=======================================================
Tools the view file and line history do work with the large uncrustify
patch series.  One impact is that the operations can be very slow due
to the large uncrustify patches.

One option to provide a faster experience is to provide an alternate
version of the edk2 repository as "documentation" that has the 
entire history re-written with uncrustify run on every commit.
The tool called git-filter-repo can be used to perform this 
transformation and runs in a reasonable period of time (a few hours) 

   https://github.com/newren/git-filter-repo
   https://github.com/newren/git-filter-repo/blob/main/contrib/filter-repo-demos/lint-history

The following steps can be used to perform this transformation.
This is the Windows version. I will add the Linux version soon.

    ** WARNING **  This operation modifies(rewrites) all the commits
                   in the local copy of the repo.  Do not perform
                   these steps on a local repo you are using for 
                   active development.

    1) Clone edk2 into a new directory (see **WARNING**)

           git clone https://github.com/tianocore/edk2.git edk2-uncrustified
           cd edk2-uncrustified

    2) Setup python virtual env, install pytools, and run stuart commands
       to setup build environment which includes installing uncrustify tools.

           https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-locally

    3) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE. 
       (e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
       EDK II specific uncrustify configuration file available when working
       with a branch that does not have those tools in its scope.

           xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck

    4) Use lint-history.py from git-filter-repo examples

           https://github.com/newren/git-filter-repo
           https://github.com/newren/git-filter-repo/blob/main/contrib/filter-repo-demos/lint-history

       Line #127 - Add try except around subprocess.check_call() with except
       being pass.  This is required because there are a few commits of C
       files in the edk2 repo that have incorrect C syntax and do not 
       build with a C compiler and break the uncrustify parser.  Skip reformat
       of C files that can not be parsed by uncrustify.  These rare instances
       are addressed in the commit that fixes the C syntax error.

       Run this slightly modified version of lint-history.  Include only
       C/H files and exclude directories that start with 'Tools' or 'BaseTools'.
       This step took about 2.2 hours on a laptop.

           lint-history.py 
               --relevant "return (not filename.startswith(b'Tools') and not filename.startswith(b'BaseTools') and (filename.endswith(b'.c') or filename.endswith(b'.h')))"
               c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\mu-uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\uncrustify.cfg --replace --no-backup --if-changed

Order of PRs to apply during extended hard freeze
==================================================
1) Update EmulatorPkg Win Host [BuildOptions] MSFT CC_FLAGS to not force debug information
   * https://bugzilla.tianocore.org/show_bug.cgi?id=3747
   * https://github.com/mdkinney/edk2/tree/Bug_3747_EmulatorPkg_WinHost_ReproducibleBuild
   * https://github.com/tianocore/edk2/pull/2215
   * Required for EmulatorPkg to pass CompareBuild for VS2019 IA32/X64 builds.
   * Status: Review complete.  PR pushed.

2) EccCheck should not revert staged and local changes
   * https://bugzilla.tianocore.org/show_bug.cgi?id=2986
   * https://github.com/mdkinney/edk2/tree/Bug_2986_EccCheckRemoveGitRevert_V2
   * https://github.com/tianocore/edk2/pull/2216
   * Required for EDK II CI to complete in a reasonable period of time when
     processing the 4000+ source file style changes made by uncrustify.
   * Also fixes critical bugs that can potentially corrupt git state when 
     EccCheck is run locally.
   * Status: Review complete.  PR pushed.

3) Update pytool LicenseCheck plugin to use temp directory for diff output file
   * https://bugzilla.tianocore.org/show_bug.cgi?id=3746
   * https://github.com/mdkinney/edk2/tree/Bug_3746_LicenseCheckUseDiffOutputFile_V2
   * https://github.com/tianocore/edk2/pull/2217
   * Required to reduce EDK II CI build times.
   * Status: Review complete.  PR pushed.

4) Update max job time from 60 min to 120 minutes in .azurepipelines/templates
   * https://bugzilla.tianocore.org/show_bug.cgi?id=3750
   * https://github.com/mdkinney/edk2/tree/Bug_3750_IncreaseAzurePipelinesTimeout
   * https://github.com/tianocore/edk2/pull/2219
   * Required to allow EccCheck of uncrustify changes to complete on Azure
     Pipelines CI agents without timing out.
   * Status: Review complete.  PR pushed.

5) Update Package YAML to ignore specific ECC files/errors
   * https://bugzilla.tianocore.org/show_bug.cgi?id=3749
   * https://github.com/mdkinney/edk2/tree/Bug_3749_EccCheckIgnoreFilesErrors
   * https://github.com/tianocore/edk2/pull/2218
   * Required to pass EccCheck
   * Status: Review complete. PR pushed

6) Uncrustify Source Changes
   * https://bugzilla.tianocore.org/show_bug.cgi?id=3737
   * https://bugzilla.tianocore.org/show_bug.cgi?id=3739
   * https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChanges_V5
   * https://github.com/tianocore/edk2/pull/2229
   * Build comparison result PASS: https://github.com/mdkinney/edk2/actions/runs/1521618836
   * EFI_D_ -> DEBUG changes required to pass PatchCheck
   * Uncrustify format changes required to pass UncrustifyCheck
   * Status:
       Waiting for review

7) UncrustifyCheck EDK II CI Plugin
   * https://bugzilla.tianocore.org/show_bug.cgi?id=3748
   * https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plugin_v6
   * Required to enforce all PRs submitted to EDK II CI match uncrustify format.
   * Unconditionally checks all packages.  Can not be committed until all C/H
     source files have been updated.
   * Status: Review complete

Combined Branch/PR for Review/Test
==================================
* Build Comparison results must pass 100% across the full set of PRs before
  the individual PRs can be pushed in the order listed above.
* Branch: https://github.com/mdkinney/edk2/tree/TestOnly_Uncrustify_PR_Series
* PR: https://github.com/tianocore/edk2/pull/2229
    Status = PASS
* CompareBuild:
    Branch: https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChanges_V5
    --ref1:        ef9a059cdb15844fe52a49af2bf7d86b9dd3e9bf
    --ref2:        Bug_3737_3739_ApplyUncrustifyChanges_V5
    Extra Options: -n 4 --quiet
    Results: https://github.com/mdkinney/edk2/actions/runs/1521618836
	30 VS2019 build comparisons PASS
	40 GCC5 build comparisons PASS
        100% PASS

The following git log shows the set of patches from --ref1 to --ref 2across
which there are no differences in any of the OBJ/LIB/DLL/EFI/FFS/FV/FD files.

--ref2
b7d4bf0675b7 (HEAD -> Bug_3737_3739_ApplyUncrustifyChanges_V5) UnitTestFrameworkPkg: Apply uncrusitify changes
7f03d25f60e7 UefiPayloadPkg: Apply uncrusitify changes    
0bfd8d9b5ac9 UefiCpuPkg: Apply uncrusitify changes                                                                                                                                                                        
e1cd9bfb9dea StandaloneMmPkg: Apply uncrusitify changes                                                                                                                                                                   
5da2f65be378 SourceLevelDebugPkg: Apply uncrusitify changes                                                                                                                                                               
95b86de07e5d SignedCapsulePkg: Apply uncrusitify changes                                                                                                                                                                  
fe71d97246c4 ShellPkg: Apply uncrusitify changes                                                                                                                                                                          
54c21c952992 SecurityPkg: Apply uncrusitify changes                                                                                                                                                                       
187a3785f12b RedfishPkg: Apply uncrusitify changes                                                                                                                                                                        
810100002a46 PcAtChipsetPkg: Apply uncrusitify changes                                                                                                                                                                    
276a695c0cf2 OvmfPkg: Apply uncrusitify changes                                                                                                                                                                           
303c0a91ab07 NetworkPkg: Apply uncrusitify changes                                                                                                                                                                        
bc80792cd1b1 MdePkg: Apply uncrusitify changes                                                                                                                                                                            
3ea86be17a2a MdeModulePkg: Apply uncrusitify changes                                                                                                                                                                      
c70ef11ed0cd IntelFsp2WrapperPkg: Apply uncrusitify changes                                                                                                                                                               
c0291221f252 IntelFsp2Pkg: Apply uncrusitify changes                                                                                                                                                                      
6a479952a690 FmpDevicePkg: Apply uncrusitify changes                                                                                                                                                                      
3a7c05b7070d FatPkg: Apply uncrusitify changes                                                                                                                                                                            
b789f98c8959 EmulatorPkg: Apply uncrusitify changes                                                                                                                                                                       
952d7a1c9220 EmbeddedPkg: Apply uncrusitify changes                                                                                                                                                                       
a1cc9881bab6 DynamicTablesPkg: Apply uncrusitify changes                                                                                                                                                                  
50654dfe5785 CryptoPkg: Apply uncrusitify changes                                                                                                                                                                         
ed965a02dfa1 ArmVirtPkg: Apply uncrusitify changes                                                                                                                                                                        
9744023fbc46 ArmPlatformPkg: Apply uncrusitify changes                                                                                                                                                                    
7a1cde5f5bba ArmPkg: Apply uncrusitify changes                                                                                                                                                                            
19d17e0913e8 UefiCpuPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                 
ffa718b4f994 SourceLevelDebugPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                        
b86cb3c5e5b4 ShellPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                   
c7c42204dc07 SecurityPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                
16b8e6f958e4 PcAtChipsetPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                             
0ac3f8b2dac5 OvmfPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                    
bc5004b8d294 NetworkPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                 
6f671a8e2377 MdePkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                     
a10c610ff9a3 MdeModulePkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                               
09a3bddba390 FatPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                     
59c61318246a EmulatorPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                
3a80367dda3b EmbeddedPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                
23eb1aaf80ca ArmVirtPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                 
875914b45c54 ArmPlatformPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                             
eb2eca82b451 ArmPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                     
f0f3f5aae7c4 (origin/master, origin/HEAD, master) UnitTestFrameworkPkg: Update YAML to ignore specific ECC files/errors                                                                                                   
c05734797790 UefiPayloadPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                              
c30c40d6c63d StandaloneMmPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                             
9944508e85f1 ShellPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                    
60fa40be458d SecurityPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                 
df790cd6b37e MdePkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                      
9deb9370766e MdeModulePkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                
d7d30e8f219f EmulatorPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                 
d5744ecba813 CryptoPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                   
c97fee87f0f9 ArmVirtPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                  
1939fc9569f2 ArmPlatformPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                              
365dced2c37a ArmPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                      
76a1ce4d5fec .azurepipelines/templates: Update max pipeline job time to 2 hours                                                                                                                                           
99f84ff47390 .pytools/Plugin/LicenseCheck: Use temp directory for git diff output                                                                                                                                         
3019f1bbabf1 .pytool/Plugin/EccCheck: Add performance optimizations                                                                                                                                                       
854462bd3479 .pytool/Plugin/EccCheck: Remove temp directory on exception                                                                                                                                                  
69877614fdee .pytool/Plugin/EccCheck: Remove RevertCode() 
--ref1                                                                                                                                                                
ef9a059cdb15 EmulatorPkg/Win/Host: Update CC_FLAGS                                                                                                                                                                        
bb1bba3d7767 (tag: edk2-stable202111) NetworkPkg: Fix invalid pointer for DNS response token on error                                                                                                                     

Best regards,

Mike












Michael D Kinney
 

Hi Maciej,

 

Thanks for the feedback.

 

  • Example #1.  This appears to be caused by the following uncrustify settings:

 

# The threshold for aligning on '=' in assignments.

# Use a negative number for absolute thresholds.

#

# 0: No limit (default).

align_assign_thresh             = 0        # number

 

              The edk2 setting for this is:

 

align_assign_thresh             = 4

 

              This means blocks of assignments that are different than more than 4 spaces will be considered a new block.

              HwAddreLen’ and ‘Xid’ are more than 4 characters in length different.  Same for ‘Xid’ and ‘Reserved’.  So

              uncrustify treats these as 3 different assignment blocks. 

 

              If we change to the default value of 0: No limit, this example is treated as a single block and all ‘=’ are aligned.

 

              Is there a reason ‘4’ was selected?  Is there is any harm in using the default of 0?

 

  • Example #2: Uncruistfy is confused by the DEBUG_CODE() macro.  This is not a traditional macro because the

contents of the macro is a block of C code.  I do not know how to convince uncrustify that the contents of a

macro like function call to be treated as a code block from an indent perspective.

 

  • Example #3/#4: Uncrustify is confused by the OPTIONAL keyword.  The edk2 config declares it as a QUALIFIER

like IN and OUT.  However, OPTIONAL only appears at the end of the line that declares a parameter to a

function.  There are 3 forms. One with comma after OPTIONAL.  One with comma before OPTIONAL, and

one with no comma if the parameter is the last parameter in the function declaration. 

 

              TYPE  ParamName OPTIONAL,

              TYPE  ParamName, OPTIONAL

              TYPE  ParamName OPTIONAL

             

              OPTIONAL is defined to nothing in edk2 builds.  From a uncrustify perspective, we really want is to be

              ignored or more correctly treated it as a token that is attached to ParamName and the combination of

              ParamName and OPTIONAL treated as one unit to determine indents.

 

Thanks,

 

Mike

   

 

 

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Maciej Rabeda
Sent: Thursday, December 2, 2021 10:27 AM
To: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@...>; Michael Kubacki <michael.kubacki@...>; Andrew Fish (afish@...) <afish@...>; Leif Lindholm <leif@...>
Subject: Re: [edk2-devel] Uncrustify Conversion Detailed Plan and Extended Hard Freeze Update #4

 

Hey Mike,

While most of the changes related to fixing coding style violations, there are a couple of changes to NetworkPkg in that PR that make the code less readable. Examples below.

Example 1:



Example 2:


Example 3:


Example 4:

On 30-Nov-21 23:34, Michael D Kinney wrote:

Hello,
 
Thank you for your patience during this extended hard freeze.
 
Just one more step to go.  There has been a delay in the review of
the patch series with the uncrustify source changes.  PR(6).  This
patch series was not sent out as patch review email because of its 
very large size.  It only contains source style changes and the 
CompareBuild tool and GitHub action has shown there are no binary 
differences introduced with these source style changes.
 
If you are a package maintainer, then please review the following
branch/PR for your package contents and review the EDK II CI results
and BuildCompare results.  I do not expect a line by line review
because we already had time to provide feedback on the source style
performed by uncrustify.  Instead, a Reviewed-by for your package
indicates that you have reviewed the EDK II CI results and CompareBuild
tool functionality and results and you accept the source style 
changes to your package.
 
   * https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChanges_V5
   * https://github.com/tianocore/edk2/pull/2229
   * https://github.com/mdkinney/edk2/actions/runs/1521618836
 
Additional details on this update below.
 
Thank you,
 
Mike
 
 
Changes from Update #3
----------------------------------------------------------------------------
* Pushed PR (5)
* Added link to PR(6). EDK II CI Status is PASS. Build Compare PASS.
* Waiting for review of PR (6)
* Review of PR (7) completed and waiting for review of PR (6)
----------------------------------------------------------------------------
 
Changes from Update #2
----------------------------------------------------------------------------
* Changed order of PRs swapping (4) and (5).  The PR that activates
  increases the max CI agent job time is independent of all the other
  PRs and its review is complete, so it can be committed now.
* Pushed PRs (1), (2), (3), (4).
* Waiting for review to complete for PRs (5) and (6)
* Reviews complete for PR (7)
* Identifies steps using git filter-branch to apply uncrustify changes to a
  code review patch series that was generated before the uncrustify changes
  avoiding manual merge.
* Identified steps using git filter-repo to generate an alternate history of
  the edk2 repo with uncrustify changes applied on every commit.  This may
  be useful when evaluating changes to files using tools like git blame
  without the large uncrustify patch series.
---------------------------------------------------------------------------
 
Changes from Update #1
----------------------------------------------------------------------------
* Changed order of PRs swapping (6) and (7).  The PR that activates
  EDK II CI check UncrustifyCheck has to be last because it unconditionally
  checks all C/H files in all packages.  Not just files that have been
  modified like some of the other checkers.
* Updated link to the branch with the UncrustifyCheck plugin that has been
  updated with a one line change and Reviewed-by and Tested-by tags.
  https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plugin_v6 
* Reviews complete for (1), (2), (3), (5), and (7)
---------------------------------------------------------------------------
 
Michael Kubacki and I have prepared the patches required to apply the 
uncrustify changes and enable EDK II CI to check all submitted
patches have been run through uncrustify.
 
We have verified through the CompareBuild GitHub Action that the
format changes performed by uncrustify have no functional changes.
All of the OBJ, LIB, DLL, EFI, FFS, FV, and FD files match 100%
across 70 VS2019/GCC5 builds of all package/platform DSC files in
the edk2 repo.
 
The hard freeze will be extended after the edk2-stable202111 tag until
all uncrustify related changes are committed.  We do not expect this 
to take more than a few days.  Do not push any PRs until the hard 
freeze is lifted.
 
The changes are broken up into 7 patch series/PRs.  The PRs are ordered
so they can be submitted using the normal submission process and EDK II
CI will pass for each one.  Details are listed below.
 
Uncrustify 73.0.3 for EDK II
=============================
* Sources: https://dev.azure.com/projectmu/_git/Uncrustify
* Documentation: https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme
* Download: https://dev.azure.com/projectmu/Uncrustify/_packaging?_a=package&feed=mu_uncrustify&package=mu-uncrustify-release&protocolType=NuGet&version=73.0.3
 
Installing Uncrustify
======================
The Uncrustify tool is installed automatically when the Pytools
environment is used and the stuart* commands are run to complete the
environment setup.  Please see:
 
    https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-locally
 
Uncrustify can also be installed from the download page listed above 
or built from sources from the source link above.
 
The Documentation link provides instruction on how to run uncrustify from 
the command line or install as a Visual Studio Code plugin.  The main
uncrustify documentation also describes how to integrate with a few other
editors.
 
We have also discussed a client side githook.  That effort has not started.
Let us know if that is a feature you would find useful.
 
Developer impact for new code reviews
======================================
Once the uncrustify checker is active in EDK II CI, developers must 
make sure their patches are run through the uncrustify tool before 
sending the patches for review.
 
Developers must install and run uncrustify against changes files before 
sending patch review emails or submitting PR for EDK II CI.  If EDK II CI
detects and differences in source formatting, then EDK II CI will fail
and the developer must run uncrustify and resubmit the patches.
 
Developer impact to patch series/PRs reviewed during edk2-stable201121 soft/hard freeze
=======================================================================================
Developers must rebase their changes after the uncrustify source changes are
committed.  The branch with a preview of the uncrustify changes can be used
to start this rebase work.
 
    https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChanges_V5
 
The following steps can be used to update an existing branch with the 
required uncrustify format.  This is the Windows version.  I will add
the Linux version soon.
 
    1) Fetch and checkout and rebase to latest edk2/master
 
           git fetch origin
           git checkout master
           git rebase origin/master
 
    2) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE. 
       (e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
       EDK II specific uncrustify configuration file available when working
       with a branch that does not have those tools in its scope.
 
           xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck
 
    3) Check out the patch series branch (e.g. MyBranch)
 
           git checkout MyBranch
 
    4) Rebase patch series against edk2-stable202111
 
           git rebase edk2-stable202111
 
    5) Create new branch for the uncrustifed version (e.g. MyBranch_Uncrustified)
 
           git checkout -b MyBranch_Uncrustified
    
    6) Use git filter-branch to uncrustify all the commits in the series
       between the rebase target from (2) and HEAD of the branch.  A filter
       can be used to scope the uncrustify operations to only the C/H files
       in the specific package the patch series is against. (e.g. DynamicTablesPkg).
       BaseTools should always be excluded.  If the package scoped filter is 
       not used, it will still work, but will take longer to run because 
       uncrustify will rescan every C/H files in the whole repo.
 
           git filter-branch --tree-filter "git ls-files DynamicTablesPkg*.c DynamicTablesPkg*.h :!BaseTools/* | c:\\Temp\\UncrustifyCheck\\mu-uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c c:\\Temp\\UncrustifyCheck\\uncrustify.cfg -F - --replace --no-backup --if-changed" edk2-stable202111..HEAD
 
    7) Now that all the individual patches in the branch are uncrustified, 
       rebase against latest edk2/master that is already uncrustified.
 
           git rebase master
 
    8) Verify the patches in this new branch.
 
Impacts to tracing history across the uncrusity changes
=======================================================
Tools the view file and line history do work with the large uncrustify
patch series.  One impact is that the operations can be very slow due
to the large uncrustify patches.
 
One option to provide a faster experience is to provide an alternate
version of the edk2 repository as "documentation" that has the 
entire history re-written with uncrustify run on every commit.
The tool called git-filter-repo can be used to perform this 
transformation and runs in a reasonable period of time (a few hours) 
 
   https://github.com/newren/git-filter-repo
   https://github.com/newren/git-filter-repo/blob/main/contrib/filter-repo-demos/lint-history
 
The following steps can be used to perform this transformation.
This is the Windows version. I will add the Linux version soon.
 
    ** WARNING **  This operation modifies(rewrites) all the commits
                   in the local copy of the repo.  Do not perform
                   these steps on a local repo you are using for 
                   active development.
 
    1) Clone edk2 into a new directory (see **WARNING**)
 
           git clone https://github.com/tianocore/edk2.git edk2-uncrustified
           cd edk2-uncrustified
 
    2) Setup python virtual env, install pytools, and run stuart commands
       to setup build environment which includes installing uncrustify tools.
 
           https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-locally
 
    3) Make a backup copy of plugin UncrustifyCheck outside WORKSPACE. 
       (e.g. C:\Temp\UncrustifyCheck) so the uncrustify tool executable and
       EDK II specific uncrustify configuration file available when working
       with a branch that does not have those tools in its scope.
 
           xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck
 
    4) Use lint-history.py from git-filter-repo examples
 
           https://github.com/newren/git-filter-repo
           https://github.com/newren/git-filter-repo/blob/main/contrib/filter-repo-demos/lint-history
 
       Line #127 - Add try except around subprocess.check_call() with except
       being pass.  This is required because there are a few commits of C
       files in the edk2 repo that have incorrect C syntax and do not 
       build with a C compiler and break the uncrustify parser.  Skip reformat
       of C files that can not be parsed by uncrustify.  These rare instances
       are addressed in the commit that fixes the C syntax error.
 
       Run this slightly modified version of lint-history.  Include only
       C/H files and exclude directories that start with 'Tools' or 'BaseTools'.
       This step took about 2.2 hours on a laptop.
 
           lint-history.py 
               --relevant "return (not filename.startswith(b'Tools') and not filename.startswith(b'BaseTools') and (filename.endswith(b'.c') or filename.endswith(b'.h')))"
               c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\mu-uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\uncrustify.cfg --replace --no-backup --if-changed
 
Order of PRs to apply during extended hard freeze
==================================================
1) Update EmulatorPkg Win Host [BuildOptions] MSFT CC_FLAGS to not force debug information
   * https://bugzilla.tianocore.org/show_bug.cgi?id=3747
   * https://github.com/mdkinney/edk2/tree/Bug_3747_EmulatorPkg_WinHost_ReproducibleBuild
   * https://github.com/tianocore/edk2/pull/2215
   * Required for EmulatorPkg to pass CompareBuild for VS2019 IA32/X64 builds.
   * Status: Review complete.  PR pushed.
 
2) EccCheck should not revert staged and local changes
   * https://bugzilla.tianocore.org/show_bug.cgi?id=2986
   * https://github.com/mdkinney/edk2/tree/Bug_2986_EccCheckRemoveGitRevert_V2
   * https://github.com/tianocore/edk2/pull/2216
   * Required for EDK II CI to complete in a reasonable period of time when
     processing the 4000+ source file style changes made by uncrustify.
   * Also fixes critical bugs that can potentially corrupt git state when 
     EccCheck is run locally.
   * Status: Review complete.  PR pushed.
 
3) Update pytool LicenseCheck plugin to use temp directory for diff output file
   * https://bugzilla.tianocore.org/show_bug.cgi?id=3746
   * https://github.com/mdkinney/edk2/tree/Bug_3746_LicenseCheckUseDiffOutputFile_V2
   * https://github.com/tianocore/edk2/pull/2217
   * Required to reduce EDK II CI build times.
   * Status: Review complete.  PR pushed.
 
4) Update max job time from 60 min to 120 minutes in .azurepipelines/templates
   * https://bugzilla.tianocore.org/show_bug.cgi?id=3750
   * https://github.com/mdkinney/edk2/tree/Bug_3750_IncreaseAzurePipelinesTimeout
   * https://github.com/tianocore/edk2/pull/2219
   * Required to allow EccCheck of uncrustify changes to complete on Azure
     Pipelines CI agents without timing out.
   * Status: Review complete.  PR pushed.
 
5) Update Package YAML to ignore specific ECC files/errors
   * https://bugzilla.tianocore.org/show_bug.cgi?id=3749
   * https://github.com/mdkinney/edk2/tree/Bug_3749_EccCheckIgnoreFilesErrors
   * https://github.com/tianocore/edk2/pull/2218
   * Required to pass EccCheck
   * Status: Review complete. PR pushed
 
6) Uncrustify Source Changes
   * https://bugzilla.tianocore.org/show_bug.cgi?id=3737
   * https://bugzilla.tianocore.org/show_bug.cgi?id=3739
   * https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChanges_V5
   * https://github.com/tianocore/edk2/pull/2229
   * Build comparison result PASS: https://github.com/mdkinney/edk2/actions/runs/1521618836
   * EFI_D_ -> DEBUG changes required to pass PatchCheck
   * Uncrustify format changes required to pass UncrustifyCheck
   * Status:
       Waiting for review
 
7) UncrustifyCheck EDK II CI Plugin
   * https://bugzilla.tianocore.org/show_bug.cgi?id=3748
   * https://github.com/mdkinney/edk2/tree/Bug_3748_add_uncrustify_ci_plugin_v6
   * Required to enforce all PRs submitted to EDK II CI match uncrustify format.
   * Unconditionally checks all packages.  Can not be committed until all C/H
     source files have been updated.
   * Status: Review complete
 
Combined Branch/PR for Review/Test
==================================
* Build Comparison results must pass 100% across the full set of PRs before
  the individual PRs can be pushed in the order listed above.
* Branch: https://github.com/mdkinney/edk2/tree/TestOnly_Uncrustify_PR_Series
* PR: https://github.com/tianocore/edk2/pull/2229
    Status = PASS
* CompareBuild:
    Branch: https://github.com/mdkinney/edk2/tree/Bug_3737_3739_ApplyUncrustifyChanges_V5
    --ref1:        ef9a059cdb15844fe52a49af2bf7d86b9dd3e9bf
    --ref2:        Bug_3737_3739_ApplyUncrustifyChanges_V5
    Extra Options: -n 4 --quiet
    Results: https://github.com/mdkinney/edk2/actions/runs/1521618836
  30 VS2019 build comparisons PASS
  40 GCC5 build comparisons PASS
        100% PASS
 
The following git log shows the set of patches from --ref1 to --ref 2across
which there are no differences in any of the OBJ/LIB/DLL/EFI/FFS/FV/FD files.
 
--ref2
b7d4bf0675b7 (HEAD -> Bug_3737_3739_ApplyUncrustifyChanges_V5) UnitTestFrameworkPkg: Apply uncrusitify changes
7f03d25f60e7 UefiPayloadPkg: Apply uncrusitify changes    
0bfd8d9b5ac9 UefiCpuPkg: Apply uncrusitify changes                                                                                                                                                                        
e1cd9bfb9dea StandaloneMmPkg: Apply uncrusitify changes                                                                                                                                                                   
5da2f65be378 SourceLevelDebugPkg: Apply uncrusitify changes                                                                                                                                                               
95b86de07e5d SignedCapsulePkg: Apply uncrusitify changes                                                                                                                                                                  
fe71d97246c4 ShellPkg: Apply uncrusitify changes                                                                                                                                                                          
54c21c952992 SecurityPkg: Apply uncrusitify changes                                                                                                                                                                       
187a3785f12b RedfishPkg: Apply uncrusitify changes                                                                                                                                                                        
810100002a46 PcAtChipsetPkg: Apply uncrusitify changes                                                                                                                                                                    
276a695c0cf2 OvmfPkg: Apply uncrusitify changes                                                                                                                                                                           
303c0a91ab07 NetworkPkg: Apply uncrusitify changes                                                                                                                                                                        
bc80792cd1b1 MdePkg: Apply uncrusitify changes                                                                                                                                                                            
3ea86be17a2a MdeModulePkg: Apply uncrusitify changes                                                                                                                                                                      
c70ef11ed0cd IntelFsp2WrapperPkg: Apply uncrusitify changes                                                                                                                                                               
c0291221f252 IntelFsp2Pkg: Apply uncrusitify changes                                                                                                                                                                      
6a479952a690 FmpDevicePkg: Apply uncrusitify changes                                                                                                                                                                      
3a7c05b7070d FatPkg: Apply uncrusitify changes                                                                                                                                                                            
b789f98c8959 EmulatorPkg: Apply uncrusitify changes                                                                                                                                                                       
952d7a1c9220 EmbeddedPkg: Apply uncrusitify changes                                                                                                                                                                       
a1cc9881bab6 DynamicTablesPkg: Apply uncrusitify changes                                                                                                                                                                  
50654dfe5785 CryptoPkg: Apply uncrusitify changes                                                                                                                                                                         
ed965a02dfa1 ArmVirtPkg: Apply uncrusitify changes                                                                                                                                                                        
9744023fbc46 ArmPlatformPkg: Apply uncrusitify changes                                                                                                                                                                    
7a1cde5f5bba ArmPkg: Apply uncrusitify changes                                                                                                                                                                            
19d17e0913e8 UefiCpuPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                 
ffa718b4f994 SourceLevelDebugPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                        
b86cb3c5e5b4 ShellPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                   
c7c42204dc07 SecurityPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                
16b8e6f958e4 PcAtChipsetPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                             
0ac3f8b2dac5 OvmfPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                    
bc5004b8d294 NetworkPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                 
6f671a8e2377 MdePkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                     
a10c610ff9a3 MdeModulePkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                               
09a3bddba390 FatPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                     
59c61318246a EmulatorPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                
3a80367dda3b EmbeddedPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                
23eb1aaf80ca ArmVirtPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                 
875914b45c54 ArmPlatformPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                             
eb2eca82b451 ArmPkg: Change use of EFI_D_* to DEBUG_*                                                                                                                                                                     
f0f3f5aae7c4 (origin/master, origin/HEAD, master) UnitTestFrameworkPkg: Update YAML to ignore specific ECC files/errors                                                                                                   
c05734797790 UefiPayloadPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                              
c30c40d6c63d StandaloneMmPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                             
9944508e85f1 ShellPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                    
60fa40be458d SecurityPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                 
df790cd6b37e MdePkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                      
9deb9370766e MdeModulePkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                
d7d30e8f219f EmulatorPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                 
d5744ecba813 CryptoPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                   
c97fee87f0f9 ArmVirtPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                  
1939fc9569f2 ArmPlatformPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                              
365dced2c37a ArmPkg: Update YAML to ignore specific ECC files/errors                                                                                                                                                      
76a1ce4d5fec .azurepipelines/templates: Update max pipeline job time to 2 hours                                                                                                                                           
99f84ff47390 .pytools/Plugin/LicenseCheck: Use temp directory for git diff output                                                                                                                                         
3019f1bbabf1 .pytool/Plugin/EccCheck: Add performance optimizations                                                                                                                                                       
854462bd3479 .pytool/Plugin/EccCheck: Remove temp directory on exception                                                                                                                                                  
69877614fdee .pytool/Plugin/EccCheck: Remove RevertCode() 
--ref1                                                                                                                                                                
ef9a059cdb15 EmulatorPkg/Win/Host: Update CC_FLAGS                                                                                                                                                                        
bb1bba3d7767 (tag: edk2-stable202111) NetworkPkg: Fix invalid pointer for DNS response token on error                                                                                                                     
 
Best regards,
 
Mike