[PATCH v6 0/5] Build cache enhancement

Steven Shi

From: "Shi, Steven" <steven.shi@intel.com>

This patch set is for the 201908 stable tag

Enhance the edk2 build cache with below patches:
Patch 01/05: Improve the cache hit rate through new cache checkpoint and hash algorithm
Patch 02/05: Print more info to explain why a module build cache miss
Patch 03/05: Fix the unsafe [self.Arch][self.Name] key usage in build cache
Patch 04/05 Add the GenFds multi-thread support in build cache
Patch 05/05 Improve the file saving and copying functions reliability in build cache

You can directly try this patch set in the branch:

In the patch 5, add error handling to skip hash calculation if find module cache
already crashed

Fix the method name typo in Misc.py from EdkLogger.quite() to EdkLogger.quiet()

Change single global lock into two locks, which are cache_lock and file_lock,
for better cache performance and IO reliability in windows

Add patch 5. To improve the autogen CopyFileOnChange() and SaveFileOnChange()
functions reliability for build cache

Enhance the SaveHashChainFileToCache() function in ModuleAutoGen.py and
not need to call f.close() in the "with open(xxx) as f:" block. The
with block will close the file automatically

Initial patch set

Shi, Steven (5):
BaseTools: Improve the cache hit in the edk2 build cache
BaseTools: Print first cache missing file for build cachle
BaseTools: Change the [Arch][Name] module key in Build cache
BaseTools: Add GenFds multi-thread support in build cache
BaseTools: Improve the file saving and copying reliability

.../Source/Python/AutoGen/AutoGenWorker.py | 27 +-
BaseTools/Source/Python/AutoGen/CacheIR.py | 29 +
BaseTools/Source/Python/AutoGen/DataPipe.py | 6 +
BaseTools/Source/Python/AutoGen/GenC.py | 0
BaseTools/Source/Python/AutoGen/GenMake.py | 233 +++---
.../Source/Python/AutoGen/ModuleAutoGen.py | 791 ++++++++++++++++--
BaseTools/Source/Python/Common/GlobalData.py | 11 +
BaseTools/Source/Python/Common/Misc.py | 44 +-
BaseTools/Source/Python/build/build.py | 182 ++--
9 files changed, 1073 insertions(+), 250 deletions(-)
mode change 100644 => 100755 BaseTools/Source/Python/AutoGen/AutoGenWorker.py
create mode 100755 BaseTools/Source/Python/AutoGen/CacheIR.py
mode change 100644 => 100755 BaseTools/Source/Python/AutoGen/DataPipe.py
mode change 100644 => 100755 BaseTools/Source/Python/AutoGen/GenC.py
mode change 100644 => 100755 BaseTools/Source/Python/AutoGen/GenMake.py
mode change 100644 => 100755 BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
mode change 100644 => 100755 BaseTools/Source/Python/Common/GlobalData.py
mode change 100644 => 100755 BaseTools/Source/Python/Common/Misc.py
mode change 100644 => 100755 BaseTools/Source/Python/build/build.py


Join devel@edk2.groups.io to automatically receive all group messages.