[EXTERNAL] Re: [edk2-discuss] failure in building unit test locally


Bret Barkelew <bret.barkelew@...>
 

Thanks for your follow-up. This is a known limitation that we have worked to resolve but ran into some unknown/unexpected errors. We definitely want to support both traditional and simplified character sets (as well as any other character sets), but if I recall there were some ambiguous OS mechanisms at work.

We would appreciate any help or contributions to get these character sets supported, but I understand if you don’t have the time.

I thought we already had an issue open to track it, but I couldn’t find it so I created a new one here:
Support non-ASCII character sets in Stuart tools · Issue #267 · tianocore/edk2-pytool-extensions (github.com)<https://github.com/tianocore/edk2-pytool-extensions/issues/267>

We can mark it as a duplicate if I find the original report.

- Bret

From: wenyi,xie via groups.io<mailto:xiewenyi2@...>
Sent: Monday, July 26, 2021 12:53 AM
To: Sean Brogan<mailto:spbrogan@...>; discuss@edk2.groups.io<mailto:discuss@edk2.groups.io>
Subject: [EXTERNAL] Re: [edk2-discuss] failure in building unit test locally

Hi, everyone

I have found the root cause of failure, it's because the language in my windows is Chinese, and the software not support Unicode will use gbk as the coding format.
After I change the language, the problem is solved.

Thanks
Wenyi

On 2021/7/21 10:56, xiewenyi (A) wrote:
Hi,Sean

I have upload the log to github, link is like below.
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fleadsama%2FShowLog%2Fblob%2Fmain%2FREADME.md&;data=04%7C01%7Cbret.barkelew%40microsoft.com%7C990acd0e770447c7a28b08d9500a7dea%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637628828276590927%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=cqhTkl5QFXsXwnq%2F0DYHfWieHhOiPsxTRgBKBkJhhsE%3D&amp;reserved=0

Thanks
Wenyi

On 2021/7/20 23:26, Sean Brogan wrote:
In the workspace/build folder there should be some logs .
Can you look at those and/or send the CI Log?

Thanks
Sean



On 7/20/2021 2:08 AM, wenyi,xie via groups.io wrote:
Hello, everyone

I want to build unit test locally, my environment is win10, python 3.9.6, VS2019.
I excute the command according to steps in the ReadMe.
1. stuart_setup -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=VS2019
2. stuart_update -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=VS2019
3. stuart_ci_build -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=VS2019 -t NOOPT -p MdePkg

In step 3, the log is just stopped at "Running Build NOOPT", how do I know the cause of the build failure ?
SECTION - Init SDE
WARNING - Using Pip Tools based BaseTools
SECTION - Loading Plugins
SECTION - Start Invocable Tool
SECTION - Getting Environment
SECTION - Loading plugins
SECTION - Building MdePkg Package
PROGRESS - --Running MdePkg: Host Unit Test Compiler Plugin NOOPT --
WARNING - Allowing Override for key TARGET_ARCH
PROGRESS - Start time: 2021-07-20 16:55:27.953832
PROGRESS - Setting up the Environment
PROGRESS - Running Pre Build
PROGRESS - Running Build NOOPT




.


wenyi,xie
 

Hi Bret,

It seems no need to change the language of windows, I found that the root cause is that the language of Visual Studio I installed is Chinese. So if any warning or error occured during building,
the VS will output error message in Chinese. This message then is passed to BaseTools, but BaseTools can't decode it using gbk, so UnicodeEncodeError happened like below.

INFO - --- Logging error ---
INFO - Traceback (most recent call last):
INFO - File "C:\Program Files\Python37\lib\logging_init_.py", line 1028, in emit
INFO - stream.write(msg + self.terminator)
INFO - UnicodeEncodeError: 'gbk' codec can't encode character '\u017b' in position 26: illegal multibyte sequence
INFO - Call stack:
INFO - File "C:\Program Files\Python37\lib\threading.py", line 890, in _bootstrap
INFO - self._bootstrap_inner()
INFO - File "C:\Program Files\Python37\lib\threading.py", line 926, in _bootstrap_inner
INFO - self.run()
INFO - File "d:\repository\edkii\ci\lib\site-packages\edk2basetools\AutoGen\AutoGenWorker.py", line 91, in run
INFO - self.InfoLogger_agent.log(log_message.levelno,log_message.getMessage())
INFO - Message: ' x64 Microsoft (R) C/C++ \u017b 19.29.30038.1 '
INFO - Arguments: ()

After I change the language of VS, I can get the real error message as below. The file cmocka.h include some character can't represented.
I try to add # pragma warning (disable:4819) to the first line of cmocka.h, and then building is successful.

ERROR - Compiler #2220 from b'd:\\\\repository\\\\edkii\\\\edk2\\\\UnitTestFrameworkPkg\\\\Library\\\\CmockaLib\\\\cmocka\\\\include\\\\cmocka.h(1849): the following warning is treated as an error'
WARNING - Compiler #4819 from b'd:\\\\repository\\\\edkii\\\\edk2\\\\UnitTestFrameworkPkg\\\\Library\\\\CmockaLib\\\\cmocka\\\\include\\\\cmocka.h(1849): The file contains a character that cannot be represented in the current code page (936). Save the file in Unicode format to prevent data loss'

Thanks
Wenyi

On 2021/7/28 0:17, Bret Barkelew wrote:
Thanks for your follow-up. This is a known limitation that we have worked to resolve but ran into some unknown/unexpected errors. We definitely want to support both traditional and simplified character sets (as well as any other character sets), but if I recall there were some ambiguous OS mechanisms at work.

 

We would appreciate any help or contributions to get these character sets supported, but I understand if you don’t have the time.

 

I thought we already had an issue open to track it, but I couldn’t find it so I created a new one here:

Support non-ASCII character sets in Stuart tools · Issue #267 · tianocore/edk2-pytool-extensions (github.com) <https://github.com/tianocore/edk2-pytool-extensions/issues/267>

 

We can mark it as a duplicate if I find the original report.

 

- Bret

 

*From: *wenyi,xie via groups.io <mailto:xiewenyi2@...>
*Sent: *Monday, July 26, 2021 12:53 AM
*To: *Sean Brogan <mailto:spbrogan@...>; discuss@edk2.groups.io <mailto:discuss@edk2.groups.io>
*Subject: *[EXTERNAL] Re: [edk2-discuss] failure in building unit test locally

 

Hi, everyone

I have found the root cause of failure, it's because the language in my windows is Chinese, and the software not support Unicode will use gbk as the coding format.
After I change the language, the problem is solved.

Thanks
Wenyi

On 2021/7/21 10:56, xiewenyi (A) wrote:
Hi,Sean

I have upload the log to github, link is like below.
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fleadsama%2FShowLog%2Fblob%2Fmain%2FREADME.md&;data=04%7C01%7Cbret.barkelew%40microsoft.com%7C990acd0e770447c7a28b08d9500a7dea%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637628828276590927%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=cqhTkl5QFXsXwnq%2F0DYHfWieHhOiPsxTRgBKBkJhhsE%3D&amp;reserved=0

Thanks
Wenyi

On 2021/7/20 23:26, Sean Brogan wrote:
In the workspace/build folder there should be some logs .
Can you look at those and/or send the CI Log?

Thanks
Sean



On 7/20/2021 2:08 AM, wenyi,xie via groups.io wrote:
Hello, everyone

I want to build unit test locally, my environment is win10, python 3.9.6, VS2019.
I excute the command according to steps in the ReadMe.
1. stuart_setup -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=VS2019
2. stuart_update -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=VS2019
3. stuart_ci_build -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=VS2019 -t NOOPT -p MdePkg

In step 3, the log is just stopped at "Running Build NOOPT", how do I know the cause of the build failure ?
SECTION - Init SDE
WARNING - Using Pip Tools based BaseTools
SECTION - Loading Plugins
SECTION - Start Invocable Tool
SECTION - Getting Environment
SECTION - Loading plugins
SECTION - Building MdePkg Package
PROGRESS - --Running MdePkg: Host Unit Test Compiler Plugin NOOPT --
WARNING - Allowing Override for key TARGET_ARCH
PROGRESS - Start time: 2021-07-20 16:55:27.953832
PROGRESS - Setting up the Environment
PROGRESS - Running Pre Build
PROGRESS - Running Build NOOPT




.