[PATCH] BaseTools: Change non-ascii character of StructurePcd comment


Yuwei Chen
 

Thanks for reminder~ : )
Thus personally think if do not want the non-ascii character in StructurePcd dsc file, the remove or change operation should be covered by ConvertFceToStructurePcd.py

Thanks,
Christine

-----Original Message-----
From: gaoliming <gaoliming@...>
Sent: Thursday, April 29, 2021 9:18 AM
To: devel@edk2.groups.io; Chen, Christine <yuwei.chen@...>; Kinney,
Michael D <michael.d.kinney@...>
Cc: Feng, Bob C <bob.c.feng@...>
Subject: 回复: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of
StructurePcd comment

Christine:
FCE prints HII question information that is from HII driver UNI file. It may
have non-ascii character.

Thanks
Liming
-----邮件原件-----
发件人: devel@edk2.groups.io <devel@edk2.groups.io> 代表 Yuwei Chen
发送时间: 2021年4月29日 8:25
收件人: Kinney, Michael D <michael.d.kinney@...>;
devel@edk2.groups.io
抄送: Feng, Bob C <bob.c.feng@...>; Liming Gao
<gaoliming@...>
主题: Re: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of
StructurePcd comment

Hi Mike,

The StructurePcd dsc file generated by our tool will have the
non-ASCII character.
The input file of ConvertFceToStructurePcd.py is generated by FCE
tool, which has the circle R non-ASCII character. This patch change
this character to ACSII character when using
ConvertFceToStructurePcd.py to generate the StructurePcd dsc file.

Best Regards,
Christine

-----Original Message-----
From: Kinney, Michael D <michael.d.kinney@...>
Sent: Thursday, April 29, 2021 12:10 AM
To: devel@edk2.groups.io; Chen, Christine <yuwei.chen@...>;
Kinney,
Michael D <michael.d.kinney@...>
Cc: Feng, Bob C <bob.c.feng@...>; Liming Gao
<gaoliming@...>
Subject: RE: [edk2-devel] [PATCH] BaseTools: Change non-ascii
character of StructurePcd comment

What file type contains the non-ASCII character?

I would prefer to see the source file with non ASCII character be
updated instead of building this conversion into the tools.

Mike

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
Yuwei Chen
Sent: Wednesday, April 28, 2021 1:45 AM
To: devel@edk2.groups.io
Cc: Feng, Bob C <bob.c.feng@...>; Liming Gao
<gaoliming@...>
Subject: [edk2-devel] [PATCH] BaseTools: Change non-ascii
character of StructurePcd comment

Currently, the ConvertFceToStructurePcd.py tool generate
StructurePcd dsc file with comments including non-ascii character
circle R. This patch changes the non-ascii character circle R to
(R) when adding the comment.

Cc: Bob Feng <bob.c.feng@...>
Cc: Liming Gao <gaoliming@...>
Signed-off-by: Yuwei Chen <yuwei.chen@...>
---
BaseTools/Scripts/ConvertFceToStructurePcd.py | 4 ++++
1 file changed, 4 insertions(+)

diff --git a/BaseTools/Scripts/ConvertFceToStructurePcd.py
b/BaseTools/Scripts/ConvertFceToStructurePcd.py
index 2052db8c4b..d029ed6a28 100644
--- a/BaseTools/Scripts/ConvertFceToStructurePcd.py
+++ b/BaseTools/Scripts/ConvertFceToStructurePcd.py
@@ -285,6 +285,10 @@ class Config(object):
comment_list = value_re.findall(line) # the string \\... in "Q...."
line
comment_list[0] = comment_list[0].replace('//', '')
comment = comment_list[0].strip()
+ comment_b = bytes(comment, encoding = "utf8")
+ if b"\xae" in comment_b:
+ comment_b = comment_b.replace(b"\xc2\xae", b"(R)") #
Change
the circle "R" character to ascii character
+ comment = str(comment_b, encoding = "utf-8")
line=value_re.sub('',line) #delete \\... in "Q...." line
list1=line.split(' ')
value=self.value_parser(list1)
--
2.26.1.windows.1








Michael D Kinney
 

So this tool converts UNI file contents into DSC file contents?

The fix only addresses a single character issue for (R), which means we will keep getting errors.

Perhaps we should convert all non-ASCII characters to '?'. If the developer using the tool
does not like the '?' characters, then they can update the strings in the UNI file to
only use ASCII characters.

BTW...I did run into this type of issue a lot when converting documents to GitBook markdown.
I recall adding many special character conversion rules to get to ASCII markdown files.
This is why I know this approach has many problems.

Mike

-----Original Message-----
From: Chen, Christine <yuwei.chen@...>
Sent: Wednesday, April 28, 2021 6:24 PM
To: gaoliming <gaoliming@...>; devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@...>
Cc: Feng, Bob C <bob.c.feng@...>
Subject: RE: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of StructurePcd comment

Thanks for reminder~ : )
Thus personally think if do not want the non-ascii character in StructurePcd dsc file, the remove or change
operation should be covered by ConvertFceToStructurePcd.py

Thanks,
Christine
-----Original Message-----
From: gaoliming <gaoliming@...>
Sent: Thursday, April 29, 2021 9:18 AM
To: devel@edk2.groups.io; Chen, Christine <yuwei.chen@...>; Kinney,
Michael D <michael.d.kinney@...>
Cc: Feng, Bob C <bob.c.feng@...>
Subject: 回复: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of
StructurePcd comment

Christine:
FCE prints HII question information that is from HII driver UNI file. It may
have non-ascii character.

Thanks
Liming
-----邮件原件-----
发件人: devel@edk2.groups.io <devel@edk2.groups.io> 代表 Yuwei Chen
发送时间: 2021年4月29日 8:25
收件人: Kinney, Michael D <michael.d.kinney@...>;
devel@edk2.groups.io
抄送: Feng, Bob C <bob.c.feng@...>; Liming Gao
<gaoliming@...>
主题: Re: [edk2-devel] [PATCH] BaseTools: Change non-ascii character of
StructurePcd comment

Hi Mike,

The StructurePcd dsc file generated by our tool will have the
non-ASCII character.
The input file of ConvertFceToStructurePcd.py is generated by FCE
tool, which has the circle R non-ASCII character. This patch change
this character to ACSII character when using
ConvertFceToStructurePcd.py to generate the StructurePcd dsc file.

Best Regards,
Christine

-----Original Message-----
From: Kinney, Michael D <michael.d.kinney@...>
Sent: Thursday, April 29, 2021 12:10 AM
To: devel@edk2.groups.io; Chen, Christine <yuwei.chen@...>;
Kinney,
Michael D <michael.d.kinney@...>
Cc: Feng, Bob C <bob.c.feng@...>; Liming Gao
<gaoliming@...>
Subject: RE: [edk2-devel] [PATCH] BaseTools: Change non-ascii
character of StructurePcd comment

What file type contains the non-ASCII character?

I would prefer to see the source file with non ASCII character be
updated instead of building this conversion into the tools.

Mike

-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
Yuwei Chen
Sent: Wednesday, April 28, 2021 1:45 AM
To: devel@edk2.groups.io
Cc: Feng, Bob C <bob.c.feng@...>; Liming Gao
<gaoliming@...>
Subject: [edk2-devel] [PATCH] BaseTools: Change non-ascii
character of StructurePcd comment

Currently, the ConvertFceToStructurePcd.py tool generate
StructurePcd dsc file with comments including non-ascii character
circle R. This patch changes the non-ascii character circle R to
(R) when adding the comment.

Cc: Bob Feng <bob.c.feng@...>
Cc: Liming Gao <gaoliming@...>
Signed-off-by: Yuwei Chen <yuwei.chen@...>
---
BaseTools/Scripts/ConvertFceToStructurePcd.py | 4 ++++
1 file changed, 4 insertions(+)

diff --git a/BaseTools/Scripts/ConvertFceToStructurePcd.py
b/BaseTools/Scripts/ConvertFceToStructurePcd.py
index 2052db8c4b..d029ed6a28 100644
--- a/BaseTools/Scripts/ConvertFceToStructurePcd.py
+++ b/BaseTools/Scripts/ConvertFceToStructurePcd.py
@@ -285,6 +285,10 @@ class Config(object):
comment_list = value_re.findall(line) # the string \\... in "Q...."
line
comment_list[0] = comment_list[0].replace('//', '')
comment = comment_list[0].strip()
+ comment_b = bytes(comment, encoding = "utf8")
+ if b"\xae" in comment_b:
+ comment_b = comment_b.replace(b"\xc2\xae", b"(R)") #
Change
the circle "R" character to ascii character
+ comment = str(comment_b, encoding = "utf-8")
line=value_re.sub('',line) #delete \\... in "Q...." line
list1=line.split(' ')
value=self.value_parser(list1)
--
2.26.1.windows.1