[RFC] UnitTestFrameworkPkg cmocka submodule alternatives


Michael D Kinney
 

Hello,

We have had at least three incidents in the last year where the link to the
cmocka submodule in the UnitTestFrameworkPkg has not been available, and this
impacted the EDK II CI system. The following submodule link is the one that
is not reliable:

https://git.cryptomilk.org/projects/cmocka.git

We have identified two potential mirrors of this repo:

https://github.com/neverware-mirrors/cmocka.git
https://gitlab.com/cmocka/cmocka.git

The following patch provided a temporary fix for the EDK II CI agents, but
does not help other consumers of the edk2 repository.

https://github.com/tianocore/edk2/commit/be746104d1766a8c363e74d6063144657820d688

I have seen one suggestion that TianoCore create its own
mirror of cmocka. This does require monitoring and maintenance
by the TianoCore community. I would prefer to use a well
maintained mirror in github as long as we do not observe any
issues with the support of that mirror.

I propose we update the submodule in the UnitTestFrameworkPkg
to use the https://github.com/neverware-mirrors/cmocka.git mirror.
By using a mirror in github, we remove one external dependency.

Please provide feedback and comments on this proposal. If there
are no objections, then we will proceed with a patch review for
this update.

Thanks,

Mike


Laszlo Ersek
 

On 12/16/20 19:45, Michael D Kinney wrote:
Hello,

We have had at least three incidents in the last year where the link to the
cmocka submodule in the UnitTestFrameworkPkg has not been available, and this
impacted the EDK II CI system. The following submodule link is the one that
is not reliable:

https://git.cryptomilk.org/projects/cmocka.git

We have identified two potential mirrors of this repo:

https://github.com/neverware-mirrors/cmocka.git
https://gitlab.com/cmocka/cmocka.git

The following patch provided a temporary fix for the EDK II CI agents, but
does not help other consumers of the edk2 repository.

https://github.com/tianocore/edk2/commit/be746104d1766a8c363e74d6063144657820d688

I have seen one suggestion that TianoCore create its own
mirror of cmocka. This does require monitoring and maintenance
by the TianoCore community. I would prefer to use a well
maintained mirror in github as long as we do not observe any
issues with the support of that mirror.

I propose we update the submodule in the UnitTestFrameworkPkg
to use the https://github.com/neverware-mirrors/cmocka.git mirror.
By using a mirror in github, we remove one external dependency.

Please provide feedback and comments on this proposal. If there
are no objections, then we will proceed with a patch review for
this update.
We could create our own fork under the <https://github.com/tianocore>
organization.

It does not require much extra maintenance or monitoring, in my opinion.
We only need to advance our fork to the actual master HEAD when we
intend to advance our submodule reference in edk2 as well. As long as
the submodule reference in edk2 does not move, the actual master HEAD of
the cmocka project may very well be ahead of our fork (mirror), without
causing issues.

I don't know who or what the <https://github.com/neverware-mirrors>
organization is, and I'd prefer not fetching code from them automatically.

Thanks
Laszlo


Laszlo Ersek
 

On 12/17/20 15:48, Laszlo Ersek wrote:

I don't know who or what the <https://github.com/neverware-mirrors>
organization is, and I'd prefer not fetching code from them automatically.
I'm sorry, this was silly.

The whole point of git is "addressing by content". Our submodule
reference in edk2 makes us check out the cmocka tree at a known hash, so
where that comes from is totally irrelevant.

I'm OK with the proposal as posted.

Thanks & sorry again,
Laszlo


Rebecca Cran <rebecca@...>
 

On 12/17/20 8:54 AM, Laszlo Ersek wrote:
On 12/17/20 15:48, Laszlo Ersek wrote:

I don't know who or what the <https://github.com/neverware-mirrors>
organization is, and I'd prefer not fetching code from them automatically.
I'm sorry, this was silly.
The whole point of git is "addressing by content". Our submodule
reference in edk2 makes us check out the cmocka tree at a known hash, so
where that comes from is totally irrelevant.
I'm OK with the proposal as posted.
Also, apparently Neverware is part of Google:

https://cloudreadykb.neverware.com/s/article/Neverware-is-now-part-of-Google-FAQ

--
Rebecca Cran


Michael D Kinney
 

Hello,

There have been a few suggestions to create a mirror of cmocka in TianoCore
org in GitHub.

I have found a GitHub action that can do a repo sync.

https://github.com/marketplace/actions/github-repo-sync

I have created a temporary mirror of cmocka in my personal GitHub area that
uses this GitHub action to sync all branches and all tags once a day.

https://github.com/mdkinney/mirror-cmocka

Here is the GitHub workflow file. It must be in the default branch for the
repo using a branch name that is not present in the repo being mirrored.
In this case, I used a branch name of 'repo-sync'.

https://github.com/mdkinney/mirror-cmocka/blob/repo-sync/.github/workflows/repo-sync.yml

Please provide feedback on this approach. If we like this approach, then
I suggest we create a new repo in TianoCore called edk2-cmocka that is a
mirror that is synced once a day and we update the cmocka submodule in the
edk2 repo to use edk2-cmocka.

Best regards,

Mike

-----Original Message-----
From: Kinney, Michael D <michael.d.kinney@intel.com>
Sent: Wednesday, December 16, 2020 10:46 AM
To: rfc@edk2.groups.io; devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>; 'Bret Barkelew'
<Bret.Barkelew@microsoft.com>
Subject: [RFC] UnitTestFrameworkPkg cmocka submodule alternatives

Hello,

We have had at least three incidents in the last year where the link to the
cmocka submodule in the UnitTestFrameworkPkg has not been available, and this
impacted the EDK II CI system. The following submodule link is the one that
is not reliable:

https://git.cryptomilk.org/projects/cmocka.git

We have identified two potential mirrors of this repo:

https://github.com/neverware-mirrors/cmocka.git
https://gitlab.com/cmocka/cmocka.git

The following patch provided a temporary fix for the EDK II CI agents, but
does not help other consumers of the edk2 repository.

https://github.com/tianocore/edk2/commit/be746104d1766a8c363e74d6063144657820d688

I have seen one suggestion that TianoCore create its own
mirror of cmocka. This does require monitoring and maintenance
by the TianoCore community. I would prefer to use a well
maintained mirror in github as long as we do not observe any
issues with the support of that mirror.

I propose we update the submodule in the UnitTestFrameworkPkg
to use the https://github.com/neverware-mirrors/cmocka.git mirror.
By using a mirror in github, we remove one external dependency.

Please provide feedback and comments on this proposal. If there
are no objections, then we will proceed with a patch review for
this update.

Thanks,

Mike


Bret Barkelew <bret.barkelew@...>
 

I like it.

- Bret

From: Kinney, Michael D<mailto:michael.d.kinney@intel.com>
Sent: Saturday, December 19, 2020 10:59 AM
To: rfc@edk2.groups.io<mailto:rfc@edk2.groups.io>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Bret Barkelew<mailto:Bret.Barkelew@microsoft.com>; Laszlo Ersek <lersek@redhat.com> (lersek@redhat.com)<mailto:lersek@redhat.com>; liming.gao<mailto:liming.gao@intel.com>
Subject: [EXTERNAL] RE: [RFC] UnitTestFrameworkPkg cmocka submodule alternatives

Hello,

There have been a few suggestions to create a mirror of cmocka in TianoCore
org in GitHub.

I have found a GitHub action that can do a repo sync.

https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmarketplace%2Factions%2Fgithub-repo-sync&;data=04%7C01%7CBret.Barkelew%40microsoft.com%7C92da18aaec1443463b2508d8a45023a2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637440011398666049%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=AX2EFVoGvtYoOZRtyFjwwTRbZkmQMgOCnjNNhWot7eo%3D&amp;reserved=0

I have created a temporary mirror of cmocka in my personal GitHub area that
uses this GitHub action to sync all branches and all tags once a day.

https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fmirror-cmocka&;data=04%7C01%7CBret.Barkelew%40microsoft.com%7C92da18aaec1443463b2508d8a45023a2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637440011398666049%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=U7ThFHC2fsgO9rVTNre3b0dI23b1Iudi1tw%2FjiFEdZc%3D&amp;reserved=0

Here is the GitHub workflow file. It must be in the default branch for the
repo using a branch name that is not present in the repo being mirrored.
In this case, I used a branch name of 'repo-sync'.

https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmdkinney%2Fmirror-cmocka%2Fblob%2Frepo-sync%2F.github%2Fworkflows%2Frepo-sync.yml&;data=04%7C01%7CBret.Barkelew%40microsoft.com%7C92da18aaec1443463b2508d8a45023a2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637440011398666049%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=CuxE3Ljy2M7APvXfnuOqb6YPnFCX%2FUkDxsiIGEUHcvY%3D&amp;reserved=0

Please provide feedback on this approach. If we like this approach, then
I suggest we create a new repo in TianoCore called edk2-cmocka that is a
mirror that is synced once a day and we update the cmocka submodule in the
edk2 repo to use edk2-cmocka.

Best regards,

Mike

-----Original Message-----
From: Kinney, Michael D <michael.d.kinney@intel.com>
Sent: Wednesday, December 16, 2020 10:46 AM
To: rfc@edk2.groups.io; devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>; 'Bret Barkelew'
<Bret.Barkelew@microsoft.com>
Subject: [RFC] UnitTestFrameworkPkg cmocka submodule alternatives

Hello,

We have had at least three incidents in the last year where the link to the
cmocka submodule in the UnitTestFrameworkPkg has not been available, and this
impacted the EDK II CI system. The following submodule link is the one that
is not reliable:

https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.cryptomilk.org%2Fprojects%2Fcmocka.git&;data=04%7C01%7CBret.Barkelew%40microsoft.com%7C92da18aaec1443463b2508d8a45023a2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637440011398666049%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=m4tHEei6OQUwJu6jcldgxycoBJoajqPb9o5aKTDra%2F0%3D&amp;reserved=0

We have identified two potential mirrors of this repo:

https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fneverware-mirrors%2Fcmocka.git&;data=04%7C01%7CBret.Barkelew%40microsoft.com%7C92da18aaec1443463b2508d8a45023a2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637440011398676045%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=19d9nV6kG4FQE3GC3ZzDTmE6%2F7EjNdMMWM%2BbSGT6bSI%3D&amp;reserved=0
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2Fcmocka%2Fcmocka.git&;data=04%7C01%7CBret.Barkelew%40microsoft.com%7C92da18aaec1443463b2508d8a45023a2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637440011398676045%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=xgErAgcMsMSKbrLZB9kgCHHl3r%2FzJM%2FJy6jhxpi0ObY%3D&amp;reserved=0

The following patch provided a temporary fix for the EDK II CI agents, but
does not help other consumers of the edk2 repository.

https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Fcommit%2Fbe746104d1766a8c363e74d6063144657820d688&;data=04%7C01%7CBret.Barkelew%40microsoft.com%7C92da18aaec1443463b2508d8a45023a2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637440011398676045%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=Nyx8mBIwNxMEu4SdHYGiQhBGcpAPxxPHXBgI%2BM0CIU0%3D&amp;reserved=0

I have seen one suggestion that TianoCore create its own
mirror of cmocka. This does require monitoring and maintenance
by the TianoCore community. I would prefer to use a well
maintained mirror in github as long as we do not observe any
issues with the support of that mirror.

I propose we update the submodule in the UnitTestFrameworkPkg
to use the https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fneverware-mirrors%2Fcmocka.git&;data=04%7C01%7CBret.Barkelew%40microsoft.com%7C92da18aaec1443463b2508d8a45023a2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637440011398676045%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=19d9nV6kG4FQE3GC3ZzDTmE6%2F7EjNdMMWM%2BbSGT6bSI%3D&amp;reserved=0 mirror.
By using a mirror in github, we remove one external dependency.

Please provide feedback and comments on this proposal. If there
are no objections, then we will proceed with a patch review for
this update.

Thanks,

Mike


Rebecca Cran
 

On 12/19/20 11:58 AM, Michael D Kinney wrote:
There have been a few suggestions to create a mirror of cmocka in TianoCore
org in GitHub.
I have found a GitHub action that can do a repo sync.
https://github.com/marketplace/actions/github-repo-sync
I have created a temporary mirror of cmocka in my personal GitHub area that
uses this GitHub action to sync all branches and all tags once a day.
https://github.com/mdkinney/mirror-cmocka
Here is the GitHub workflow file. It must be in the default branch for the
repo using a branch name that is not present in the repo being mirrored.
In this case, I used a branch name of 'repo-sync'.
https://github.com/mdkinney/mirror-cmocka/blob/repo-sync/.github/workflows/repo-sync.yml
Please provide feedback on this approach. If we like this approach, then
I suggest we create a new repo in TianoCore called edk2-cmocka that is a
mirror that is synced once a day and we update the cmocka submodule in the
edk2 repo to use edk2-cmocka.
I'd suggest just using the Gitlab mirror. Unlike cryptomilk.org, Gitlab should be just as reliable as Github and won't introduce another potential failure point.

--
Rebecca Cran