Windows: sign the DLLs (inside the .tar.gz and .zip archives)

[DilumAluthge]

Both archives now have bin/julia.exe signed, so we are definitely back to where we were a while ago!

But none of the dlls are signed, and I'm wondering whether some of the problems in JuliaLang/julia#54366 might go away if we just sign everything, including all the dlls in bin as well as the sysimage, i.e. lib/julia/sys.dll. The theory there being that Windows Defender is slowing things down somehow, and maybe it would take a less involved path for files that have a valid signature on it.

Also the files in libexec/julia are also not signed, but that is probably ok?

Oh, and I just realized that there are also lots of shared libraries in share/julia/compiled/v1.12/... If this theory about Windows Defender is correct, then we should probably sign all of those as well...

Originally posted by @davidanthoff in #352 (comment)

[staticfloat]

I believe what needs to happen here is this section needs to have some rules added to tell the Inno setup to sign the.dll files we want signed (example). We could technically sign every .dll in the directory, but I think a more measured whitelisted approach is probably the right thing to do for now.

[davidanthoff]

I think we'll need to rework the whole Windows code signing a bit more in any case: the current certificate is expiring in August, and the new guidelines that are in place now require private keys to be stored in hardware devices. So, the whole design here of storing the certificate in the repo encrypted won't work anymore (also: from a security point of view that seems like a, ugh, not ideal way of doing things...).

I looked around a bit, and it turns out MS has a new code signing service that looks really good to me: https://azure.microsoft.com/en-us/products/trusted-signing (https://techcommunity.microsoft.com/t5/security-compliance-and-identity/trusted-signing-is-in-public-preview/ba-p/4103457 is also interesting). One of the main benefits is that we would never even have to deal with certificates anymore, they handle all of that (and in fact issue a new certificate every day, that is then only valid for 72 hours). All in all it sounds way easy. Maybe the only question is whether the 5000/month are enough for us? Not clear to me whether every file counts, in which case we could presumably hit that limit very quickly, at least if we were to sign each nightly build... But maybe the right thing to do there is to only sign properly released builds in any case.

Probably also worth looking what other options there are...

But if we have to change things in any case, I would also suggest that we move the signing step out of the inno setup thing, and just add a "signing" stage between the build and the packaging stage? That seems way easier?

Whatever we do, we need to coordinate with Juliaup also: I am also using the current code signing certificate to sign the Juliaup things. What I did originally was to upload the certificate to Azure Key Vault, and am using that. That also has the benefit that the private key never ends up on any of the build machines, but overall it is way less automated than this new trusted signing thing that MS has.

It might be a good idea to start sorting out all of this now, as these things typically tend to take a while, and we probably don't want to be stuck in August without the ability to sign and release builds anymore :)

CC @ViralBShah

[ViralBShah]

cc @aviks @StefanKarpinski

[DilumAluthge]

I think the easiest short-term solution here is to switch to using Azure Key Vault and AzureSignTool.exe (or something similar). IIUC, we can pass the invocation for AzureSignTool.exe into Inno Setup, and thus minimize the amount of changes we need to make.

The new "Trusted Signing" beta looks interesting, but it sounds like it'll be more work to implement? I'm also concerned about the files-per-month limit. I think we definitely want to continue codesigning nightlies; that way, we can catch any codesigning problems as early as possible, instead of not detecting problems until release time. So I'm thinking we should probably not pursue the "Trusted Signing" thing just yet, particularly since the Azure Key Vault + AzureSignTool approach shouldn't be too hard to implement.

[DilumAluthge]

Note: we won't store the (encrypted) codesigning certificate as a cryptic secret in this repo.

But we will need to store an (encrypted) Azure Service Principal Secret, so that we can authenticate to Azure.

[davidanthoff]

I read up a bit more on this, and the new "Trusted Signing" service integrates with the standard signtool (https://learn.microsoft.com/en-us/azure/trusted-signing/how-to-signing-integrations), so my best guess is that would actually be the least effort to implement.

It is not clear to me how one would get one of these novel "only in hardware" certificates into azure key vault. Probably possible somehow, but that sounds definitely more involved than the "trusted signing" service, just because now one has to deal with MS and a company that issues certificates...

@DilumAluthge I think you are right, though, that this is a distinct question of disentangling this from innosetup. I think we should for now just prioritize to get a new certificate up and running, because we do have a deadline of August for that and even if we stick with the inno stuff we will have to change how things work for that because of the new requirements.

[DilumAluthge]

I read up a bit more on this, and the new "Trusted Signing" service integrates with the standard signtool (https://learn.microsoft.com/en-us/azure/trusted-signing/how-to-signing-integrations), so my best guess is that would actually be the least effort to implement.

Ah interesting.

[DilumAluthge]

It is not clear to me how one would get one of these novel "only in hardware" certificates into azure key vault.

I believe that whichever company you buy the certificate from will automate that process, e.g. https://www.sectigo.com/resource-library/sectigo-integrates-microsoft-azure-key-vault-with-certificate-manager-platform

[davidanthoff]

The sectigo site also has:

Sectigo will also be releasing integration with Azure Key Vault HSM

Which is what would need to work, I think. Now that press release is from 2020, but I didn't find any more information on their website, which suggests that this might not work?

[DilumAluthge]

Ah, hmmm. I can't seem to find it for Sectigo, but this vendor seems to support Azure Key Vault HSM: https://signmycode.com/azure-key-vault-code-signing (Control-F for Cloud HSM (Azure) in that page).

[DilumAluthge]

I think the two big concerns with Trusted Signing are:

1. It's a beta, so there might still be bugs.
   • But after some time, I'd expect the product to get more stable, so maybe this isn't an issue in the long term
2. The 5,000 files per month limit.
   • If we can get Azure to give the JuliaLang project free credits, then maybe this isn't an issue. I do want us to continue codesigning all nightly builds.

[ViralBShah]

So if it is a beta, we should at least get a new certificate for the time being.

On credits, @Keno Were you in touch with Azure folks ever?

[Keno]

I have no contacts at Azure

[Keno]

Actually, scratch that. We do have a contact at Azure and we do have credits, but they said we aren't using them. Someone needs to investigate that at some point.

[davidanthoff]

So there is a 5,000 signature per month included in the price of $10/month for this Azure Trusted Signing, and then every additional signature is $0.005. I think if we were to sign nightly builds and proper releases and pre-release builds it would most likely still come out cheaper (or roughly similar) to the cost of buying a new certificate. Plus, the whole management of this seems just way, way easier than buying a certificate, importing it into key vault and then going through exactly the same thing again in three years.

My understanding is that this Azure Trusted Signing is in public preview until June (and free until then), and then in June they will start charging for it and presumably that will also mean it won't be in preview anymore, at least that is how they seem to do that.

I would propose the following: someone at JuliaHub signs us up for this "trusted signing" thing, and then I can try to use it for Juliaup right now and see how it goes. If it is all smooth sailing, and the program exits out of preview by June, then the timing of this all seems to be just fine, i.e. then we should be able to migrate the CI story over before mid August (when the old cert expires) as well, right?

[giordano]

If the count is one certificate per archive (rather than all binary files in the archive), then we're now at build number 36k-something, build 31k was in December, so we're at about 1k builds per months, but probably need two certificates per build, for the two platforms, so we'd need about 2k certificates per month.

[davidanthoff]

I think the count is per file signed, and if we want to file all exe and dll files (which I think we should) then we are at about 170 files per release. But my math actually left out that we have different processor architectures, i.e. x64 and x86, so my cost math was pretty off, by at least a factor of 2 :)

[DilumAluthge]

Hmmm, so 2k tarballs per month * 170 signatures per tarball = 340k signatures per month? And if we get 5k free signatures per month, that means we need to pay for 335k signatures per month * $0.005 per signature = $1675 per month = $20,100 per year. Is that math right? Because that seems far too expensive, unless Azure gives us enough credits every year to cover that.

[DilumAluthge]

@giordano That 2k tarballs per month includes pull request CI, right? We currently don't codesign the pull request tarballs, so we can subtract PR builds from that 2k per month number.

[davidanthoff]

And we probably don't want to sign PR builds, as signing essentially signals that JuliaHub is standing behind that code, which I assume is not the case for every PR.

The right math is probably something like 2 builds per nightly (x64 and x86) at 170 files, 31 nightlies per month? So roughly 10k signatures per month? But that of course can also easily change, if more binaries are added... There is also a premium tier at $100/month that includes 100k signatures per month... Juliaup will also need some signatures, but probably a lot less.

I don't know what kind of budget there is, but to me even the premium tier seems not a bad deal, if that actually means we won't have to deal with certificate renewals in the future and all the person-time that sucks up...

I still think we should just give it a try and see whether it actually works or not.

[DilumAluthge]

31 nightlies per month

It's not actually one nightly per night. We build a "nightly" for each commit to master, and for each commit to all release-* branches. So the count of builds would be:

1. Number of commits to master in one month.
2. Number of commits to release-* branches in one month.
3. Number of tag builds in one month.

[DilumAluthge]

Oh wait, we also have the from-source builds. Those we actually do run once every 24 hours, so there will be 31 (* 2 architectures) of those per month.