Digicert EV certs and CI/CD

Scott

Our arc CLI tooling is written in Go and distributed as a binary+install script for macOS and Linux. On Windows, we provide an installer (we use go-msi, powered by the wix toolset) to move our .exe to Program Files and add an entry to the PATH.

Microsoft specifies that Windows binaries/installers are to be signed using their signtool program with an Extended Validation Code Signing Certificate (EV cert).

This all seems pretty reasonable so far. We'll just buy an EV token, add a signing stage to the build server, and move on with our lives. Right?

Digicert EV Token

We sourced our EV token from Digicert. After some authorisation steps, and waiting a week, this cute little blue USB HSM arrived in the mail. This one is a SafeNet eToken 5110.

Blue Digicert Token attached to build server

The key activation process is pretty easy and documented by Digicert, so we'll skip to the interesting bits.

We tell Vagrant to configure a VirtualBox USB filter during setup which passes control over the token hardware to our Windows VM.

v.customize ["usbfilter", "add", "0",
    "--target", :id,
    "--name", "Digicert Token",
    '--vendorid', '0x0529',
    '--productid', '0x0620']

There are two software tools available for the token, both hosted by Digicert through links in their Knowledgebase:

  1. the SafeNet Authentication Client software - from the HSM OEM, used to view/manage the token.
  2. Digicert Certificate Utility for Windows - their own tool which lets you manually sign files.
  3. Windows will use it's internal SmartCard suport to unlock/reference the token when invoked with signtool.

I found that the SafeNet Authentication Client had to be installed for Windows to correctly handle the token as a SmartCard, while the Digicert tool functions as a 'portable' self-contained tool.

After that, I could manually invoke signtool from CMD and sign with the token - with a catch! When signtool runs, it unlocks/accesses the token forceing a 2FA token pin prompt. This prompt is only presented via GUI, with no official (known) manner to bypass this for CLI use.

The CI Struggle

Because 2FA behaviour for code signing is a 'feature' of the EV token it's somewhat difficult to bypass.

As mentioned on StackOverflow, there are a few possible workarounds:

  • Configure the key (through SafeNet's tool) to require unlock once per session, but this still requires a human to login to use graphical pin entry on the Windows VM every fresh boot.
  • There are tools such as SafeNetTokenSigner which use Windows calls to unlock the certificate before use, but it didn't work for me (github thread).
  • Create a wrapper program to automatically respond to token pin requests, but it didn't work on Windows 10 1903 or newer for me

In the end, the solution was to export the certificate and pass the credentials to signtool with a specific set of arguments (this has since been detailed in this SO post).

Our Windows install has windows-sdk installed which provides signtool.exe at a path similar to C:\Program Files (x86)\Windows Kits\10\bin\10.0.18362.0\x64\signtool.exe, I've added an environment variable to our VM which makes finding signtool easier. I found the Windows SDK had to be newer than June 2019 to use this method.

  • Export the token's public certificate with SafeNet Authentication Client

    • Advanced View
    • User Certificates - select the cert
    • Right click - Export Certificate, save it somewhere safe

  • Get the Container Name

    • In the certificate info toolbar, there's a "Copy" icon which copies the fields to clipboard.
    • We want the Container Name and Reader Name strings.
    • Treat the Container Name string as carefully as your token password (no plaintext git copy etc).

  • I keep the certificate file electricui.cer in a special place on the VM host, which is shared with the Windows VM through vagrant's NFS share functionality.

  • On our GoCD build "Environment Variables" tab, we add two secret strings, EV_PASS and EV_CONTAINER_NAME GoCD environment variable edit page

  • The GoCD build stage is this command: sign /tr http://timestamp.digicert.com /td sha256 /fd sha256 /f C:\ev-sign\electricui.cer /csp "eToken Base Cryptographic Provider" /kc [{{%EV_PASS%}}]=%EV_CONTAINER_NAME% /n "Electric UI Pty Ltd" ./arc.exe

    • %STRING% is how Windows Environment variables are referenced.

  • We now have a CI compatible signing pipeline which doesn't require human intervention.

A note on token passwords

Don't randomly generate a pin for your token which includes a caret ^ character - this is an escape character in Windows.

This cost me a few hours...

Wrapping up

Our GoCD build process looks like this:

GoCD pipeline chart - green ticks for each job
  1. Builds the binaries for Windows, macOS and Linux,
  2. Sign the Windows executable with signtool,
  3. Runs the packaging stage where go-msi makes our installer,
  4. We sign the installer msi with signtool,
  5. Upload our built binaries to Amazon S3