Background

Recently I have been working on a project that was setup using a brand new Github Repository. This repo would be used to store desired state templates for the project’s infrastructure. The way I wanted to structure all of the templates was to re-use modules that our team maintains in a separate private repository. I also wanted to ensure that everything should be desired state managed, even the Github repository itself. This way we can approach all of the lower layers of the infrastructure, albeit the cloud resources, repo settings or the infrastructure delivery pipelines, in the same exact way, by reading code. Also, if something where to happen, it is all recorded in the git history.

We are using Terraform and for our modules we source all of our modules using SSH.

# example.tf

module "cluster" {
  source = "git@github.com:benkoben/benkoben-terraform-modules.git//modules/kubernetes-cluster?ref=dawd35f"
  
  # Omitted 
}

Insert caveat here 🤖! Because the module repo is private, clients need to present proof that they are authorized to download the content of said repo. For user accounts this boils down to permissions set in Github. But for autonomous identities such as action runners we can use SSH keys; Deployment keys to the rescue!

You can launch projects from a repository on GitHub.com to your server by using a deploy key, which is an SSH key that grants access to a single repository. GitHub attaches the public part of the key directly to your repository instead of a personal account, and the private part of the key remains on your server. For more information, see “Delivering deployments.” - Github docs

Sounds awesome, but there is one catch I had to take into consideration. Our infrastructure delivery pipelines use runners that come and go. So the “server” (or rather servers) are ephemeral, meaning that the key is not persisted between workflow executions (if not included in the base image of runner machine). A common pattern is that consumers use runners provided by a central party (I.e. Github themselves or an org team responsible for managing Github enterprise resources). So control over what is included in the base image is out of the question. However, Including a SSH key installation step in our workflows of solves this particular issue, so that is what we opted doing.

While this particular case is all about installing SSH keys on action runners, what I found out is not necessarily specific to this particular use case. Lets dive into how to configure sensitive secrets on Github repositories using desired state templates using Terraform and libsodium encryption

Problem statement/breakdown (TLDR;)

Infrastructure delivery pipelines / Github actions have to setup a trust with another private (in my case modules) repository
This can be done using Github Deployment keys. Where the public key is configured on the other repository and private key installed on the servers that want to checkout that repository.
Our server are ephemeral. We are using self-hosted runners centrally managed by an external team.
We cannot ask the external team to pre-install all of our SSH keys for us because that wont scale well for them if all departments start demanding the same thing. So we need to ensure the SSH key is configured as a repository secret which our workflows then use to install the key every execution.
How do we configure Github repository secret(s) using desired state templates that are checked into version control without compromising the secret’s content?

Solution

The main ingredient for this recipe is the Github Terraform provider. Using this provider we can easily define our repository settings. Below is an oversimplified example:

# example.tf

# Example template that creates a github repository secret containing a private SSH key.
provider "github" {
  owner = "benkoben"
}

variable "secrets" {
	type = list(object({
		name = string
		value = string
	}))
	default = []
	description = "List of secrets that should be created in the Github repository"
}

data "github_repository" "mycoolrepo" {
  full_name = "egg-bacon-spam-terraform"
}

resource "github_actions_secret" "repo_scope" {
  for_each= {for secret in var.secrets : secret.name => secret}
  
  repository      = data.github_repository.mycoolrepo.name
  secret_name     = each.value.name
  encrypted_value = each.value.value
}

Then I will use a .tfvars file containing the actual value for our key.

# example.dev.tfvars

# SSH key
secrets = [
	{
		name = "SSH_KEY_GITHUB_ACTIONS"
		value = "KEY HERE"
	}
]

But wait? If this file is checked into version control, our private key is compromised and will be a part of version control history. Not to mention it will also be recorded as plaintext in our state file! Well, if you noticed the encrypted_value attribute in our github_actions_secret you might already understand that we will not put in a plaintext value. Instead we can encrypt all secrets we want to add. This is what the provider docs have to say about this:

Libsodium is used by GitHub to decrypt secret values.

Cool! What is libsodium and how do we encrypt our plaintext value with it in such a way that it is safe for us to upload to version control?

Encryption

Github exposes a public key for our repository via its REST API. This public key can be used to encrypt a plaintext value that we want to configure as a repository secret using libsodium. Github’s API docs outline how this key can be retrieved and also provides sample scripts. Based on the provided sample scripts I will be using Python for this example for transparency reasons and for the sake of explaining what is going on. We will be using PyNacl, a libsodium library for Python.

NOTE: Github CLI has a built-in feature for this as well. gh secret -R benkoben/egg-bacon-spam-terraform set --no-store -b - <<< (cat ~/.ssh/id_deploymeny_key | base64 -w0

#!/bin/env python3

import subprocess
import json
import sys

from base64 import b64encode
from nacl import encoding, public

# This script is a wrapper around Github CLI to get OIDC authentication. Therefore this script assumes
# that you have Github CLI intalled and have run "gh auth login"

def get_github_public_key(org, repo):
    try:
        # Construct the command
        command = [
            "gh", "api",
            "-H", "Accept: application/vnd.github+json",
            "-H", "X-GitHub-Api-Version: 2022-11-28",
            f"/repos/{org}/{repo}/actions/secrets/public-key"
        ]
        
        # Run the command and capture output
        result = subprocess.run(
            command,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            text=True,
            check=True
        )
        
        # Parse the output with jq-like functionality
        output_json = json.loads(result.stdout)
        public_key = output_json.get("key")
        
        return public_key
    except subprocess.CalledProcessError as e:
        print("Error executing command:", e.stderr)
        return None
    except json.JSONDecodeError:
        print("Error decoding JSON output.")
        return None

def encrypt(key, value):
    """Encrypt a Unicode string using the public key."""
    key = public.PublicKey(key.encode("utf-8"), encoding.Base64Encoder())
    sealed_box = public.SealedBox(key)
    encrypted = sealed_box.encrypt(value.encode("utf-8"))
    return b64encode(encrypted).decode("utf-8")

if __name__ == "__main__":
    ORG = "benkoben"
    REPO = "egg-bacon-spam-terraform"

    value_to_encrypt = sys.stdin.read()
    public_key = get_github_public_key(ORG, REPO)
    
    if not public_key:
        print("Failed to retrieve the public key.")
        sys.exit(1)

    secret = encrypt(public_key, value_to_encrypt)
    print(secret)

Our script starts with reading the plaintext value from stdin. Then retrieves the public key for benkoben/egg-bacon-spam-terraform before proceeding with creating a ciphertext using the encrypt function. The encrypt function initializes a nacl.public.SealedBox object which is used to encrypt value using the public key provided to the key argument. The encrypted value is then returned base64 encoded to preserve cipher text’s original format.

The code itself it pretty straight forward but apparently the SealedBox encryption has a pretty neat security feature that I was not expecting when I started working on this. When I ran my script with the same input and key I expected a deterministic output, but that is not the case.

(encrypt_secrets) 0 kooijman@~/Documents/encrypt_secrets$: echo "hello world" | ./encrypt_secret.py
/NflRn9s7pktRwM77dWX28DSRbzEqsp521/JIRdknEmBV0SmBtk1vXFELtGQaWXlZasNKVCDHmyP4WvX

(encrypt_secrets) 0 kooijman@~/Documents/encrypt_secrets$: echo "hello world" | ./encrypt_secret.py
Qu6fuoVuzpMlWY3ufmHHh7K3Zay7O77hULZ/qOytbztbSAOiHf0PWWPQ++Ued6htsyaYeBBQforY1Sfm

(encrypt_secrets) 0 kooijman@~/Documents/encrypt_secrets$: echo "hello world" | ./encrypt_secret.py                                                                                                                          
^[[AJWI82upYwMhYM5P/BUhfHcgOrDoQRL5kzYnKM4PksEauW0JXvGgzMsCgB2RlUV07hck94h9W12/lbnVZ

(encrypt_secrets) 0 kooijman@~/Documents/encrypt_secrets$: echo "hello world" | ./encrypt_secret.py                                                                                                                          
tKjrm7n9slpsZBBtWdne8Ql0yKhbReUfHXCpwiGQ6kZENGLRDmYMUy2qt3qeQqIiOG0OoIkapHd0/DbI

(encrypt_secrets) 0 kooijman@~/Documents/encrypt_secrets$: echo "hello world" | ./encrypt_secret.py                                                                                                                          
Q/sW5QYrITL8T+NFhge5qbu9I7Na7FAwfNiBgLlH+krhpc8qrZ0TT/kB0rpP5eHpPAS85uEgV9uc+Mtp

(encrypt_secrets) 0 kooijman@~/Documents/encrypt_secrets$: echo "hello world" | ./encrypt_secret.py                                                                                                                          
D7XYdVzOqjZiUNQiJxf1dQ6FtFMui9pB5vyPHxMpIjDrLNzmBterB3Xkg8j07dtTvrGA436Y20dBC6Pk

From the SealedBox reference docs:

The SealedBox class encrypts messages addressed to a specified key-pair by using ephemeral sender’s key pairs, which will be discarded just after encrypting a single plaintext message.

So SealedBox generates an ephemeral key-pair which is included in the ciphertext, this is what is causing non-deterministic output. The private key is discarded by the client once the encryption process is done, this is what makes these keys ephemeral. However, because the public key is included and sent to the receiver, it can be decrypted fully using both the ephemeral public key and the private key that the receiver already has. A more detailed description can be found here.

sealex-box-diagram

The only way of decrypting the ciphertext requires access to both the public (embedded ephemeral key) and the Receivers’s private key. I’d say the chances of someone hacking into Github to receive your repositories key-pair is pretty low. So in practice only Github knows how to decrypt the ciphertext! Anways, it seems we sidetracked a little bit. But its interesting knowing what going on under the hood!

Lets continue with bootstrapping our Github Environment.

Applying

I use the following command to encrypt my SSH key (again, the private key that pairs with the public key configured as a Deployment key on my module repo).

# preseve the plaintext value original format (newlines etc.) using base64 encoding
cat ~/.ssh/id_deployment_key | base64 -w 0 | ./encrypt_secret.py
RnQzlgomP6Zcgf0yLH8IjWqZVFk10dpQYcrMhgAxDDH4UgjPkyDVyPXjn/WE4ccx0iNeluvoJE9Mil0Huy5yV067NUC975FDFO0ZbhkOD2E3lYveP+qLd9LEeNUowWx7me7Wa2gKyQ6Y5Bm2K4GMcJirFVdwK/cBSlriAHQByKJ2D/Xy1Z2vZbPqtIitQOdmzrA8DwnnZDqOcjjTfJ9sYGlFm93pao6vC22hdEIfaoWnLBCZQPXJjFcXsnRb1YK5OPHwLBlHQu0oQeWbGXbQBvI714fSAFmxLdgNREzjaOrQQffNhfhdImleLGWgaCfEMn7wxTisFhnPF/cherIwJtygIqalrBE7n0LjSE81J7aCJX+WtBjnGyzeQc6v1xiVVmttestE4msupAfphIWFHW6jqkjEhrLg4tuWf4MatHrDjg0E0sIs65HUS07QbjvDbZbOFegJo1xv0uqFO65ZJKff4tB4P9KSDKMDxJfIXZwxCKXK5IFbSKz4Th8L0i9jf3OKqVs0uX9FuzOCURLgDsmdeLmV3sOLlJxVFcJiG3JSLASYPngg2m6Pc+zR2+t3w4LADnjl1Cqrrrl211oc7HsdPtPabkt0GRvszbZhrYKXk0n4SqVX3Fi3J4cH2qRaFrCcQElglhEzY1MZLWWL64PtMoDcHOqUKzxYglD+kmSw8C90MpkdH2ezmQExhiBtkEuKcIwkdMwjrUTCS5llpIRSSH3Px+KHEqanuK4rV0Q85GpEozPokBZfqJgH2pcrXiyBZXxmPzs+oeRXPivAFgcQyBKJZ2zXkDmNoptmhsk=

The contents of my .tfvars file is now

# param.dev.tfvars

# SSH key
secrets = [
	{
		name = "SSH_KEY_GITHUB_ACTIONS"
		value = "RnQzlgomP6Zcgf0yLH8IjWqZVFk10dpQYcrMhgAxDDH4UgjPkyDVyPXjn/WE4ccx0iNeluvoJE9Mil0Huy5yV067NUC975FDFO0ZbhkOD2E3lYveP+qLd9LEeNUowWx7me7Wa2gKyQ6Y5Bm2K4GMcJirFVdwK/cBSlriAHQByKJ2D/Xy1Z2QZbPqtIitQOdmzrA8DwnnZDqOcjjTfJ9sYGlFm93pao6vC22hdEIfaoWnLBCZQPXJjBcXsnRb1YK5OPHwLBlHQu0oQeWbGXbQBvI714fSAFmxLdgNREzjaOrQQffNhfhdImleLGWgaCfEMn7wxTisFhnPF/cherIwJtygIqalrBE7n0LjSE81J7aCJX+WtBPnGyzeQc6v1xiVVmttestE4msupAfphIWFHW6jqkjEhrLg4tuWf4MatHrDjg0E0sIs65HUS07QbjvDbZbOFegJo1xv0uqFO65ZJKff4tB4P9KSDKMDxJfIXZwxCKXK5IFbSKz4Th8L0i9jf3OKqVs0uX9FuzOCURLgDsmdeLmV3sOLlJxVFcJiG3JSLASYPngg2m6Pc+zR2+t3w4LADnjl1Cqrrrl211oc7HsdPtPabkt0GRvszbZhrYKXk0n4SqVX3Fi3J4cH2qRaFrCcQElglhEzY1MZLWWL64PtMoDcHOqUKzxYglD+kmSw8C90MpkdH2ezmQExhiBtkEuKcIwkdMwjrUTCS5llpIRSSH3Px+KHEqanuK4rV0Q85GpEozPokBZfqJgH2pcrXiyBZXxmPzs+oeRxPivAFgcQyBKJZ2zXkDmNoptmhsk="
	}
]

After a successful apply we should see a new secret for each secret defined in our template!

github-secret-upload-success

Github actions

We do not need to add more fancy steps in order to decrypt our secrets, Github takes care of that. In my case I just specified the SSH_KEY_GITHUB_ACTIONS using the official workflow syntax. For installing the key onto a runner I used the following steps:

# ...
# Triggers omitted
# ...

env:
  SSH_KEY: ${{ secrets.SSH_KEY_GITHUB_ACTIONS }}

jobs:
  terraform-plan:
    environment: lab
    runs-on: self-hosted-runners
    steps:
      - name: 'Checkout code'
        uses: actions/checkout@v4
        
	  - name: Validate SSH key
        run: |
          echo "Adding private SSH key to keychain."
          echo "Make sure the following public key is added to module repository Settings>Deployment Keys"
          echo "${SSH_KEY}" | base64 -d | ssh-keygen -y -f /dev/stdin          

      - name: Add SSH key to runner keychain
        working-directory: ${{ fromJson(steps.absPath.outputs.path).terraformWorkDir }}
        run: |
          ls -la
          eval `ssh-agent -s`
          ssh-add - <<< $( echo "${{ secrets.SSH_KEY_GITHUB_ACTIONS }}" | base64 -d)           
    # ...
    # ...
    # Terraform workflow omitted
    # ...

Conclusion

It’s possible to encrypt plaintext values using a built-in feature of the Github API. For each repository a key-pair is generated which consumers can use to to encrypt and commit secrets to version control using libsodium SealedBoxes and the public key. I wrote a quick wrapper around the Github CLI for retrieving my repo public keys, there are multiple other methods available because we are essentially just communication with a REST API.

While my particular use case might be a somewhat of a corner case, I do think it does demonstrates the encryption feature pretty well. I hope you found this as interesting as I did. Thanks for reading anyways!

Happy coding! :)

Background#

Problem statement/breakdown (TLDR;)#

Solution#

Encryption#

Applying#

Github actions#

Conclusion#