Manage kubernetes secrets in git with GPG and SOPS

10 min readJun 22, 2021

GitOps

We use Git for everything now, from code source to organization, history, and even for Kubernetes Cluster Management (aka GitOps).

But, there is still something not widely adopted. Managing our secrets in Git. Some tools like HashiCorp Vault, Google Secret Management, or AWS Secret Manager provide us a solution to manage our secrets in a dedicated system, but they are still not in sync with our source code.

We will see here, thanks to Mozilla SOPS how to integrate our secrets management directly in Git.

The main benefits of using git for everything, from dev to operation, is to be able to follow the evolution of something, find how and why something has changed (thanks to conventional-changelog) and ability to revert it when we want.

But secrets, like passwords, api keys or sensitive information, should not be stored in the repository mainly because git, by design, doesn't allow file restriction access. What if we can now store them in git, leveraging the benefit of the versioning system without exposing our data to everyone? This is the purpose SOPS, which can be used to encrypt the Kubernetes secrets.

SOPS Installation

Sops is very simple to install, like every golang application, you just have to download the binary for your specific Operating System (Linux, Mac, Windows) directly from the release page on GitHub.

By the way, you can install it thanks to brew on Mac & Linux (SOPS Formuale)

GPG Configuration

To be able to store secrets in your git repository, you will need to encrypt them before committing them. To do so, SOPS relies on PGP keys and on your local gpgbinary. You can also use keys stored in external system like Google KMS or Amazon KMS . In this article, we will focus only on local PGP keys.

We need to provide dedicated keys to our CI, so I advise you to create a new pair of keys following this pretty good documentation provided by Gitlab.

But, for the sake of simplicity, I will mention all the steps in this doc.

Generating a GPG key

Install GPG for your operating system. If your Operating System has gpg2 installed, replace gpg with gpg2 in the following commands.
Generate the private/public key pair with the following command, which will spawn a series of questions:

 $ gpg — full-gen-key

Note: In some cases like Gpg4win on Windows and other macOS versions, the command here may be gpg --gen-key.

3. The first question is which algorithm can be used. Select the kind you want or press Enter to choose the default (RSA and RSA):

$ Please select what kind of key you want: 
(1) RSA and RSA (default) 
(2) DSA and Elgamal 
(3) DSA (sign only) 
(4) RSA (sign only) 
Your selection? 1

4. The next question is key length. We recommend you choose 4096:

$ RSA keys may be between 1024 and 4096 bits long. 
What keysize do you want? (2048) 4096 
Requested keysize is 4096 bits

5. Specify the validity period of your key. This is something subjective, and you can use the default value, which is to never expire:

$ Please specify how long the key should be valid. 0 = key does not expire 
<n> = key expires in n days 
<n>w = key expires in n weeks 
<n>m = key expires in n months 
<n>y = key expires in n years Key is valid for? (0) 0 
Key does not expire at all

6. Confirm that the answers you gave were correct by typing y:

$ Is this correct? (y/N) y

7. Enter your real name, the email address to be associated with this key (should match a verified email address you use in GitLab) and an optional comment (press Enter to skip):

$ GnuPG needs to construct a user ID to identify your key. Real name: Mr. Robot 
Email address: <your_email> 
Comment: 
You selected this USER-ID: 
    "Mr. Robot <your_email>" Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O

8. Pick a strong password when asked and type it twice to confirm.

9. Use the following command to list the private GPG key you just created:

$ gpg --list-secret-keys --keyid-format LONG <your_email>

Replace <your_email> with the email address you entered above.

10. Copy the GPG key ID that starts with sec. In the following example, that’s 30F2B65B9246B6CA:

$ sec rsa4096/30F2B65B9246B6CA 2017-08-18 [SC] 
D5E4F29F3275DC0CDA8FFC8730F2B65B9246B6CA uid            [ultimate] Mr. Robot <your_email> 
ssb   rsa4096/B7ABC0813E4028C0 2017-08-18 [E]

11. Export the public key of that ID (replace your key ID from the previous step):

$ gpg --armor --export 30F2B65B9246B6CA

12. Finally, you can copy the public key and add it in your profile settings if it is your personal mail, else you can also use it in the GitLab CI, which we will be discussing in the following sections

At the end, you should have at least one key of type S and one of type E:

$ gpg --list-secret-keys --keyid-format LONG ci@gitlab.com
sec   rsa4096/OAEFIJEFAOIJOEFA 2019-08-23 [SC] # Type 'S'
      OIPAEFFKJAJOIEA564FEAF4EA6G54EA65G4A654A
uid                 [ultimate] ci-gitlab <ci@gitlab.com>
ssb   rsa4096/OJA21FE56A456AAC 2019-08-23 [E] # Type 'E'

In this command result, you can see keys attached to my (fake) email ci@gitlab.com

Encrypt a secret

We will work with a standard secret for this example coming from the kubernetes.io secret documentation :

$ cat secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  username: YWRtaW4=
  password: MWYyZDFlMmU2N2Rm

SOPS provides an handy parameter to choose which keys should be encrypted. This could be used directly from the command line :

$ sops --encrypt --encrypted-regex '^(data|stringData)$' secret.yaml

The file looks like this now:

apiVersion: ENC[AES256_GCM,data:gEA=,iv:2jeZ0y0SEPXZTgrsmYXc7LgPydoPwnKzF4GVqykSw5M=,tag:YzEMrctU19sUIqat+LUFIw==,type:str]
data:
    secret: ENC[AES256_GCM,data:uJIDP3yMdFyW/7bnBAU0MJp07xmF0nlh,iv:ardoVW+p9HLNpaWfXOWrevZFdNJqGJvt00jPbkrr7p4=,tag:mB3wqa7kNE7gZ9J87IboNg==,type:str]
kind: ENC[AES256_GCM,data:tu+OHbXI,iv:QfqZeYOkTPTzJ618gh+/zGWPMDBJfjp+GwM8yBZCD+Y=,tag:XrJfNhmyDhOQNncd/uNvxA==,type:str]
metadata:
    name: ENC[AES256_GCM,data:X8WMTByIcUrjag==,iv:pQ6xt5DPchrxwXtt98l4mgCixpcpaA2ewST0lAqYY4E=,tag:fqj/1PbzbhjIhRx9Ogitxg==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    lastmodified: '2020-06-13T12:21:16Z'
    mac: ENC[AES256_GCM,data:dU1bm2GBwdkqMQqEa+b0EfuMG5BRiiHDxNjB3tvB4JTVxqamQ17A47tzMld93/lZnJWRb4BtDFUI2xHh/5BCpxdI4J67AvWdv91usRgpgh7z7SF4pK4UUVah3WDYd3tvo+VzugvN6b4V+oGZlfJM789dJrTDOjSCAyyaaa/Qfb0=,iv:Cdaet+2Sowq50TDIkDO1RXi4vgaVqe7rYz08PHybAv0=,tag:1aeR/f4NnydiP2Vuuvljvw==,type:str]
    pgp:
    -   created_at: '2020-06-13T12:21:16Z'
        enc: |
            -----BEGIN PGP MESSAGE-----            hQGMA5/a4uV8wP1EAQwAshoGsWCKiYnsi1Rr/HoMtvp8Tx5cmUWLPz0OsGsQUFc/
            HQ48h7KUAjemTBnyQ6SpZUK2uI5YOLnOIqNcQJvNb4o30o0M3kuD2KNzSbsRGt3j
            vODVKcs699sT1mX/rc6/EzpLy0NUgrAlXn7IUG+rhnk97OvfMdjP6+i77OpSU9Uc
            5l6FWokBzFBvL6EENay6EO54C17MBXijoM4h6x7YajOjk5VNFtUl7wh039VGWZpQ
            7vMuTbla06DWAHIFKIOs54yEh+vMTMgnsg8NrPzz4xqKDMcmZoQIm1jwc8EvxYK4
            nGI4LpXO8VMgCyHBEcMjCGhZ5xXAiMRzF9M8XAWRfDFgYjDosL8QOQkl4vVnsf4e
            5bhITFz1ZGaJwEeVzlt82cxWy4+BNq+lfAghqEnHGAjLGFIgTiRtVB47NLk9WPIT
            T7Qv5gvATdW0rNHecfoGCKWx15ZJezXSo8Vtt5eVPgdYd3wO76Uw6XWhkxwSgQNT
            vE7W5qvLoqxZrJs4Kyd20l4BQzSEiHJRg7wmWx9Y0BhyzF1Zh6F0oQBxTzGjP0f1
            J9THfwIOhyOBwm8bWkEz7ac84o2NEItdrvlhrB0Y61ZfRl6xZKWqFxJ3N1yVHxxM
            MWRYzaQcuoh5670lRhY5
            =KjAd
            -----END PGP MESSAGE-----
        fp: 57E6DA39E907744429FB07871141FE9F63986243
    unencrypted_suffix: _unencrypted
    version: 3.5.0

The following result should be stored in your git repository. In this example, we will name it secrets.enc.yaml

We can do better, because SOPS provides us a solution to store configuration required for every secret directly into the .sops.yaml. We can add the key encrypted-regex to simplify the command line. This is helpful when you need to add another secret key in an existing secret.

Every team member can now read and edit Kubernetes secret simply

If you want to test a decrypt operation and apply the secret into your namespace, you can run the following command:

$ sops -d secrets.enc.yaml | kubectl apply -f -You shall get back the previous secret.yaml fileapiVersion: v1
data:
  username: YWRtaW4=
  password: MWYyZDFlMmU2N2Rm
kind: Secret
metadata:
  name: mysecret
type: Opaque

Multi user access to secrets

What we will try to achieve is to store secrets in Git but with restrictions on “who can access what”.

For example, if we need multiple users to access a secret then each of them need to already have his/her GPG key pairs, as described above.

User1 will generate a file containing a secret:

sops -pgp 5844C613B763F4374BAB2D2FC735658AB38BF93A -e dev_secret.yaml > dev_secret.encrypted.yaml

Now, we would like every developers (user1, user2, user3) to be able to read this file. But, only developers from the project and not everyone with access to the git repository. So we still have to encrypt this file.User1 has encrypted the file dev_secret.yaml and stored the result in dev_secret.encrypted.yaml and is the only one able to decrypt it.

To do so, the admin , lets say User1 can execute the following commands:

User1 has to create the secret with the command

sops --pgp 57E6DA39E907744429FB07871141FE9F63986243,\
           5844C613B763F4374BAB2D2FC735658AB38BF93A,\
           AE0D6FD0242FF896BE1E376B62E1E77388753B8E \
           -e dev_secret.yaml > dev_secret.encrypted.yaml

We can check that both user2 and user3 can decrypt the dev_secret.encrypted.yaml file:This command contains every public key ids, comma separated.

user2 $ sops -d dev_secret.encrypted.yaml 
        secret=5Tz2QNxki789YFDauser2 $ sops -d dev_secret.encrypted.yaml 
        secret=5Tz2QNxki789YFDa

All the *.encrypted.yaml files are now stored in Git and can be managed like any other resources, with history and diff in commits. Only those defined during encryption can read them edit them.

Configure automatic key selection

Like we saw before, sops commands often requires references to key-id of people concerned by the modification and this is error prone and hard to manage if you share access with a lot of people.

To simplify this, the team can create a file, named .sops.yaml and placed it in the root of our Git repository.

creation_rules:  # Specific to `dev_a` env
  - path_regex: dev_a\.encrypted\.yaml$
    # Here, only the `user1` key-id
    pgp: >-
      5844C613B763F4374BAB2D2FC735658AB38BF93A  # Specific to `int` env
  - path_regex: int\.encrypted\.yaml$
    # Here, we have :
    # * `user1` key-id: 5844C613B763F4374BAB2D2FC735658AB38BF93A
    # * `user2` key-id: AE0D6FD0242FF896BE1E376B62E1E77388753B8E
    # * `user3` key-id: 57E6DA39E907744429FB07871141FE9F63986243
    pgp: >-
      5844C613B763F4374BAB2D2FC735658AB38BF93A,
      AE0D6FD0242FF896BE1E376B62E1E77388753B8E,
      57E6DA39E907744429FB07871141FE9F63986243  # Specific for new env `dev_a_and_b`
  - path_regex: dev_a_and_b\.encrypted.yaml$
    # Here, we have only `Alice` and `Bobby` :
    # * `user1` key-id: 5844C613B763F4374BAB2D2FC735658AB38BF93A
    # * `user2` key-id: AE0D6FD0242FF896BE1E376B62E1E77388753B8E
    pgp: >-
      5844C613B763F4374BAB2D2FC735658AB38BF93A,
      AE0D6FD0242FF896BE1E376B62E1E77388753B8E

Keys are selected by matching a regex against the path of the file, so possibilities are wide and this is simpler than using parameters in command line!So, no more ` — pgp <key-id>`, sops automatically selects the closest .sops.yaml file from the CWD

Add or Remove access with .sops.yaml

If a .sops.yaml file is used, user1 can simplify the add-pgp or rm-pgp command previously seen. he/she just need to change the .sops.yaml and use the command sops update keys dev_secret.encrypted.yaml to update who can decrypt the file.

Gitlab CI Configuration

This example shows how to configure Gitlab in order to decrypt secrets but this is totally compatible with other kinds of CI.

First, you need to export the private PGP key in order to add it as a Gitlab variable. You can do that by using the following command:

$ gpg --export-secret-key --armor > ci.private.key

After that, you will have access to the private key of your CI system. Be careful with it, this private key is the only element allowing decryption of your data, you should not loose it, nor share it widely.

Now, the CI is ready to be configured. Next, we can import into GitLab secret pane (in Settings > CI/CD > Variables) the content of the ci.private.key created previously and the passphrase defined for it.

NOTE: The variable Type is defined to File for the KEY. For more information about this, look at the official GitLab documentation about file variable

Then, we can define a GitLab CI job able to read the value from the encrypted secret. For our example, this will just do a cat on the standard output. Of course, you can do what you want with those values in your CI pipeline.

We need to take care of the image used in our Continuous Integration. We should be able to access the gpg command. In this example, we will install it.

deploy int:
  image: google/cloud-sdk # <1>
  before_script:
  - apt-get update && apt-get install -y curl gnupg # <2>
  - curl -qsL https://github.com/mozilla/sops/releases/download/v3.5.0/sops-v3.5.0.linux -o /usr/local/bin/sops # <3>
  - chmod +x /usr/local/bin/sops
  - cat $PGP_PRIVATE_KEY | gpg --batch --import # <4> 
  - echo $GPG_PASSPHRASE | gpg --batch --always-trust --yes --passphrase-fd 0 --pinentry-mode=loopback -s # <5>
  script: 
  - sops -d int.encrypted.env > int.env # <6>
  - cat int.env

Define a job image based required for our deploy phase. it should fit team needs, this is just for example here.
We install gpg and curl thanks to apt on debian distribution. Should be adapted if you are using another distribution.
We use curl to download sops binary from GitHub. Take care to get the latest version.
Import the ci.key into our gpg toolchain.
Provide the PASSPHRASE to the gpg toolchain to be totally "not interactive".
Then, the CI can decrypt the dev_secret.encrypted.yaml file and use it where we want in our deploy phase.

The problem with the above configuration is that, every time the pipeline is triggered, it will re-install every software, from gpg to sops, which can be time-consuming and source of error.

GitLab CI Optimization

Now let’s create another project just for CI tooling image, and he will prepare the image of the CI to be able to have gpg and sops installed, alongwith helm, and kubectl. Here is for example, the Dockerfile:

FROM alpine:3.11.2ARG HELM_VERSION="v3.2.4"
ARG HELM_TARGET="/usr/bin/helm"ARG KUBECTL_VERSION="v1.17.7"
ARG KUBECTL_TARGET="/usr/bin/kubectl"ARG SOPS_VERSION="v3.5.0"
ARG SOPS_TARGET="/usr/local/bin/sops"RUN apk update && apk add --no-cache gnupg ca-certificates curl git bash python3 && \
    curl -f -o ${KUBECTL_TARGET} https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl && \
    curl -f https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz | tar xvz --strip-components=1 linux-amd64/helm && \
    curl -qsL https://github.com/mozilla/sops/releases/download/${SOPS_VERSION}/sops-${SOPS_VERSION}.linux -o ${SOPS_TARGET} && \
    mv helm ${HELM_TARGET} && \
    chmod +x ${HELM_TARGET} && \
    chmod +x ${SOPS_TARGET} && \
    chmod +x ${KUBECTL_TARGET}ENV KUBECONFIG=/root/.kube/config
COPY kubeconfig $KUBECONFIGENTRYPOINT [""]

Then, he will activate a schedule to build this image every day and publish it in their own docker registry. Thanks to this, he can improve the previous .gitlab-ci.yml:

deploy:
  image:
    name: snigdhasambit/helm3 # Image created by the previous Dockerfile 🐳
    # entrypoint: ["/bin/sh", "-l", "-c"]
  only:
    - master
    - tags
  variables:
    CONTEXT: cluster1-aws-us-east-1
    KUBECONFIG: $CI_PROJECT_DIR/kubeconfig
    NAMESPACE: my-namespace-dev
    RELEASE_NAME: sample-dev
  
  environment:
    name: $CONTEXT  before_script:
    - cat $PGP_PRIVATE_KEY | gpg --batch --import 
    - echo $GPG_PASSPHRASE | gpg --batch --always-trust --yes --passphrase-fd 0 --pinentry-mode=loopback -s  script:    - echo $HA_CREDENTIALS
    - kubectl config use-context $CONTEXT
    - kubectl config set-context $CONTEXT --namespace $NAMESPACE
    - sops -d $CI_PROJECT_DIR/secrets/secret.enc.yaml | kubectl apply -f -
    - helm upgrade -i $RELEASE_NAME ./mychart

You can see, the job execution is much straightforward now:

gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key 772B6D0C9376C9D5: public key "sre-gpg-sops <snigdha.sambit@email.com>" imported
gpg: key 772B6D0C9376C9D5: secret key imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg:       secret keys read: 1
gpg:   secret keys imported: 1
[32;1m$ echo $GPG_PASSPHRASE | gpg --batch --always-trust --yes --passphrase-fd 0 --pinentry-mode=loopback -s[0;m
�M���
�w+m�v���b�_� �3��!��o3��j�^��Iw+m�v��_� �
	w+m�v���#�>�j��W�|!bc���U+��|��u*��c	�1���pQ9vs�� A�!,�%�r�:�ey��hE4R=�?e(#�+��sg���(�(Կ�ټ�v}F� ���f6 %�!iX|��^�ApOK���=�g�R����?���%��n���U|TqRI/�u ��\�K�缗�D����r�>���t�૬Ĩ$F��0�]����s��!��v�|X���:9���It�0K��RdO�yO>&�sg�Vp+"u-���<��a����m'� ���}y�v�U�L��0P3N4O�=9e��bm�#�7J�SCk��m�1HY��@�\8��6�����@��9&^KbY�	��9���;�n���[{�.;�-�`+%@I!�
�$�hV4䙏�}M=p&���H�����҅�]�X��F�3�3�gs"��|���1$���6i$$����dE�+���9:��Aj�N��A����Cӏ��[eA-�t�[32;1m$ echo 
[32;1m$ kubectl config use-context $CONTEXT[0;m
Switched to context "cluster1-aws-us-east-1".
[32;1m$ kubectl config set-context $CONTEXT --namespace $NAMESPACE[0;m
Context "cluster1-aws-us-east-1" modified.
[32;1m$ sops -d $CI_PROJECT_DIR/secrets/secret.enc.yaml | kubectl apply -f -[0;m
secret/mysecret unchanged
[32;1m$ helm upgrade -i $RELEASE_NAME ./mychart[0;m
Release "sample-dev" has been upgraded. Happy Helming!
NAME: sample-dev
LAST DEPLOYED: Tue Jul 14 18:57:38 2020
NAMESPACE: my-namespace-dev
STATUS: deployed
REVISION: 4
TEST SUITE: None
section_end:1594753058:step_script
[0K[32;1m
Job succeeded
[0;m

NOTE: You can use this method with any docker registry, not only the one from GitLab

Of course, this is just an extract of the real .gitlab-ci.yaml, which includes linting, test, build & deploy.

Conclusion

Here, we learn how to configure our CI identity and configure the GitLab CI to be able to decrypt secrets. We also see how to optimize it in a GitLab environment leveraging usage of home made docker image.