OPA on Kubernetes: Security policies

https://blog.openpolicyagent.org/opas-full-stack-policy-language-caeaadb1e077

Problem statement

Every organisation now are onboarded into kubernetes and terraform and use a lot of kubernetes objects, GCP resources and other infrastructure components. Therefore adhering to some best practices and guidelines is of utmost priority. We need to define some policy which would check the state of objects and either allow or deny the creation or mutate(edit) the resources during run time.

Some of the usecases where we need a policy adherance tool are as follows:

1. Scan terraform resources before applying to check some labels or best practices. Ex: All resources should have either (dev, stg, prd, tooling labels)
2. Check and deny any pods running with priviledged access
3. Only allow pods to run images from specific registries like gcr, k8s.gcr.io/ etc
4. Ensure all kubernetes objects have a team lable or some specific annotations
5. Enforce each deploymemt had PDB’s i.e disallow PodDisruptionBudgets with .spec.maxUnavailable == 0
6. Requires Ingress resources to be HTTPS only
7. …

Introduction

OPA Gatekeeper is a policy-based control system for Kubernetes that allows you to define and enforce policies for resource validation and admission control. In this page, we will explore how to install OPA Gatekeeper using Helm charts and create policies to require labels on Kubernetes objects.

Installation with Helm Charts

Prerequisites

Before you begin, make sure you have Helm installed and configured with your Kubernetes cluster. You can install Helm by following the official documentation.

Install OPA Gatekeeper

To install OPA Gatekeeper using Helm charts, follow these steps:

1. Add the OPA Gatekeeper Helm repository:

helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts

2. Update your Helm repositories:

helm repo update

3. Install OPA Gatekeeper:

helm install gatekeeper/gatekeeper --name-template=gatekeeper --namespace gatekeeper-system --create-namespace

This will deploy OPA Gatekeeper to your cluster.

Verify the installation

We can verify the installation now:

k get all -n gatekeeper-system
NAME                                                READY   STATUS    RESTARTS      AGE
pod/gatekeeper-audit-856d7d4c5f-xcpww               1/1     Running   1 (82m ago)   82m
pod/gatekeeper-controller-manager-b559dccc5-9v6m2   1/1     Running   0             82m
pod/gatekeeper-controller-manager-b559dccc5-gcxd5   1/1     Running   0             82m
pod/gatekeeper-controller-manager-b559dccc5-hbk2c   1/1     Running   0             82m

NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/gatekeeper-webhook-service   ClusterIP   172.24.34.193   <none>        443/TCP   82m
NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/gatekeeper-audit                1/1     1            1           82m
deployment.apps/gatekeeper-controller-manager   3/3     3            3           82m
NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/gatekeeper-audit-856d7d4c5f               1         1         1       82m
replicaset.apps/gatekeeper-controller-manager-b559dccc5   3         3         3       82m

These two deployments create the pods that run the Gatekeeper controllers:

• gatekeeper-controller-manager implements the webhooks that the Kubernetes API will call for validation and mutation
• gatekeeper-audit checks for policy compliance on objects that already exist in the cluster

And also see these new custom resource definitions (CRDs) that have been created in the cluster:

k get crd -l gatekeeper.sh/system=yes

NAME                                                 CREATED AT
assign.mutations.gatekeeper.sh                       2023-10-17T14:24:39Z
assignimage.mutations.gatekeeper.sh                  2023-10-17T14:24:39Z
assignmetadata.mutations.gatekeeper.sh               2023-10-17T14:24:40Z
configs.config.gatekeeper.sh                         2023-10-17T14:24:40Z
constraintpodstatuses.status.gatekeeper.sh           2023-10-17T14:24:40Z
constrainttemplatepodstatuses.status.gatekeeper.sh   2023-10-17T14:24:40Z
constrainttemplates.templates.gatekeeper.sh          2023-10-17T14:24:40Z
expansiontemplate.expansion.gatekeeper.sh            2023-10-17T14:24:41Z
expansiontemplatepodstatuses.status.gatekeeper.sh    2023-10-17T14:24:41Z
modifyset.mutations.gatekeeper.sh                    2023-10-17T14:24:41Z
mutatorpodstatuses.status.gatekeeper.sh              2023-10-17T14:24:41Z
providers.externaldata.gatekeeper.sh                 2023-10-17T14:24:41Z

Using the constrainttemplates.templates.gatekeeper.sh CRD you can create general policy definitions that Gatekeeper will use.

All code and examples can be found in the below repo:

https://github.com/snigdhasambitak/opa-policies

Creating Constraint Templates

Constraint Templates are the building blocks for defining policies in OPA Gatekeeper. Let’s create one that requires labels on Kubernetes objects.

https://dev.to/ashokan/kubernetes-policy-management-ii-opa-gatekeeper-465g

Example Constraint Template

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        openAPIV3Schema:
          properties:
            labels:
              type: object
              properties:
                required_labels:
                  type: array
                  items:
                    type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels
        violation[{"msg": msg}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels.required_labels[_]}
          not provided[label]
          msg := sprintf("Required label '%v' is missing", [label])
        }

This Constraint Template defines a requirement for labels on Kubernetes objects.

Creating K8sRequiredLabels Objects

Now, let’s create a K8sRequiredLabels object that specifies which labels are required for pods and namespaces.

Example K8sRequiredLabels Object

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: required-pod-labels
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    labels:
      required_labels:
        - "app"
        - "env"

This K8sRequiredLabels object enforces that all pods must have “app” and “env” labels.

Example K8sRequiredLabels Object for Namespaces

You can create a similar object for namespaces:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: required-namespace-labels
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Namespace"]
  parameters:
    labels:
      required_labels:
        - "team"

This enforces that all namespaces must have a “team” label.

Mutation Webhooks

Kubernetes also provides a mechanism where the admission controller can mutate (that is, update) the contents of the request to the Kubernetes API in order to satisfy the constraint.

Mutation of this sort can be a solution to the issue of having Kubernetes API requests rejected at the admission stage, which then requires the requesting entity to modify and re-submit the request.

k get crd -l gatekeeper.sh/system=yes  | grep -i mutation

assign.mutations.gatekeeper.sh                       2023-10-17T14:24:39Z
assignimage.mutations.gatekeeper.sh                  2023-10-17T14:24:39Z
assignmetadata.mutations.gatekeeper.sh               2023-10-17T14:24:40Z
modifyset.mutations.gatekeeper.sh                    2023-10-17T14:24:41Z

The function of these CRDs is:

• AssignMetadata — defines changes to the metadata section of a resource
• Assign — any change outside the metadata section
• ModifySet — adds or removes entries from a list, such as the arguments to a container

In the previous example Gatekeeper was used to deny the creation of a namespace without the owner label set, you can use the AssignMetadata CRD to set the label to a specific value.

Here is the YAML for this CRD instance. This will cause any attempt to create a namespace without a value for the label team to have that label set to the value syseng. Save it to the file example-assign-metadata.yaml.

apiVersion: mutations.gatekeeper.sh/v1alpha1
kind: AssignMetadata
metadata:
 name: ns-label-owner
spec:
   match:
     kinds:
       - apiGroups: ["*"]
         kinds: ["Namespace"]
   location: "metadata.labels.team"
   parameters:
     assign:
       value:  "syseng"

Now once you create any namepsace without any labels, it should also add the team label with the value syseng

kubernetes.io/metadata.name=test-opa-mutation,team=syseng

Advantages of OPA Gatekeeper

• Policy Enforcement: OPA Gatekeeper enforces policies across your Kubernetes cluster, ensuring that resources comply with defined rules.
• Dynamic Policies: Policies can be updated and modified without affecting the core Kubernetes cluster, allowing for dynamic policy management.
• Customization: OPA Gatekeeper provides flexibility in defining policies tailored to your specific requirements.

Use Cases

OPA Gatekeeper can be used for various purposes, including:

1. Security: Enforce security policies, such as requiring specific labels, pod affinity/anti-affinity, and network policies.
2. Compliance: Ensure compliance with industry standards and regulations by enforcing specific configurations.
3. Resource Optimization: Define policies for optimizing resource allocation and utilization.
4. Cost Management: Enforce policies that control costs by limiting resource usage.

Conclusion

OPA Gatekeeper is a powerful tool for enforcing policies in your Kubernetes environment. By creating Constraint Templates and K8sRequiredLabels objects, you can ensure that Kubernetes objects adhere to your desired labeling standards. This improves security, compliance, and resource management within your cluster.

For more information, consult the official OPA Gatekeeper documentation.

Resources

Open Policy Agent

GitHub - open-policy-agent/conftest: Write tests against structured configuration data using the…

https://github.com/open-policy-agent/gatekeeper-library/tree/master/library