yesod-mirror/k8s/configs

k8s/configs

WARNING: Provisional Readme

This directory contains kubernetes configurations, which are defined in jsonnet and which can also depend upon loaded helm charts.

Manifests are built using custom Tanka-like build rules, as in this example:

load("@rules_jsonnet//jsonnet:jsonnet.bzl", "jsonnet_to_json")
load("//tools:tanka.bzl", "tanka_environment")

# Generate a json manifest containing all of the manifests
jsonnet_to_json(
    name = "main",
    src = "main.jsonnet",
    outs = ["main.json"],
    data = [
        # Depend on a helm chart (transitively, in this case)
        "@helm_coderv2_coder//:chart",
    ],
    visibility = ["//visibility:public"],
    deps = [
        "//k8s/configs/templates",
    ],
)

# Defines three targets
# - example.show: Prints the list of output entities
# - example.diff: Diffs the list of entities against the live entities
# - example.apply: Applies the changes to Kubernetes.
tanka_environment(
    name = "example",
    main = ":main",
    spec = "spec.json",
)

Secret Management with SOPS

We use SOPS to manage secrets in this repository. Encrypted files are checked into version control, and Bazel handles decryption during the build process, keeping secrets in memory or temporary build artifacts (which are not committed).

Prerequisites

• SOPS: The sops binary is automatically managed by Bazel (fetched via MODULE.bazel), so you don't strictly need it installed on your system to build, but you do need it to edit or create secrets.
  • Install: brew install sops (macOS) or download from GitHub Releases.
• Encryption Key: You must have a configured Age key or PGP key that matches the .sops.yaml configuration (if one exists at the repo root) or pass the keys explicitly via command line.

Workflow

1. Create/Edit Encrypted File: Create a file (e.g., secrets.sops.yaml or secrets.sops.json) and encrypt it.

   # Example: Encrypting a new file
   sops --encrypt --age <your-age-public-key> secrets.json > secrets.sops.json
   
   # Example: Editing an existing encrypted file
   SOPS_AGE_KEY_FILE="./key.txt" sops secrets.sops.json
   

2. Define Bazel Target: In the BUILD.bazel file of your environment (e.g., k8s/configs/environments/media/BUILD.bazel), use the sops_decrypt rule to decrypt the file at build time.

   load("//tools:sops.bzl", "sops_decrypt")
   load("@rules_jsonnet//jsonnet:jsonnet.bzl", "jsonnet_library")
   
   # 1. Decrypt the secrets file
   sops_decrypt(
       name = "secrets",
       src = "secrets.sops.yaml", # The encrypted source file
       out = "secrets.json",      # The decrypted output filename
   )
   
   # 2. Wrap it in a jsonnet_library so it can be imported
   jsonnet_library(
       name = "secrets_lib",
       srcs = [":secrets"],
   )
   

3. Use in Jsonnet: Update your jsonnet_to_json target to depend on the library, and import the secrets in your Jsonnet code.

   BUILD.bazel:

   jsonnet_to_json(
       name = "main",
       src = "main.jsonnet",
       deps = [
           ":secrets_lib",
           # ... other deps ...
       ],
       # ...
   )
   

   main.jsonnet:

   local secrets = import "k8s/.../secrets.json";
   
   {
       secrets: {
           examplePostgres: postgres.Secret(postgres.SecretParams{
               name: "example-postgres",
               namespace: "example",
               password: secrets.example_psql_db_pwd,
           }), 
       }
   }
   

Safety

• Do not commit decrypted files. The sops_decrypt rule places files in the bazel-out directory, which is ignored by git.
• Ensure your .gitignore includes *.json or specific secret patterns if you are working with them locally outside of Bazel.