117 lines
3.9 KiB
Markdown
117 lines
3.9 KiB
Markdown
|
# k8s/configs
|
||
|
|
|
||
|
|
WARNING: Provisional Readme
|
||
|
|
|
||
|
|
This directory contains kubernetes configurations, which are defined in jsonnet and which can also depend upon loaded helm charts.
|
||
|
|
|
||
|
|
Manifests are built using custom Tanka-like build rules, as in this example:
|
||
|
|
|
||
|
|
```BUILD
|
||
|
|
load("@rules_jsonnet//jsonnet:jsonnet.bzl", "jsonnet_to_json")
|
||
|
|
load("//tools:tanka.bzl", "tanka_environment")
|
||
|
|
|
||
|
|
# Generate a json manifest containing all of the manifests
|
||
|
|
jsonnet_to_json(
|
||
|
|
name = "main",
|
||
|
|
src = "main.jsonnet",
|
||
|
|
outs = ["main.json"],
|
||
|
|
data = [
|
||
|
|
# Depend on a helm chart (transitively, in this case)
|
||
|
|
"@helm_coderv2_coder//:chart",
|
||
|
|
],
|
||
|
|
visibility = ["//visibility:public"],
|
||
|
|
deps = [
|
||
|
|
"//k8s/configs/templates",
|
||
|
|
],
|
||
|
|
)
|
||
|
|
|
||
|
|
# Defines three targets
|
||
|
|
# - example.show: Prints the list of output entities
|
||
|
|
# - example.diff: Diffs the list of entities against the live entities
|
||
|
|
# - example.apply: Applies the changes to Kubernetes.
|
||
|
|
tanka_environment(
|
||
|
|
name = "example",
|
||
|
|
main = ":main",
|
||
|
|
spec = "spec.json",
|
||
|
|
)
|
||
|
|
```
|
||
|
|
|
||
|
|
## Secret Management with SOPS
|
||
|
|
|
||
|
|
We use [SOPS](https://github.com/getsops/sops) to manage secrets in this repository. Encrypted files are checked into version control, and Bazel handles decryption during the build process, keeping secrets in memory or temporary build artifacts (which are not committed).
|
||
|
|
|
||
|
|
### Prerequisites
|
||
|
|
|
||
|
|
* **SOPS**: The `sops` binary is automatically managed by Bazel (fetched via `MODULE.bazel`), so you don't strictly need it installed on your system to *build*, but you do need it to *edit* or *create* secrets.
|
||
|
|
* Install: `brew install sops` (macOS) or download from [GitHub Releases](https://github.com/getsops/sops/releases).
|
||
|
|
* **Encryption Key**: You must have a configured Age key or PGP key that matches the `.sops.yaml` configuration (if one exists at the repo root) or pass the keys explicitly via command line.
|
||
|
|
|
||
|
|
### Workflow
|
||
|
|
|
||
|
|
1. **Create/Edit Encrypted File**:
|
||
|
|
Create a file (e.g., `secrets.sops.yaml` or `secrets.sops.json`) and encrypt it.
|
||
|
|
```bash
|
||
|
|
# Example: Encrypting a new file
|
||
|
|
sops --encrypt --age <your-age-public-key> secrets.json > secrets.sops.json
|
||
|
|
|
||
|
|
# Example: Editing an existing encrypted file
|
||
|
|
SOPS_AGE_KEY_FILE="./key.txt" sops secrets.sops.json
|
||
|
|
```
|
||
|
|
|
||
|
|
2. **Define Bazel Target**:
|
||
|
|
In the `BUILD.bazel` file of your environment (e.g., `k8s/configs/environments/media/BUILD.bazel`), use the `sops_decrypt` rule to decrypt the file at build time.
|
||
|
|
|
||
|
|
```python
|
||
|
|
load("//tools:sops.bzl", "sops_decrypt")
|
||
|
|
load("@rules_jsonnet//jsonnet:jsonnet.bzl", "jsonnet_library")
|
||
|
|
|
||
|
|
# 1. Decrypt the secrets file
|
||
|
|
sops_decrypt(
|
||
|
|
name = "secrets",
|
||
|
|
src = "secrets.sops.yaml", # The encrypted source file
|
||
|
|
out = "secrets.json", # The decrypted output filename
|
||
|
|
)
|
||
|
|
|
||
|
|
# 2. Wrap it in a jsonnet_library so it can be imported
|
||
|
|
jsonnet_library(
|
||
|
|
name = "secrets_lib",
|
||
|
|
srcs = [":secrets"],
|
||
|
|
)
|
||
|
|
```
|
||
|
|
|
||
|
|
3. **Use in Jsonnet**:
|
||
|
|
Update your `jsonnet_to_json` target to depend on the library, and import the secrets in your Jsonnet code.
|
||
|
|
|
||
|
|
**BUILD.bazel**:
|
||
|
|
```python
|
||
|
|
jsonnet_to_json(
|
||
|
|
name = "main",
|
||
|
|
src = "main.jsonnet",
|
||
|
|
deps = [
|
||
|
|
":secrets_lib",
|
||
|
|
# ... other deps ...
|
||
|
|
],
|
||
|
|
# ...
|
||
|
|
)
|
||
|
|
```
|
||
|
|
|
||
|
|
**main.jsonnet**:
|
||
|
|
```jsonnet
|
||
|
|
local secrets = import "k8s/.../secrets.json";
|
||
|
|
|
||
|
|
{
|
||
|
|
secrets: {
|
||
|
|
examplePostgres: postgres.Secret(postgres.SecretParams{
|
||
|
|
name: "example-postgres",
|
||
|
|
namespace: "example",
|
||
|
|
password: secrets.example_psql_db_pwd,
|
||
|
|
}),
|
||
|
|
}
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
### Safety
|
||
|
|
|
||
|
|
* **Do not commit** decrypted files. The `sops_decrypt` rule places files in the `bazel-out` directory, which is ignored by git.
|
||
|
|
* Ensure your `.gitignore` includes `*.json` or specific secret patterns if you are working with them locally outside of Bazel.
|