Stacks
Pre-requisites
Afon entry point
You can choose one of the Afon entry points.
Setup Local Env
You need to have set up you local env.
Stack deployment
To deploy a stack, using the CLI command, we run manage-stack with the deploy action, eg:
afon-cli.sh manage-stack deploy aws-full
afon-cli.sh manage-stack deploy azure-polaris
afon-cli.sh manage-stack deploy gcp-polaris
afon-cli.sh manage-stack deploy gcp-keycloak
NOTE: To run manage-stack you need to be logged in with the corresponding cloud cli.
IMPORTANT: manage-stack uses $AFON_CDKTF as its tofu state dir, so if you deploy or setup a stack (ie deploy a stack plus configure the Polaris catalogs) you must take care not move your $AFON_CDKTF until the teardown of the stack, other tofu will not know the information to destroy the stack.
Of the actions, deploy just deploys the stack (waiting for DNS to propagate) and setup does the deployment and configures the Polaris catalogs.
Manage a stack
A single command manages any stack, given an action and a stack id:
afon-cli.sh manage-stack <action> <stack-id> [<step>] [--replace] # run without args to list the stack ids and their actions
where:
<action>is one ofdelete,deployorsetup;<stack-id>is<cloud>-<name>, egaws-keycloak, named asvalidate-stacklists the stacks. It can also be<cloud>-<role>, the stack of the cloud with that role instacks.yaml(catalog,identityortre), egaws-tre, unless a stack is defined under that id, so that it works whatever the ids are.validate-stack,generate-cleanup-scriptand the other commands taking a stack id accept it too, and so do the scripts (aws-create,aws-cluster,tofu-plan, the kube estimator and so on), which, as their name has the cloud, also take the role alone, egaws-create tre. Every defined stack is listed: with the bundledstacks.yaml,aws-full,aws-keycloak,aws-polaris,aws-snowflake,azure-full,azure-keycloak,azure-polaris,gcp-keycloakandgcp-polaris, and, to delete every stack of a cloud,<cloud>-all, egaws-all;<step>, if given, is the step from which to resume a failed run (see Resume a failed stack deployment);--replace, if given, letsdeployandsetupdelete the other stacks conflicting with the stack, see stacks side by side. Withafon.sh, pass it after--, eg./afon.sh -a manage-stack -- deploy aws-full --replace.
For example:
afon-cli.sh manage-stack setup aws-keycloak
afon-cli.sh manage-stack deploy aws-full wait_on_polaris_and_keycloak_for_cloud_aws
afon-cli.sh manage-stack delete aws-full
With afon.sh, pass the same args positionally, eg ./afon.sh -a manage-stack setup aws-keycloak.
To run a scenario on a stack that is set up, ie what its components can be asked to run beyond setting up, use
afon-cli.sh manage-stack run <stack-id> [<scenario> [<step>]]
Without a <scenario> it lists those of the stack, and <step> resumes a failed run. For example aws-snowflake has reading-aws, reading-azure, reading-gcp and writing-azure, which replace the deploy-snowflake-* commands. Setting up the stack and running a scenario is the flow snowflake-<scenario>:
afon-cli.sh manage-stack run snowflake-reading-azure --replace
The previous per-stack commands, deploy-*-stack, setup-*-stack and delete-*-stack, are gone, as are those spanning two clouds: use manage-stack, eg manage-stack delete <stack-id>, or manage-stack delete <cloud>-all for every stack of a cloud, eg aws-all. What spans stacks is a scenario, eg manage-stack run aws-full external-azure, run once the stacks it needs are set up, or a flow, eg manage-stack run cross-cloud-azure, which sets them up first.
Run a flow of stacks
A flow is a composed run: a named, ordered list of actions on stacks and scenarios, run in order with one command, stopping at the first failure. run takes either a stack id and a scenario, or a flow name, which is never a stack id:
afon-cli.sh manage-stack run # lists the flows and their items
afon-cli.sh manage-stack run <flow> [<item>:<step>] [--replace] [--dry-run]
--replaceapplies to everydeployandsetupitem;--dry-runlists the steps of the whole flow, grouped by item, without running them;<item>:<step>resumes from a step, as step names repeat across items. A failed flow reports the step to resume from, and the items that did not run, egafon-cli.sh manage-stack run cross-cloud-azure 2:deploy_complete_for_azure_polaris.
The flows are:
aws-full-and-azure-polaris:setup aws-full,setup azure-polaris;aws-keycloak-and-polaris:setup aws-keycloak,setup aws-polaris;gcp-keycloak-and-polaris:setup gcp-keycloak,setup gcp-polaris;azure-and-gcp-polaris:setup azure-polaris,setup gcp-polaris;azure-keycloak-and-polaris:setup azure-keycloak,setup azure-polaris;cross-cloud-azure:setup aws-full,setup azure-polaris, thenrun aws-full external-azure;snowflake-<scenario>, iesnowflake-reading-aws,snowflake-reading-azure,snowflake-reading-gcpandsnowflake-writing-azure:setup aws-snowflake, thenrun aws-snowflake <scenario>;snowflake-reading-all: just the three reading runs in a row, on anaws-snowflakethat is already set up.
They are defined in stack-flows.yaml (afon-ops/src/main/resources/stack-flows.yaml, bundled in the jar), and validated when used: the stacks, scenarios and actions must exist, a scenario runs on a stack no earlier item deleted (a flow may be just a chain of runs on stacks already set up), and a later item must not delete a stack an earlier one leaves running, by conflicting with it. Each flow has a generated document, under flow_*.md.
Define your own flows
A stack-flows.yaml in the config dir ($AFON_HOME/config) replaces the bundled one entirely, rather than adding to it, so start from a copy, which this exports there (without overwriting an existing file):
afon-cli.sh manage-stack run --export-flows
Each flow is a name, which is not the id of a stack, and a list of items, each one action (delete, deploy, setup or run) on a stack, and a scenario for run:
flows:
my-flow:
- setup: {cloud: aws, role: tre} # the stack of the cloud with the role in stacks.yaml
- setup: {stack: aws-lab} # a stack by its id
- run: {cloud: aws, component: Snowflake, scenario: reading-aws} # the stack of the cloud that deploys the component
A stack is named by its id (stack), or by a cloud with a role (catalog, identity or tre) or a component it deploys. The bundled flows name stacks by role or component, so that they work whatever the ids in stacks.yaml. A flow whose stacks stacks.yaml does not define cannot run: it is not listed, and running it fails, naming the missing stack. A file that is not valid, eg an unknown action, role, cloud or component, a run with no scenario, or a repeated flow, makes manage-stack fail before doing anything, naming the file and the flow. The generated documents (flow_*.md) are made when Afon is built, from the bundled flows only, so a flow of your own has none: list its steps with manage-stack run <flow> --dry-run.
Stacks side by side
NOTE: The AKS cluster, network and identity of an Azure stack are named <stack-type>-<name>, eg keycloak-aks-vnet and keycloak-<CLUSTER_NAME>, as on AWS, so the stacks of a resource group don’t collide. The GKE cluster, VPC, subnet and router of a GCP stack are likewise named after the stack type, eg keycloak-<CLUSTER_NAME>, so the stacks of a project don’t either.
Each stack has its own tofu state, so stacks run side by side, unless they conflict, ie they hold a common exclusive resource: the DNS record of a domain name, eg OIDC_DOMAIN_NAME, or the pinned egress IP, AWS_EGRESS_IP, AZURE_EGRESS_IP or GCP_EGRESS_IP, which only one running stack can hold. So deploy and setup first delete the stacks deployed in the cloud that conflict with the stack, including the stack itself. They only delete the stack itself, if deployed, unless given --replace: if any other stack conflicts, they fail before deleting any, saying what the other stack already deploys, eg
Cannot deploy aws-polaris, as stack aws-full is running and already deploys Polaris at emryspolaris.click (AWS_POLARIS_DOMAIN_NAME).
To deploy aws-polaris, delete it first:
afon-cli.sh manage-stack delete aws-full
or rerun with --replace to delete it as part of this run.
Only a deployed stack, ie with a tofu state, can make them fail: a conflicting stack dir without one, eg left from validate-stack, has nothing deployed, so they just delete it. The pinned egress IP, AWS_EGRESS_IP, AZURE_EGRESS_IP or GCP_EGRESS_IP, is only held, and so only conflicts, when it is set. The AWS stack workflow always runs with --replace.
delete deletes just the stack, while delete <cloud>-all, eg delete aws-all, deletes every stack deployed in the cloud. The legacy stack of a cloud, deployed before each stack had its own tofu state (see upgrading to per-stack state), may be any stack, so it conflicts with every stack of the cloud, and delete of any stack of the cloud deletes it too. Likewise, a stack not listed above, eg one deployed from validate-stack aws-full and tofu-apply, conflicts with every stack of the cloud.
| Stack | Holds | Runs side by side with |
|---|---|---|
aws-keycloak | OIDC_DOMAIN_NAME | aws-polaris |
aws-polaris | AWS_POLARIS_DOMAIN_NAME, AWS_EGRESS_IP | aws-keycloak |
aws-full | OIDC_DOMAIN_NAME, AWS_POLARIS_DOMAIN_NAME, AWS_JUPYTER_DOMAIN_NAME, AWS_EGRESS_IP | none |
aws-full, with AWS_KEYCLOAK_EXTERNAL=true | AWS_POLARIS_DOMAIN_NAME, AWS_JUPYTER_DOMAIN_NAME, AWS_EGRESS_IP | aws-keycloak |
azure-keycloak | OIDC_DOMAIN_NAME | azure-polaris |
azure-polaris | AZURE_POLARIS_DOMAIN_NAME, AZURE_EGRESS_IP | azure-keycloak |
azure-full | OIDC_DOMAIN_NAME, AZURE_POLARIS_DOMAIN_NAME, AZURE_JUPYTER_DOMAIN_NAME, AZURE_EGRESS_IP | none |
azure-full, with AZURE_KEYCLOAK_EXTERNAL=true | AZURE_POLARIS_DOMAIN_NAME, AZURE_JUPYTER_DOMAIN_NAME, AZURE_EGRESS_IP | azure-keycloak |
gcp-keycloak | OIDC_DOMAIN_NAME | gcp-polaris |
gcp-polaris | GCP_POLARIS_DOMAIN_NAME, GCP_EGRESS_IP | gcp-keycloak |
The Polaris stacks, aws-polaris, azure-polaris and gcp-polaris, deploy no KeyCloak of their own, but use the KeyCloak server at OIDC_DOMAIN_NAME, which the keycloak stack of the cloud, eg aws-keycloak, deploys, so deploy (or set up) it first, eg:
afon-cli.sh manage-stack setup aws-keycloak
afon-cli.sh manage-stack setup aws-polaris
Their deploy and setup check first that KeyCloak answers at OIDC_DOMAIN_NAME, and fail fast if it doesn’t.
The aws-full stack deploys its own KeyCloak, so it conflicts with both aws-keycloak and aws-polaris, unless AWS_KEYCLOAK_EXTERNAL=true is set in afon-aws.properties (it is false when empty): it then uses the KeyCloak aws-keycloak deploys, like aws-polaris, so it runs side by side with aws-keycloak, and its deploy and setup likewise check first that KeyCloak answers. Only the Polaris stacks pin their egress IP to AWS_EGRESS_IP, if set: aws-keycloak always gets a fresh one. The same holds on Azure with AZURE_EGRESS_IP: azure-keycloak always gets an address from Azure. On GCP gcp-polaris pins its egress IP to GCP_EGRESS_IP, if set, and gcp-keycloak always gets addresses from Google.
On Azure it is the same with azure-keycloak and AZURE_KEYCLOAK_EXTERNAL=true in afon-azure.properties (it is false when empty): azure-full then uses the KeyCloak azure-keycloak deploys, so it runs side by side with it, and its deploy and setup check first that KeyCloak answers. The KeyCloak domain OIDC_DOMAIN_NAME must then have its DNS zone in the Azure resource group, which afon-cli.sh dns-create-zones azure azure-keycloak creates, once azure-create has created the resource group.
A KeyCloak domain name per cloud
OIDC_DOMAIN_NAME in afon.properties is the domain name of the KeyCloak of every cloud, so the KeyCloak stacks of different clouds, eg aws-keycloak and azure-keycloak, conflict, and moving an env from one cloud to another means moving its DNS to the zone of the new cloud. To give a cloud a KeyCloak of its own, set <CLOUD>_OIDC_DOMAIN_NAME, ie AWS_OIDC_DOMAIN_NAME, AZURE_OIDC_DOMAIN_NAME or GCP_OIDC_DOMAIN_NAME, in its afon-<cloud>.properties. The domain name of a cloud is that property if it is set and not blank, else OIDC_DOMAIN_NAME, so existing configs behave as before. Everything of the cloud then uses it: its KeyCloak stacks, which hold it, so those of two clouds with different domain names no longer conflict; the Polaris stacks that use that KeyCloak; the DNS zone and certificate of the cloud (each domain needs its own, eg on AWS a hosted zone named after it and an ACM certificate, see DNS config); and the checks, check-oidc <cloud>, which also shows a token with --token, and the status and smoke scripts.
OIDC_ISSUER_URL is a template, in which [OIDC_DOMAIN_NAME] stands for the domain name of the cloud, so the issuer follows it. The host of the issuer must be the domain name of the cloud, else the stack fails to synthesise.
Each cloud’s KeyCloak has its own users and tokens, so a client authenticates with the KeyCloak of the cloud hosting the Polaris it calls, whichever cloud it runs from: the ops commands, which take the cloud, and the smoke tests use the domain name of that cloud, and a federated (external) catalog, eg azure_external of the AWS Polaris, gets its token from the KeyCloak of the cloud hosting the remote Polaris, here Azure. The users the flow authenticates as must therefore be configured in that KeyCloak too (configure-users <cloud>).
Upgrading to per-stack state (one-off)
Stacks used to be deployed per cloud: all the stacks of a cloud shared one plan $AFON_CDKTF/stacks/<cloud> and one tofu state $AFON_CDKTF/terraform.<cloud>.tfstate, eg stacks/azure and terraform.azure.tfstate. They are now deployed per stack id, each with its own plan stacks/<stack-id> and state terraform.<stack-id>.tfstate, eg stacks/azure-polaris and terraform.azure-polaris.tfstate.
A stack deployed before the change, ie with a terraform.<cloud>.tfstate, is the legacy stack of its cloud. manage-stack cannot tell which stack it is, so it treats it as conflicting with every stack of the cloud. Hence the first deploy or setup of any stack of a cloud with a legacy stack fails, before deleting anything, eg:
Cannot deploy azure-polaris, as the legacy stack azure, deployed before each stack had its own tofu state, may be any stack.
To deploy azure-polaris, delete it first:
afon-cli.sh manage-stack delete azure-polaris
or rerun with --replace to delete it as part of this run.
This is a one-off per cloud, per cdktf dir: once the legacy stack of a cloud is deleted, it is gone for good, and from then on every stack of the cloud has its own state. Each cdktf env (see cdktf-switch) has its own legacy stacks, so it happens once in each env. Either:
- rerun with
--replace, which deletes the legacy stack and then deploys the stack, egafon-cli.sh manage-stack setup azure-polaris --replace, or - delete it first, as
deleteof any stack of the cloud deletes the legacy stack too, egafon-cli.sh manage-stack delete azure-polaris, then deploy the stack as usual.
To see whether a cdktf dir still has legacy stacks, look for terraform.<cloud>.tfstate files, where <cloud>={aws,azure,gcp}:
ls $AFON_CDKTF/*.tfstate
A legacy stacks/<cloud> dir without a terraform.<cloud>.tfstate, eg left from synthesising a stack without deploying it, has nothing deployed, so it doesn’t make deploy or setup fail: they just delete it. On AWS, the stacks now also name their components from <stack>-<CLUSTER_NAME> rather than CLUSTER_NAME, see AWS: the legacy stack keeps its old names until it is deleted, which its own plan and state take care of. The AWS stack workflow runs with --replace, so it deletes the legacy stack of its runner’s cdktf dir on its first run, without failing.
Validate a stack’s config
To validate a stack’s config without deploying it, eg that the config dir has all the properties the stack needs, you can synthesise just the cdktf plan of a stack, which needs no cloud login:
afon-cli.sh validate-stack <stack> # eg azure-polaris or aws-full, run without <stack> to list the stacks
This fails fast, eg with Property X not found, if the config is incomplete. It overwrites the plan in $AFON_CDKTF/stacks/<stack-id> (ie cdk.tf.json and assets), but not the state file $AFON_CDKTF/terraform.<stack-id>.tfstate, as it doesn’t run tofu. To leave the plan of a running or previous deploy of the same stack untouched, synthesise into another cdktf dir, eg afon-cli.sh --cdktf /tmp/cdktf validate-stack azure-polaris.
A stack is a list of components, eg Polaris, Keycloak, Jupyter, Spark, Trino, deployed to its cloud: see Stack composition for what each stack is made of, and how to define a new one.
Define your own stacks
The stacks are defined by stacks.yaml, bundled in the jar (afon-cdk/src/main/resources/stacks.yaml). To define them differently, eg to drop a component from a stack or add a stack, put a stacks.yaml in the config dir ($AFON_HOME/config). It replaces the bundled one entirely, rather than adding to it, so start from a copy of it, which this exports to the config dir (without overwriting an existing file):
afon-cli.sh validate-stack --export-stacks
Then edit $AFON_HOME/config/stacks.yaml. Nothing else needs to change: validate-stack, manage-stack and its docs (manage-stack <action> <stack-id> -h) take their stacks from it.
Each stack is <cloud>-<name>, and the ordered list of the components it deploys, each after those before it:
stacks:
aws-lab:
role: tre # optional, see below: the role the stack plays in its cloud, if any
pinnedEgressIp: true # optional, default true (or as the stack it extends): holds <CLOUD>_EGRESS_IP, eg AWS_EGRESS_IP, which only one running stack can
components:
- name: Keycloak # deployed unless AWS_KEYCLOAK_EXTERNAL=true, ie the property <CLOUD>_KEYCLOAK_EXTERNAL
unless: KEYCLOAK_EXTERNAL # or 'when', to deploy it only if the property is true
- Polaris
- Jupyter
- NetworkPolicies
- SparkOperator
- Spark
- CrdCleanup
aws-lab-trino:
extends: aws-lab # starts with the components of aws-lab, which must be of the same cloud
components: [Trino]
A stack may have a role, one of catalog (the catalog provider, eg Polaris), identity (the identity provider, eg KeyCloak) or tre (the trusted research environment, ie all the components). The code that depends on what a stack is for, rather than on its id, looks for the stack of the cloud with the role: the flows of manage-stack run (eg cross-cloud-azure sets up the tre stack of AWS and the catalog stack of Azure), and the checks that say which stack to set up (eg for KeyCloak to answer). So the ids are free to name as you like, but a cloud has at most one stack per role, and a flow whose stacks the file does not define cannot run. A stack with no role, like aws-lab, is managed all the same; the role is not inherited by a stack that extends another.
The components are those of the code, eg Polaris, Keycloak, Jupyter, Spark, Trino; a file cannot define a new one. The stacks of a cloud are independent of those of the others.
A stack is validated when loaded, and every command fails before doing anything, naming the file and the entry, if the file is not valid: a YAML syntax error, an unknown component, cloud or property, a malformed or repeated stack id, an extends of an unknown stack, of another cloud or in a cycle, or components composed wrongly (a component repeated, or missing one it needs before it).
Try the new stack, which like the others conflicts with those holding the same domain names or egress IP:
afon-cli.sh validate-stack aws-lab # synthesises its plan, and checks that it is the same in each JVM
afon-cli.sh manage-stack setup aws-lab
validate-stack prints the role of the stack, if it has one, and its components as the file defines them, for the config (those left out by a when or unless property are not listed), and without a stack lists the stacks of the file.
Define your own Docker images
The Docker images that the stacks deploy (Polaris, Keycloak, Spark, JupyterHub, Guacamole, …) are defined by docker-images.yaml, bundled in the jar, so that a version bump or an image pinned by digest is a change of one file, not of the code. As with the stacks, a docker-images.yaml in the config dir ($AFON_HOME/config) replaces the bundled one entirely, rather than adding to it, so start from a copy of it, which this exports to the config dir (without overwriting an existing file):
afon-cli.sh docker-image --export
Then edit $AFON_HOME/config/docker-images.yaml. A file that is not valid, eg an unknown key or category, a malformed digest, or an image the code needs that it does not define, fails the commands that use it, naming the file and the problem.
Each image is under the name the code looks it up by:
images:
keycloak:
image: quay.io/keycloak/keycloak # the repository, without the tag
tag: 26.6.3
digest: sha256:<64 hex digits> # optional: pins the image, which is then pulled as image:tag@digest
category: upstream # optional: first-party, upstream or third-party
polaris:
image: apache/polaris
tag: ${polaris.version} # the version of Polaris that the project builds against
The names are those of the bundled file, which the code needs all of; an entry of another name is accepted but nothing uses it. The trino entry only sets the image tag, the image being that of the vendored Helm chart.
A shell script that needs the reference of an image asks the ops command docker-image <name>, eg afon-cli.sh docker-image postgres-alpine prints AFON_DOCKER_IMAGE=postgres:16-alpine, as scripts/stack/docker-image-source.sh does for kube-polaris-db. (Run by afon.sh, which has no JVM, the script asks it the same way, a Maven run, so a few seconds slower.) The system property afon.docker-images names a file to use instead, or bundled for only the bundled one.
Image pinning
The bundled images are pinned by digest, so that what is pulled is what was pinned, not whatever the tag points to by then (the digest only fixes the content, it does not say who built it: that is what a signature is for). The digest is that of the manifest list, so that it holds for each platform. An image is pulled by its digest, the tag then only documenting it, so when bumping a tag, update the digest with it, or the old version is still pulled. docker-image --pin looks up in the registries the digest that each tag points to and writes it to the file, leaving the rest of the file, comments included, as it is (it needs Docker and network access, and writes nothing if a lookup fails):
afon-cli.sh docker-image --pin keycloak # after changing its tag: pins it, adding the digest if it has none
afon-cli.sh docker-image --pin # refreshes the digest of each image that has one
It pins the docker-images.yaml of the config dir, so export the bundled one to it first (docker-image --export), unless --file gives another. An image that is not to be pinned says why under unpinned, eg unpinned: only its tag is used, the image being that of the Helm chart, in place of its digest: --pin refuses it with that reason, and --verify lists it as not checked with the reason. docker-image --verify only checks, writing nothing, that the digest of each image that has one is that its tag points to now, and exits with 1, naming the images, if not: that is, if a tag was changed without its digest, or published again by its registry. It checks the images in effect, or those of --file, or the named ones:
afon-cli.sh docker-image --verify
afon-cli.sh docker-image --verify --report verify.md # also writes a report, as Markdown, or as JSON if named *.json
The report is written even when the check fails, for a CI job to keep: it has the result, each image checked with its status (ok, mismatch, unpinned or error) and the pinned and registry digests, and the images not checked with the reason. The Polaris images are unpinned, as their tag is the version of Polaris the project builds against, which a pinned digest of another version would outlive, and so is the Trino one, only its tag being set, the image being that of the Helm chart.
To check instead that the images are signed, see docker-image --verify --signatures in Image signing.
The images that the Helm charts of the stacks pull are the charts’ own, so they are not in the file. docker-image --charts reports them, from the plans synthesised for the stacks (validate-stack <stack-id> makes one), so with the values the stack gives each chart, eg the Trino tag it overrides: it renders every chart with helm template, which needs helm installed but no cluster, and lists the images of each release, saying if each is pinned by digest, has a tag only or none, and which the file lists. The images that only helm test pods use are counted, not listed.
afon-cli.sh docker-image --charts aws-full azure-full
afon-cli.sh docker-image --charts --report charts.md # the stacks that are synthesised, also as a report, JSON if named *.json
Image signing
Sign the images
docker-image --sign signs the images with cosign, so that the cluster can be made to admit only images that we signed. Each signature covers the digest the image is pinned to, not its tag, so an image with no digest cannot be signed, and nothing is done if any of the named images cannot be.
It works for one cloud, --cloud aws|azure|gcp or else AFON_CLOUD, whose properties give:
<CLOUD>_IMAGE_REGISTRY: the registry, with the path in it, to keep the images in, eg123456789012.dkr.ecr.eu-west-2.amazonaws.com/emrys;<CLOUD>_COSIGN_KEY: the reference of a key in the KMS of the cloud, ascosign sign --keytakes it, egawskms:///alias/afon-image-signing, which only the deployer role should be able to use.
The first-party images are signed where they are. The others (upstream and third-party) are first copied to the registry: the exact content that their pinned digest identifies, whatever their tag points to by now. The copy goes under the path of the image without its own registry (quay.io/keycloak/keycloak becomes <registry>/keycloak/keycloak) and is signed, after checking that it has the digest of the source. So what the cluster pulls is in our registry, signed with our key.
It needs Docker, cosign 3 (the flags are its own), network access and the credentials of the registry and the key. For a key file, its password goes in COSIGN_PASSWORD, empty if it has none, as the command asks for no input.
The signatures are not uploaded to a transparency log, and are stored in the legacy layout, a sha256-<digest>.sig tag of the repository, which Kyverno reads, rather than cosign 3’s new bundle format.
--report writes a record of the signing, for a CI job to keep: the cloud, registry and key reference, when, and each image signed with its source, the reference of the image that was signed, with its digest, and whether it was copied. It is written even when signing stops on an error, with what was signed before.
afon-cli.sh docker-image --sign --cloud aws # every image that has a digest
afon-cli.sh docker-image --sign --cloud aws keycloak # one image
afon-cli.sh docker-image --sign --cloud aws --report signed.md # also records the result, as Markdown, or JSON if named *.json
To check that the images are signed, without a cluster, docker-image --verify --signatures runs cosign verify on each image with the public key of the cloud, <CLOUD>_COSIGN_PUBLIC_KEY, where --sign put the signature: the image itself if it is first-party, else its copy in the registry. It needs cosign and access to the registry, not the private key. Every image is checked even if one fails, and it exits with 1, naming the images and why (not signed, or signed with another key), if any does, so a CI job can run it before a deploy:
afon-cli.sh docker-image --verify --signatures --cloud aws # every image that has a digest
afon-cli.sh docker-image --verify --signatures --cloud aws keycloak # one image
afon-cli.sh docker-image --verify --signatures --cloud aws --report signatures.md # also writes a report, as Markdown, or JSON if named *.json
Verify the signatures in the cluster
The Kyverno component of the full stacks installs Kyverno in the kyverno namespace, and applies two ImageValidatingPolicys that check that the images of the pods, the containers and the initContainers, are signed with the key of the cloud. Which namespaces are checked, and how, is decided by the label afon.emrys.health/verify-images of the namespace:
| Label | Policy | An image that is not signed | Kyverno down |
|---|---|---|---|
audit | emrys-verify-images-audit | is recorded in the policy reports (kubectl get policyreport -A), the pod is admitted | pods are admitted (failurePolicy: Ignore) |
enforce | emrys-verify-images-enforce | the pod is rejected | pods are rejected (failurePolicy: Fail), as the image cannot be checked |
| none | not checked, as for the namespaces of Kyverno and the system |
The default namespace, where the workloads run, is labelled audit, so the stacks only audit by default. To enforce in a namespace, list it in <CLOUD>_IMAGE_POLICY_ENFORCED_NAMESPACES (comma-separated, eg default) and deploy the stack again, or relabel it:
kubectl label namespace default afon.emrys.health/verify-images=enforce --overwrite # audit -> enforce, one namespace at a time
Audit the images of a namespace until the reports have no violations (every image of its pods mirrored and signed, see docker-image --sign) before enforcing it. Break-glass: if pods are rejected that must run, remove the label of the namespace, which stops it being checked, and put it back after:
kubectl label namespace default afon.emrys.health/verify-images-
If Kyverno itself is down, an enforced namespace rejects its pods too, but the namespace of Kyverno is never checked, so it can recover; to let the pods in meanwhile, remove the label, or delete the policy with kubectl delete imagevalidatingpolicies.policies.kyverno.io emrys-verify-images-enforce.
The public key
afon-cli.sh signing-public-key <aws|azure|gcp> sets it for you: it generates a key pair (once, in $AFON_HOME/cosign/<cloud>) unless <CLOUD>_COSIGN_KEY is set, in which case it uses that key, and writes the public key to <CLOUD>_COSIGN_PUBLIC_KEY in afon-<cloud>.properties, keeping a .bak copy, with --print only printing the line. A generated key has no password unless COSIGN_PASSWORD is set, and a key file of <CLOUD>_COSIGN_KEY needs its password in COSIGN_PASSWORD (empty if it has none). It reads the config as text, so it also repairs a config where the property was written as a PEM with spaces, as it used to be, which the scripts cannot source.
Kyverno verifies with the public key alone, <CLOUD>_COSIGN_PUBLIC_KEY, so it needs no access to the key. docker-image --public-key prints it from <CLOUD>_COSIGN_KEY, as the line to put in the config, the base64 of the key (a single word, as the scripts source the config files and a PEM, with its spaces, would break them):
afon-cli.sh docker-image --public-key --cloud aws # AWS_COSIGN_PUBLIC_KEY=MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
Without that property Kyverno is installed and no policy is applied. The signatures are not uploaded to a transparency log, so to verify an image the admission controller only needs to reach the registry (and, as always, DNS and the API server), not the KMS or a log. That is what an egress-restricted TRE must allow it: the acceptance test restricts it to exactly that with a NetworkPolicy and checks that images are still verified (and, without the registry, that they are not).
The signing report
afon-cli.sh signing-report reads the policy reports that Kyverno writes on the current kubectl context (it needs kubectl and jq), and tells which pods the policies checked, whether each passed, and with --images, the images of the pods that failed, with the number of pods: that is what is left to sign. The pods are those of the namespaces labelled afon.emrys.health/verify-images, and Kyverno finds them by scanning the namespace in the background, so it also reports the pods that were running before the policy, within moments.
afon-cli.sh signing-report # the pods checked and their images, in all the namespaces
afon-cli.sh signing-report default --images # only the images of the pods in default that failed
afon-cli.sh signing-report --json | jq .summary
A pod that failed may have images that are signed: the report is of the pod, not of each image. The reports of the controllers (Deployments, ReplicaSets…) repeat those of their pods, so they are left out.
The ephemeralContainers are not checked yet, and images that no one has signed are reported, such as those of the Helm charts, until they are mirrored and signed.
Setup AWS full stack
The AWS full stack includes Polaris, Spark (via Spark Connect), and Jupyter (behind Guacamole with OAuth2).
afon-cli.sh manage-stack setup aws-full
Steps executed:
delete_stacks_conflicting_with_aws_full— delete the AWS stacks conflicting with it, ie any existing AWS stackdeploy_aws_full_stack— deploy AWS full stackwait_on_emryspolaris— wait for DNS to propagatecreate_all_catalogs_for_cloud_aws— create all catalogs in AWS Polarisaws_spark_talks_to_cloud_aws— smoke test: Spark writes and reads via Polaris
Access Polaris and Jupyter
For development purposes, we’ve registered for aws domains emryspolaris.click and emrysjupyter.click.
Once the AWS full stack is deployed, we can access Polaris and Jupyter via the URLs:
https://emryspolaris.clickhttps://emrysjupyter.click/guacamole
Deploy AWS Keycloak only
The AWS Keycloak stack deploys only the Keycloak server, ie the EKS cluster with Keycloak behind its HTTPS ingress at OIDC_DOMAIN_NAME, without Polaris, eg to serve as a standalone OIDC provider, such as for aws-polaris, which runs side by side with it.
afon-cli.sh manage-stack deploy aws-keycloak
Steps: delete_stacks_conflicting_with_aws_keycloak → deploy_aws_keycloak_stack → wait_on_emrysoauth → deploy_complete_for_aws_keycloak
It requires the Keycloak properties (OIDC_DOMAIN_NAME, KEYCLOAK_*) and the AWS cluster properties, but none of the AWS_POLARIS_* properties. Once deployed, you can check the OIDC provider is ready with afon-cli.sh check-oidc <cloud>.
Setup AWS Keycloak only
To deploy the AWS Keycloak stack and configure it with the realm, the POLARIS_ID client and the users (and their confidential clients), plus the Jupyter browser client if AWS_JUPYTER_DOMAIN_NAME is set:
afon-cli.sh manage-stack setup aws-keycloak
Steps: delete_stacks_conflicting_with_aws_keycloak → deploy_aws_keycloak_stack → wait_on_emrysoauth → configure_oidc_users_for_cloud_aws → setup_complete_for_aws_keycloak
Setup Azure full stack
afon-cli.sh manage-stack setup azure-full
NOTE: azure-full does not fit on the default nodes (3 × Standard_D2s_v3), nor on 4 of them: pods such as the Spark Connect driver stay Pending (Insufficient cpu). Set larger nodes in afon-azure.properties first, eg AZURE_AKS_NODE_POOL=vmSize:Standard_D4s_v6,nodeCount:2, and check with afon-cli.sh kube-estimator azure-full <nodes>.
NOTE: azure-full deploys the network policies, which limit what the notebooks and the Spark master and driver can resolve to the domains of whitelist.yaml. Its azure section, as in the template, lists the Azure names: the OIDC, Polaris and Jupyter domain names and ${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net, else Spark fails with `UnknownHostException: