Kubernetes API: freckle.id sign-in¶
Status: proposal
Nothing here is applied yet. This page is the design for review; the rollout at the end starts once it's approved.
Both clusters' API servers learn to accept ID tokens issued by freckle.id
(Pocket ID), so people can run kubectl as their freckle.id user. RBAC binds
to freckle.id groups, so a membership change in Pocket ID takes effect on
the next token, with no change in git.
This is additive. The Tailscale API-server proxies stay exactly as they are, and so do the Talos admin kubeconfigs.
People only, by design
freckle.id sign-in on the API servers is for people. Agents never touch the Kubernetes API: they use MCP tools. The MCP servers that do call the API (the cluster operate server, C-436, and the observe reads) use plain in-cluster ServiceAccounts in the cluster they serve, with ClusterRoles/Roles as their personas. So there are no persona clients, federated identities or API resources in Pocket ID for Kubernetes.
What exists today¶
| fairy-k8s01 | atlantis-k8s01 | |
|---|---|---|
| Kubernetes / Talos | v1.37.2 / v1.14.5 | v1.37.2 / v1.14.5 |
| API endpoint | https://api.fairy-k8s01.freckle.services:6443 |
https://api.atlantis-k8s01.freckle.services:6443 |
| Tailscale API-server proxy | ProxyGroup fairy-k8s01-kube-apiserver, mode: auth |
ProxyGroup atlantis-k8s01-kube-apiserver, mode: auth |
| API server config | KubeAPIServerConfig document (talos/control-plane/01-cluster.yaml) |
same |
freckle.id runs Pocket ID v2.19.1 on atlantis (freckle-id namespace),
served at https://freckle.id on the public gateway with a publicly
trusted certificate. OIDC clients and groups are managed in OpenTofu
(frecklelabs/tofu platform/) with the trozz/pocketid provider 2.5.0;
group membership is set in the Pocket ID admin UI.
What sign-in needs from Pocket ID is all standard OIDC, already used by other apps here:
- Public clients with PKCE, so
kubectl oidc-loginneeds no client secret. - ID token
audis the client ID. kubelogin hands the API server the ID token, so the audience check is just the cluster's client ID. groupsclaim with Pocket ID group names, when thegroupsscope is requested (paperless, nautobot and the cockpit read it today).allowed_user_groupson the client, so onlyk8s-adminsmembers can get a token at all.
No Pocket ID upgrade is needed.
Pocket ID clients and group¶
| Object | Settings |
|---|---|
Group k8s-admins |
Members: nicole and alt-nicole. Kubernetes is only ever used as an admin, so the everyday account is in it on purpose. |
Client kubernetes-fairy |
Custom client_id, is_public = true, pkce_enabled = true, callbacks http://localhost:8000 and http://localhost:18000 (kubelogin's defaults), allowed_user_groups = [k8s-admins]. |
Client kubernetes-atlantis |
Same, for atlantis. |
One client per cluster means a token's aud names exactly one cluster, so a
fairy token is never accepted by atlantis. A k8s-viewers group isn't
created; it can be added later if something needs read-only people access.
API server configuration¶
Use Kubernetes structured authentication configuration, not the legacy
--oidc-* flags. Talos 1.14 has a first-class document for it,
KubeAuthenticationConfig. Both clusters already use the KubeAPIServerConfig
document, so Talos already runs the API server with --authentication-config
(today holding only the anonymous health-check rules). Adding the JWT
authenticator is a change to that document, which Talos applies live.
# kubernetes/clusters/fairy-k8s01/talos/control-plane/03-authentication.yaml
# (atlantis: the same file with kubernetes-atlantis)
---
apiVersion: v1alpha1
kind: KubeAuthenticationConfig
configuration:
# Talos's default: unauthenticated health checks only. Restated so the
# patch can't drop it.
anonymous:
enabled: true
conditions:
- path: /livez
- path: /readyz
- path: /healthz
jwt:
- issuer:
url: https://freckle.id
audiences:
- kubernetes-fairy
claimValidationRules:
- expression: "has(claims.preferred_username) && claims.preferred_username != ''"
message: freckle.id token has no preferred_username
claimMappings:
username:
expression: "'freckle-id:' + claims.preferred_username"
groups:
expression: "has(claims.groups) ? claims.groups.map(g, 'freckle-id:' + g) : []"
uid:
expression: claims.sub
- Names. A person is
freckle-id:<preferred_username>and their groups arefreckle-id:<group>. The prefix keeps them apart fromsystem:names, ServiceAccounts and the Tailscale proxy's identities.uidis the stable Pocket ID user ID, for the audit log. - Audience. Only this cluster's client is listed. With a single audience,
audienceMatchPolicyis not needed. - Prefixes and expressions. CEL expression mappings carry no
prefixfield; the prefix is part of the expression. - Rendering. Talos writes the file itself (with the
apiserver.config.k8s.io/v1beta1type header, which 1.37 still serves; CEL mappings are in v1beta1). Before applying, diff the rendered config (topfdiff ortalosctl apply-config --dry-run) and check that theanonymousblock and thejwtentry both survive the merge.
Issuer reachability, TLS and break-glass¶
- The issuer is the public
https://freckle.id, the URL every other client uses. Its certificate is publicly trusted, so the config sets nocertificateAuthority. fairy fetches discovery and JWKS over the internet; atlantis reaches its own public gateway the way its pods already do. - An unreachable issuer doesn't stop the API server: it starts and serves
every other authenticator, and only freckle.id logins fail. Tokens already
issued keep validating against the cached JWKS until they expire. Check
after the first node with the API server log and the
apiserver_authentication_jwt_authenticator_*metrics. - Break-glass: atlantis depending on a workload it runs is known and
accepted. The Talos
admin@<cluster>kubeconfigs, and the Talos configs that can mint a new one (talosctl kubeconfig), don't depend on freckle.id.
How people reach the API server¶
Clients reach https://api.<cluster>.freckle.services:6443 directly by
default. That name is the cluster endpoint, so it is already in the API
server certificate, and the kubeconfig trusts the cluster CA (the same
certificate-authority-data as the Talos admin kubeconfig). The network
path to it is Nicole's to arrange; this design adds no Tailscale ProxyGroup,
tag or grant.
The Tailscale API-server proxies (<cluster>-apiserver.<tailnet>, auth
mode) are unchanged. An auth-mode proxy impersonates the tailnet identity
and doesn't forward the caller's bearer token, so freckle.id contexts don't
go through it.
RBAC in git¶
A shared app dir kubernetes/apps/auth/kubernetes-rbac/, with a Flux
Kustomization pointing at it from each cluster's kube-system apps:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: freckle-id-k8s-admins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: freckle-id:k8s-admins
Bindings name groups only, never freckle-id:<user>, so who has access is
decided by group membership in Pocket ID. The binding is inert until the API
servers trust the issuer, so it can merge first.
Nicole's kubeconfig¶
New contexts sit next to the existing ones:
clusters:
- name: fairy-k8s01-direct
cluster:
server: https://api.fairy-k8s01.freckle.services:6443
certificate-authority-data: <cluster CA, from the Talos admin kubeconfig>
users:
- name: freckle-id-fairy
user:
exec:
apiVersion: client.authentication.k8s.io/v1
command: kubectl
args:
- oidc-login
- get-token
- --oidc-issuer-url=https://freckle.id
- --oidc-client-id=kubernetes-fairy
- --oidc-extra-scope=profile
- --oidc-extra-scope=groups
contexts:
- name: fairy-freckle-id
context:
cluster: fairy-k8s01-direct
user: freckle-id-fairy
Rollout (after approval)¶
- Pocket ID (frecklelabs/tofu PR, Nicole reviews): group
k8s-admins, clientskubernetes-fairyandkubernetes-atlantis. After apply, Nicole addsnicoleandalt-nicoleto the group in the admin UI. - Probe the kubectl login on fairy before the API server trusts
anything:
kubectl oidc-login setup --oidc-issuer-url=https://freckle.id --oidc-client-id=kubernetes-fairy --oidc-extra-scope=profile --oidc-extra-scope=groups, and check the decoded claims (never the token itself):iss,aud = kubernetes-fairy,preferred_username, andgroupscontainingk8s-admins. Adjust the mapping if a claim differs. - RBAC (public repo, merges on green): the shared dir and both pointers.
- Talos patch for fairy (public repo):
03-authentication.yaml. Nicole applies it one control-plane node at a time with topf, checkingkubectl get --raw /readyzwith the admin context and the API server log after each node. Thenkubectl --context fairy-freckle-id auth whoamishowsfreckle-id:<user>withfreckle-id:k8s-admins. - atlantis: the same patch with
kubernetes-atlantis, applied the same way, then the sameauth whoamicheck. A fairy token sent to atlantis's API server gets a 401.