Changing the cluster¶
INSERT, UPDATE and DELETE are real API calls: each row is one create,
update or delete, sent as the gateway's identity and checked by the API server
like any other client's. This page shows them working, with output from a CI run against the shop
namespace the Examples use.
Change a ConfigMap¶
UPDATE k8s.core_configmaps
SET data = data || '{"LOG_LEVEL":"debug"}'
WHERE namespace = 'shop' AND name = 'checkout-config'
RETURNING name, data;
name | data
-----------------+-------------------------------------------
checkout-config | {"CURRENCY": "EUR", "LOG_LEVEL": "debug"}
(1 row)
kubectl sees the change at once, because it is in the cluster, not in
Postgres:
$ kubectl -n shop get configmap checkout-config -o jsonpath='{.data.LOG_LEVEL}'
debug
Insert a whole manifest¶
raw is a whole object on INSERT too. A complete manifest can be inserted
as one jsonb value, and a typed column given beside it overrides the same
field, so raw read from one object is a template for another:
INSERT INTO k8s.core_configmaps (namespace, name, raw) VALUES ('shop', 'flags',
'{"metadata":{"labels":{"team":"payments"}},"data":{"NEW_CHECKOUT":"off"}}')
RETURNING name, labels, data;
-- a copy under a new name, with data overridden
INSERT INTO k8s.core_configmaps (namespace, name, data, raw)
SELECT 'shop', 'flags-canary', '{"NEW_CHECKOUT":"on"}', raw
FROM k8s.core_configmaps WHERE namespace = 'shop' AND name = 'flags'
RETURNING name, labels, data;
name | labels | data
-------+----------------------+-------------------------
flags | {"team": "payments"} | {"NEW_CHECKOUT": "off"}
(1 row)
name | labels | data
--------------+----------------------+------------------------
flags-canary | {"team": "payments"} | {"NEW_CHECKOUT": "on"}
(1 row)
A column the INSERT leaves NULL takes its value from raw. Metadata the API
server assigns (uid, resourceVersion, creationTimestamp, managedFields)
is dropped from raw rather than sent, and a raw whose apiVersion or kind
names another kind is refused rather than relabelled.
Writes are never served from cache and never batched: each row is one call.
What Axiom refuses¶
Pods are read-only at the SQL layer, whatever RBAC allows. Deleting a pod
by WHERE clause is too easy to do by accident and too hard to undo:
DELETE FROM k8s.core_pods WHERE namespace = 'shop' AND name = 'checkout-worker';
psql:<stdin>:1: ERROR: foreign table "core_pods" does not allow deletes
A conflicting write does not silently win. An UPDATE carries the
resourceVersion of the row it read. If something else changed the object in
between, the API server rejects it and the statement fails with 40001, the
SQLSTATE Postgres itself uses for a serialization failure: re-read and retry.
Identity and server-managed fields are refused. Changing name,
namespace, uid or resource_version raises 0A000 rather than doing
something surprising.
Find it and fix it in one statement¶
The shipped RBAC grants writes per resource, and not on Deployments, so grant that first:
kubectl patch clusterrole axiom-gateway --type=json -p='[{"op":"add","path":"/rules/-","value":
{"apiGroups":["apps"],"resources":["deployments"],"verbs":["get","list","watch","update"]}}]'
Then annotate each Deployment in shop using under a fifth of the memory it
requests. It is scoped to one namespace on purpose: drop that line and it
annotates kube-system too.
It reads live usage and live spec, walks pod → ReplicaSet → Deployment, and
writes the answer back:
WITH used AS (
SELECT m.namespace, m.name AS pod, sum(axiom_quantity(c->'usage'->>'memory')) AS bytes
FROM k8s.metrics_k8s_io_pods m, jsonb_array_elements(m.containers) c
GROUP BY 1, 2),
requested AS (
SELECT p.namespace, p.name AS pod,
r.metadata->'ownerReferences'->0->>'name' AS deployment,
sum(axiom_quantity(c->'resources'->'requests'->>'memory')) AS bytes
FROM k8s.core_pods p
JOIN k8s.apps_replicasets r
ON r.namespace = p.namespace AND r.name = p.metadata->'ownerReferences'->0->>'name',
jsonb_array_elements(p.spec->'containers') c
GROUP BY 1, 2, 3),
ratio AS (
SELECT q.namespace, q.deployment, round(100 * sum(u.bytes) / sum(q.bytes)) AS pct
FROM requested q JOIN used u USING (namespace, pod)
GROUP BY 1, 2
HAVING sum(q.bytes) > 0)
UPDATE k8s.apps_deployments d
SET annotations = coalesce(d.annotations, '{}')
|| jsonb_build_object('axiom/memory-used-pct', r.pct::text)
FROM ratio r
WHERE d.namespace = r.namespace AND d.name = r.deployment AND r.pct < 20
AND d.namespace = 'shop' -- one namespace at a time: this writes to the cluster
RETURNING d.namespace, d.name, r.pct;
namespace | name | pct
-----------+---------+-----
shop | catalog | 0
shop | web | 1
(2 rows)
$ kubectl -n shop get deploy catalog -o jsonpath='{.metadata.annotations.axiom/memory-used-pct}'
0
Annotating changes no pod template, so nothing rolls. Rewriting
resources.requests instead would right-size the workload in the same
statement, and would also restart every pod it touched, which is why the
example annotates.