Aji Arya

Everything a post can render

2026-11-07T00:00:00+00:00

This post exercises everything the site can generate. Use it as a reference when you write, and as a visual test when you change the styles.

## Headings

The post title at the top of this page is the only heading level 1. Each page keeps one h1, so a post body starts at level 2. The table of contents lists level 2 headings only.

## Heading level 3

Level 3 starts a subsection.

## Heading level 4

Level 4 divides a subsection.

## Heading level 5

Level 5 is for short labelled groups.

## Heading level 6

Level 6 is the smallest heading.

## Text formats

A paragraph can mix bold, italic, bold italic, ~~strikethrough~~, underline, inline code, and a link to the writing page. External links, such as Zola, open in a new tab.

Format	Source	Result
Bold	`text`	text
Italic	`text`	text
Bold italic	`*text*`	text
Strikethrough	`~~text~~`	~~text~~
Underline	`<u>text</u>`	text
Superscript	`x<sup>2</sup>`	x²
Subscript	`H<sub>2</sub>O`	H₂O
Highlight	`<mark>text</mark>`	text
Small	`<small>text</small>`	text
Keyboard	`<kbd>Ctrl</kbd>`	`Ctrl`
Abbreviation	`<abbr title="Object Storage Daemon">OSD</abbr>`	OSD
Inline code	`code`	`code`

Smart punctuation converts “straight quotes”, dashes – like this – and ellipses… as you type.

A footnote keeps a side comment out of the sentence.^[1] It also works with a named label.^[2]

## Lists

An unordered list can nest:

Linux hosts
OpenStack control planes
- Nova
- Neutron
Ceph storage clusters

An ordered list is a numbered checklist:

Map the system.
Define done.
Keep a rollback path.

A task list tracks state:

Write the post
Render the diagrams
Replace the demo content

A definition list pairs a term with its meaning:

Failure domain: A set of components that can fail together.
Placement group: A Ceph unit that maps objects to OSDs.

## Special blocks

## Markdown alerts

Five alert types use standard Markdown syntax, for example > [!NOTE].

A note adds useful context that a reader can skip.

A tip shows an easier way to do something.

An important block states what a reader must know to succeed.

A warning flags a risk that needs attention now.

A caution flags an action with negative consequences.

## Notice blocks

Four more types use the notice component. Their body accepts Markdown.

## Blockquote

A plain blockquote carries a key statement:

A boring upgrade is built in advance: map the system, write down what success means, and rehearse the failures.

## Code

A code block sets its language for syntax highlighting:

fn main() {
    println!("Calm infrastructure is designed.");
}

nodes:
  - id: client
    label: Client
    tone: blue

cargo run -- diagrams check

A code block can also show line numbers and highlight lines:

1[extra]
2demo = true
3highlight = "a post"

A long line scrolls inside the code box, and the language label and Copy button stay in place:

openstack --os-cloud production-region-one compute service set --disable --disable-reason "planned-maintenance-window" compute-node-0042.example.internal nova-compute

## Tables and media

Table columns can align left, centre, or right. Wide tables scroll inside their own box.

Component	Role	Replicas
MON	Quorum	3
OSD	Data	6
RGW	Gateway	2

An image can sit on its own or in a figure with a caption:

A figure with a caption, using the site footer art.

A horizontal rule separates two parts of a post:

## Diagrams

Each diagram compiles from a YAML file in diagrams/. Every diagram has a toolbar to zoom, fit, reset, and copy as PNG. Animated diagrams also have play, speed, and direction controls.

## Detailed style

The detailed style is the default. This is the smallest useful diagram:

Minimal diagram: two nodes and one labelled edge, with automatic left-to-right layout.

Source: diagrams/showcase-minimal.yaml

Edges have three line styles, three routes, and three arrow options:

Edge options: solid, dashed, and dotted lines; elbow, straight, and curved routes; arrows on both ends or none. The default arrow is on the end only.

Source: diagrams/showcase-edges.yaml

Edges have four animation types, and flow has three directions:

Animation options: none, draw, pulse, and flow in alternating, forward, and reverse directions.

Source: diagrams/showcase-animation.yaml

Manual layout places each node at explicit coordinates. This diagram also shows all eight tones and both node shapes:

Manual layout: the eight tones, box and pill shapes, and two groups.

Source: diagrams/showcase-manual.yaml

Automatic layout can also run top to bottom, with rank overrides and groups:

Top-to-bottom layout: groups and rank overrides in a real diagram.

Source: diagrams/ceph-data-path.yaml

## Architecture detailed style

The architecture-detailed style draws physical infrastructure with switch, server, and VM nodes, NIC and switch port grids, port-to-port links, and named bonds. Hover a node to trace its links.

Architecture detailed: two switches, a server with two bonds, and a VM, linked port to port.

Source: diagrams/showcase-arch-detailed.yaml

## Architecture simple style

The architecture-simple style draws logical network membership. Networks are rails and nodes attach to them. Hover a legend entry to highlight its network.

Architecture simple: three logical networks as rails, with no ports or physical links.

Source: diagrams/showcase-arch-simple.yaml

Footnotes collect at the end of the post, with a link back to the reference. ↩
The label in the source does not appear in the output. The site numbers footnotes in order. ↩

A Ceph storage fabric with dual-leaf redundancy

2026-10-28T00:00:00+00:00

A Ceph cluster is only as reliable as the fabric underneath it. The monitor quorum needs low, predictable latency, and the OSDs need enough independent paths that losing a NIC, a cable, or a leaf switch does not turn into a storage outage.

The logical view is deliberately simpler: it shows which hosts participate in the Ceph public network and which participate in the cluster network. Monitors stay on the public network; OSDs use both.

Ceph logical networks: the public network connects clients, monitors, and OSDs, while the cluster network carries OSD replication and recovery traffic.

Source: diagrams/ceph-storage-fabric-simple.yaml

The physical view below expands that into NICs, ports, leaf switches, and individual links.

Three MON hosts and three OSD hosts, each with two dual-port NICs, dual-homed to two leaf switches. bond0 and bond1 are MLAG bonds; each one has a member on leaf-01 and leaf-02.

Source: diagrams/ceph-storage-fabric.yaml

## The topology

Six physical hosts sit below a pair of 48-port leaf switches. Each host has two dual-port NICs:

NIC 1 has two ports.
NIC 2 has two ports.
Every port is connected: one port from each NIC goes to leaf-01, and one port from each NIC goes to leaf-02.

That gives every host four independent paths to the fabric, arranged as two two-member bonds:

bond0 = nic1/p1 → leaf-01 + nic2/p1 → leaf-02
bond1 = nic2/p2 → leaf-01 + nic1/p2 → leaf-02

Each bond has one member on each NIC and one member on each leaf. A failed NIC, cable, or leaf removes one member from each bond, not a whole bond. The dashed badges and matching colors in the physical diagram mark the two bond groups.

The leaf pair must be configured as an MLAG (or a stacked pair) so both switches present one logical LAG endpoint. Without that, the two bond members on different leaves would not form a single LAG.

## Logical networks

The public network is the client-facing path: clients reach RGW/MON services and the OSDs accept front-end traffic. The cluster network is the private back-end used by OSDs for heartbeat, object replication, and recovery. Ceph Monitors always operate on the public network, so the dedicated MON hosts are shown only on the public rail.

An OSD host is normally on both networks. In the logical diagram, the two colored rails show that membership directly; hovering a legend entry highlights that network’s rail and stubs. In the physical diagram, the same host has separate NICs and ports for each path.

## Reading the port grid

Each leaf shows its full 48-port grid. Port numbers 1–12 are in use: six hosts contribute two links each. The highlighted ports are the ones carrying links in this diagram.

On a host, the two NIC blocks show the port-level wiring directly:

nic1/1 → leaf-01
nic1/2 → leaf-02
nic2/1 → leaf-02
nic2/2 → leaf-01

Port cells and links are colored by bond: bond0 is blue, bond1 is amber. Each bond gets a dashed badge below its host, labeled bond0 or bond1, so the bond grouping is visible without hiding the individual NIC/port mapping. The NIC labels still identify nic1 and nic2; the port numbers identify the member port. Hovering any host highlights its four links and both bond badges, and dims the rest; the diagram is also pan/zoomable for detail.

## What I check first

MLAG health on the leaf pair. A split MLAG is worse than a single switch.
Bond member state on every host. A bond that has silently lost half its members still passes traffic, but no longer has the redundancy you designed.
LACP hashing and link utilization. Four links do not help if one flow can only use one member.
Monitor network latency. Keep the MON quorum on the lowest-latency path, and do not let bulk OSD recovery saturate it.
Failure-domain reality. Two leaf switches are not two failure domains if they share power, management, or an upstream link.

Operating OpenStack without fear

2026-10-26T00:00:00+00:00

OpenStack upgrades have a reputation for drama. Most of the drama I have seen did not come from the upgrade itself; it came from the assumptions around it — undocumented local changes, untested database migrations, and a rollback plan that existed only in someone’s head.

## Start with a map

Before touching a control-plane node, draw the system as it actually is:

Which services run on which hosts?
Which database and message queue do they depend on?
Where does the API traffic enter, and what checks the health of each endpoint?
Which tenant networks cross which physical interfaces?
What is the storage path for a volume that is attached to a running instance?

The drawing does not need to be beautiful. It needs to be true. A good diagram makes the upgrade order obvious, and it exposes the components that have no obvious owner.

## Define done

An upgrade is not done when the packages install. It is done when:

Every API endpoint reports the expected version.
A new instance can boot, attach a volume, and reach the network.
An existing instance survives a live migration.
The control-plane services survive a restart of one node.
The monitoring is quiet for the right reasons, not because it stopped collecting.

## Keep a rollback path

A rollback is not a time machine. Database migrations, message queue schemas, and agent versions can make a clean rollback impossible. Decide in advance which parts are reversible, which parts are forward-only, and which parts need a maintenance window.

## The habit

The best upgrade is the one that feels boring. That does not happen by accident. It happens when the system is mapped, the success criteria are written down, and the team has practiced the failure modes before they are real.

That is the work I care about: not making infrastructure look complicated, but making it legible enough to operate calmly.

Ceph’s data path is simpler than its operations

2026-10-19T00:00:00+00:00

Ceph can feel like a system with too many daemons. The data path, though, is remarkably direct: clients authenticate, receive a cluster map, and then talk to the OSDs that own the objects.

The Ceph object data path: authentication through MON, placement through the cluster map, and I/O directly to OSDs.

Source: diagrams/ceph-data-path.yaml

## The parts that matter

The monitor quorum is the source of truth for cluster membership and the cluster map. If the monitors cannot agree, clients cannot get a current view of the cluster. That is why monitor latency and quorum health deserve attention before the first OSD fills up.

The manager is not in the I/O path. It collects telemetry, exposes the dashboard, and runs modules. Losing it should not stop writes; losing the monitor quorum will.

RGW is the object gateway. It authenticates S3 and Swift requests and translates them into RADOS operations. Once a client has a map, reads and writes go to OSDs, not through the gateway again.

## The operational surface

The complexity is in placement, recovery, and failure domains:

Placement groups decide how objects map to OSDs. A poorly planned pool can make recovery slow or impossible to reason about.
Failure domains are only useful if they match reality. A “rack” that shares a power feed is not a failure domain.
Recovery is a first-class workload. A cluster that is healthy at steady state can still be unusable during a rebuild.
BlueStore puts data and metadata on the same device. That is simpler to operate, but capacity planning still needs headroom.

## What I check first

ceph -s
ceph osd df tree
ceph pg stat
ceph health detail

The first command tells me whether the cluster is healthy. The second tells me whether the capacity headroom is real. The third tells me whether recovery is making progress. The fourth tells me whether the health is a warning I can ignore or a symptom of the next outage.

The diagram does not make Ceph simple. It makes the conversation honest: the data path is short, and the operational work is everything around it.

The Kubernetes control plane, as it actually runs on OpenStack

2026-10-12T00:00:00+00:00

Most Kubernetes diagrams start at the API server. That is a reasonable place to start reading, but it is not where a request starts, and it is not where the interesting failures live.

Here is the path I use when I need to explain a cluster to someone who will operate it:

A request path through Kubernetes on OpenStack, ending at Ceph-backed storage.

Source: diagrams/k8s-control-plane.yaml

## Follow the request path

The client speaks HTTPS to an ingress gateway. The gateway terminates TLS and forwards to the API server. The API server authenticates the request, applies admission, and writes the resulting object to etcd.

From there, the control plane is a set of watches and reconciliations. The scheduler binds a pod to a node. The controller manager notices that the desired state and observed state disagree. The kubelet on the chosen node sees the pod, asks the container runtime to start it, and reports status back.

The storage path is separate. The Ceph CSI driver turns a persistent volume claim into a RADOS-backed volume. The kubelet mounts it. The application writes to a filesystem that is actually replicated across OSDs.

## What the diagram leaves out

It leaves out the parts that make production interesting:

etcd latency. A slow etcd is a slow API server. Everything upstream inherits that latency.
The network between the control plane and the workers. Overlay networks, MTU, and security groups are not visible in a box-and-arrow diagram, but they decide whether the line is real.
The difference between “the API server is up” and “the cluster is healthy.” A control plane can accept writes while nothing schedules.
Storage recovery. Ceph may be healthy from Kubernetes’ point of view while it is busy rebuilding.

## Three failure modes I watch

A control-plane node loses its etcd member. The remaining members keep quorum, but the API server on the failed node is now a trap for load balancers that only check TCP.
The CSI driver cannot reach the Ceph monitors. Pods stay pending, or worse, existing volumes fail to mount on a new node.
A worker has a full disk. The kubelet evicts pods, but the container runtime and the Ceph OSD may disagree about what “full” means.

The diagram is useful because it gives us a shared vocabulary for those conversations. It is not useful if we pretend the arrows are guarantees.

## A small operational rule

When a request is slow, walk the path once and write down the latency budget at each hop. When a request fails, walk it again and write down the failure domain at each hop. The boxes do not change; the questions do.

# A quick sanity check from a worker node
curl -sk https://kubernetes.default.svc/healthz
systemctl is-active kubelet containerd
ceph -s

The next step is not another dashboard. It is a runbook that says what to check first, what to ignore, and who owns the boundary between the platform and the storage cluster.