etcfs-cloudformation-template

CloudFormation for provisioning EtcFS clusters on AWS — the CloudFormation counterpart to etcfs-terraform-modules.

One stack builds a complete cluster: an Auto Scaling Group of compute nodes running colocated etcd, a shared io2 Multi-Attach volume they all open as a raw block device, the DynamoDB row they elect a cluster-forming seed through, and the lifecycle hook that lets a node leave without stranding an etcd member behind it.

No dependency on other stacks. It needs an existing VPC, one subnet, and the matching AZ.

Architecture

Why a seed row at all

etcd's --initial-cluster wants the membership known at start-up, and an Auto Scaling Group is the opposite of that: three instances launch at once, none of them knows whether it is the first, and EC2 reports all three running well before any of them has etcd listening on 2379. Picking a peer by instance state alone means picking one with nothing serving yet — and every co-booting node makes that same mistake simultaneously.

So exactly one node has to be elected to form the cluster while the rest join it. The election is a conditional PutItem against a DynamoDB row keyed by cluster name: the write succeeds for one node and fails for everyone else, atomically, with no coordinator. The losers read the row and join through the winner.

That row is a bootstrap hint, not a liveness oracle, and everything subtle about this stack follows from that one distinction.

The recorded seed will eventually be terminated — scaled in, replaced, or simply killed — and the row then names an address nobody answers on. A node booting at that moment must tell apart two situations that look identical from the row alone: a cluster that has not formed yet, and a healthy cluster whose seed has moved on. Reclaiming the row in the second case forms a second etcd cluster against the same block device, which is precisely the split brain the rest of the design exists to prevent.

It tells them apart by asking the survivors directly. That is what the ClusterName and Role=etcfs-node tags are for, and why they are load-bearing rather than descriptive.

How the join protocol stays single-sourced

This template contains no node bootstrap of its own. Its user-data exports the stack's parameters as ETCFS_* environment variables, downloads terraform/modules/etcfs-asg/scripts/node-bootstrap.sh at the ref given by pBootstrapRef, and runs it. That script is plain bash configured entirely through the environment, so nothing has to be rendered or unescaped on the way in — the whole shim is about twenty lines.

Everything that decides how a node joins therefore has one implementation, shared by both deployment paths: the seed election, etcd member add against the elected seed, clearing a crashed node's member entry before adding a replacement, and writing /etc/etcfs/etcfuse-meta.yaml. Discovery logic is where divergence between two copies would be most expensive.

pBootstrapRef defaults to refs/heads/main, which fetches whatever is on that branch at each instance launch. Pin a tag for a reproducible stack.

Requirements

• An existing VPC and a subnet, with the AZ passed separately in pAvailabilityZone. It must be the subnet's own AZ: an io2 Multi-Attach volume is a single-AZ object and every node has to attach the same one, so the whole cluster lands in one subnet.
• CAPABILITY_NAMED_IAM at deploy time. The stack creates IAM roles and an instance profile with explicit names, which CloudFormation refuses to do under plain CAPABILITY_IAM.
• io2 Multi-Attach availability in the target region, and instance types that support it. The template's AllowedValues are all Nitro families, which is the requirement.
• Deploy permissions for the principal running the deployment. AdministratorAccess covers it; a scoped deploy role needs cloudformation:* on the stack plus create, read, update and delete on the resource types the stack owns — IAM roles and instance profiles, EC2 security groups, volumes and launch templates, Auto Scaling groups and lifecycle hooks, DynamoDB tables and SSM parameters — and iam:PassRole for the role it creates.
• Stack name of 48 characters or fewer. See Limitations.

Deploying

Or via CLI:

aws cloudformation deploy \
  --template-file etcfs-asg-template.yml \
  --stack-name etcfs \                      # max 48 chars
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides \
      pVpcId=vpc-0abc \
      pSubnetId=subnet-0abc \
      pAvailabilityZone=eu-west-1a          # must be pSubnetId's AZ

Watch a node come up in /var/log/etcfs-cfn-bootstrap.log (the shim) and /var/log/etcfs-bootstrap.log (the script it runs). A healthy node ends with node <id> up and mounted and has EtcFS on /mnt/etcfuse.

Parameters

Parameter	Required	Default	Description
pVpcId	yes	—	VPC to deploy the cluster into.
pSubnetId	yes	—	Subnet for the ASG. All nodes land here, so it must be in the volume's AZ.
pAvailabilityZone	yes	—	AZ for the io2 Multi-Attach volume. Must match pSubnetId's AZ.
pClusterName	no	stack name	Cluster identity: scopes etcd node IDs, the seed row, and the ClusterName tag peers are discovered by. Two stacks must not share one.
pAsgDesiredCapacity	no	3	Node count. One of 1, 3, 5 — odd, for Raft.
pAsgMinSize	no	3	Floor a scale-in policy may take the group to.
pAsgMaxSize	no	5	Ceiling for scale-out.
pEc2InstanceType	no	t3.large	Node instance type. Restricted to families that support Multi-Attach.
pInstanceAMI	no	AL2023	Resolved from SSM Public Parameter Store; tracks the latest Amazon Linux 2023 AMI.
pRootVolumeSize	no	20	Root gp3 volume size in GB. Holds etcd's data dir and WAL.
pEbsVolumeSize	no	10	Shared io2 volume size in GB.
pEbsVolumeIops	no	1000	Provisioned IOPS. io2 caps at 64000 and a 1000:1 IOPS-to-GiB ratio.
pBootstrapRepo	no	etcfs/etcfs-terraform-modules	Repository holding node-bootstrap.sh.
pBootstrapRef	no	refs/heads/main	Ref to fetch it from. Pin a tag for reproducibility.
pEnvironmentTag	no	production	Value for the Environment tag.

Scaling

aws autoscaling set-desired-capacity \
  --auto-scaling-group-name etcfs-asg --desired-capacity 5

Keep the desired capacity odd. etcd tolerates (n-1)/2 failures, so an even count buys no extra fault tolerance over the odd number below it and adds a member that can only widen the majority a write has to reach.

Scale-in

An EC2_INSTANCE_TERMINATING lifecycle hook holds a departing instance in Terminating:Wait, and the node itself does the leaving. A watcher installed at boot, etcfs-leave.service, polls IMDS for autoscaling/target-lifecycle-state; the moment that reads anything other than InService the node does three things and then releases the hook:

• etcd member remove for itself, run against a surviving peer rather than its own endpoint — a member that removes itself has to apply the configuration change that stops it, on a node being torn down underneath it. Localhost is kept as a last resort for a lone node or a partition. A member that will never answer again still counts toward every majority, so it consumes fault tolerance without supplying any; one is survivable, but a group that churns accumulates them until they cost quorum for real.
• Repointing the seed row, if it is the node the row names. The row is written once at cluster formation and nothing else refreshes it, so terminating the original seed otherwise leaves it naming a dead address for the rest of the cluster's life. The repoint runs under the same conditional PutItem a joining node reclaims with, so losing it to a concurrent writer is a correct outcome rather than a failure.
• complete-lifecycle-action, unconditionally — including after every removal attempt has failed. A dangling member is repaired by the next joiner's scrub; an unreleased hook stalls the group until the heartbeat expires. For the same reason the hook's DefaultResult is CONTINUE: a node too wedged to reach IMDS or the AWS API degrades to an ungraceful termination rather than holding the ASG for the full timeout.

IMDS is the signal here, not a systemd shutdown unit, because Terminating:Wait does not stop the operating system — nothing on the instance would run until the hook had already been released. Polling metadata costs no API call and needs no credentials.

The degraded path is still safe: EtcFS fences a node that disappears, and the next node to join repairs both the membership and the seed row on its own. What is lost is timing — the repair waits until something joins, and the seed-row half of it waits out STALE_SECONDS first.

A node killed with ec2:TerminateInstances directly, or one that simply dies, still reaches the hook — the ASG runs any instance it finds unhealthy through the same transition, whoever initiated the termination — but by then there is nothing left to act on it. The watcher is gone with the instance, nothing calls complete-lifecycle-action, and the hook runs out its HeartbeatTimeout to CONTINUE. Measured on a hard-killed node: it sat in Terminating:Wait for the full timeout and then drained, with the group back at 3 of 3. The only cost is that each hard kill holds an ASG slot for that long, which is what HeartbeatTimeout is now sized against — the watcher itself finishes in about 45 seconds when it can run at all.

The membership repair in that case falls to the next joiner, and it happens well before the hook gives up. Measured end to end: a non-seed node was hard-killed with its watcher disabled, leaving a dangling voter within ~40 s; the ASG launched a replacement at ~90 s; the replacement logged member ... has no matching live instance — removing before joining at ~170 s, cleared the dead voter and joined in its place. Quorum held at 2 of 3 throughout and the filesystem stayed readable and writable. When it is the seed that dies this way the repair is slower, because the recorded row must first cross STALE_SECONDS before a joiner will reclaim it.

Earlier revisions did this from a Lambda invoked by an EventBridge rule, acting on a survivor over SSM Run Command. Moving it onto the node removed four resources — the function, its IAM role, the rule and the invoke permission — along with the node role's SSM Run Command grant, and left the protocol in one place instead of split between a shell script and an inlined Python function that had to be kept in step with terraform/modules/etcfs-asg/lambda/graceful_leave.py.

Boot stagger

Nodes sleep for a delay derived from the last octet of their private IP before starting the election. This is jitter and nothing more: it spreads the conditional writes and health probes of a simultaneously-launched group so the losers back off against a seed that is already answering. Which node may form a cluster is decided entirely by the conditional PutItem, so two nodes drawing the same delay — or one booting slowly enough to overtake the order — changes nothing. An ASG tag cannot substitute for the row here, however appealing the idea: CreateTags is last-writer-wins with no conditional form, so two nodes that both read it empty would both go on to form a cluster.

Resources

Nine resources, all unconditional.

Logical ID	Type	Purpose
EtcfsSecurityGroup	AWS::EC2::SecurityGroup	Cluster SG. All outbound; ingress added separately.
SgIngressEtcdClient	AWS::EC2::SecurityGroupIngress	Self-referencing 2379–2380, so members reach each other and nothing else does.
iamEc2Role	AWS::IAM::Role	Node role: fencing attach/detach, peer and volume reads, seed-row access, lifecycle-hook completion, SSM and CloudWatch agent.
iamEc2InstanceProfile	AWS::IAM::InstanceProfile	Binds the above to the launch template.
SharedDataVolume	AWS::EC2::Volume	The io2 Multi-Attach volume, raw. Retain on delete and replace — it holds the filesystem.
CloudWatchAgent	AWS::SSM::Parameter	Agent config fetched at boot: memory, CPU, and disk on / and /mnt/etcfuse.
SeedElectionTable	AWS::DynamoDB::Table	One row per cluster, keyed by cluster_name. The conditional-write election runs against it.
asgLaunchTemplate	AWS::EC2::LaunchTemplate	IMDSv2-required, encrypted root, and the user-data shim that fetches node-bootstrap.sh.
asg	AWS::AutoScaling::AutoScalingGroup	The cluster. Propagates ClusterName and Role at launch and carries the terminating hook, which costs no resource of its own.

Limitations and things to know

The instance tags are load-bearing. node-bootstrap.sh finds peers with one EC2 filter — tag:ClusterName plus tag:Role=etcfs-node — and three things depend on the result: deciding whether a cluster is alive when the recorded seed has gone, clearing etcd members with no instance behind them, and building the endpoint list etcfuse-meta talks to. Removing either tag does not degrade discovery, it inverts it: the search comes back empty and a joining node reads that as "no cluster here". The scrub fails closed against this case, but the split-brain path cannot be defended from inside the script.

Stack names are capped at 55 characters. IAM role names are limited to 64, and the node role is ${StackName}-instance. A longer stack name fails partway through creation. The old 48-character cap came from the EventBridge rule and Lambda role, both now gone.

Deleting the stack leaves the shared volume behind, still billing. SharedDataVolume carries DeletionPolicy: Retain and UpdateReplacePolicy: Retain on purpose — it holds the filesystem, and CloudFormation must never take it with the stack. The cost is that every delete-stack orphans a volume that nothing will ever reclaim, and an io2 volume bills for its provisioned IOPS whether or not anything is attached: at the default 1000 IOPS that is roughly $73/month each, indefinitely. Three runs of this stack left three orphans totalling ~$220/month before anyone noticed. After deleting a stack, find them and remove them:

aws ec2 describe-volumes \
  --filters Name=status,Values=available Name=tag:Name,Values='*-data' \
  --query 'Volumes[].[VolumeId,Size,Iops,CreateTime]' --output table

One subnet, one AZ, no AZ redundancy. Multi-Attach volumes do not cross AZs, so the whole cluster shares the AZ's failure domain. This is a property of the shared-block-device design, not of the template.

Nothing enforces an odd DesiredCapacity after deploy. The parameter restricts it at deploy time; a scaling policy is free to set anything between min and max.

No alarms. The CloudWatch agent ships metrics but the template creates no alarms on them, so a cluster running one member below quorum looks fine from CloudFormation.

pInstanceAMI tracks latest by default. A stack update re-resolves it, which replaces instances whenever Amazon publishes a new AL2023 image. Pin an AMI ID for a stable fleet.

Regenerating the diagrams

The two figures have different sources, because they are different kinds of picture.

docs/architecture.png comes from docs/figures/architecture.drawio, drawn with the AWS 2023 icon set. Edit it in draw.io or the desktop app (pacman -S drawio-desktop on Arch), then re-export headlessly:

drawio --export --format png --scale 2 --border 10 \
  --output docs/architecture.png docs/figures/architecture.drawio

Edge waypoints in that file are absolute page coordinates, chosen so no connector crosses a resource label. Moving a node means re-checking them.

docs/join-flow.png comes from docs/figures/join-flow.tex. It is a decision tree rather than an inventory of AWS resources, so it stays in TikZ where the branch layout is part of the source:

tectonic -X compile docs/figures/join-flow.tex --outdir docs/figures/out
pdftoppm -png -r 220 docs/figures/out/join-flow.pdf docs/join-flow

License

Apache 2.0. See LICENSE.