jonnify·codemojex /codemojex/production/the-pragmatic-valkey-node

Codemojex · dive C8.2

The pragmatic Valkey node.

Valkey runs commands on one thread, and that single fact sizes the machine: the node is provisioned for a steady latency tail, not for throughput headroom. The whole competitive state and the EchoMQ bus live on one small, sharply-configured machine — the dedicated echo-valkey Fly node — private on the org’s 6PN, upgraded only on evidence.

Grounding — design §The pragmatic Valkey node · echo/config/runtime.exs (the echo-valkey wiring) · application.ex (conn_opts/0 · valkey_host/0).

One machine, four sizing choices — select a choice
echo-valkey Fly shared-cpu-2x · 1 GB · Valkey 9.1 · jemalloc vCPU 1 the command thread io-threads 1 512 MB · noeviction working set · single-digit MB 1 GB vCPU 2 threads that sleep fsync · purge · lazy free the app 6PN · IPv6 public internet no route in × bound to the IPv6 wildcard — answers only on the org’s 6PN
io-threads 1 → Valkey runs every command on one thread. Extra I/O threads busy-wait and would drain the shared class’s burst balance even while idle, so the setting stays at 1 and the second vCPU is left for the background threads that sleep — the everysec AOF fsync on its own thread, jemalloc’s background purge, the lazy frees.
§1

Four choices, one budget

design §The pragmatic Valkey node

The recommendation is short because the constraint is sharp. The launch node is Valkey 9.1 over its bundled jemalloc on a Fly shared-cpu-2x, one-gigabyte machine, eviction off. The command thread is the budget every choice protects: a guess costs a handful of Lua steps on that thread, and the tail stays steady as long as nothing else competes for it — not an idle I/O thread burning the burst balance, not a fork the machine cannot absorb, not an eviction scan.

maxmemory is a loud 512-megabyte guardrail under noeviction — far above a working set that is single-digit megabytes, because player balances live in Postgres, not here. The point of the guardrail is the failure mode it buys: a runaway keyspace rejects writes rather than being killed. Durability is AOF alone, one fork source; and jemalloc is the allocator the resident-size budget depends on. The node is private by construction — bound to the IPv6 wildcard so it answers only on the org’s 6PN, with requirepass on top.

Every figure in this section is the design’s: §The pragmatic Valkey node records the machine class, the 512-megabyte guardrail, the io-threads 1 rationale, and the AOF-alone durability posture. One caveat travels with the node: its volume lives on one host in one region — the single point of failure to retire with a replica before real money flows.

§2

The wiring — env in, IPv6 out

runtime.exs · application.ex

The connection is wired from the environment at release boot. runtime.exs reads three variables — the host, the password, and a port that defaults to 6390. Unset locally, the connectors fall back to 127.0.0.1:6390 with no auth, so dev and test run against a local engine unchanged.

echo/config/runtime.exs · the echo-valkey wiring source
# The shared Valkey — the EchoMQ bus + the real-time competitive state — lives on the dedicated
# echo-valkey Fly node (private 6PN, requirepass). Wire host/port/password from the env. If
# VALKEY_HOST is unset (local dev/test), the connectors fall back to 127.0.0.1:6390 with no auth.
if valkey_host = System.get_env("VALKEY_HOST") do
  config :codemojex, valkey_host: String.to_charlist(valkey_host)
end

if valkey_password = System.get_env("VALKEY_PASSWORD") do
  config :codemojex, valkey_password: valkey_password
end

config :codemojex, valkey_port: String.to_integer(System.get_env("VALKEY_PORT") || "6390")

The app side dials over Fly’s IPv6-only 6PN. Codemojex.Application builds one shared conn_opts list — RESP3 plus the port, then the optional host and password — for every Valkey lane: the Bus, the near-cache tables, and each consumer’s private lane. The host goes through valkey_host/0, which resolves the configured hostname to its AAAA record before the connector ever dials.

echo/…/codemojex/application.ex · the IPv6 dial source
# The shared connector options for every Valkey lane (the Bus, the near-cache tables, and each
# consumer's private lane). host/password come from prod config (runtime.exs reads the env);
# unset locally → omitted, so the connector uses its 127.0.0.1:6390 no-auth default (dev/test).
defp conn_opts do
  [protocol: 3, port: Application.get_env(:codemojex, :valkey_port, 6390)]
  |> maybe_put(:host, valkey_host())
  |> maybe_put(:password, Application.get_env(:codemojex, :valkey_password))
end

# Resolve the configured Valkey host to an IPv6 address tuple so the connector dials echo-valkey
# over Fly's IPv6-only 6PN: gen_tcp infers inet6 from an 8-element address tuple, so this needs
# no global :kernel/inet config (which a release cannot set at compile time) and no connector
# change. Unset (local dev/test) → nil → the connector falls back to 127.0.0.1. A charlist
# hostname is resolved to its AAAA; if resolution fails the charlist passes through (best effort).
defp valkey_host do
  case Application.get_env(:codemojex, :valkey_host) do
    nil ->
      nil

    host ->
      case :inet.getaddr(host, :inet6) do
        {:ok, addr} -> addr
        {:error, _} -> host
      end
  end
end

The trick is in the return shape: gen_tcp infers inet6 from an 8-element address tuple, so the release needs no global :kernel/inet configuration — which a release cannot set at compile time — and no connector change. On a resolve failure the charlist passes through, best effort.

§3

The upgrade ladder

each step on evidence

The launch node is deliberately small, and the path out of it is a ladder, not a rebuild. Each step is taken on evidence, never ahead of it — and the order matters, because the first step retires the only risk that touches money.

Four steps, each on evidence — select a step
① a replica in the primary region ② split web / worker Fly process groups ③ performance machine sustained guess throughput ④ hash-tag shard one thread is the bottleneck each step on evidence — never ahead of it
step ① → the launch node’s volume lives on one host in one region — the single point of failure. A replica in the primary region retires it before real money flows; no other step on the ladder is taken first.

The last step is the one the keyspace prepared for on day one: the bus’s keys are braced — emq:{q}: — so every key of a queue hashes to one slot, and a cluster split moves whole queues, never fragments of one. Grounding — design §The pragmatic Valkey node (the ladder) · the v2 keyspace law taught in C3.

§4

Two version-9 touchpoints

forward · not shipped

Expiry at close — forward. The closed-game keys — the board, the hashes, the counter, the marker, the lock hashes — are not expired by the close today, so under noeviction they accumulate. A key-level expiry at close would fix it on any version, and HEXPIRE would give per-field control where a hash should outlive some of its fields. The lock plane’s marker could collapse to one self-expiring hash field, released by token with DELIFEQ rather than an unconditional delete. None of this is shipped surface; the design records it as the maintenance the guardrail eventually collects.

The 9.1 fixes. Two land on paths the bus uses directly: the stream-trim null-deref on EchoMQ.Stream.trim, and the rehashing latency reduction on the lane sorted sets and the job-hash dictionary as they grow. The version choice is the same discipline as the sizing: taken for a named path the system exercises, not for a headline.

Grounding — design §The pragmatic Valkey node (the two touchpoints) · EchoMQ.Stream.trim/4 in echo/apps/echo_mq/lib/echo_mq/stream.ex.

§5

References

sources · related

Sources

Related