Pinterest: task queues & partitioning.

“PinLater is a Thrift service to manage scheduling and execution of asynchronous jobs.” Written in Java, Apache-2.0, open-sourced in 2014 and archived in February 2018 (succeeded by Pacer). Its surface is three core actions — enqueue, dequeue, ACK — over three components: a stateless Thrift service for job submission and scheduling, a storage backend that holds the jobs and their state, and worker pools that execute the jobs.

Reliability is explicit acks with automatic retry at a configurable delay: a worker replies a positive or negative ACK depending on whether execution succeeded or failed, and a failed job is retried later. Per-job knobs are set at enqueue time — “Priorities, delayed execution and flexible retry policies that can be specified per job at enqueue time” — plus checkpointing for long-running jobs.

Two storage backends sit behind it. The README guidance: services should “default to use the MySQL backend as long as the QPS is in the lower to mid range (no more than 1000 QPS per shard). If the QPS is expected to be higher than this, then the Redis implementation should be used.” Redis was the high-throughput backend; MySQL was the durable default.

Functional partitioning — polyglot by access pattern. Pinterest keeps different data on different stores, chosen by how it is read: “We keep pin_id → pin object cache in a memcache cluster, but we keep board_id → pin_ids in a redis cluster.” Different data, different store — not forced into one engine. And the id itself carries placement: the sharded fleet packs the shard into the high bits of a 64-bit id, ID = (shard ID << 46) | (type ID << 36) | (local ID<<0) (shard id 16 bits, type id 10 bits, local id 36 bits), so the id tells you where the row lives — no lookup table.

The list-based reliable queue. A consumer atomically moves a job off the work list onto a processing list as it claims it (RPOPLPUSH source processing, blocking BRPOPLPUSH), then LREM removes it from the processing list once the job is acknowledged. If the worker crashes mid-job, the job is still on the processing list — a recovery worker scans for items stuck too long and re-queues them. No job is lost to a crash between receive and ack. RPOPLPUSH/BRPOPLPUSH were deprecated in Redis 6.2.0 in favour of LMOVE/BLMOVE (which take an explicit LEFT/RIGHT direction); LMOVE is today’s primitive, RPOPLPUSH the original.

The fleet behind PinLater grew through a wall. Pinterest started from “eight EC2 servers running one MySQL instance each”; by September 2011 “every piece of our infrastructure was over capacity.” They finished launching the sharding approach “in early 2012, and it’s still the system we use today,” carrying 50 billion Pins across one billion boards. PinLater’s own sharding is a free sharding scheme — a shard is chosen when a job is enqueued — so adding a shard is a configuration change, not a data migration, and load spreads evenly. The shard id rides in the job descriptor itself, [queue_name][shard_id][priority][local_id].

The case study’s punch line is the durability turn. Pinterest later got “5X more throughput on the MySQL backend, which allows us to do over 2,000 enqueues per second with a single i2.2xl MySQL EC2 instance” and so chose to “move all our workloads to MySQL and deprecate the use of the Redis backend” — for MySQL’s durability and replication. A list-in-RAM queue is fast, but a job store that must not lose work eventually wants a durable record. That is the same tension R7.01 (Redis as a primary database) and the persistence floor name.

The Pinterest case is the lens; the bus the reader is building is EchoMQ, backed by Valkey. Each idea ties back to one verified surface. Partitioning is the braced keyspace: per-queue keys are emq:{q}:<type> and every key of one queue lands on one slot, computed client-side as slot/1 = rem(crc16(hashtag(key), 0), 16384) — CRC16-XMODEM over the {…} substring, mod 16384, the cluster-spec algorithm — with the on-disk vector slot(“123456789”) == 12739. Crash-safety is a server-clock lease plus an attempts counter, not a processing list: the claim stamps redis.call('TIME') and the job HASH carries state / attempts / payload (jobs.ex: HSET KEYS[1] 'state' 'pending' 'attempts' '0' 'payload' …). A crash lets the lease expire; the job is re-claimed and attempts increments.

• The lane ring is not a processing list. EchoMQ.Lanes does run redis.call('LMOVE', KEYS[1], KEYS[1], 'LEFT', 'RIGHT'), but that is a single-key ring rotation — it rotates the list of lane names round-robin so service spreads across lanes; it is not a source → processing move. The owned wire still knows the family — EchoWire.Cmd.rpoplpush/2 (“Open an RPOPLPUSH builder”) and blmove exist in echo_wire/cmd.ex. PinLater and classic-Redis recover via a processing list and a recovery worker; EchoMQ reaches the same goal through the lease.
• Placement is computed from identity, not looked up. Pinterest embeds the shard in the id’s high bits (shard ID << 46, a bit-extract). The branded id is hashed to a placement — the contract vector EchoData.BrandedId.hash32(274557032793636864) == 234878118 (the id USR0KHTOWnGLuC) — and a queue’s co-location comes from the {q} hashtag, not from the snowflake’s minting-node field. The shared principle is the one to keep: placement is a property of the identity, computed rather than stored in a registry — Pinterest by bit-extraction, echo by hash plus the hashtag.

The worked consumer is codemojex. Codemojex.Guesses.submit/3 mints a branded JOB (EchoData.BrandedId.generate!(“JOB”)) and runs Lanes.enqueue(Bus.conn(), “cm”, player, job, payload) — a guess is a job on the player’s lane, so the bus rotates service across players and one keyboard masher cannot starve the field.

The module takes the two ideas apart and then follows the scale arc: partition first, then make the queue reliable, then scale it out. Read them in order.