Distributed rendering

1Preview-then-handoff

Spectral path tracing through a high-index gemstone with dozens of internal bounces is expensive, and streaming every interactive frame to a remote machine would add visible input latency. Pleochroma instead renders locally while the camera moves, and hands off to its one configured remote endpoint once the camera settles: the client sends a compact SceneState (serialized with postcard) and the remote streams traced radiance back, which is folded into the client's own accumulation buffer. That endpoint is always a coordinator, pleochroma-worker serve, which renders the request itself, splits it across render workers that have joined it, or both (section 3); the desktop app never talks to a worker directly.

The settings dialog's Live Compute choice decides what happens at the settle: Local Only, Remote Only, or Local + Remote. With Local + Remote both sides trace the same image against one shared sample budget — the live view's single target sample count, there is no separate remote setting — through one sample cursor per settle (LiveEpoch, apps/pleochroma-cut/src/bridge/sample_cursor/live.rs), so their samples are combined, never duplicated. The remote side claims short chunks sized to about 1.5 s of its measured rate (8 samples for the first chunk, before a rate is known; LIVE_CHUNK_TARGET_SECS, bridge/remote/live_lane), so a camera drag wastes at most a second or two of remote time. A chunk that fails keeps the prefix the remote already traced and returns the untraced remainder to the local tracer; after two consecutive failures the remote sits out the rest of that settle. Each settle is stamped with the scene it was dispatched for, so a scene change during the handoff can never merge samples of the old scene into the new image.

2Sample-range additivity

Splitting a frame into screen-space tiles load-balances poorly for a gemstone render: background tiles finish almost instantly while tiles covering the stone itself carry the full bounce budget. Because each sample's RNG seed is a deterministic function of pixel index and sample number, sample contributions are additive and order-independent, so Pleochroma instead partitions work by sample-index range across the whole frame — one node traces samples [0, 64), another [64, 128), and so on — with no tile-seam stitching and linear load balancing across heterogeneous machines.

Disjointness is the whole correctness condition, so it is enforced in one place: a SampleCursor in the GUI-free crates/pleochroma-dispatch hands out disjoint absolute sample ranges to every contributor of one image — the export's local and remote lanes, the live view's local tracer and remote chunks, and the coordinator's own lane and joined workers alike. Every backend also applies the same non-finite rule (add_finite_sample, crates/pleochroma/src/optics/raytracer/sampling.rs, with its twin in the GPU reduction): a sample with a NaN or infinite component is dropped but still counted, so one bad sample can never poison a pixel's sum and the divisor of a merge is always the number of samples actually traced. A statistical test renders one range on the CPU and the next on the GPU, merges them the production way, and checks the result against an independent CPU reference (renderer/gpu/merge_tests.rs).

3The coordinator and joining workers

pleochroma-worker serve is the coordinator (apps/pleochroma-worker/src/serve, src/coordinator). It always serves the read-only design library to viewers. On a build with the worker feature it also opens a separate worker port that render workers dial into, and it traces samples with its own CPU/GPU only when started with --render (--only-gpu/--only-cpu imply it); without that flag it acquires no GPU and traces nothing itself. Its WELCOME advertises exactly that: no render capability for a bare coordinator with no joined workers (the desktop app then sees a library-only remote and renders locally), the plain CPU or GPU backend for --render alone, and a Coordinator { workers, threads, gpus } backend as soon as a worker has joined. A capability change between requests is announced to the viewer with CAPABILITY_CHANGED.

A worker is pleochroma-worker join <coordinator-host:port> (apps/pleochroma-worker/src/join): it connects out to the coordinator's worker port over mutual TLS with a worker certificate, reports its render capability in its HELLO, and then serves the coordinator's requests on that connection. No inbound port is needed on the worker, so a machine behind NAT or a cloud VM can join. --slots K opens K parallel connections (default 2, at most 64), each serving one request stream at a time; every slot reconnects forever with jittered backoff, 1 s doubling to 60 s. The second slot is there for fast workers: it keeps a chunk in flight while the first one's transfer and decode would otherwise leave the GPU idle.

The worker port sits two ports above --bind and each enrollment listener one above its own port, on --bind's host unless overridden (WORKER_PORT_OFFSET, apps/pleochroma-worker/src/cli/mod.rs). Every listener binds loopback only unless --allow-remote is passed. --no-workers skips the worker port and its enrollment listener; --no-enroll skips both enrollment listeners.

How a request runs depends on its kind (src/coordinator/job). An export-type request — a still export, a tilt-video frame — becomes a job: a lane pool (pleochroma-dispatch's LanePool) over the request's sample range, with one lane for every idle joined worker whose pixel cap accepts the image plus the coordinator's own lane, each claiming chunks from the shared sample cursor sized to its measured rate. A live-view request takes the own lane plus every idle joined worker, fastest first (--interactive-workers all, the default); a coordinator without --render serves it from the joined workers alone. If a worker dies mid-job, the samples it had not finished go back to the cursor and the remaining lanes trace them, so the viewer still receives every sample of the request exactly once. Only when no lane at all is left does the viewer get ALL_WORKERS_LOST, and it then renders the unfinished part locally. The coordinator pings every idle joined connection after 10 s without traffic and drops it after 30 s of silence (LivenessConfig, src/coordinator/registry.rs); a joined worker that hears nothing for 45 s reconnects (JOIN_IDLE_TIMEOUT).

Two advanced flags change the live-view rule: --interactive-workers N caps the joined workers a live-view request may take (0 keeps it on the own lane alone, the lowest-latency choice on a slow link; the default is all), and --pin-interactive-worker <label> prefers one named worker while it is connected, idle and accepts the image. Throughput jobs — an export streamed final-only, or any final-picture request — wait in a per-viewer queue, one active job per viewer certificate. A small picture (its estimated time on the fastest worker under --whole-image-secs, default 2 s) goes whole to that one worker instead, and up to --jobs-per-viewer of them (default 8) run at once, so a catalogue batch keeps a fast worker busy. All in-flight jobs share one memory budget, --max-job-memory-mib (default 2048; a job is charged width × height × 48 bytes, plus width × height × 36 bytes for every lane that works on it, plus its HDR map and any share the viewer renders itself; a job whose lanes do not all fit runs on fewer, and one past the cap even with a single lane is refused, not queued).

4What travels on the wire

• Lossless result compression. A radiance delta (FRAME/PREVIEW payload) can travel compressed, negotiated per connection: the client lists the encodings it can decode in its HELLO and the server picks the first entry of its own preference list the client accepts (negotiate, crates/pleochroma-net/src/messages/encoding.rs). The default preference is a byte shuffle (four planes: every value's first byte, then every second byte, …) followed by Zstandard level 1, then shuffle plus LZ4, then raw; a loopback peer gets raw only, and raw is also the fallback whenever the two sides share nothing. Each frame header names its own encoding, so a payload that would not shrink is simply sent raw. The decoded bytes are bit-identical to the raw f32 buffer, NaN payloads and signed zeros included. On captured real frames the default came out 1.54–1.99× smaller — 35–50 % fewer bytes on the link — encoding at 465–621 MB/s and decoding above 1 GB/s on one laptop core. Decoding is bounded against decompression bombs: the declared raw length must equal width × height × 12 before anything is decoded, the output buffer is exactly that size, and output that is short or would overflow it is rejected (radiance/payload.rs). Both hops negotiate it: viewer to coordinator, and coordinator to each joined worker. The negotiated encoding is only a connection's starting point: every connection also measures how fast its frame writes drain and compresses frames adaptively to that measured link speed (Zstandard or LZ4 on slow links, raw on fast ones and on loopback), switching on the fly when the link changes. --payload-encoding on serve and join (auto by default) pins one encoding instead (raw, lz4, zstd or zstd:LEVEL with a level from 1 to 22) on every link, loopback included, when the other side accepts it.
• Final Picture Only. The export dialog and the tilt-video export have a Transfer choice (its default is set in the Remote Coordinator form): Full Data sends float radiance that is merged with the local lanes; Final Picture Only has the coordinator render, tone-map with the desktop export's own function (tonemap_accumulation) and send one PNG per image, the same PNG the desktop would have written from the same samples. With Compute set to Local + Remote, the setting Final-Picture Exports: This Machine Renders a Share Too (on by default) keeps the local lanes busy: the desktop reserves a tail of the sample budget (viewer_samples), traces it itself, and uploads the float sum as one CONTRIBUTION message (added in protocol version 16) for the coordinator to fold in before it tone-maps. If that share does not arrive in time the coordinator traces the tail itself, so a slow local machine never stalls the export. With the setting off, or with Remote Only, the local lanes stay idle. The Live Transfer setting offers the same trade for the viewport: Final Picture streams finished, denoised 8-bit display frames (denoised on the coordinator with the desktop's own À-Trous denoiser, its guides from the coordinator's own primary-ray prepass) instead of float deltas. A finished picture cannot be summed with anything, so this mode is remote-only by construction: the local GPU/CPU pauses after the handoff. A remote that does not implement it answers UNSUPPORTED_REQUEST, and the app falls back to full data with one note.
• HDR environment maps by content hash. A scene lit by a loaded HDR panorama names the map only by the SHA-256 of its file bytes (crates/pleochroma-net/src/messages/asset.rs). A node that lacks it answers NEED_ASSET; the requester sends the bytes once (ASSET, at most 256 MiB, checked before anything is allocated); the receiver verifies the hash and keeps the bytes in a bounded on-disk cache (asset-cache next to --db or --cert-dir, or $PLEOCHROMA_ASSET_CACHE_DIR; size cap $PLEOCHROMA_ASSET_CACHE_MIB, default 2048 MiB), and a later request naming the same hash is never asked again. The coordinator asks the viewer once and forwards the map to each joined worker that asks. A remote advertises HDR support in its WELCOME only when some lane can render it, and HDR jobs only go to workers whose own cache works. For a remote without HDR support, or a map that cannot be sent (not loaded from a file, or over the size limit), HDR scenes render locally, with one status note. The asset is a Radiance .hdr file or, since protocol version 25, an IXENVMAP version 1 float asset (crates/pleochroma-net/src/messages/envmap_asset.rs), told apart by its first bytes. The Android app sends the map exactly as the device renders with it, as a float asset, and an endpoint that advertises HDR support decodes both.
• Concave tools. A scene carries the stone's tool volumes beside its planes (SceneState::tools, protocol version 19; empty for a convex stone), at most 128 of them (MAX_TOOL_PRIMITIVES), and the worker side validates each one before tracing. The GPU kernels trace the same 128 (GPU_MAX_TOOL_PRIMITIVES), so a coordinator or worker with a GPU traces such a stone on the GPU and in the hybrid CPU/GPU split, and the split it measures is kept per tool-count bucket (tool_bucket, render_core/hybrid.rs). Local, remote and combined renders trace the same stone.
• Frosted facets. A scene names the facets a design frosts (SceneState::frosted_planes, protocol version 26: the ascending indices of those planes, empty for a design without frosting) beside the girdle_frosted flag, and the worker side re-derives each facet's finish from them (facet_finishes_with_frosted), so a frosted design renders as frosted on a remote. A list with more entries than the stone has planes is refused. The design-library protocol carries a library design's frosting too (FrostedTierWire), so a mirror and a remote editor open keep it.
• Durable server jobs. A final-picture request is tied to its connection: if the viewer drops, the work stops. A job is not. Protocol version 25 adds ClientMessage::Job (crates/pleochroma-net/src/messages/job.rs): the client submits a scene with a random 16-byte token, a label, a sample count and a color space; the server stores it as a file, renders it on its own lane, one job at a time, and keeps the PNG until its retention ends (7 days by default, $PLEOCHROMA_JOB_RETENTION_HOURS; the directory is $PLEOCHROMA_JOB_DIR, or jobs next to the library database). The client reconnects at any time to list, fetch, cancel or delete it. The server derives the job's id from the client certificate's fingerprint and the token, so repeating a submit finds the same job and no certificate can address another's jobs. A restart puts a running job back in the queue, to render again from sample 0. The limits are 8 unfinished jobs per certificate, 256 jobs and 8 GiB of results in total, and pictures of up to 16,777,216 pixels (4096 × 4096) with up to 32768 samples per pixel. A scene with color zones is not part of a job; a client must not submit one. A coordinator without --render answers every job request with UNSUPPORTED_REQUEST. The Android app uses jobs for its Send To Server action (mobile page).

5Mutual TLS, roles and token enrollment

The wire protocol lives in crates/pleochroma-net, typed message framing with no networking of its own. pleochroma-worker's serve and join commands carry the actual sockets and TLS.

• TLS 1.3 only, via rustls with the ring crypto provider; TLS 1.2 is excluded from the feature set at compile time, not merely disabled at runtime (tls::server_config/tls::client_config).
• Mutual authentication: both sides present a certificate signed by the same private CA (pleochroma-worker cert subcommands manage that CA). The coordinator's own certificate is used on both of its ports.
• Every client certificate carries a role: a worker certificate's Common Name is worker:<label>, a viewer's is the plain label (which may not start with worker:). Each listener checks the role and its own fingerprint allowlist — allowlist-viewers.txt and allowlist-workers.txt next to the CA, with a pre-role allowlist.txt still honoured as the viewers list while it is the only one — so a viewer certificate is never accepted on the worker port, nor a worker certificate on the viewer port. The HELLO carries the peer's role too, and a mismatch is refused with ROLE_REFUSED. --trust-any-client-cert skips both allowlists, never the role check.
• One-time token enrollment (the wire protocol and claim-side verification in crates/pleochroma-net/src/enroll; the server-side registry that mints and expires tokens in apps/pleochroma-worker/src/enroll): pleochroma-worker cert issue-token asks a running serve process's enrollment listener, from a loopback connection only, to mint a 180-second, single-use token for a viewer, or with --role worker for a worker. The desktop app's Redeem button, cert claim, or join --token redeems it — verifying the listener's certificate chain against the CA fingerprint carried inside the token itself, before sending the token's secret — and writes the same three-file certificate bundle cert issue-client would, without a manual copy. A listener refuses a token request for the other role.
• Plaintext serving (--insecure-no-tls) is available for local debugging, refused on any non-loopback bind address, and disables the worker port and both enrollment listeners.
• Every connection negotiates a protocol version during its HELLO/WELCOME handshake (PROTOCOL_VERSION, 28 at the time of writing; the constant in crates/pleochroma-net/src/messages/mod.rs is authoritative), together with a build hash and, where both sides can establish it, a content hash of the pleochroma source tree; a mismatched peer is refused outright (BUILD_MISMATCH) rather than risking a misread frame or a merge of samples from different physics.
• Every message is length-prefixed and bounded before a payload is even allocated (framing::MAX_FRAME_LEN, 512 MiB; the enrollment listener caps its own request further, at 4 KiB), and a just-accepted socket carries a 20-second read/write timeout through its TLS handshake until its first HELLO arrives (HANDSHAKE_TIMEOUT, serve/socket.rs) — an idle or slow-loris connection cannot pin a thread indefinitely. --max-connections (default 64) bounds authenticated connections, counted separately for viewers and joined workers, and a second limiter at four times that cap bounds connections still in their handshake. A per-address cap on the viewer port, --max-preauth-per-ip (default 8), closes a connection from an address that already has that many unauthenticated ones open.

6The pleochroma-worker CLI

pleochroma-worker has four subcommands: render (one-shot scene-to-PNG, no networking), serve (the coordinator: the design-library protocol over TLS, the worker port, and with --render its own render lane), join (a render worker dialling a coordinator), and cert (manages the private CA: init, issue-server, issue-client, issue-token, claim). render, join and serve --render need a build with the worker feature (gpu implies it). Flags below are copied from the CLI's own --help text in apps/pleochroma-worker/src/cli/mod.rs.

pleochroma-worker render \
  --scene scene.json \
  --out render.png \
  --width 3840 \
  --height 2160 \
  --samples 8192 \
  --threads 16

--only-gpu / --only-cpu force a backend; the default is a hybrid split across both when it measures as worthwhile.

pleochroma-worker serve \
  --bind 0.0.0.0:7878 --allow-remote \
  --ca pki/ca.pem --cert pki/server.pem --key pki/server.key \
  --render

--bind defaults to loopback only (127.0.0.1:7878); --allow-remote is required for any other address, on any of the four listeners. --db points at the design-library SQLite file (default facet_diagrams.sqlite in the working directory). Leave out --render for a coordinator that only distributes work to joined workers. --threads and --only-gpu/--only-cpu govern the own lane; --max-connections defaults to 64.

pleochroma-worker join coordinator.local:7880

The address is the coordinator's worker port (also accepted as --coordinator). --cert-dir names the worker certificate bundle (ca.pem, client.pem, client.key; default worker-cert in the working directory); --threads and --only-gpu/--only-cpu work as for render. --slots defaults to 2 (the second slot keeps a fast GPU worker busy between chunks); raise it further mainly on CPU-only machines that juggle many small chunks.

7Setting up a coordinator, its workers and the desktop app

# 1. create a CA
pleochroma-worker cert init --dir pki/

# 2. issue the coordinator's own certificate (used on the viewer and worker ports)
pleochroma-worker cert issue-server --dir pki/ --host coordinator.local --ip 192.0.2.10

# 3. start the coordinator; drop --render if this machine should not trace samples itself
pleochroma-worker serve --bind 0.0.0.0:7878 --allow-remote \
  --ca pki/ca.pem --cert pki/server.pem --key pki/server.key --render

# 4. mint one-time, 180-second enrollment tokens (loopback only)
pleochroma-worker cert issue-token --ca pki/ca.pem --admin-addr 127.0.0.1:7879 --name my-laptop
pleochroma-worker cert issue-token --ca pki/ca.pem --admin-addr 127.0.0.1:7881 --name gpu-box --role worker

# first run: claim the worker token into worker-cert/, then join
pleochroma-worker join --coordinator coordinator.local:7880 --token GW1-...

# every later run reuses the claimed bundle
pleochroma-worker join coordinator.local:7880

serve's viewer enrollment listener defaults to the same host as --bind, one port up, and the worker enrollment listener to one port above the worker port (--enroll-bind and --worker-enroll-bind override them), but cert issue-token only ever succeeds from a loopback peer regardless of that address — the operator runs it on the coordinator itself, not remotely. join --token claims one port above the worker port it was given (7881) unless --enroll-addr says otherwise; a token is single-use, so later runs leave it out. For an air-gapped machine, cert issue-client --role worker writes the same bundle for copying by hand.

In pleochroma-cut, open the Remote panel (Remote Coordinator) and choose Set Up Remote Coordinator: enter the coordinator's viewer address (coordinator.local:7878), paste the enrollment address (coordinator.local:7879) and the printed token, and press Redeem. The client redeems it (cert claim's wire equivalent), stores the returned certificate bundle, fills in the certificate folder, and completes the mutual-TLS handshake on the next Test Connection or render. The same form holds the live stream preferences and the default export Transfer; the app keeps exactly one such endpoint, and a settings file from an older build that listed several workers keeps only the first.

Ray-Traced and Path-Traced

Protocol 28 appends EngineSpec as the final SceneState field; protocol 29, the current version, renders the backdrop as a lit tray. All viewers, coordinators and workers must update together. Ray-Traced sample ranges are fixed AA tap ranges, CPU/GPU lanes use the selected engine, and DisplayOnly skips denoising. Frozen Ray-Traced jobs use format 2; historical format-1 jobs remain Path-Traced. The CLI accepts --engine path|ray.

Ray-Traced is a deterministic approximation: one d-line index, or three fixed indices with Fire, RGB absorption tint and 1/4/8/16 AA taps. It finishes at the tap count without denoising. It omits polarization, birefringent doubling, scattering, zoning and edge rounding. Path-Traced retains the spectral model. Optical metrics, tilt sweeps and diagnostics keep their separate analytic tracer.