Surviving spot executors: How Celeborn saved a shuffle-heavy Spark pipeline
Who this is for: Engineers running shuffle-heavy Spark/Kyuubi pipelines—large table rebuilds (CTAS) or incremental-merge dbt models on spot-heavy Kubernetes clusters—who are seeing frequent executor churn, preemption, or Out of Memory (OOM) during shuffle write.
Key takeaway: For shuffle-heavy dbt models failing on spot executors, evaluate remote shuffle before expanding the spot fleet.
Abstract
Cloud providers offer spot-market compute at deep discounts, but those machines can disappear with little warning. For shuffle-heavy Spark jobs that means losing in-progress intermediate data and triggering expensive retries—because shuffle durability is tied to executor lifetime.
Apache Celeborn changes that failure model by moving shuffle data to a replicated remote tier, decoupling executor lifetime from shuffle durability. In these tests, Celeborn's main value was reliability: it completed the full job on 13 spot executors, where local shuffle at the same sizing failed and needed 34 executors to succeed. This reliability can enable lower-cost spot usage, but actual cost efficiency still depends on fleet size, Celeborn infrastructure cost, runtime overhead, and preemption rate.
Key Terminology
- Shuffle—Redistributing the data between worker machines while a job runs.
- Executor—A worker process that executes part of a Spark job.
- Spot (preemptible) instance—A discounted cloud machine the provider can reclaim with little warning.
- DBT—Framework that handles transformation in data pipelines.
- Remote Shuffle Service (RSS)—Stores intermediate shuffle data on a dedicated, durable tier instead of executor local disk.
- AQE—adaptive query execution; Spark's runtime query optimization.
- OOM—out of memory.
- Decommission—gracefully shutting down an executor or worker node.
- PVC—persistent volume claim; a piece of durable storage mounted to a pod.
1. The Spot-Shuffle Problem
Recurring data transformation pipelines (dbt on Spark/Kyuubi) are shuffle-intensive by nature—JSON parsing, array flattening, and Iceberg CTAS/incremental merges all generate large intermediate datasets. Spot instances commonly offer 60–90% cost savings, but the widely recommended pattern of running the driver on-demand and executors on spot breaks down for these workloads. dbt is our example workload here, but the pattern applies to any shuffle-heavy Spark pipeline.
The reason is shuffle-loss amplification: Spark can recompute lost tasks, but lost shuffle files force larger stage retries. Shuffle data normally lives on executor-local disk. When an executor is preempted, OOM-killed, or scaled in, those blocks disappear with it. One executor's loss can invalidate work across hundreds of downstream tasks. AWS typically gives about two minutes of spot interruption warning, depending on interruption type and provider, while Kubernetes evictions may arrive with no useful warning.
The central hypothesis: disaggregating shuffle durability with Celeborn converts spot pricing from a risky discount into a repeatable production cost advantage.
2. How Celeborn Makes Shuffle Durable
Apache Celeborn decouples intermediate shuffle data from compute engine executors. Instead of writing shuffle outputs to local executor disks, map tasks push partitioned data over the network to Celeborn workers, which aggregate all records for a given partition and persist them—optionally replicated across two workers. Reducers then read from Celeborn, not from executors.
Two consequences follow. First, once task data is pushed and acknowledged, the executor can be preempted, decommissioned, or scaled down without invalidating anything. Second, because shuffle output no longer lives on executor disk, Spark's shuffle tracking stops pinning executors, so dynamic allocation can scale in aggressively mid-job.
| Property | Local shuffle | Celeborn RSS |
|---|---|---|
| Shuffle durability | Bound to executor / PVC lifetime—blocks vanish when the node dies | Independent of executor lifetime |
| Executor loss impact | Stage retry risk | Mapper retry only |
| Dynamic allocation | Blocked without ESS/PVC | Executors release after push |
| Replication | Single replica (default) | Dual-replica on workers |
3. Why Existing Mitigations Fall Short
The below mitigations help, but they still leave shuffle data too closely tied to executor's lifetime. YARN had External Shuffle Service for this problem, but vanilla Spark does not provide an equivalent built-in option on Kubernetes. Teams usually reach for Spark-native mitigations before considering a separate shuffle tier. Each of these approaches helps, but all of them leave shuffle data tied to the executor's lifetime—which is the gap Celeborn fills.
| Mitigation | Mechanism | Limitation under spot |
|---|---|---|
| Lineage recomputation | Re-run lost map tasks from lineage | Full shuffle rewrite—expensive |
| spark.decommission.enabled | Gracefully shut down and decommission executors | 2-min window; not guaranteed |
| PVC + dynamic allocation | Persist shuffle blocks on a mounted PVC so they survive executor loss | Reported: bind latency; idle PVC cost (not measured here) |
| Push-based shuffle (Magnet) | Merge pushed blocks from multiple mappers into larger, consolidated blocks | Still co-located; limited K8s support |
| On-demand during shuffle | Fall back to on-demand executors during shuffle-heavy stages | Defeats cost optimization |
| YARN External Shuffle Service | Dedicated shuffle daemon on each node; keeps shuffle blocks beyond executor lifetime | YARN-only; no vanilla Spark equivalent on Kubernetes |
Celeborn is not competing with push-based shuffle (Magnet) on merge efficiency—it competes with local shuffle durability as the blocker to throwaway compute.
4. Test Setup and Results
Pipeline: a dbt project for semi-structured data that parses nested JSON, flattens repeated arrays, and merges results into incremental Iceberg tables. Each run executes 41 Spark jobs through Kyuubi on Spark 3.5.6 and Kubernetes. The first shuffle-heavy CTAS model acts as the canary: if it fails, the pipeline stops.
| Dimension | R1: Spot + Celeborn | R2: Spot + local shuffle | R3: On-demand + local shuffle |
|---|---|---|---|
| Executor profile | 12 min / 30 max; 13 peak | 12 min / 30 max; 13 peak at failure | 30 min / 50 max; 34 peak |
| Driver memory | 100 GB | 100 GB | 160 GB |
| Outcome | Success | Failed | Success |
| Jobs completed | 41/41 | 5/41 | 41/41 |
| Wall-clock | 71.4 min | 35.4 min before abort | 51.4 min |
| Shuffle I/O | 883 GB | N/A | 1,592 GB |
| Key interpretation | Reliable at constrained spot sizing | Local shuffle failed at same sizing | Completed faster, but with a larger on-demand fleet |
Three configurations were executed (C1, C2, C4); C3 (decommission + PVC) was designed but not run. Each config is a single run.
Note: R1 and R3 changed several variables at once—executor count, memory sizing, shuffle manager, and AQE settings—so treat the speed and sizing differences as directional, not as a precise measurement of Celeborn overhead.
5. What the Results Show
Key finding: At the same executor sizing, Celeborn completed the pipeline, and local shuffle did not.
R1 (Celeborn) and R2 (local shuffle) ran the same 13-executor profile and the same 100 GB driver memory. R1 finished all 41 jobs; R2 aborted at model 5 with executor OOM during shuffle write. Local shuffle only completed the DAG in R3, after scaling to 34 executors—2.6× the fleet. The variable that changed the outcome was where shuffle data lived, not how much compute was available.
The takeaway: Celeborn trades some wall-clock speed for reliability at smaller, discounted sizing.
Illustrative cost analysis
This compares executor-only compute spend under a simple unit-price assumption; it is illustrative, based on single run, and not a benchmark.
| R1 (Celeborn, spot) | R3 (local shuffle, on-demand) | |
|---|---|---|
| Executors | 13 | 34 |
| Runtime | 71.4 min (1.19 h) | 51.4 min (0.86 h) |
| Executor-hours | 15.5 | 29.2 |
| Unit price | $0.25 (75% spot discount) | $1.00 |
| Executor cost | ~$3.90 | ~$29.20 |
| Excluded | Celeborn worker tier | — |
6. When to Use Celeborn—and When Not To
| Signal | Recommendation |
|---|---|
| First CTAS/shuffle-heavy model OOMs or retries under spot | Evaluate Celeborn RSS |
| Pipeline completes on spot with decommission + PVC, no retries | Spark-native mitigations may suffice |
| Dynamic allocation blocked by shuffle tracking | RSS enables aggressive scale-in |
| No-preemption, on-demand executors, no sizing pressure | Local shuffle is faster; RSS adds cost |
7. Conclusion
Spot instances are one of the largest levers for reducing batch analytics cost, but shuffle fragility often prevents teams from using them safely. Apache Celeborn externalizes shuffle durability, decoupling executor lifetime from intermediate data. In this pipeline, that changed the failure mode: a constrained local-shuffle run aborted on executor OOM, while the Celeborn run completed the full DAG on spot executors. The primary value demonstrated here is reliability at constrained sizing, not raw performance. Whether spot executors with Celeborn beat a right-sized on-demand local shuffle configuration on cost per run depends on executor sizing, RSS infrastructure cost, runtime overhead, and the observed preemption rate.