Comparing Parquet Compression Codecs: Snappy vs ZSTD Benchmark

>GLB_ // tools · Apache Parquet

Snappy is the default Parquet codec in Spark and PyArrow. ZSTD keeps showing up as “the better default”. I wanted numbers on a realistic dataset instead of vendor charts, so I ran the comparison myself. Script at the end; run it on your data, because results depend heavily on the data.

Setup

• Data: 5,000,000 rows of synthetic event data, 334 MB as Arrow in memory:

• Writer: PyArrow 25.0.1, pq.write_table, row groups of 1M rows, defaults otherwise (dictionary encoding on).
• Machine: 2 vCPU Intel Xeon @ 2.80 GHz, Linux, local disk.
• Method: median of 3 runs for most configs; GZIP-6 and ZSTD-19 single run (they’re slow). Read = pq.read_table of the full file.

Results

What the numbers say

ZSTD-1 vs Snappy: 45% smaller files for ~25% more write time, same read time. 164 MB → 91 MB. Write went from 1.67 s to 2.08 s; read was 0.57 s vs 0.53 s, within noise. On object storage, where you pay for bytes stored, bytes scanned (Athena, BigQuery external tables) and network transfer, that’s a clear win.

Higher ZSTD levels mostly don’t pay off here. Levels 1, 3 and 9 land at ~91–92 MB. Level 19 gets to 83 MB (another ~9%) at 50× the write time of level 1. Only worth it for write-once, read-for-years archives. Note level 3 came out marginally larger than level 1 on this data; not every dataset rewards higher levels.

Decompression speed is flat across ZSTD levels. Read times were ~0.48 s for levels 3 to 19. ZSTD’s decode cost doesn’t grow with the compression level, which is why heavy levels are viable for cold data.

GZIP loses on every axis. Bigger than ZSTD-1, 7× (level 6) to 57× slower to write, and slowest to read. The default-level run took almost 2 minutes. I see no reason to choose GZIP for new Parquet data unless a consumer can’t read ZSTD.

LZ4 ≈ Snappy on size, slightly faster read. If you’re on Snappy for speed, LZ4 is equivalent; ZSTD-1 is the better trade.

Encoding does a lot of the work before the codec runs. Uncompressed Parquet is already 234 MB vs 334 MB in Arrow memory, thanks to dictionary encoding on low-cardinality strings and Parquet’s encodings. The codec compresses what’s left. Columns like session_hash (high-entropy strings) barely compress with anything; columns like event_ts (sorted) compress extremely well.

Caveats

• Synthetic data. Real data usually has more repetition (compresses better) or more entropy in strings (worse). Run this on a sample of your own tables.
• Small machine, local disk. On S3, read time is dominated by network; smaller files then read faster end-to-end, which favors ZSTD even more.
• Single-threaded comparison per file. Engines like Spark parallelize across files and row groups; relative codec costs stay similar.
• Compatibility. ZSTD in Parquet is supported by current Spark, Trino, DuckDB, Athena engine v3, BigQuery, Snowflake and PyArrow. Very old readers may not support it. Verify every consumer.

How to switch

PyArrow:

pq.write_table(table, "out.parquet", compression="zstd", compression_level=1)

Spark:

spark.conf.set("spark.sql.parquet.compression.codec", "zstd")
# or per write:
df.write.option("compression", "zstd").parquet("s3://lake/out/")

Delta Lake (data files are Parquet, so the same Spark conf applies):

spark.conf.set("spark.sql.parquet.compression.codec", "zstd")

Existing files keep their codec. New writes and OPTIMIZE rewrites use the new one, so a table converges as it’s compacted. Mixing codecs in one table is fine; the codec is stored per column chunk.

DuckDB:

COPY tbl TO 'out.parquet' (FORMAT parquet, COMPRESSION zstd, COMPRESSION_LEVEL 1);

My default

• Hot/warm analytical data: ZSTD level 1–3.
• Cold archives, write once: ZSTD 9+ if you’ve measured a gain on your data.
• Snappy: only where a consumer requires it.
• GZIP: avoid for new data.

Reproduce it

import time, os, statistics, numpy as np, pyarrow as pa, pyarrow.parquet as pq

N = 5_000_000
rng = np.random.default_rng(42)
start = np.datetime64("2026-01-01T00:00:00")
table = pa.table({
    "event_id": np.arange(N, dtype=np.int64),
    "event_ts": start + np.sort(rng.integers(0, 86400 * 90, N)).astype("timedelta64[s]"),
    "user_id": rng.integers(1, 200_000, N, dtype=np.int64),
    "country": pa.array(rng.choice(["US", "AR", "DE", "BR", "IN", "GB", "FR", "MX"], N)),
    "event_type": pa.array(rng.choice(["view", "click", "add_to_cart", "purchase"], N, p=[.7, .2, .07, .03])),
    "amount": np.round(rng.lognormal(3, 1, N), 2),
    "session_hash": pa.array([f"{x:016x}" for x in rng.integers(0, 2**63, N)]),
})

configs = [("NONE", None), ("SNAPPY", None), ("LZ4", None), ("GZIP", 6),
           ("ZSTD", 1), ("ZSTD", 3), ("ZSTD", 9), ("ZSTD", 19)]
os.makedirs("out", exist_ok=True)
print("codec,level,size_MB,write_s,read_s")
for codec, lvl in configs:
    path = f"out/{codec}_{lvl}.parquet"
    w, r = [], []
    for _ in range(3):
        t = time.perf_counter()
        pq.write_table(table, path, compression=codec, compression_level=lvl,
                       row_group_size=1_000_000)
        w.append(time.perf_counter() - t)
        t = time.perf_counter(); pq.read_table(path); r.append(time.perf_counter() - t)
    print(f"{codec},{lvl or '-'},{os.path.getsize(path)/1e6:.1f},"
          f"{statistics.median(w):.2f},{statistics.median(r):.2f}")

References

• Apache Parquet — Compression codecs: https://parquet.apache.org/docs/file-format/data-pages/compression/
• Apache Parquet — Encodings: https://parquet.apache.org/docs/file-format/data-pages/encodings/
• PyArrow — pyarrow.parquet.write_table (compression, compression_level): https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html
• Spark SQL — Parquet configuration (spark.sql.parquet.compression.codec): https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#configuration
• DuckDB — Parquet export options: https://duckdb.org/docs/data/parquet/overview.html
• Zstandard — Reference and benchmarks: https://facebook.github.io/zstd/