Tag: Data Engineering
All the articles with the tag "Data Engineering".
-
Delta Lake Schema Evolution Explained with Real Examples
Schema enforcement vs. evolution in Delta Lake: mergeSchema, overwriteSchema, MERGE evolution, column mapping for renames and drops, and type widening.
-
Small Files Problem in Delta Lake: Detection and Compaction Strategy
How to detect small files in a Delta table from the transaction log, and a compaction strategy using OPTIMIZE, optimized writes and auto compaction.
-
Comparing Parquet Compression Codecs: Snappy vs ZSTD Benchmark
A reproducible PyArrow benchmark of Parquet codecs on 5M rows: Snappy, LZ4, GZIP and ZSTD levels 1–19. File size, write and read time.
-
S3 Eventual Consistency Edge Cases in Data Pipelines
S3 has been strongly consistent since 2020. The edge cases that still break data pipelines: no atomic rename, concurrent writers, LastModified, events.
-
Designing Event Schemas for Product Analytics in ClickHouse
Fixed columns vs Map vs JSON, ORDER BY for funnels, dedup, TTL and materialized views: a practical event schema for product analytics in ClickHouse.
-
Why ClickHouse Is Fast: Understanding MergeTree Internals
Parts, granules, sparse primary indexes, marks and background merges: a hands-on look at how MergeTree lets ClickHouse skip most of your data.
-
AWS Glue Job Fails Due to Dependency Version Conflict: How to Fix It
Your Glue job works locally but fails with ImportError or numpy binary incompatibility. Why it happens and how to pin dependencies against the Glue runtime.
-
CCLF is not eligibility: where dual status gets lost in Medicare ACO data
How a coalesce and a latest-month rule in Tuva's CCLF + ALR connectors can turn full-dual Medicare beneficiaries into non-duals in CMS-HCC.