Tag: Spark
All the articles with the tag "Spark".
-
Partitioning Strategy in Data Lakes: When It Hurts Performance
Partitioning speeds up queries only when it matches filters and data volume. When over-partitioning backfires, and what to use instead.
-
Delta Lake Schema Evolution Explained with Real Examples
Schema enforcement vs. evolution in Delta Lake: mergeSchema, overwriteSchema, MERGE evolution, column mapping for renames and drops, and type widening.
-
Small Files Problem in Delta Lake: Detection and Compaction Strategy
How to detect small files in a Delta table from the transaction log, and a compaction strategy using OPTIMIZE, optimized writes and auto compaction.
-
Debugging AWS Glue Job Bookmarks Not Working as Expected
Glue job bookmarks fail silently: reprocessed data, skipped files, or no state at all. A checklist of the root causes and how to verify each one.
-
Window Functions vs JOIN in Spark: A Physical Plan Perspective
When solving analytical queries in Spark SQL, there are often multiple correct formulations. However, they do not produce equivalent execution plans. This article compares two approaches to the same
-
Hiding Personal Information in AWS Glue with Spark
Protecting personal data before analytics consumption is a core requirement in modern data platforms. In AWS-based lake architectures, this is typically achieved through data de-identification during
-
Handling Boolean vs IntegerType Mismatches Between MySQL and Spark (Glue JDBC)
When integrating MySQL data into Apache Spark (for example, through AWS Glue), you might encounter schema mismatches caused by how MySQL represents TINYINT(1) fields. This issue often surfaces when
-
Managing Evolving Schemas in Apache Spark: A Strategic Approach
Schema management is one of the most overlooked yet critical aspects of building reliable data pipelines. In a fast-moving environment, schemas rarely remain static: new fields are added, data types