Storage

File and table formats, and how a data lake is laid out: Parquet, Iceberg, Delta, Hive, Trino, Databricks, partitioning, small files.

Tech

All posts (25)

• Partitioning Strategy in Data Lakes: When It Hurts Performance

  8 Oct, 2026

  Partitioning speeds up queries only when it matches filters and data volume. When over-partitioning backfires, and what to use instead.

• Delta Lake Schema Evolution Explained with Real Examples

  8 Oct, 2026

  Schema enforcement vs. evolution in Delta Lake: mergeSchema, overwriteSchema, MERGE evolution, column mapping for renames and drops, and type widening.

• Small Files Problem in Delta Lake: Detection and Compaction Strategy

  8 Oct, 2026

  How to detect small files in a Delta table from the transaction log, and a compaction strategy using OPTIMIZE, optimized writes and auto compaction.

• Comparing Parquet Compression Codecs: Snappy vs ZSTD Benchmark

  8 Oct, 2026

  A reproducible PyArrow benchmark of Parquet codecs on 5M rows: Snappy, LZ4, GZIP and ZSTD levels 1–19. File size, write and read time.

• S3 Eventual Consistency Edge Cases in Data Pipelines

  8 Oct, 2026

  S3 has been strongly consistent since 2020. The edge cases that still break data pipelines: no atomic rename, concurrent writers, LastModified, events.

• Modern Table Formats: Iceberg, Delta Lake, and Hudi

  1 Oct, 2025

  Data Lakes made it possible to store raw data at scale, but they lacked the reliability and governance of data warehouses. Files could be dropped into storage (S3, HDFS, MinIO), but analysts struggled

• Trino in Modern Architectures: SQL Queries on S3 and MinIO

  30 Sep, 2025

  The rise of cloud object storage has transformed how organizations build data platforms. Hadoop Distributed File System (HDFS) once dominated, but today services like Amazon S3, Google Cloud Storage

• Hive Metastore: The Glue Holding Big Data Together

  23 Sep, 2025

  When people think of Hive, they often remember the early days of Hadoop and MapReduce. But while Hive as a query engine has largely faded, one of its components remains critical to the modern data

• Why Parquet Became the Standard for Analytics

  23 Sep, 2025

  In the early days of Big Data, data was often stored in simple formats such as CSV, JSON, or text logs. While these formats were easy to generate and understand, they quickly became inefficient at

• The History of Hive and Trino: From Hadoop to Lakehouses

  19 Sep, 2025

  The evolution of Big Data architectures is deeply tied to the history of two projects born at Facebook: Hive and Trino . Both emerged from real engineering pain points, but at different times and for

• What Is a Data Lake and What Is a Data Lakehouse?

  19 Sep, 2025

  Over the last decade, the world of data architecture has gone through several transformations. From traditional data warehouses to Hadoop-based data lakes and now to the emerging Lakehouse paradigm,

• Choosing Between saveAsTable and Iceberg’s writeTo in AWS Glue and Athena

  3 Sep, 2025

  When working with Spark on AWS Glue , there are multiple ways to persist DataFrames as tables and make them queryable in Amazon Athena . Two common approaches are: Using Spark’s Hive-style saveAsTable

• Optimizing Partition Strategies in Apache Iceberg on AWS

  24 Jul, 2025

  When working with large-scale analytical datasets, efficient partitioning is critical for achieving optimal query performance and cost savings. Apache Iceberg, a modern table format designed for big

• How Transactions Work in Databricks Using Delta Lake

  22 Jul, 2025

  Databricks is a powerful platform for big data analytics and machine learning. One of its key features is the ability to run transactional workloads over large-scale data lakes using Delta Lake . This

• When Should You Use Iceberg with Athena? Partitioning Strategies and Best Practices

  5 Jul, 2025

  As data lakes grow in size and complexity, tools like Amazon Athena combined with table formats like Apache Iceberg become essential for scalability, data governance, and performance. In this post,

• When Should You Use Parquet and When Should You Use Iceberg?

  18 Jun, 2025

  In modern data architectures, selecting the right storage and management solution is essential for building efficient, reliable, and scalable pipelines. Two popular choices that often come up are

• How to Fix 'DataFrame' object has no attribute 'writeTo' When Working with Apache Iceberg in PySpark

  17 Jun, 2025

  If you’re working with Apache Iceberg in PySpark and encounter this error: Failed to write to Iceberg table: 'DataFrame' object has no attribute 'writeTo' You’re not alone. This is a common mistake

• Apache Cassandra vs Apache Parquet: Understanding the Differences

  14 May, 2025

  In modern data architectures, it's common to encounter both Apache Cassandra and Apache Parquet , particularly when dealing with large-scale, distributed systems. Both technologies are associated with

• From HDFS to S3: The Evolution of Data Lakes in the Cloud

  29 Apr, 2025

  For years, HDFS (Hadoop Distributed File System) was the default choice for building data lakes in on-premises and Hadoop-based environments. But as cloud computing gained momentum, a new player took

• Is S3 the New HDFS? Comparisons and Use Cases in Big Data

  29 Apr, 2025

  Over the past decade, the way organizations store and manage big data has shifted dramatically. Once dominated by the Hadoop Distributed File System (HDFS) , the field is now led by Amazon S3 and

• Understanding How Hive Converts SQL Queries into Hadoop Jobs

  31 Mar, 2025

  When you execute a SQL query in Apache Hive, the query is not directly run on a traditional database. Instead, Hive translates it into a Hadoop job, which is then executed across a distributed system.

• Delta Lake vs. Traditional Data Lakes: Key Differences and Vendor Options

  30 Mar, 2025

  Introduction As data-driven organizations scale their analytics and machine learning workloads, the limitations of traditional data lakes become more apparent. Delta Lake is an open-source storage

• Understanding the Differences Between Parquet, Avro, JSON, and CSV

  9 Feb, 2025

  When working with data, choosing the right file format can significantly impact performance, storage efficiency, and ease of use. In this post, we will compare four widely used data formats: Parquet,

• Why to choose a Data Lake?

  7 Feb, 2023

  There are some reason that will take you to choose to use a Data Lake as solution for your Data Operations. The most importants are: Increase operational efficiency Make data available faster Lower

• Set secrets in Databricks

  29 Dec, 2022

  If you add as plain text the user and password of your connections you are making a mistake that it's easy to solve. In order to solve You have to install the data bricks cli with pip: pip install