Distributed systems

Design and theory behind data systems: GFS, MapReduce, Dynamo, Spark, HDFS internals, CAP, sharding, replication.

Tech

All posts (59)

• Partitioning Strategy in Data Lakes: When It Hurts Performance

  8 Oct, 2026

  Partitioning speeds up queries only when it matches filters and data volume. When over-partitioning backfires, and what to use instead.

• Delta Lake Schema Evolution Explained with Real Examples

  8 Oct, 2026

  Schema enforcement vs. evolution in Delta Lake: mergeSchema, overwriteSchema, MERGE evolution, column mapping for renames and drops, and type widening.

• Small Files Problem in Delta Lake: Detection and Compaction Strategy

  8 Oct, 2026

  How to detect small files in a Delta table from the transaction log, and a compaction strategy using OPTIMIZE, optimized writes and auto compaction.

• Debugging AWS Glue Job Bookmarks Not Working as Expected

  27 Sep, 2026

  Glue job bookmarks fail silently: reprocessed data, skipped files, or no state at all. A checklist of the root causes and how to verify each one.

• Daily Failure Reporting in DynamoDB Using Lambda, EventBridge Scheduler, and SES

  1 Mar, 2026

  Operational monitoring requires structured visibility into failures. If your processes write execution logs to DynamoDB and mark failed executions with status = FAILED , you can implement a

• Window Functions vs JOIN in Spark: A Physical Plan Perspective

  1 Mar, 2026

  When solving analytical queries in Spark SQL, there are often multiple correct formulations. However, they do not produce equivalent execution plans. This article compares two approaches to the same

• Hiding Personal Information in AWS Glue with Spark

  25 Jan, 2026

  Protecting personal data before analytics consumption is a core requirement in modern data platforms. In AWS-based lake architectures, this is typically achieved through data de-identification during

• Handling Boolean vs IntegerType Mismatches Between MySQL and Spark (Glue JDBC)

  12 Nov, 2025

  When integrating MySQL data into Apache Spark (for example, through AWS Glue), you might encounter schema mismatches caused by how MySQL represents TINYINT(1) fields. This issue often surfaces when

• Facebook and Big Data: The Open Source Projects That Changed the Industry

  19 Sep, 2025

  When people talk about the history of Big Data, a few companies come to mind: Google, Yahoo, and Facebook. Each of them faced unique challenges that forced them to build large-scale distributed

• HDFS vs. Object Storage: The Battle for Distributed Storage

  19 Sep, 2025

  Distributed storage has always been the foundation of Big Data. In the early days, Hadoop Distributed File System (HDFS) was the de facto standard. Today, however, object storage systems like Amazon

• The History of Hive and Trino: From Hadoop to Lakehouses

  19 Sep, 2025

  The evolution of Big Data architectures is deeply tied to the history of two projects born at Facebook: Hive and Trino . Both emerged from real engineering pain points, but at different times and for

• Google Bigtable vs. Amazon DynamoDB: Understanding the Differences

  17 Sep, 2025

  When choosing a NoSQL database for scalable, low-latency applications, two major options stand out: Google Cloud Bigtable and Amazon DynamoDB . While both are managed, highly available, and

• Managing Evolving Schemas in Apache Spark: A Strategic Approach

  16 Sep, 2025

  Schema management is one of the most overlooked yet critical aspects of building reliable data pipelines. In a fast-moving environment, schemas rarely remain static: new fields are added, data types

• Debugging Spark DataFrame .show() Timeouts in PyCharm and VSCode

  3 Sep, 2025

  When working with PySpark , one of the first commands developers use to quickly inspect data is: raw_df.show() However, in certain environments (especially when running inside PyCharm or VSCode with a

• Choosing Between DynamoDB and Cassandra for a Crypto Exchange

  13 Jul, 2025

  When designing the backend of a crypto exchange, selecting the right database architecture is crucial. Two common NoSQL databases often considered for this type of application are Amazon DynamoDB and

• Handling Python datetime Objects in Amazon DynamoDB

  13 Jul, 2025

  When developing data pipelines or applications that store time-based records in Amazon DynamoDB , developers frequently encounter serialization errors when working with Python's datetime objects.

• How Hadoop Made Specialized Storage Hardware Obsolete

  6 Jul, 2025

  In the early 2000s, enterprise data processing was dominated by high-end hardware. Organizations relied heavily on centralized storage systems such as SAN (Storage Area Networks) and NAS (Network

• How Google Changed Big Data: The Story of GFS, MapReduce, and Bigtable

  4 Jul, 2025

  In the early 2000s, Google faced a unique challenge: how to store, process, and query massive amounts of data across thousands of unreliable machines. The traditional systems of the time—designed for

• Are NoSQL Databases Really Schema-less?

  19 Jun, 2025

  A Perspective from the MERN Stack When we first start learning about NoSQL databases, one of the most common things we hear is that they are "schema-less." At first glance, this seems like a huge

• How Network Topology Shapes Distributed Computing and Big Data Systems

  19 Jun, 2025

  When discussing distributed systems and Big Data, people often focus on storage, processing frameworks, and scalability—but one foundational concept underlies it all: network topology . It’s the

• How to Fix 'DataFrame' object has no attribute 'writeTo' When Working with Apache Iceberg in PySpark

  17 Jun, 2025

  If you’re working with Apache Iceberg in PySpark and encounter this error: Failed to write to Iceberg table: 'DataFrame' object has no attribute 'writeTo' You’re not alone. This is a common mistake

• What Is Sharding and Why It Matters

  21 May, 2025

  As our world becomes increasingly digital, the amount of data we create every day is staggering. Think about all the emails, messages, orders, and photos uploaded every second. How do big companies

• From Tables to Partitions: Designing NoSQL Databases with Cassandra

  20 May, 2025

  As data professionals transition from relational databases to NoSQL systems like Apache Cassandra, one of the most important mindset shifts is understanding that you don't model data for storage, but

• Apache Cassandra vs Apache Parquet: Understanding the Differences

  14 May, 2025

  In modern data architectures, it's common to encounter both Apache Cassandra and Apache Parquet , particularly when dealing with large-scale, distributed systems. Both technologies are associated with

• Fixing Spark Ivy Error in Docker: "basedir must be absolute"

  9 May, 2025

  If you're running Apache Spark inside Docker using Bitnami's images and suddenly encounter an Ivy error that says: Exception in thread "main" java.lang.IllegalArgumentException: basedir must be

• How Dynamo Reshaped the Internal Architecture of Amazon S3

  9 May, 2025

  Introduction Amazon S3 launched in 2006 as a scalable, durable object storage system. It avoided hierarchical file systems and used flat key-based addressing from day one. However, early versions of

• How HDFS Achieves Fault Tolerance Through Replication

  6 May, 2025

  One of the core strengths of the Hadoop Distributed File System (HDFS) is its fault tolerance . In a world of distributed computing, failures are not rare—they're expected. HDFS tackles this by using

• Summary: Teaching HDFS Concepts to New Learners

  6 May, 2025

  Introducing Hadoop Distributed File System (HDFS) to newcomers can be both exciting and challenging. To make the learning experience structured and impactful, it’s helpful to break down the core

• How Clients Know Where to Read or Write in HDFS

  5 May, 2025

  Hadoop Distributed File System (HDFS) is designed to decouple metadata management from actual data storage . But how does a client—like a Spark job or command-line tool—know where to read or write the

• How HDFS Avoids Understanding File Content

  5 May, 2025

  One of the defining features of Hadoop Distributed File System (HDFS) is that it doesn’t understand the contents of the files it stores . This is not a limitation—it's an intentional design choice

• How Spark and MapReduce Handle Partial Records in HDFS

  5 May, 2025

  When working with large-scale data processing frameworks like Apache Spark or Hadoop MapReduce, one common question arises: What happens when a record (e.g., a line of text or a JSON object) is split

• How HDFS Tracks Block Size and File Boundaries

  4 May, 2025

  When dealing with massive files, Hadoop Distributed File System (HDFS) doesn't read or store them as a whole. Instead, it splits them into large, fixed-size blocks . But how does it know where each

• How Metadata Works in HDFS and What It Stores

  4 May, 2025

  HDFS stores metadata separately from the actual file content to optimize performance and scalability. This metadata is managed entirely by the NameNode , which allows clients to quickly locate and

• The Architecture of HDFS: NameNode, DataNodes, and Metadata

  3 May, 2025

  HDFS (Hadoop Distributed File System) was built to support the reliable storage and access of large datasets distributed across commodity hardware. To make this possible, HDFS relies on a master/slave

• What Happens When HDFS Splits Files Mid-Word or Mid-Row?

  2 May, 2025

  HDFS is designed to store and process massive amounts of data efficiently. One of its key design decisions is to split files into large, fixed-size blocks , typically 128MB or 256MB. But what happens

• How HDFS Handles File Partitioning and Block Distribution

  1 May, 2025

  One of the key innovations behind the Hadoop Distributed File System (HDFS) is how it breaks down large files and distributes them across multiple machines. This mechanism, called partitioning and

• What is HDFS and Why Was It Revolutionary for Big Data?

  30 Apr, 2025

  In the early 2000s, the world was generating data at a scale never seen before—web logs, social media, sensors, and more. Traditional storage systems simply couldn't keep up with the volume, velocity,

• From HDFS to S3: The Evolution of Data Lakes in the Cloud

  29 Apr, 2025

  For years, HDFS (Hadoop Distributed File System) was the default choice for building data lakes in on-premises and Hadoop-based environments. But as cloud computing gained momentum, a new player took

• The History and Evolution of Amazon S3: Was It Ever Based on HDFS?

  28 Apr, 2025

  When discussing cloud storage today, Amazon S3 is almost synonymous with scalable, reliable object storage. However, a common question among those familiar with big data technologies like Hadoop is:

• MapReduce: A Framework for Processing Unstructured Data

  27 Apr, 2025

  MapReduce is both a programming model and a framework designed to process massive volumes of data across distributed systems. It gained popularity primarily due to its efficiency in handling

• Understanding .master() in Apache Spark

  27 Apr, 2025

  In Apache Spark, the .master() method is used to specify how your application will run, either on your local machine or on a cluster. Choosing the correct option is essential depending on your

• Running PySpark on Google Colab: Do You Still Need findspark?

  5 Apr, 2025

  Introduction For a long time, using Apache Spark in Google Colab required manual setup, including installing Spark and configuring Python to recognize it. This was often done using the findspark

• Understanding the Relationship Between Database Replication and the CAP Theorem

  26 Mar, 2025

  Introduction Database replication is a fundamental strategy in distributed systems that ensures data is duplicated across multiple nodes. However, when designing a replicated database, one must

• Understanding the CAP Theorem in NoSQL Databases

  9 Feb, 2025

  The CAP theorem (Consistency, Availability, and Partition Tolerance) plays a crucial role in designing and selecting NoSQL databases. This theorem states that in a distributed system, it is impossible

• Calculating Levenshtein Distance in Apache Spark Using a UDF

  7 Feb, 2025

  When working with text data in big data environments, measuring the similarity between strings can be essential. One of the most commonly used metrics for this is the Levenshtein distance , which

• Creating a PySpark DataFrame for Sentiment Analysis

  4 Feb, 2025

  When working with sentiment analysis, having structured data in a PySpark DataFrame can be very useful for processing large datasets efficiently. In this post, we will create a PySpark DataFrame

• Ranking Products Using Window Functions in PySpark

  30 Jan, 2025

  Introduction Window functions are powerful tools in SQL and PySpark that allow us to perform calculations across a subset of rows related to the current row. In this blog post, we'll explore how to

• Grouping Data in PySpark with Aliases for Aggregated Columns

  13 Jan, 2025

  When working with large datasets in PySpark, grouping data and applying aggregations is a common task. In this post, we’ll explore how to group data by a specific column and use aliases for the

• Counting Word Frequency in a SQL Column

  3 Jan, 2025

  Sometimes, you may need to analyze text data stored in a database, such as counting the frequency of words in a text column. This blog post demonstrates how to achieve this in SQL using a practical

• Understanding Distributed System - Maintainability

  6 Mar, 2024

  Introduction It’s widely recognized that the bulk of software costs arise after its initial development in maintenance tasks like bug fixes, feature additions, and day-to-day operation. Therefore,

• Understanding Distributed System – Resiliency

  6 Mar, 2024

  Introduction Chapter 24 - Common Failure Causes Hardware Faults Incorrect Error Handling Configuration Changes Single Points of Failure Network Faults Resources Leaks Load Pressure Cascading Failures

• Understanding Distributed System – Scalability

  3 Mar, 2024

  Introduction Scaling an application involves maintaining performance as load increases. The long-term solution for increasing capacity is to architect for horizontal scalability. In this section,

• Understanding Distributed System – Coordination

  2 Mar, 2024

  Introduction Our ultimate goal is to build a distributed application consisting of a group of processes that gives its users the illusion they are interacting with one coherent node. While achieving a

• Understanding Distributed System - Communication

  25 Feb, 2024

  Part I - Communication Introduction Interprocess communication (IPC) is fundamental to distributed systems, enabling processes to exchange data over networks. This communication relies on agreed-upon

• Understanding Distributed Systems - Introduction

  28 Dec, 2023

  Chapter 1: Introduction In the realm of modern technology, the need for distributed systems has become increasingly apparent. But why invest time and resources in building such intricate

• Matei Zaharia - Spark: The Definitive Guide - Architecture of a Spark Application

  23 Oct, 2022

  The Architecture of a Spark Application The Spark driver The driver is the process “in the driver seat” of your Spark Application. It is the controller of the execution of a Spark Application and

• Matei Zaharia - Spark: The Definitive Guide - Life Cycle of a Spark Application

  23 Oct, 2022

  The Life Cycle of a Spark Application (Inside Spark) The SparkSession The first step of any Spark Application is creating a SparkSession. In many interactive modes, this is done for you, but in an

• Matei Zaharia - Spark: The Definitive Guide. Common Operations

  22 Oct, 2022

  Define Schemas manually When using Spark for production Extract, Transform, and Load (ETL), it is often a good idea to define your schemas manually, especially when working with untyped data sources

• Kleppmann - Designing Data Intensive Applications

  17 Oct, 2022

  A data-intensive application is typically built from standard building blocks that provide commonly needed functionality. For example, many applications need to: • Store data so that they, or another