Posts
All the articles I've posted.
-
How to Disable an AWS Glue Trigger from the CLI
When working with AWS Glue , triggers are an important mechanism to orchestrate jobs or workflows. Sometimes, however, you may need to temporarily disable a trigger without deleting it—for example, to
-
Orchestrating Multiple AWS Glue Workflows with Step Functions
In modern data architectures, it is common to manage multiple ETL pipelines that must run in sequence or in parallel. AWS Glue provides a robust framework for building workflows, but when we need to
-
Understanding the Strategy Design Pattern
In the landscape of software design, maintaining flexibility and scalability is crucial. One of the most effective ways to achieve these qualities is by leveraging design patterns. Among the
-
Choosing Between saveAsTable and Iceberg’s writeTo in AWS Glue and Athena
When working with Spark on AWS Glue , there are multiple ways to persist DataFrames as tables and make them queryable in Amazon Athena . Two common approaches are: Using Spark’s Hive-style saveAsTable
-
Debugging Spark DataFrame .show() Timeouts in PyCharm and VSCode
When working with PySpark , one of the first commands developers use to quickly inspect data is: raw_df.show() However, in certain environments (especially when running inside PyCharm or VSCode with a
-
Incremental Data Loads: Choosing Between resource_version and created_at/updated_at
Incremental data loading is a cornerstone of modern data engineering pipelines. Instead of re-ingesting entire datasets on each execution, incremental strategies focus on retrieving only records that
-
Optimizing Amazon Athena Queries with Partitions: A Practical Example
When working with Amazon Athena, one of the most effective strategies to improve query performance and reduce costs is partitioning your data . Partitions allow Athena to scan only the relevant
-
Running Apache Airflow Across Environments
Apache Airflow has become a de facto standard for orchestrating data workflows. However, depending on the environment, the way Airflow runs can change significantly. Many teams get confused when