Posts
All the articles I've posted.
-
Managing Evolving Schemas in Apache Spark: A Strategic Approach
Schema management is one of the most overlooked yet critical aspects of building reliable data pipelines. In a fast-moving environment, schemas rarely remain static: new fields are added, data types
-
Orchestrating Multiple AWS Glue Workflows: A Practical Guide
AWS Glue provides a robust environment for building and managing ETL pipelines, but many data engineers face the challenge of chaining or coordinating multiple workflows . This article explores
-
Secure Ways to Share Private Data on AWS: Beyond Public Buckets
When building data platforms in the cloud, it is common to share data with partners, clients, or internal teams outside your own. AWS provides several mechanisms to grant secure, granular access — far
-
Designing a Semantic Layer for Athena + Power BI
Modern data architectures benefit from a clear separation of layers: Ingesta , Staging , and Semantic (Presentation) . When using Amazon Athena as the query engine and Power BI as the visualization
-
Querying JSONB in PostgreSQL Efficiently
In modern applications, it is common to store semi-structured data in JSON format inside a relational database like PostgreSQL. However, to analyze this data properly, you need a way to transform it
-
Understanding Window Functions in SQL: Beyond Simple Aggregations
When we think about SQL functions, we often start with scalar functions ( UPPER() , ROUND() , NOW() ) or aggregate functions ( SUM() , AVG() , COUNT() ). But there is a third type that is essential
-
Automating Data Extraction with Airflow, BeautifulSoup, and MinIO
In the data engineering ecosystem, a common task is to automate the extraction of data from external sources, perform minimal processing, and store it in a data lake for further analysis. In this
-
How to Set CloudWatch Log Retention Policies with Terraform
AWS CloudWatch is a powerful service for monitoring applications and infrastructure. However, by default, CloudWatch Logs are configured to never expire . This can lead to excessive storage costs and