Tag: Spark

All the articles with the tag "Spark".

• Hiding Personal Information in AWS Glue with Spark

  25 Jan, 2026

  Protecting personal data before analytics consumption is a core requirement in modern data platforms. In AWS-based lake architectures, this is typically achieved through data de-identification during

• Managing Evolving Schemas in Apache Spark: A Strategic Approach

  16 Sep, 2025

  Schema management is one of the most overlooked yet critical aspects of building reliable data pipelines. In a fast-moving environment, schemas rarely remain static: new fields are added, data types

• Debugging Spark DataFrame .show() Timeouts in PyCharm and VSCode

  3 Sep, 2025

  When working with PySpark , one of the first commands developers use to quickly inspect data is: raw_df.show() However, in certain environments (especially when running inside PyCharm or VSCode with a

• Fixing Spark Ivy Error in Docker: "basedir must be absolute"

  9 May, 2025

  If you're running Apache Spark inside Docker using Bitnami's images and suddenly encounter an Ivy error that says: Exception in thread "main" java.lang.IllegalArgumentException: basedir must be

• How Spark and MapReduce Handle Partial Records in HDFS

  5 May, 2025

  When working with large-scale data processing frameworks like Apache Spark or Hadoop MapReduce, one common question arises: What happens when a record (e.g., a line of text or a JSON object) is split

• Understanding .master() in Apache Spark

  27 Apr, 2025

  In Apache Spark, the .master() method is used to specify how your application will run, either on your local machine or on a cluster. Choosing the correct option is essential depending on your

• Running PySpark on Google Colab: Do You Still Need findspark?

  5 Apr, 2025

  Introduction For a long time, using Apache Spark in Google Colab required manual setup, including installing Spark and configuring Python to recognize it. This was often done using the findspark

• Matei Zaharia - Spark: The Definitive Guide - Architecture of a Spark Application

  23 Oct, 2022

  The Architecture of a Spark Application The Spark driver The driver is the process “in the driver seat” of your Spark Application. It is the controller of the execution of a Spark Application and