Tag: Spark
All the articles with the tag "Spark".
-
Ranking Products Using Window Functions in PySpark
Introduction Window functions are powerful tools in SQL and PySpark that allow us to perform calculations across a subset of rows related to the current row. In this blog post, we'll explore how to
-
Grouping Data in PySpark with Aliases for Aggregated Columns
When working with large datasets in PySpark, grouping data and applying aggregations is a common task. In this post, we’ll explore how to group data by a specific column and use aliases for the
-
Matei Zaharia - Spark: The Definitive Guide - Architecture of a Spark Application
The Architecture of a Spark Application The Spark driver The driver is the process “in the driver seat” of your Spark Application. It is the controller of the execution of a Spark Application and
-
Matei Zaharia - Spark: The Definitive Guide - Life Cycle of a Spark Application
The Life Cycle of a Spark Application (Inside Spark) The SparkSession The first step of any Spark Application is creating a SparkSession. In many interactive modes, this is done for you, but in an
-
Matei Zaharia - Spark: The Definitive Guide. Common Operations
Define Schemas manually When using Spark for production Extract, Transform, and Load (ETL), it is often a good idea to define your schemas manually, especially when working with untyped data sources