Dataframes

Recently, following users’ feedback, I started solving more puzzles in my blog. To choose a problem for today I went to Project Euler page, which provides hundreds of very nice puzzles. I wanted to avoid solving some very easy problem, so I decided to pick the first puzzle…

So we learned two methods to remove duplicates. distinct() dropDuplicates() The distinct method is similar to DISTINCT in the SQL, and it does not support complex scenarios. The dropDuplicates() is more powerful. It takes a list of columns, compares records only using the given columns, and eliminates duplicates.

The withColumn() is super convenient. However, it has got some problems. Each withColumn() method introduces a new projection in the SQL execution plan. Therefore, calling it multiple times can generate big SQL execution plans, which can cause performance issues and even StackOverflowException. …

Spark gives you two methods to select one or more columns from your dataframe. select() selectExpr() The select() method takes only two types of arguments. Column Name Column Object df=spark.read.format(“csv”).option(“inferschema”,”true”).option(“samplingRation”,”0.0001").option(“header”,”true”).load(“/databricks-datasets/airlines/part-00000”)

If you are tired of learning “pandas-killers” then I hear you. I’ve myself spend way too many earnings trying to push through the limits of pandas by reaching out for Dask/Modin/Vaex/cuDF, etc… but hear me out — I think this might be it! Why should I learn Polars? There are two main reasons for this: …

So we have two count() methods in Spark: Dataframe.count() GroupedData.count() If you are using a count before the groupBy(), it will be an action, and you cannot chain any other transformation after the count () Action. However, if you use a count() method after the groupBy(), it will be a transformation, and you can chain further transformations.

Vaex is an incredibly powerful DataFrame library in Python capable of processing hundreds of millions or even billions of rows per second, without loading the entire dataset in memory. …

We often perform join, union, difference, intersection, etc. operations between python DataFrames. Throughout my data science journey, Understanding the concepts of Set Theory has helped me perform these tasks efficiently. In this article, I would like to use the set theory to implement DataFrame operations. The topic of discussion! 1…