Skip to content
Home ยป python

python

Spark 3.0: Solving the “dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z” error

  • by
  • 2 min read

In the past couple of weeks, I’ve been working on a project which users Spark pools in Azure Synapse. However, this appears to be a general Spark issue. I was unable to write to delta lake using Spark because I received the following error. You may get a different result… 

Solve TypeError: ‘dict’ object does not support indexing when running SQL queries in Python

  • by
  • 2 min read

I ran into another silly error, for which I wanted to share the solution in order to save you some time. It occurs when trying to run a query using Python’s SQLAlchemy libary. Let’s dive right in. The problem When you’re trying to run a query, either by using Pandas’… 

How to copy the first value of a group to all rows in the group of a Pandas DataFrame

  • by
  • 2 min read

Recently, I needed to calculate the relative change of a value compared to the first value within a DataFrame group, a GroupBy object. With SQL, I would use a window function (e.g. PARTITION BY in PostgreSQL). In this article, we’ll do it in Pandas (Python). To demonstrate, let’s start with…