I recently got more interested in observability, logging, data quality, etc. For many use cases, you don’t need full-blown observability solutions. Python has robust tools that will help you achieve your logging goals. In this blog post, I discuss a small trick I encountered for using Python’s logging module properly.…
In the past couple of weeks, I’ve been working on a project which users Spark pools in Azure Synapse. However, this appears to be a general Spark issue. I was unable to write to delta lake using Spark because I received the following error. You may get a different result…
It’s been a couple of years since I first used NetworkX in Python. Something that I expected to be truly obvious was adding node attributes to your network, preferably when calling a from_pandas_… function. However, that wasn’t the case. In this article, we’ll create a network from an edge list…
Today, I tried a data transformation that seemed so obvious: splitting the string values of a Pandas column on a delimiter and one-hot encode the resulting strings. However, it took me quite some time to figure out how to do it elegantly. Here’s what I wanted to achieve. I had…
This a brief overview on how to find the dependencies of a Python package. You might need this in case you have to upload packages to a custom package repository or conda channel. Let’s dive in. Find package dependencies with PyPi JSON endpoint To find a Python package’s dependencies, you…
In this article, you’ll learn how you can get metadata from your pipeline run inside your pipeline’s notebooks (or other activities). The documentation on this subject is relatively limited, so I hope it will prevent a lot of wasted time. In Airflow, you have macros, to refer to the metadata…
Coming from Airflow, I expected macros to be all over the place in Kubeflow. Surprisingly, it took me a lot of time to scramble some pieces of documentation on how to use them. In this blog post, I’ll try to give an overview of how and where you can use…
I ran into another silly error, for which I wanted to share the solution in order to save you some time. It occurs when trying to run a query using Python’s SQLAlchemy libary. Let’s dive right in. The problem When you’re trying to run a query, either by using Pandas’…
In this blog post, we’ll make a treemap in Python using Plotly Express. While this seems like a trivial task, there are some intricacies involved. I ran into multiple error, and I want to make sure you don’t. First, let’s load all relevant packages that you’re gonna need throughout this…
Recently, I needed to calculate the relative change of a value compared to the first value within a DataFrame group, a GroupBy object. With SQL, I would use a window function (e.g. PARTITION BY in PostgreSQL). In this article, we’ll do it in Pandas (Python). To demonstrate, let’s start with…