Coming from Airflow, I expected macros to be all over the place in Kubeflow. Surprisingly, it took me a lot of time to scramble some pieces of documentation on how to use them. In this blog post, I’ll try to give an overview of how and where you can use…
Sometimes you just want to know if a particular external table or schema exists in Amazon Redshift (Spectrum). This article outlines various alternatives to achieve that. First, if you’d like to list the tables that match your criteria, you can do that by querying SVV_EXTERNAL_TABLES. This table has some interesting…
I ran into another silly error, for which I wanted to share the solution in order to save you some time. It occurs when trying to run a query using Python’s SQLAlchemy libary. Let’s dive right in. The problem When you’re trying to run a query, either by using Pandas’…
In this blog post, we’ll make a treemap in Python using Plotly Express. While this seems like a trivial task, there are some intricacies involved. I ran into multiple error, and I want to make sure you don’t. First, let’s load all relevant packages that you’re gonna need throughout this…
In this article, you’ll learn about keeping a Docker container alive, so that it doesn’t exit. We’ll use Ubuntu as an example. When you use docker run, a process is ran in an isolated container. This means that, when you run an a container with a Ubuntu image, the operating…
Recently, I needed to calculate the relative change of a value compared to the first value within a DataFrame group, a GroupBy object. With SQL, I would use a window function (e.g. PARTITION BY in PostgreSQL). In this article, we’ll do it in Pandas (Python). To demonstrate, let’s start with…
In this blog post, I explore the options for taking the row-wise sum of a subset of columns from a Pandas DataFrame. Let’s try to take the row-wise sum of the columns first_column, second_column, and third_column. This means we’re leaving other_column out. But first, let’s take a step back. If…
I’ve been working with Terraform recently for deploying the required architecture for a data pipeline in Google Cloud Platform (GCP). It’s my first project with Terraform and I decided to dive into its random_something resources, because I couldn’t have duplicate resource names. Here are some findings. Randomize a resource name…
In this blog post we’ll do a small thing to remove trailing or leading whitespaces from a DataFrame’s column names. This is something that can occur when working with files that haven’t been properly formatted. There are multiple things you can do to fix this. First, let’s create some dummy…
In this article, you’ll learn how to execute shell commands using the subprocess package in Python. This article is part of a two-part series related to running shell commands from within Python. Part 1: Execute shell commands with the os package Part 2: Execute shell commands with the subprocess package…