Surprisingly, the Pandas error “columns overlap but no suffix specified”, is one I ran into surprisingly late. Solving it is usually very straightforward. We’ll tackle it in this blog post. First, let’s take a closer look at the error: “ValueError: columns overlap but no suffix specified: Index([<list of columns>], dtype=’object’)”…
Creating a tracking solution has never been easier. There are numerous (free) tools available to collect and store event data. In this blog post I explain how to collect data using Fivetran webhooks and storing it in ElephantSQL, a managed PostgreSQL database. In this blog post, we will: Set up…
In this short blog post, I tackle an error related to a classic problem within machine learning: how to treat unseen categorical values and solve the “found unknown categories” error. Imagine you have a train and a test data set with the following values in a column: Train: [‘green’, ‘red’,…
In the past couple of weeks, I started to use sklearn pipelines more intensively. It seemed like a good project to find out more about them and share my experiences in a blog post. So here it is: a sklearn pipeline tutorial. Why pipelines? In most machine learning projects, you…
In a previous post, I explained how you can sample two Pandas DataFrame exactly the same way. In this blog post, I want to use that helper function to undersample your predictors and target variable. When you are working with an imbalanced data set, it’s often good practice to under-…
I’ve been trying to build a model using machine learning today and I bumped into an error when I wanted to dummify my categorical predictors. It seemed I didn’t really know how Scikit’s OneHotEncoder worked. But I do now. And I want to share it with you. I had a…
In this blog post, two quick solutions to sample two pandas data frames in the same way. Why? Because sometimes you load your predictors (X) and your target (y) from two different files and you don’t want to put them together in one table. Sampling two data frames with a…
The SettingWithCopyWarning message is a confusing warning to many who are new to Pandas. If you’ve ever taken a computer science course, you might be aware of passing/copying by value or by reference. Well, it very much applies to pandas DataFrames too. Let’s go. Basically, when you are slicing a…
This is a cool one I used for a feature engineering task I did recently. I had multiple documents in a Pandas DataFrame, in long format. These documents belonged to people and it had an n:1 relation: people could have multiple documents. I was wondering how to concatenate each person’s…
Coming from R, I’m not a big fan of MultiIndex DataFrames in Pandas. I definitely see the merits, but it just doesn’t feel right within a machine learning and feature engineering context. However, sometimes you will end up with a MultiIndex DataFrame, after some ninja line of code. In this…
