Reading CSVs is always a little bit living on the edge, especially when multiple regions are involved in producing them. In this blog post, we’re solving UnicodeDecodeError: ‘utf-8’ codec can’t decode byte […] in position […]: invalid continuation byte. Important, I’m assuming you got the error when you used Pandas’…
There are many situations that you have an 1:n relationship, where you would have wanted an 1:1 relationship. Sometimes, it makes sense to concatenate the strings from rows belonging to the same group. Let’s do that in Python, using Pandas. Imagine, you have the following table that contains an ID…
This blog post is the first post in a two-part series on subsetting Pandas DataFrame rows using chained conditions. In this post, we tackle the following ValueError. Part 1: Bitwise operators Part 2: Parentheses Filtering (or subsetting) a DataFrame can easily be done using the loc property, which can access…
This blog post is the second post in a two-part series on subsetting Pandas DataFrame rows using chained conditions. In this post, we tackle the following TypeError. Part 1: Bitwise operators Part 2: Parentheses Filtering (or subsetting) a DataFrame can easily be done using the loc property, which can access…
Recently, I’ve been working with Pandas DataFrames that had a DateTime as the index. When I tried reindexing the DataFrame (using the reindex method), I bumped into an error. Let’s find out what causes it and how to solve it. The Python error I’m talking about is: A “duplicate axis”?…
JSON or JavaScript Object Notation is a popular file format for storing semi-structured data. Transforming it to a table is not always easy and sometimes downright ridiculous. However, Pandas offers the possibility via the read_json function. If you are not familiar with the orient argument, you might have a hard…
Pandas can be somewhat puzzling, sometimes. It has a ton load of functionalities, but that can make the syntax and methods obscure. Simply judging from the method name, the ‘join’ and ‘merge’ method could be the same thing. However, they aren’t. Here’s an error that I used to run into…
I’ve spent hours trying to find an elegant solution for this problem, and I’m ashamed about how easy it eventually was. I had a column with comma-separated values that I wanted to create features from. Each value should represent one feature. I got started with building loops over the values…
Surprisingly, the Pandas error “columns overlap but no suffix specified”, is one I ran into surprisingly late. Solving it is usually very straightforward. We’ll tackle it in this blog post. First, let’s take a closer look at the error: “ValueError: columns overlap but no suffix specified: Index([<list of columns>], dtype=’object’)”…
In a previous post, I explained how you can sample two Pandas DataFrame exactly the same way. In this blog post, I want to use that helper function to undersample your predictors and target variable. When you are working with an imbalanced data set, it’s often good practice to under-…