The title of this blog post says it all: let’s solve a warning. It’s one that occurs when you’re using Pandas’ read_csv. Let’s start by just doing that. Let’s say you run into the following error: Solve DtypeWarning: Columns (X,X) have mixed types. Specify dtype option on import or set…
As an R-native, switching to Python and Pandas has its little annoyances. In R, when you print a data frame, or its head, you get to see the whole shebang. In Pandas, this is not the case: rows and columns get truncated to keep the output manageable. However, there are…
Without a doubt, Pandas is one of the most popular Python packages within Data Science. However, coming from an R background, discovering the difference between the join and merge method was mind-boggling. Especially if it generates confusing errors that urge you to use pd.concat, which you shouldn’t. This is the…
When you’re an R poweruser, pivoting tables in pandas feels unnecessarily complex. Why are there two pivot functions? Why does it return an index when you wanted a column? Why does it generate multi index columns? Those are the questions I tackle in this blog post. 💥 This blog post…
Getting a firm understanding of NaNs in your dataset ensures you don’t draw wrong conclusions from an incomplete dataset. In this blog post I show how you can list the amount of NaNs per column, per row, and per group. First, let’s create some dummy data, and add some NaNs.…
Without going into detail, here’s something I truly hate in R: replacing multiple values. In Python’s pandas, it’s really easy. In this blog post I try several methods: list comprehension, apply(), replace() and map(). First, let’s create some dummy data. First, let’s try with list comprehension. The get() function tries…
Here’s another post about seemingly convenient things in pandas. Let’s produce a DataFrame from partially matched columns from another DataFrame. Let’s create some dummy data. Let’s say we are trying to select the columns that contain the world ‘color’. Here’s a solution I found on the web. By converting the…
Having R as my data science lingua franca, some things in pandas feel less intuitive. As I’m slowly transitioning to pandas, I’ll be posting a lot of examples over the next couple of weeks. Here’s the first one: concatenating columns values of a pandas DataFrame into another (new) column. First…
In this blog post I decompose a time series of monthly data using the pandas and statsmodels package in Python. You can find the data that I use in this blog post in my github repo. It is a monthly average of daily car counts on different hubs on the…
Pandas is the go-to package for anything data science in Python. However, if you’re used to R and the convenience of dplyr or data.table, pandas can be confusing, now and then. For example, the following error is a real newb issue. ValueError: You are trying to merge on object and…