All of us data hoarders get to a point where we need to circumvent in-place rules that prevent us from scraping the web. Enter: Selenium. The world’s favorite tool for automating tasks in a browser. Selenium uses ChromeDriver, initially created for automated testing, but also a splendid scraping tool. In…
The reason people love tools like Dataform so much, is that it allows them to automate parts of the ELT workflow. In this blog post we will set our destination dataset, depending on the branch we’re running the definition on. A really interesting use case is to keep the resulting…
One of Google’s latest acquisitions, ELT data orchestration tool Dataform, supports JavaScript. This is a great way to chain Dataform’s actions. In this blog post we explore how to automatically create a destination table in your data warehouse, before an incremental definition is run. If you are interested in a…
With the acquisition of Dataform by Google, it is steadily becoming the ELT data orchestration tool par préférence. In this blog post we explore how to automatically create a table in your data warehouse that will function as an incremental data destination in Dataform. If you are interested in a…
One of the most fundamental tools for evaluating classifiers is the confusion matrix. In this blog post, I elaborate on how to generate a confusion matrix in R using the tidymodels ecosystem. First, I create a classifier to predict the income class for the very popular adult data set. From…
In this blog post, I discuss the difference between several BigQuery functions that appear to do the same: JSON_EXTRACT and JSON_QUERY — and their value counterparts JSON_EXTRACT_SCALAR and JSON_VALUE. MPP Data Warehouses are increasingly becoming data lakes. In this light, extracting strings/values from semistructured data (stored as a JSON STRING)…
I’m a data scientist. Deploying is not part of my core skills, but something I need to be able to do when working on hobby projects. I recently embraced DigitalOcean to deploy my Django (Python) Project and want to elaborate on a problem I ran into. The error that appeared…
I recently got my hands on an invitation for Hex. Not only do they produce great blog posts, they also offer a product for a specific data science niche. The common issue with data science notebooks is that they are hard to translate to a compelling story for a not-so-tech-savvy…
This is a blog post about a very specific topic. I wanted to filter a data frame on a set of strings that I wanted to match partially. Let’s dive right in. Matching partially is fairly easy, and there are many libraries to choose from, with grepl and str_detect the…
Selligent is an ecosystem of marketing tools, mostly known for it’s marketing automation platform formerly known as Campaign. I was never a big fan of Selligent because it used to be a very closed ecosystem. However, it made available a wide range of APIs recently, allowing you to query its…