I’ve been working in Django a lot lately and needed to dump my database using Django’s dumpdata command. However, that produced an error. Here’s how I solved it. However, halfway the dump, my terminal returned an error. Apparently, there was a Byte Order Mark or BOM somewhere in my SQLite…
In this brief article, I explain how you can run a Selenium server, right from within your R code. This allows you to not manually open and run a Selenium command in your command prompt. This code has been tested on a Win11 machine. Prerequisites To run Selenium from within…
In the past half decade, there have been huge shifts in the land of personal data and privacy. There have been numerous legal as well as technological evolutions. Both for analytical and advertising purposes, digital professionals rely on collecting and linking personal data. Consequently, these evolutions can have a solid…
As a data scientist, I usually get a data warehouse to work with. The data’s there and I don’t have to worry about the architecture. However, now and then, I need to get my hands dirty. Lastly, I needed SnowSQL and I couldn’t get it to work, due to an…
One of the most frequent used Excel functions is probably SUMIF and its SUMIFS variant. In this article, you’ll learn how to do exactly the same in PySpark. What is the sumif function? In Excel, the SUMIF function is an aggregation function for summing values from a column, but only…
This article covers the transfer of files over SFTP (or Secure/SSH File Transfer Protocol). While there are quite some options to choose from, the Python package Paramiko is used to achieve it. Paramiko is a pure-python implementation of the SSH2 protocol. Although Paramiko is the foundation for some other packages…
Every since I’ve started using Synapse for an assignment, I’ve preferred using Spark Notebooks to get anything done. However, they take time to spin up, something I wanted to mitigate by using Synapse-native components. In this article, we’ll generate a date range without Spark notebooks. It’s unnecessarily complicated, but you…
I recently got more interested in observability, logging, data quality, etc. For many use cases, you don’t need full-blown observability solutions. Python has robust tools that will help you achieve your logging goals. In this blog post, I discuss a small trick I encountered for using Python’s logging module properly.…
In the past couple of weeks, I’ve been working on a project which users Spark pools in Azure Synapse. However, this appears to be a general Spark issue. I was unable to write to delta lake using Spark because I received the following error. You may get a different result…
It’s been a couple of years since I first used NetworkX in Python. Something that I expected to be truly obvious was adding node attributes to your network, preferably when calling a from_pandas_… function. However, that wasn’t the case. In this article, we’ll create a network from an edge list…