In this blog post, you’ll learn how you can run scraping jobs using Selenium from Google Cloud Run, a fully managed solution that eliminates the burden of configuring and managing a Kubernetes cluster. It simplifies your production pipeline and fully automates your scraping process. Take notice, I’m not an expert…
This is a brief overview of the possibilities for setting environment variables in a virtual environment (venv). Set environment variables in Python code Environment variables can be set inside your Python script. All child processes will inherit the environment variables and their values. Set environment variables in activate.bat or activate.ps1…
I was running a scraping job from a Docker container using Python and the psycopg2 adapter that wrote the results to a Postgres database managed on DigitalOcean (DO). On random occasions, the container died, for some reason. I started debugging and these are my findings. The error I ran into…
One of the most widely used technical indicators in technical analysis is the Relative Strength Index. It measures the magnitude of recent price evolutions to evaluate if a price of an asset is overbought or oversold. In this blog post, I list the many ways you can calculate the RSI…
Using TA-Lib in a Jupyter notebook isn’t straightforward, and using it in a hosted solution like Deepnote could use a tutorial too. In this blog post, we’ll set up the popular Python wrapper for TA-lib. A prerequisite for installing the TA-lib Python wrapper is that you have the TA-lib library…
The recent crypto bull market got me interested in chartist approaches to trading strategies. I’ve always wanted to use Deepnote‘s notebook solution, so I decided to combine my newfound hobby of algorithmic trading with my new tool craving. I started using Backtrader in Deepnote, but that produced some plotting issues.…
Recently, I’ve been working with Pandas DataFrames that had a DateTime as the index. When I tried reindexing the DataFrame (using the reindex method), I bumped into an error. Let’s find out what causes it and how to solve it. The Python error I’m talking about is: A “duplicate axis”?…
Sometimes, it’s not entirely convenient to scrape data from the web: sometimes one needs to hover the mouse pointer over a particular element for another one to be rendered. In this blog post, we’ll simulate a mouseover in Selenium and Python. To mimic a mouseover event to trigger the display…
All of us data hoarders get to a point where we need to circumvent in-place rules that prevent us from scraping the web. Enter: Selenium. The world’s favorite tool for automating tasks in a browser. Selenium uses ChromeDriver, initially created for automated testing, but also a splendid scraping tool. In…
I’m a data scientist. Deploying is not part of my core skills, but something I need to be able to do when working on hobby projects. I recently embraced DigitalOcean to deploy my Django (Python) Project and want to elaborate on a problem I ran into. The error that appeared…
