In this blog post, you’ll learn how you can run scraping jobs using Selenium from Google Cloud Run, a fully managed solution that eliminates the burden of configuring and managing a Kubernetes cluster. It simplifies your production pipeline and fully automates your scraping process. Take notice, I’m not an expert…
I’ve had to brush up my Docker skills again due to some side projects I’m working on. So I ran into some basic issues I wanted to elaborate on. In this blog post: “image is referenced in multiple repositories.” I had to remove a Docker image from my machine using…
I’ve been dockerizing web scrapers in order to add them in a data pipeline. Web scraping is always a little bit of living on the edge because you never know which elements might have changed and which errors will pop up. In this blog post I would like to elaborate…
I was running a scraping job from a Docker container using Python and the psycopg2 adapter that wrote the results to a Postgres database managed on DigitalOcean (DO). On random occasions, the container died, for some reason. I started debugging and these are my findings. The error I ran into…
Taking a Docker course, building your first image and nothing happens? Here’s why and how to solve this issue. When you run the docker build command, the Docker client will send the “context” of your Dockerfile via REST to the Docker daemon. This “context” is the content of what is…
When taking Docker tutorials, one of the first things that pops up is running a web server in a container via Nginx. However, when you map ports (e.g. 80 on 80) you might get the following error: Error response from daemon: driver failed programming external connectivity on endpoint <your container>:…
Recently, I jumped on the serverless hype train. I basically sold my soul to Microsoft and Azure, so the possibility of running Python in Azure Functions is pretty damn cool. But since it’s still in preview mode (beta), it has proven to be a nerve-wrecking experience. Here’s some brief instructions…