Solve Postgres error “Error: SSL SYSCALL error: EOF detected”

I was running a scraping job from a Docker container using Python and the psycopg2 adapter that wrote the results to a Postgres database managed on DigitalOcean (DO). On random occasions, the container died, for some reason. I started debugging and these are my findings.

The error I ran into was the following:

Error: SSL SYSCALL error: EOF detected

I literally had no idea what to expect when Googling this error. Here’s what I found.

Make the queries run faster

It tends to happen with Postgres deployments that have very little RAM allocated to them. For example, I’m using the cheapest Postgres hosting on DO, it only has 512mb of memory attached to it .

In other words: the lower the memory, the longer it takes to run complex queries, with a higher probability that the connection times out.

If you can spare the bucks: add more memory.

Adjust the connection timeout

If adding more memory is not an option, try changing the keepalives parameters of your Postgres connection. For example, in psycopg2, you can do that as follows.

keepalive_kwargs = {
  "keepalives": 1,
  "keepalives_idle": 60,
  "keepalives_interval": 10,
  "keepalives_count": 5
}

conn = psycopg2.connect(
  host = 'YOUR_HOST>',
  database = '<DB_NAME>',
  user = '<USER>',
  password = '<PASSWORD>',
  port = 25060,
  **keepalive_kwargs
)

Let’s go over these four parameters quickly:

• keepalives (boolean): By setting this to 1, you indicate you want to use your own client-side keepalives.
• keepalives_idle (seconds): Sets the number of seconds of inactivity after which a keepalive message such be sent.
• keepalives_interval (seconds): Sets the number of seconds to wait before resending a message that has not been acknowledged by the server.
• keepalives_count (count): Sets the number of non-acknowledged keepalives to determine that the connection is dead.

So, solve the problem by making your queries run faster, or by making sure your connection doesn’t time out.

Great success!