Strange connection issues - Let's Encrypt R3 expiration

A few days ago I started my Gatsby development environment as usual, but for some reason the page did not start. Learn what happened.

· 3 min read
Strange connection issues - Let's Encrypt R3 expiration

The Problem

A few days ago, around September 29th, 2021, I started my Gatsby development environment as usual, I ran gatsby develop command to continue working on my Gatsby Ghost Simply theme conversion, but for some reason the page did not start.

The Investigation

There were quite a few errors about Gatsby not being able to query certain fields, and sure enough the GraphQL Explorer was showing no data that I was expecting from my Ghost CMS instance. Looking again at the top of the console I found this:

...
warn The gatsby-source-ghost plugin has generated no Gatsby nodes. Do you need it?
...

So indeed Gatsby couldn’t get data from my Ghost instance. I proceeded to debug this further, trying to pinpoint where it was failing in the code but I was unsuccessful. The Ghost instance itself and its API was accessible through the web and through the terminal that Gatsby was ran in… This was really puzzling and I proceeded to investigate Gatsby config, plugins, Ghost CMS docker, reverse-proxy, etc. - all in vain.

For a moment I had to drop that activity, and at around the same time I noticed that I can’t access my Home Assistant instance from Android App. I used to have such access issues before, when my firewall (OPNSense) would permaban the app for using okhttp, which apparently is hard-coded to be blocked in OPNsense. But now I had this blocking disabled for a while, so this couldn’t be that issue. So again I proceeded to investigate - first I started checking nginx/reverse-proxy logs and strangely there were no traces of the app ever connecting - neither in access nor in error logs. That was again puzzling… Home Assistant app is able to check its logs on the Android device, and that’s where I had my first lead at the issue:

...
10-02 14:05:00.464  4821  5579 E NetworkSM: Caused by: android.system.ErrnoException: isConnected failed: ECONNREFUSED (Connection refused)
10-02 14:05:00.464  4821  5579 E NetworkSM:  at libcore.io.IoBridge.isConnected(IoBridge.java:274)
10-02 14:05:00.464  4821  5579 E NetworkSM:  ... 28 more
10-02 14:05:00.586  4821  5121 E LocBroadcastReceiver: Error receiving zones from Home Assistant
10-02 14:05:00.586  4821  5121 E LocBroadcastReceiver: io.homeassistant.companion.android.common.data.integration.IntegrationException: javax.net.ssl.SSLHandshakeException: Unacceptable certificate: CN=R3, O=Let's Encrypt, C=US // [!code focus]
10-02 14:05:00.586  4821  5121 E LocBroadcastReceiver:  at io.homeassistant.companion.android.common.data.integration.impl.IntegrationRepositoryImpl.getZones(IntegrationRepositoryImpl.kt:294)
10-02 14:05:00.586  4821  5121 E LocBroadcastReceiver:  at io.homeassistant.companion.android.common.data.integration.impl.IntegrationRepositoryImpl$getZones$1.invokeSuspend(Unknown Source:14)
...

Unacceptable certificate: CN=R3, O=Let's Encrypt, C=US -> and that’s when I remembered reading something about Let’s Encrypt certificate expiring that would require everyone to re-issue.

I checked my own certificate, it was valid, so hmm, that was not it. I started reading on it, and quickly it all became clear:

Help thread for DST Root CA X3 expiration (September 2021)
If you have any questions about whether you need to do anything special for the upcoming DST Root CA X3 expiration in September 2021, please post them here. A staff member may split out some conversations into their own threads. Note: Your first step in debugging should be to update your operating system. Most problems are solved by running the latest operating system available for your machine, and staying up to date will also make you more secure. Update 30 September 2021 Yesterday, the R3…
Link thumbnail

The Solution

At that moment I realized that both my issues - Gatsby/Ghost connection and Home Assistant connection from Android had to be both caused by the Root CA expiration . I checked SSL Server test for my site and indeed Certification Path was not trusted due to expired DST Root CA X3 certificate.

Now, there are/were several ways to fix this:

opnsense using wrong letsencrypt R3 intermediate certificate
opnsense using wrong letsencrypt R3 intermediate certificate
Link thumbnail

but since my OPNSense version was anyway old, I just proceeded to upgrade to the latest version, which also included updated ACME client. After that it was just a matter of renewing (forcefully - just in case) the certificates followed by a reboot and voilà!

Both Gatsby could connect back to Ghost CMS and my phone could connect back to Home Assistant!