Error: connection failed / Certificate mismatch / Broken chain of trust
As some of you might have noticed and others will run into later. We have had some serious TLS issues. The last 24 hours have been a dark period in IOTAppStory.com history.
Straight to the point: We “lost” the root certificate that we have all been using to callhome over https.
Before diving into details, consequences and remedies. Let’s refresh our TLS knowledge below.
TLS, Certificate Authorities & root certificates
TLS Stands for Transport Layer Security.
TLS 1.2 Consists out of 2 round trips: Client to server and back, Client to server and back
CA Stands for Certificate Authority
If you are interested in the history of this protocol checkout this video.
In video #232 Andreas Spiess Explains how TLS works on a basic level and how to implement it in your projects. It's become bit outdated but still brings across the point.
And if you are interested in a more detailed explanation on how the TLS handshake works check this video.
The ESP32 side of our library runs TLS 1.2 using mbedTLS provided by the WiFiClientSecure library.
The esp8266 on the other hand runs TLS 1.2 using BearSSL provided by the WiFiClientSecureBearSSL library.
In both cases our library stores the IOTAppStory.com root certificate as a file in SPIFFS on your device. Which is then used to callhome() and check for updates.
As a comparison: browsers have a dedicated system that run in the background and store all root certificates for verifying server certificates presented by the websites you visit.
The root certificate stored in SPIFFS is used for 2 things:
- Authenticate the server (is this really IOTAppStory.com)
- Encrypt and decrypt messages (in our case firmware updates)
The main advantage of using root certificates compared to the server certificate (or fingerprint of the server certificate) is that the server certificate usually has a limited lifetime. In our case 2 years.
Root certificates usually have a lifetime of 10-20 years. Our CA Comodo's certificate had an expected lifetime from 2004 to 2029.
So even when the server certificate is renewed the root certificate is still valid.
So what happened?
Early 2019 Comodo CA was rebranded to Sectigo CA. At that time, reading the documentation, we did not realise the impact this would have on IOT devices calling IOTAppStory.com for updates. We quote:
This rebrand will not affect existing products and Comodo Authority roots will remain trusted.
For Comodo certificate owners the impact will be small; all Comodo certificates will continue to work and stay trusted by browsers and applications. The root structure doesn’t change, so no adjustments or new installations are needed.
In our mind this was confirmed by the fact that even until yesterday before the renewal (10-11-2020) if you inspected the root certificate it would state valid from 2004-2029.
But this did not take into account that IOT devices use a static certificate and once our server certificate would be renewed that this would be done by a slightly changed root certificate provided by Sectigo CA instead of Comodo CA.
We still had the Comodo root certificate installed after the server certificate renewal but when using inspection tools it now said expired 5 months and 11 days ago compared to the day before. And even thought browsers had no issues visiting IOTAppStory.com, for IOT devices this broke the chain of trust and all devices stopt calling home.
After talking to all the TLS specialist we could get a hold of we came to the sad conclusion that there was no way to fix this or go back. All we could do was continue and replace the root certificate with the new one from Sectigo.
What would we have done differently if we knew?
We would have warned about the coming change 6, 2 and 1 month in advance and provided the new root certificate ahead of time.
What can I do to get my device calling home again?
Well for those of you who have their devices within reach (you are in the same network) it’s quite easy.
- Go into config mode by pressing the mode button for 7-8 seconds
- Go to the certificate page
- Click the “get IAS root cert” button
- Exit config mode and reboot
For those of you who have their devices on a remote location. There is no easy way to put this…. You will have to visit your device to perform the steps above or ask someone local to do it for you.
Our apologies & offers
We would like to offer our apologies. This should not have happened. And we realise this might impact your future trust in our service.
Starting from tomorrow morning (12-11-2020) we are going to double up all limits for accounts that were created before 12-11-2020 UTC free of charge. This means that if your current account limit is set to 20 devices, 20mb storage, 2880 calls / day and 250mb data / month. You will end up with 40 devices, 40mb storage, 5760 calls / day and 500mb data / month.
And for all of the business users (of IOTAppStory.com), current and those considering the business offers we made. (you might read this before we speak in person) Whatever we offered you is going to be free for the next 12 months.
