I recently discovered Google was sending all MPLSART's search traffic to https://www.mplsart.com and Android users were getting a big ugly security warning, but not desktop users. After sifting through a lot of Googling, here is how I made Godaddy's SSL Certs play nice on Google AppEngine and make Android users actually load our site on https.
tl;dr: If you are getting an SSL error on your AppEngine App from Android or are getting a Chain Issues:Incomplete error when running your site via SSL Labs SSL Test, skip down to Step 4. If you are renewing your Cert or don't have your private key used originally, start from step 1 below.
We've had SSL on MPLSART.COM as optional for about a year as we prep for Google Signing flow for our admin panel. Google seemed to pickup on this and started directing traffic to the https version of the site. Google preferes secure sites vs insecure site for results. This is controllable via Webmaster tools, but that is a seprate issue.
My neighbor recently brought it to my attention that she punched our site into Google search and when she clicked on the first result, it showed a bit ugly security error. I immediatly checked on my computer and it was fine and generally ignored it. Works on my machine, right? Then it happened to me. We put a link to a https url in a newsletter and I clicked on it from my phone and the dread hit me. We depend on mobile traffic and search traffic and I'm sure a lot of visitors were falling off. Ouch.
So I started to Googling. Numerous blog posts and Stack Overflow questions provide instructions on how to use SSL certs for App Engine. Several of them even talk about using GoDaddy. Separately, numerous sites talk about Android's SSL issue, but I couldn't find any that worked together. Even AppEngine's own excellent docs on Using Custom Domains And SSL seemed to get Step 4 below wrong-ish. After several hours of pulling my hair out, I got it to work.
- I'm writing this as a quick reference to recreate these steps next time I renew my cert as well as help others dealing with the same issue.
- I found it very helpful to name the files the way I did below. It helped me understand what I was converting to PEM and concatenating and uploading, etc. I'm also notoriously verbose in naming things.
- I don't really go into what intermediary certs or cert chains are, but it is at the core of the problem. GoDaddy discusses their use here. I'm not an SSL expert.
- This problem isn't 100% Android, GoDaddy, or AppEngine specific, so it may help other situations as well.
- If you find typos or glaring errors or better ways to do this, hit me up on twitter or in the comments.
Steps to Fixing the Issue
Step 0: The Issue and The Goal
The warning on Android is due to GoDaddy's use of intermediary certificates in combination with most instructions on how to convert Godaddy's cert bundle into PEM format AppEngine needs. Specifically, the issue is most help instructions concat them incorrectly. This is covered in Step 4 below.
AppEngine Ultimately needs two things from use regardless of if we correctly accounted for the intermediary certs or not. To be clear, most instructions will give you a working https in the browser but not on android.
- PEM encoded X.509 public key certificate - Created from non-PEM encoded certs we download from Godaddy.
- Unencrypted PEM encoded RSA private key - The RSA version of our non-PEM private key created when you originally set up your cert. If you no longer have this, start at step 1.
Step 1: Generate an Unencrypted Private Key (If you still have your original Key and CSR skip to Step 4 below)
openssl req -new -newkey rsa:2048 -nodes -keyout mplsart2017.unencrypted.priv.key -out mplsart2017.csr
You will be prompted to fill in several fields. Here is what I put so I can remember next time I need to do this
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Minnesota
Locality Name (eg, city) :Minneapolis
Organization Name (eg, company) [Internet Widgits Pty Ltd]:MPLSART.COM
Organizational Unit Name (eg, section) :IT
Common Name (e.g. server FQDN or YOUR name) :mplsart.com
Email Address :firstname.lastname@example.org
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password : <redacted>
An optional company name :MPLSART.COM
This will output 2 files mplsart2017.unencrypted.priv.key and mplsart2017.csr which are your non-PEM encoded private key and your Certificate Signing Request (CSR) respectively
Step 2: Re-Key Cert with GoDaddy
Login to GoDaddy and navigate to where you can manage your CERT. You should see screen similar to below (TODO: Attach screenshot)
Click "Rekey and Manage" and expand the "Re-Key certificate" panel
Switch back over to the terminal and copy the contents of the CSR you generated above.
cat mplsart2017.csr | pbcopy
Switch back over to GoDaddy and paste the CSR into the CRS field. Save and Submit All Saved Changes.
Step 3: Wait...
GoDaddy is verifying your changes and will email you when they are complete. For me this took about 10 minutes.
Step 4: Download Certificate Zip File.
From the Cert managment screen (in the screenshot above) click download. I selected "Apache" as the Server Type, however I'm not sure it really matters.
Download the Zip file and unzip. The remainder of these steps assume you unzipped them to directory ./mplsart.com/
In the Zip will be 2 files:
- gd_bundle-g2-g1.crt - This is the bundled intermediate certificates including the CA Root Certificate
- asdf1234.crt - The certificate. Your filename will be different.
Note: You can find the bundle as well as the unbundled GoDaddy Certs in their repository. You don't need anything here, but it was helpful to understand what the bundle was all about.
Step 4: Convert Godaddy's Certs to PEM format and THEN Concatenate them
Important: If you concat before converting to PEM like most articles suggest, your SSL will work in browsers but not on Android Devices
Individually convert the cert files from the GoDaddy download to PEM format needed by AppEngine
openssl x509 -inform PEM -in mplsart.com/asdf1234.crt > mplsart.com/asdf1234.crt.pem
openssl x509 -inform PEM -in mplsart.com/gd_bundle-g2-g1.crt > mplsart.com/gd_bundle-g2-g1.crt.pem
Now Concatenate them into one PEM format file
cat mplsart.com/asdf1234.crt.pem mplsart.com/gd_bundle-g2-g1.crt.pem > ./mplsart2017.combine.public.pem
We're done with the public key. We now have a combined public key in PEM format which we'll upload to Google AppEngine in Step 7 below.
Debug Note: Go ahead and open this file to see the contents. There should be two sets of certs starting with -----BEGIN CERTIFICATE----- and ending with -----END CERTIFICATE-----
If this is what you see, then we're good to go. If you see more, you likely concatenated the non PEM version of gd_bundle-g2-g1.crt or didn't correctly convert IT to PEM
Step 5: Create the Unencrypted PEM encoded RSA private key
Next we need to convert our private key we generated in step 1 above (or retained from last time we did this) into an RSA private key
openssl rsa -in mplsart2017.unencrypted.priv.key -out mplsart2017.unencrypted.priv.key.pem
Step 6: Run a few Validations to ensure we did it all correcty
These checks are slightly modified from right from the Appengine Documentation for Using Custom Domains And SSL
To verify that the private key and certificate match, you can use the
openssl x509 and
openssl rsa -noout -modulus -in mplsart2017.unencrypted.priv.key | openssl md5
openssl x509 -noout -modulus -in mplsart.com/asdf1234.crt.pem | openssl md5
These 2 should output the same md5 hash
Next verify that a certificate and it's CA chain are valid. Note: This will pass even if you got step 4 out of order potentially.
openssl verify -verbose -CAfile mplsart.com/asdf1234.crt mplsart.com/asdf1234.crt
I received this output, which seemed ok...
./mplsart2017combined_pre_pem.crt: /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
error 2 at 1 depth lookup:unable to get issuer certificate
Otherwise I get an error 20 at depth 0 and that seems bad
Step 7: Upload to Google AppEngine via Google Cloud Console
Under AppEngine -> Settings -> SSL Certificates -> Upload New Certficate
Name your "Cert2017" or some indicator of the upgrade
For PEM encoded X.509 public key certificate select ./mplsart2017.combine.public.pem created in Step 4
For Unencrypted PEM encoded RSA private key select ./mplsart2017.unencrypted.priv.key.pem created in Step 5
If you have not correctly formatted your keys as PEM, you'll get an error. Otherwise, click "Upload"
Step 8: Enable the new cert for your domain and disable the other certs.
Click into each cert in the AppEngine Consol. Under Enable SSL for the following custom domains uncheck your domain for all old certs and ensure it is checked for the new cert.
Step 9: Validate
Check your site out on android (you may need to restart your browser and/or device if you get impatient)
Check your site out on the SSLShopper SSL Checker and SSL Labs SSL Test (you may need to clear the cache on the site). The later should give you big green A
Step 10: Go to Bed
Step 11: Back Up Your Files
Back up your files, especially your non-PEM private key and CSR file from step 1. You'll need it next time you renew your SSL Cert next year. Otherwise, you'll have to start at step 1 again.
Common Search Phrases I entered:
- SSL Error. Failed to validate the certificate chain
- Setting up GoDaddy SSL Certs on Appengine
- GoDaddy Cert Not Trusted on Android
- SSL Certificate Not Trusted on Android only
References and Handy Links: