How to Get AppEngine + GoDaddy + Android SSL Certs to Actually Work

I recently discovered Google was sending all MPLSART's search traffic to https://www.mplsart.com and Android users were getting a big ugly security warning, but not desktop users. After sifting through a lot of Googling, here is how I made Godaddy's SSL Certs play nice on Google AppEngine and make Android users actually load our site on https.

Update: I have since quit using 3rd Party SSL certs and just used Google's managed SSL. It's really simple, free, and they auto renew so you don't have to deal with this mess annually. Learn More.
 

tl;dr: If you are getting an SSL error  on your AppEngine App from Android or are getting a Chain Issues:Incomplete error when running your site via SSL Labs SSL Test, skip down to Step 4. If you are renewing your Cert or don't have your private key used originally, start from step 1 below.

Back Story

We've had SSL on MPLSART.COM as optional for about a year as we prep for Google Signing flow for our admin panel. Google seemed to pickup on this and started directing traffic to the https version of the site. Google preferes secure sites vs insecure site for results. This is controllable via Webmaster tools, but that is a seprate issue.

My neighbor recently brought it to my attention that she punched our site into Google search and when she clicked on the first result, it showed a bit ugly security error. I immediatly checked on my computer and it was fine and generally ignored it. Works on my machine, right? Then it happened to me. We put a link to a https url in a newsletter and I clicked on it from my phone and the dread hit me. We depend on mobile traffic and search traffic and I'm sure a lot of visitors were falling off. Ouch.

So I started to Googling. Numerous blog posts and Stack Overflow questions provide instructions on how to use SSL certs for App Engine. Several of them even talk about using GoDaddy. Separately, numerous sites talk about Android's SSL issue, but I couldn't find any that worked together. Even AppEngine's own excellent docs on Using Custom Domains And SSL seemed to get Step 4 below wrong-ish. After several hours of pulling my hair out, I got it to work.

Quick Notes

• I'm writing this as a quick reference to recreate these steps next time I renew my cert as well as help others dealing with the same issue.
• I found it very helpful to name the files the way I did below. It helped me understand what I was converting to PEM and concatenating and uploading, etc. I'm also notoriously verbose in naming things.
• I don't really go into what intermediary certs or cert chains are, but it is at the core of the problem. GoDaddy discusses their use here. I'm not an SSL expert. 
• This problem isn't 100% Android, GoDaddy, or AppEngine specific, so it may help other situations as well.
• If you find typos or glaring errors or better ways to do this, hit me up on twitter or in the comments.

Steps to Fixing the Issue

Step 0: The Issue and The Goal
The warning on Android is due to GoDaddy's use of intermediary certificates in combination with most instructions on how to convert Godaddy's cert bundle into PEM format AppEngine needs. Specifically, the issue is most help instructions concat them incorrectly. This is covered in Step 4 below.

AppEngine Ultimately needs two things from use regardless of if we correctly accounted for the intermediary certs or not. To be clear, most instructions will give you a working https in the browser but not on android.

• PEM encoded X.509 public key certificate - Created from non-PEM encoded certs we download from Godaddy. 
• Unencrypted PEM encoded RSA private key - The RSA version of our non-PEM private key created when you originally set up your cert. If you no longer have this, start at step 1.

Step 1: Generate an Unencrypted Private Key (If you still have your original Key and CSR skip to Step 4 below)

openssl req -new -newkey rsa:2048 -nodes -keyout mplsart2017.unencrypted.priv.key -out mplsart2017.csr

You will be prompted to fill in several fields. Here is what I put so I can remember next time I need to do this
 

Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Minnesota
Locality Name (eg, city) []:Minneapolis
Organization Name (eg, company) [Internet Widgits Pty Ltd]:MPLSART.COM
Organizational Unit Name (eg, section) []:IT
Common Name (e.g. server FQDN or YOUR name) []:mplsart.com
Email Address []:info@mplsart.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []: <redacted>
An optional company name []:MPLSART.COM

This will output 2 files mplsart2017.unencrypted.priv.key and mplsart2017.csr which are your non-PEM encoded private key and your Certificate Signing Request (CSR) respectively

Step 2: Re-Key Cert with GoDaddy

Login to GoDaddy and navigate to where you can manage your CERT. You should see screen similar to below (TODO: Attach screenshot)

Click "Rekey and Manage" and expand the "Re-Key certificate" panel 

Switch back over to the terminal and copy the contents of the CSR you generated above.

cat mplsart2017.csr | pbcopy 

Switch back over to GoDaddy and paste the CSR into the CRS field. Save and Submit All Saved Changes.

Step 3: Wait...
GoDaddy is verifying your changes and will email you when they are complete. For me this took about 10 minutes.

Step 4: Download Certificate Zip File.
From the Cert managment screen (in the screenshot above) click download. I selected "Apache" as the Server Type, however I'm not sure it really matters.
Download the Zip file and unzip. The remainder of these steps assume you unzipped them to directory ./mplsart.com/

In the Zip will be 2 files:

• gd_bundle-g2-g1.crt - This is the bundled intermediate certificates including the CA Root Certificate
• asdf1234.crt - The certificate. Your filename will be different.  
   

Note: You can find the bundle as well as the unbundled GoDaddy Certs in their repository. You don't need anything here, but it was helpful to understand what the bundle was all about.

Step 4: Convert Godaddy's Certs to PEM format and THEN Concatenate them
Important: If you concat before converting to PEM like most articles suggest, your SSL will work in browsers but not on Android Devices

Individually convert the cert files from the GoDaddy download to PEM format needed by AppEngine
 

openssl x509 -inform PEM -in mplsart.com/asdf1234.crt > mplsart.com/asdf1234.crt.pem
openssl x509 -inform PEM -in mplsart.com/gd_bundle-g2-g1.crt > mplsart.com/gd_bundle-g2-g1.crt.pem

Now Concatenate them into one PEM format file

cat mplsart.com/asdf1234.crt.pem mplsart.com/gd_bundle-g2-g1.crt.pem > ./mplsart2017.combine.public.pem

We're done with the public key. We now have a combined public key in PEM format which we'll upload to Google AppEngine in Step 7 below.

Debug Note: Go ahead and open this file to see the contents. There should be two sets of certs starting with -----BEGIN CERTIFICATE----- and ending with -----END CERTIFICATE-----

If this is what you see, then we're good to go. If you see more, you likely concatenated the non PEM version of gd_bundle-g2-g1.crt or didn't correctly convert IT to PEM

Step 5: Create the Unencrypted PEM encoded RSA private key
Next we need to convert our private key we generated in step 1 above (or retained from last time we did this) into an RSA private key

openssl rsa -in mplsart2017.unencrypted.priv.key -out mplsart2017.unencrypted.priv.key.pem

Step 6: Run a few Validations to ensure we did it all correcty
These checks are slightly modified from right from the Appengine Documentation for Using Custom Domains And SSL

To verify that the private key and certificate match, you can use the openssl x509 and openssl rsacommands. Examples:

openssl rsa -noout -modulus -in mplsart2017.unencrypted.priv.key | openssl md5
openssl x509 -noout -modulus -in  mplsart.com/asdf1234.crt.pem | openssl md5

These 2 should output the same md5 hash

Next verify that a certificate and it's CA chain are valid. Note: This will pass even if you got step 4 out of order potentially.

openssl verify -verbose -CAfile mplsart.com/asdf1234.crt mplsart.com/asdf1234.crt

I received this output, which seemed ok...
./mplsart2017combined_pre_pem.crt: /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
error 2 at 1 depth lookup:unable to get issuer certificate

Otherwise I get an error 20 at depth 0 and that seems bad

Step 7: Upload to Google AppEngine via Google Cloud Console
Under AppEngine -> Settings -> SSL Certificates -> Upload New Certficate

Name your "Cert2017" or some indicator of the upgrade

For PEM encoded X.509 public key certificate  select ./mplsart2017.combine.public.pem created in Step 4
For Unencrypted PEM encoded RSA private key select ./mplsart2017.unencrypted.priv.key.pem created in Step 5

If you have not correctly formatted your keys as PEM, you'll get an error. Otherwise, click "Upload"


Step 8: Enable the new cert for your domain and disable the other certs.
Click into each cert in the AppEngine Consol. Under Enable SSL for the following custom domains uncheck your domain for all  old certs and ensure it is checked for the new cert.

Step 9: Validate
Check your site out on android (you may need to restart your browser and/or device if you get impatient)
Check your site out on the SSLShopper SSL Checker and SSL Labs SSL Test (you may need to clear the cache on the site). The later should give you big green A
 

Step 10: Go to Bed
 

Step 11: Back Up Your Files
Back up your files, especially your non-PEM private key and CSR file from step 1. You'll need it next time you renew your SSL Cert next year. Otherwise, you'll have to start at step 1 again.

Common Search Phrases I entered:

• SSL Error. Failed to validate the certificate chain
• Setting up GoDaddy SSL Certs on Appengine
• GoDaddy Cert Not Trusted on Android
• SSL Certificate Not Trusted on Android only

References and Handy Links:

• What is an intermediate certificate? - GoDaddy

• how to set-up SSL on google app engine (custom domain name ) Stack Overflow - Note: Jill Russek's comment got me nearly there, but still had Step 4 incorrect

• Godaddy SSL certificate is not trusted by Android browsers (HTC)  - Quora - Not AppEngine specific

• SSL Certificate Not Trusted Error Deep Explaination - SSLShopper

• Using Custom Domains and SSL - Google Cloud Docs - Using SSL with Appengine. Note: GoDaddy is not listed, but step 2 on concatenating your CA .crt files only works if you assume the files are already PEM formatted