Lessons Learned While Moving From Python 2.7 to Python 3.7 Beta on Google Cloud AppEngine Standard

The deprecation of python 2.7 has me starting to think about moving away from the nice walled garden of AppEngine first generation runtimes into the somewhat fog of war that is second generation runtimes

January 5th, 2019 by Blaine Garrett

Note: This is a living article as I make the transition from python 2.7 to python 3.7

Background

I've been using Google AppEngine for close to 10 years. Most of that time has been on the Python 2.7 first generation standard environment runtime. Like many of the first generation runtimes, it provided a sort of walled garden with direct access to lots of great tools like memcache, task queues, and more. With a local sdk that made unit testing a breeze, straight forward pay-for-what-you-use billing with a great free quota, and with years of experience serving production SaaS products, AppEngine has been my go-to for building sites like this blog, the DivRods project, and MPLSART. I've also built or contributed to several libraries designed for AppEngine Python 2.7.

While Google claims they have no plans to deprecate first generation runtimes, the maintainers of python are deprecating 2.7 Jan 1st, 2020. Additionally, many awesome features of Google Cloud (such as pub/sub) are not being added to the Python 2.7 runtimes. I also have several important projects that are at risk of becoming unusable if I don't get this figured out. All that said, I think now is the time to get up to speed with the Python 3.7 AppEngine Standard Environment.

Quick Notes

This post only considers the AppEngine standard environment - not the flexible environment.
This is not a migration guide per se. As of the initial writing of this, I'm fully expecting to have to start from scratch for many things.
~~At the time of writing, Python 3.7 support for AppEngine is in Beta with no SLA, etc. However, it is my experience that Google's Betas are very close to production grade.~~ Update: Python 3.7 left Beta December 12th, 2018
I'm also toying with the idea of straight up switching to Node runtime for AppEngine. I even have a bit of experience using it. However, I've decided to more forward with python 3.7 for the time being.

Summary of Things Missing or Made More Difficult in the Python 3.7 Environment

Again, the Python 2.7 runtime environment was more of a walled garden with a nice sdk for local dev. This is also a summary of things I liked about the Python 2.7 runtime.

The environment was sandboxed and could be emulated almost entirely locally which made unit testing a breeze - especially with testbed. Running the SDK locally is the primary means of functional testing for many.
Datastore access was built in and had a nice ORM by way of the db and later ndb package. Also, the ndb package had memcache built in to seamlessly prevent excessive datastore calls.
Things like task queues, search api, etc were also built in to the sandbox.
Many 3rd party libraries were built in to the deployed environment so you didn't have the deploy bloat (I'm looking at you pytz)

Questions I Need to Answer

As I embark on this journey, I have several key things to keep in mind. Many great first generation features are now harder to use and there are many python 3.7 specific things I need to also figure out. This will likely turn into a sort of table of contents for the rest of this post.

How do I run my application now that there is no dev_appserver.py and webapp2 is unnecessary
How do I access the datastore without the ndb orm
What are my memcache options
Does pay-as-you-go billing and free quotas still work in the Standard environment?
In general, what are the cost differences between Python 2.7 and Python 3.7 Standard Environments?

👍