Skip to content

Latest commit

 

History

History
373 lines (294 loc) · 43.9 KB

README.md

File metadata and controls

373 lines (294 loc) · 43.9 KB
Raw
💥 ALERT!!
This repo will soon be relocating to GoogleCloudPlatform as we better organize these code samples! Stay tuned as more info is coming soon.

Modernizing Google Cloud serverless compute applications

To the latest Cloud services and serverless platforms

This is the corresponding repository to the Serverless Migration Station video series whose goal is to help users on a Google Cloud serverless compute platform modernize to other Cloud or serverless products. Modernization steps generally feature a video, codelab (self-paced, hands-on tutorial), and code samples. The content initially focuses on App Engine and Google's earliest Cloud users. Read more about the codelabs in this announcement as well as this one introducing the video series. This repo is for Python developers; there is another repo for Java developers.

Google App Engine (Standard) has undergone significant changes between the legacy and next generation platforms. To address this, we've created a set of resources showing developers how to perform individual migrations that can be applied to modernize their apps for the latest runtimes, meaning to Python 3 even though Google expressed long-term support for legacy runtimes like Python 2. The content falls into one of these topics:

  1. Migrate from a legacy App Engine service to a similar Cloud product
  2. Shift to another Cloud serverless compute platform (e.g., from App Engine to Cloud Run)
  3. General app, data, or service migration steps and best practices

Each codelab begins with a sample app in a "START" repo folder then walks developers through that migration, resulting in code in a "FINISH" repo. If you make mistakes along the way, you can always go back to START or compare your code with what's in the corresponding FINISH folder. The baseline apps are in Python 2, and since we also want to help you port to Python 3, some codelabs contain additional steps to do so.

NOTEs:

  1. These migrations are typically aimed at our earliest users, e.g., Python 2
  2. Python 3.x App Engine users: You're already on the next-gen platform, so you would focus on migrating away from the legacy bundled services
  3. Python 2.5 App Engine users: to revive apps from the original 2.5 runtime, deprecated in 2013 and shutdown in 2017, you must migrate from db to ndb and get those apps running on Python 2.7 before attempting these migrations.

Prerequisites

  • A Google account and Cloud (GCP) project with an active billing account
  • Familiarity with operating system terminal/shell commands
  • Familiarity with developing & deploying Python 2 apps to App Engine
  • General skills in Python 2 and 3

Cost

App Engine, Cloud Functions, and Cloud Run are not free services. While you may not have enabled billing in App Engine's early days, all applications now require an active billing account backed by a financial instrument (usually a credit card). Don't worry, App Engine (and other GCP products) still have an "Always Free" tier and as long as you stay within those limits, you won't incur any charges. Also check the App Engine pricing and quotas pages for more information.

Furthermore, deploying to GCP serverless platforms incur minor build and storage costs. Cloud Build has its own free quota as does Cloud Storage. For greater transparency, Cloud Build builds your application image which is than sent to the Cloud Container Registry, or Artifact Registry, its successor; storage of that image uses up some of that (Cloud Storage) quota as does network egress when transferring that image to the service you're deploying to. However you may live in region that does not have such a free tier, so be aware of your storage usage to minimize potential costs. (You may look at what storage you're using and how much, including deleting build artifacts via your Cloud Storage browser.)

Why

App Engine initially launched in 2008 (video), providing a suite of bundled services making it convenient for developers to access a database (Datastore), caching (Memcache), independent task execution (Task Queue), large "blob" storage (Blobstore) to allow for end-user file uploads or to serve large media files, and other companion services. However, apps leveraging those services can only run their apps on App Engine.

To increase app portability, its 2nd-generation service launched in 2018, initially removing those legacy bundled services. The main reason to move to the 2nd generation service is that it allows developers to upgrade to the latest language runtimes, such as moving from Python 2 to 3 or Java 8 to 17. Unfortunately, it was mutually exclusive to do so, meaning while you could upgrade language releases, you lost access to those bundled services, making it a showstopper for many users.

However, due to their popularity and to help users upgrade, the App Engine team restored access to many (but not all) of those services in Fall 2021. For more on this, see the Legacy services section below. As Google is continually striving to have the most open cloud on the market, and while many of those services are now available again, apps can still be more portable if they migrated away from the legacy services to similar Cloud or 3rd-party offerings. Another issue with the bundled services is that they're only available in 2nd generation runtimes that have a 1st generation service (Python, Java, Go, PHP), excluding 2nd generation-only runtimes like Ruby and Node.js.

Once apps have moved away from App Engine bundled services to similar Cloud or 3rd-party services. apps are portable enough to:

  1. Run on the 2nd generation App Engine service
  2. Shift across to other serverless platforms, like Cloud Functions or Cloud Run (with or without Docker), or
  3. Move to VM-based services like GKE or Compute Engine, or to other compute platforms

NOTEs:

  1. App Engine (Flexible) is a next-gen service but is not within the scope of these tutorials. Curious developers can compare App Engine Standard vs. Flexible to find out more.
  2. Many use cases for Flexible or a desire for containerization can be handled by Cloud Run.
  3. Small apps or large monolithic apps broken up into multiple, independent microservices can consider migrating to Cloud Functions.

Progression (what order to do things)

"START" and "FINISH" repo folders

All codelabs begin with code in a START repo folder and end with code in a FINISH folder, implementing a single migration. Upon completion, users should confirm their code (for the most part) matches what's in the FINISH folder. The baseline migration sample app (Module 0; link below) is a barebones Python 2.7 App Engine app that uses the webapp2 web framework plus the ndb Datastore library.

  1. With Module 0 as the STARTing point, the Module 1 codelab migrates from the webapp2 web framework to Flask, FINISHing at code matching the Module 1 repo.
  2. Next, STARTing with the Module 1 application code (yours or ours), Module 2 migrates from ndb to Cloud NDB, ending with code matching the (Module 2) FINISH repo folder. There's also has a bonus migration to Python 3, resulting in another FINISH repo folder, this one deployed on the next-generation platform.
  3. Your Python 2 apps may be using other built-in services like Task Queues or Memcache, so additional migration modules follow, some more optional than others, and not all are available yet (keep checking back here for updates).

The order of migrations

Beyond Module 2, with some exceptions, there is no specific order of what migrations modules to tackle next. It depends on your needs (and your applications'). However, there are related migrations where one or more modules must be completed beforehand. This table attempts to put an order on module subsets.

Topic Module ordering Description
Baseline 0 ⇒ 1 Not a migration but a description of the baseline application (review this material before doing any migrations)
Web framework 1 ⇒ everything else Current App Engine runtimes do not come with a web framework, so this must be the first migration performed. All migrations below can be performed after this one.
Bundled services 17 and 22 These modules are for those who want to continue using Python bundled services from Python 3 App Engine.
Datastore 2 [⇒ 3 [⇒ 6]] Moving off App Engine ndb makes your apps more portable, so the Module 2 Cloud NDB migration is recommended. Module 3: Migrating to Cloud Datastore (Firestore in Datastore mode) is optional and only recommended if you have other code using Cloud Datastore. Module 6: Migrating to Cloud Firestore (Native mode) is generally not recommended unless you must have the Firebase features it has, and those features will eventually be integrated into Cloud Datastore.
(Push) Task Queues [7 ⇒] 8 [⇒ 9] Moving off App Engine taskqueue makes your apps more portable, so the Module 8 Cloud Tasks migration is recommended for those using push tasks. Those unfamiliar with push tasks should do Module 7 first to add push tasks to the sample app. Module 9: Migrating to Cloud Datastore (Firestore in Datastore mode), Cloud Tasks (v2), and Python 3 is optional and only recommended if you have other code using Cloud Datastore and considering upgrading to Python 3.
(Pull) Task Queues [18 ⇒] 19 Moving off App Engine taskqueue makes your apps more portable, so the Module 19 Cloud Pub/Sub migration is recommended for those using pull tasks. The app is also ported to Python 3. Those unfamiliar with pull tasks should do Module 18 first to add pull tasks to the sample app.
Memcache [12 ⇒] 13 Moving off App Engine memcache makes your apps more portable, so the Module 13 Cloud Memorystore (for Redis) migration is recommended for those using memcache. Those unfamiliar with memcache should do Module 12 first to add its usage to the sample app.
Cloud Functions 11 Cloud Functions does not support Python 2, so after the Module 1 migration, you need to upgrade your app to Python 3 before attempting this migration, recommended if you have a very small App Engine app, or it has only one function/feature.
Cloud Run 4 or 5 Module 4 covers migrating to Cloud Run with Docker. Those unfamiliar with containers or do not wish to create/maintain a Dockerfile should do Module 5. Those doing Module 4 will get additional information about Cloud Run in Module 5 not covered in Module 4.
Blobstore [15 ⇒] 16 Moving off App Engine blobstore makes your apps more portable, so the Module 16 Cloud Storage migration is recommended for those using blobstore. Those unfamiliar with blobstore should do Module 15 first to add its usage to the sample app.
Users [20 ⇒] 21 Moving off App Engine users makes your apps more portable, so the Module 21 Cloud Identity Platform migration is recommended for those using users. Those unfamiliar with users should do Module 20 first to add its usage to the sample app.
General migration 6 ⇒ 10 ⇒ 14 This series is more generic and not targeting a specific feature migration, but rather if you need to migrate your App Engine apps from one running project to another. It starts with Module 6 if you need to migrate your code, say from Datastore to Firestore. Module 10 is if you need to migrate your data from one project to another, and finally, Module 14 is after you're done migrating your app, your data, or both, and need to migrate a running service on one GCP project to another.

Migration modules

The table below summarizes migration module resources currently available along with a more detailed table of contents below. Be sure to check back for updates as more resources are planned.

Summary table

Module Topic Video Codelab START here FINISH here
0 Baseline app link N/A (no tutorial; just review the code) N/A Module 0 code (2.x)
1 Migrate to Flask link link Module 0 code (2.x) Module 1 code (2.x) (and code (3.x))
2 Migrate to Cloud NDB link link Module 1 code (2.x) Module 2 code (2.x) & code (3.x)
3 Migrate to Cloud Datastore link link Module 2 code (2.x) & code (3.x) Module 3 code (2.x) & code (3.x)
4 Migrate to Cloud Run with Docker link link Module 2 code (2.x) & Module 3 code (3.x) Module 4 code (2.x) & code (3.x)
5 Migrate to Cloud Run with Buildpacks link link Module 2 code (3.x) Module 5 code (3.x)
6 Migrate to Cloud Firestore N/A N/A Module 3 code (3.x) no work required; Datastore upgrade automatic
7 Add App Engine taskqueue push tasks link link Module 1 code (2.x) Module 7 code (2.x) & code (3.x)
8 Migrate to Cloud Tasks link link Module 7 code (2.x) Module 8 code (2.x)
9 Migrate to Python 3, Cloud Datastore & Cloud Tasks v2 TBD link Module 8 code (2.x) Module 9 code
10 Migrate Datastore/Firestore data to another project TBD N/A N/A TBD
11 Migrate to Cloud Functions link link Module 2 code (3.x) Module 11 code (3.x)
12 Add App Engine memcache link link Module 1 code (2.x) Module 12 code (2.x) & code (3.x)
13 Migrate to Cloud Memorystore link link Module 12 code (2.x) & code (3.x) Module 13 code (2.x) & code (3.x)
14 Migrate service between projects TBD TBD TBD TBD
15 Add App Engine blobstore link link Module 0 code (2.x) Module 15 code (2.x)
16 Migrate to Cloud Storage link link Module 15 code (2.x) Module 16 code (2.x & 3.x)
17 Migrate to Python 3 bundled services (Part 1) link link Module 1 code (2.x) Module 1 code (3.x)
18 Add App Engine taskqueue pull tasks link link Module 1 code (2.x) Module 18 code (2.x)
19 Migrate to Cloud Pub/Sub link link Module 18 code (2.x) Module 19 code (2.x & 3.x)
20 Add App Engine users link link Module 1 code (2.x) Module 20 code (2.x)
21 Migrate to Cloud Identity Platform link link Module 20 code (2.x) Module 21 code (2.x) & code (3.x)
22 Migrate to Python 3 bundled services (Part 2) link N/A Module 22 code (2.x & 3.x) (⇐ same folder)

Table of contents

If there is a logical codelab to do immediately after completing one, they will be designated as NEXT. Other recommended codelabs will be listed as RECOMMENDED, and the more optional ones will be labeled as OTHERS (and usually in some kind of priority order).

Migrations from legacy App Engine APIs/bundled services

  • Module 1 codelab: Migrate from webapp2 to Flask

  • Module 2 codelab: Migrate from App Engine ndb to Cloud NDB

    • Python 2
    • Codelab bonus port to Python 3.x
    • RECOMMENDED:
      • Module 7 - add App Engine Task Queue push tasks (and migrate to Cloud Tasks in Module 8)
      • Module 18 - add App Engine Task Queue pull tasks (and migrate to Cloud Pub/Sub in Module 19)
      • Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13)
      • Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16)
      • Module 20 - add App Engine Users (and migrate to Cloud Identity Platform in Module 21)

  • Module 7 codelab: Add App Engine Task Queues push tasks to existing sample app

  • Module 8 codelab: Migrate from App Engine Task Queues push tasks to Cloud Tasks v1

    • Python 2
    • RECOMMENDED:
      • Module 9 - migrate to Python 3 and Cloud Datastore
      • Module 18 - add App Engine Task Queue pull tasks (and migrate to Cloud Pub/Sub in Module 19)
      • Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13)
      • Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16)
      • Module 20 - add App Engine Users (and migrate to Cloud Identity Platform in Module 21)

  • Module 9 codelab: Migrate a Python 2 Cloud NDB & Cloud Tasks (v1) app to a Python 3 Cloud Datastore & Cloud Tasks (v2) app

    • Optional migrations
      • Migrating to Python 3 is not required but recommended as Python 2 has been sunset
      • Migrating to Cloud Datastore is optional as Cloud NDB works on 3.x
    • Python 2
    • Python 3
      • FINISH: TBD
    • RECOMMENDED:
      • Module 18 - add App Engine Task Queue pull tasks (and migrate to Cloud Pub/Sub in Module 19)
      • Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13)
      • Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16)
      • Module 20 - add App Engine Users (and migrate to Cloud Identity Platform in Module 21)

  • Module 18 codelab: Add App Engine Task Queues pull tasks to existing sample app

  • Module 19 codelab: Migrate from App Engine Task Queues pull tasks to Cloud Pub/Sub

    • Python 2
    • Python 3
    • RECOMMENDED:
      • Module 7 - add App Engine Task Queue push tasks (and migrate to Cloud Tasks in Module 8)
      • Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13)
      • Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16)
      • Module 20 - add App Engine users (and migrate to Cloud Identity Platform in Module 21)

  • Module 12 codelab: Add App Engine Memcache to existing sample app

  • Module 13 codelab: Migrate from App Engine Memcache to Cloud Memorystore (for Redis) v1

    • Python 2
    • Python 3
    • RECOMMENDED:
      • Module 7 - add App Engine Task Queue push tasks (and migrate to Cloud Tasks in Module 8)
      • Module 18 - add App Engine Task Queue pull tasks (and migrate to Cloud Pub/Sub in Module 19)
      • Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16)
      • Module 20 - add App Engine users (and migrate to Cloud Identity Platform in Module 21)

  • Module 15 codelab: Add App Engine Blobstore to existing sample app

  • Module 16 codelab: Migrate from App Engine Blobstore to Cloud Storage (for Redis) v1

    • Python 2
    • Python 3
    • RECOMMENDED:
      • Module 7 - add App Engine Task Queue push tasks (and migrate to Cloud Tasks in Module 8)
      • Module 18 - add App Engine Task Queue pull tasks (and migrate to Cloud Pub/Sub in Module 19)
      • Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13)
      • Module 20 - add App Engine Users (and migrate to Cloud Identity Platform in Module 21)

  • Module 20 codelab: Add App Engine Users to existing sample app

    • Not a migration: add GAE Users to prepare for migration to Cloud Identity Platform/Firebase Auth
    • Python 2
    • NEXT:
      • Module 21 - migrate App Engine Users to Cloud Identity Platform/Firebase Auth

  • Module 21 codelab: Migrate from App Engine Users to Cloud Identity Platform/Firebase Auth

    • Python 2
    • Python 3
    • RECOMMENDED:
      • Module 7 - add App Engine Task Queue push tasks (and migrate to Cloud Tasks in Module 8)
      • Module 18 - add App Engine Task Queue pull tasks (and migrate to Cloud Pub/Sub in Module 19)
      • Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13)
      • Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16)

  • Module 3 codelab: Migrate from Cloud NDB to Cloud Datastore

    • Optional migration
      • Recommended only if using Cloud Datastore elsewhere (GAE or non-App Engine) apps
      • Helps w/code consistency & reusability, reduces maintenance costs
    • Python 2
    • Python 3
    • RECOMMENDED:
      • Module 7 - add App Engine Task Queue push tasks (and migrate to Cloud Tasks in Module 8)
      • Module 18 - add App Engine Task Queue pull tasks (and migrate to Cloud Pub/Sub in Module 19)
      • Module 12 - add App Engine Memcache (and migrate to Cloud Memorystore in Module 13)
      • Module 15 - add App Engine Blobstore (and migrate to Cloud Storage in Module 16)
      • Module 20 - add App Engine Users (and migrate to Cloud Identity Platform in Module 21)

Migrations to other Cloud serverless platforms

  • Module 4 codelab: Migrate from App Engine to Cloud Run with Docker

  • Module 5 codelab: Migrate from App Engine to Cloud Run with Cloud Buildpacks

  • Module 11 codelab: Migrate from App Engine to Cloud Functions

    • Optional migration
      • Recommended for small apps or for breaking up large apps into multiple microservices
    • Python 3 only
    • RECOMMENDED:
      • Module 7 - add App Engine Task Queue push tasks (and migrate to Cloud Tasks in Module 8)
      • Module 18 - add App Engine Task Queue pull tasks (and migrate to Cloud Pub/Sub in Module 19)
      • Module 12 - add App Engine memcache (and migrate to Cloud Memorystore in Module 13)
      • Module 15 - add App Engine blobstore (and migrate to Cloud Storage in Module 16)
    • OTHER OPTIONS (in somewhat priority order):
      • Module 5 - migrate to Cloud Run container with Cloud Buildpacks

Considerations for mobile developers

If your original app users does not have a user interface, i.e., mobile backends, etc., but still uses webapp2 for routing, some migration must still be completed. Your options:

Canonical code samples

Accessing legacy services in second generation

Many legacy App Engine first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) services are available (as of Sep 2021 for second generation runtimes (Python 3, Java 11/17, PHP 7/8, and Go 1.12 & newer) in a public preview. There are no videos or codelabs yet, however the Module 1 Flask migration using App Engine ndb Python 2 sample is available in Python 3 if you have access. Similarly, Python 3 editions are also available for Modules 7 and 12 which add usage of App Engine taskqueue and memcache, respectively. Also see the documentation on accessing bundled services from Python 3.

Community

Python App Engine developers hang out in various online communities, including these:

References