Using Google Cloud Storage from Google App Engine’s Golang Runtime in Development

Terminology

Google Cloud Storage (GCS): object store (for storing files). Like Amazon S3.
Google App Engine (GAE): a platform-as-a-service (PaaS) offering. It is somewhat similar to Heroku or Amazon’s Elastic Beanstalk.
Golang: programming language. Commonly known as Go.
Golang Runtime: One of Google App Engine’s runtimes (Python, Java, PHP, Ruby and Go).

Trying the Golang Runtime on Google App Engine

I tried Google App Engine many years ago back when it was announced (way before the days of Google Cloud) and when Python was the only runtime. I’ve been using Heroku more lately but recently came back to checkout GAE because I wanted to try the Go runtime for a project (sHAR – share HAR files) and the free 5GB GCS bucket sealed the deal.

Once I followed the Quickstart for Go App Engine tutorial and had the Hello World app up and running, I setup the essential routes (using mux), put in the bare-bones logic to get the POST body, and I was ready to store a file to GCS.

package shar

import (
	"io"
	"io/ioutil"
	"net/http"
	"os"

	"github.com/gorilla/mux"
	"google.golang.org/appengine"
	"google.golang.org/appengine/log"
)

// serve static file
func defaultPageHandler(w http.ResponseWriter, r *http.Request) {
	f, err := os.Open("static/index.html")
	if err != nil {
		ctx := appengine.NewContext(r)
		log.Errorf(ctx, "error opening file: %v", err)
		http.Error(w, "", http.StatusInternalServerError)
	}

	io.Copy(w, f)
}

func uploadHandler(w http.ResponseWriter, r *http.Request) {
	ctx := appengine.NewContext(r)
	payload, err := ioutil.ReadAll(r.Body)
	if err != nil {
		log.Errorf(ctx, "reading request body failed with error: %v", err)
	}

	// write request body back as response body, for no reason
	w.Write(payload)

	// write payload to Google Cloud Storage
	// ...
}

func init() {
	router := mux.NewRouter()
	router.HandleFunc("/", defaultPageHandler)
	router.HandleFunc("/upload", uploadHandler).Methods("POST")
	http.Handle("/", router)
}

Storing Files to Google Cloud Storage

There is a page dedicated to reading and writing data to GCS from a Go App Engine app. There were a couple issues with getting a write to GCS to work when I tried it for the first time.

First, the sample code in the docs seemed like a bad copy-pasta job from the examples folder of the gcloud-golang public Github repository. I managed to put together an example GAE app that successfully wrote to GCS, but only in production. The docs have since been improved after some discussion in a Github issue.

The second issue is that the write to GCS would not work in dev and I kept running into an authentication error. I saw errors like the following:

INFO     2016-05-16 01:49:48,294 devappserver2.py:769] Skipping SDK update check.
INFO     2016-05-16 01:49:48,328 api_server.py:205] Starting API server at: http://localhost:52382
INFO     2016-05-16 01:49:48,330 dispatcher.py:197] Starting module "default" running at: http://localhost:8080
INFO     2016-05-16 01:49:48,331 admin_server.py:116] Starting admin server at: http://localhost:8000
2016/05/16 01:49:55 ERROR: createFile: unable to close bucket &{ %!q(*storage.ACLHandle=&{0xc8201ce810 app_default_bucket  false }) %!q(*storage.ACLHandle=&{0xc8201ce810 app_default_bucket  true}) %!q(*storage.Client=&{0xc8201cdb30 0xc82023c000}) "app_default_bucket"}, file "file.txt": googleapi: Error 401: Invalid Credentials, authError
INFO     2016-05-16 01:49:55,493 module.py:787] default: "GET / HTTP/1.1" 200 62
2016/05/16 01:49:56 ERROR: createFile: unable to close bucket &{ %!q(*storage.ACLHandle=&{0xc82016e160 app_default_bucket  false }) %!q(*storage.ACLHandle=&{0xc82016e160 app_default_bucket  true}) %!q(*storage.Client=&{0xc820a862a0 0xc82018c000}) "app_default_bucket"}, file "file.txt": googleapi: Error 401: Invalid Credentials, authError



2016/05/16 02:02:36 ERROR: createFile: unable to close bucket "app_default_bucket", file "demo-testfile-go": googleapi: Error 401: Invalid Credentials, authError

As it turns out, the reason for this is because there is no Google Cloud Storage emulator for the Go dev env.

GCS from GAE Go in Dev Workaround

In addition to the regular free 5GB GCS bucket, each App Engine app also gets a separate free 5GB staging bucket. The name of this bucket would have staging. prefixed to the default bucket name: staging.<YOUR-APP-NAME>.appspot.com. You need to activate the cloud storage buckets for each GAE app you create, they are not availalbe by default.

By using a function like the following to use the staging bucket when in dev, you can keep running the same code in dev and not clobber your prod bucket with data generated during development.

func bucketName(ctx context.Context) (string, error) {
	if appengine.IsDevAppServer() {
		return "staging.<YOUR-APP-NAME>.appspot.com", nil
	} else {
		// determine the default bucket name
		bucketName, err := file.DefaultBucketName(ctx)
		if err != nil {
			log.Errorf(ctx, "failed to get default GCS bucket name: %v", err)
			return "", err
		}

		return bucketName, nil
	}
}

The DefaultBucketName(…) method is available from the ”google.golang.org/appengine/file” package:

func DefaultBucketName(c context.Context) (string, error)

Calling DefaultBucketName(...) would return the proper bucket name in prod, but returns the same default name for all apps in dev; hence the staging bucket name is hard coded as a string in the bucketName() func.

Call our custom bucketName() function to get the right bucket name based on the environment (dev/prod) and pass it to the bucket initialization code like so:

func doingSomething() error {
	// …

	bName, err := bucketName(ctx)
	if err != nil {
		return err
	}
	bucket := client.Bucket(bName)
	wc := bucket.Object(fileName).NewWriter(ctx)

	if _, err := wc.Write(fInfo.Data); err != nil {
		log.Errorf(ctx, "unable to write data to bucket %q, file %q: %v", bucket, fInfo.Name, err)
		return err
	}

	if err := wc.Close(); err != nil {
		log.Errorf(ctx, "unable to close bucket %q, file %q: %v", bucket, fInfo.Name, err)
		return err
	}

	return nil
}

At this time, there does not appear to be any plans to add GCS dev emulation for the Go runtime. Let me know if you have found a better hack than this to get GCS uploads working in dev for the GAE Go runtime. An alternative to this is to use the blobstore service, which has a proper dev emulator.

TOUGH TYPE