Appengine Files Api - part 1: Storing/Fetching remote images in Blobstore using Django

Here is a quick tutorial on how to fetch remote images with the new Files API on Google Appengine 1.4.3 released March 30th. These examples use Django. Before we begin, read up on how to write files to blobstore using the Files API. Prior to the 1.4.3 release of the SDK, the only way to get files into the blobstore (and thus leverage things like BlobInfo) was to upload them via a POST request. To do what I am attempting below required a really confusing multistep process to fetch the file data and then generate a post request to the blobstore upload url. This was very error prone and awkward, but it worked. With 1.4.3, the solution is fairly straight forward. Also, note, the below example works with any file data, not just images. First lets set up a simple url to handle the fetching and displaying. [source=python] # Put this in your /urls.py from django.conf.urls.defaults import * urlpatterns = patterns('', ... (r'^remote_fetch_image/$', 'cdn.views.remote_fetch_image'), (r'^remote_display_image/(?P[^/]+)/$', 'cdn.views.remote_display_image'), ... ) [/source] Next create a folder in your main application called "cdn". Inside of it, add and empty __init__.py file as well as a views.py. You may also need to add to add "cdn" to your list of installed apps in settings.py [source=python] INSTALLED_APPS = ( ... 'cdn', ... ) [/source] Next, open up cdn/views.py and add the following code: [source=python] from __future__ import with_statement # Note this MUST go at the top of your views.py from google.appengine.ext import blobstore from django import http from google.appengine.ext import db def remote_fetch_image(request): from google.appengine.api import files, urlfetch from django.core import urlresolvers from django import http # Fetch image content image_url = request.GET.get('image_url', None) if not image_url: return http.HttpResponse('Please provide a query argument named "image_url" that is the full url to an image. ') fetch_response = urlfetch.fetch(image_url) file_data = fetch_response.content content_type = fetch_response.headers.get('content-type', None) # Create the file file_name = files.blobstore.create(mime_type=content_type) # Open the file and write to it with files.open(file_name, 'a') as f: f.write(file_data) # Finalize the file. Do this before attempting to read it. files.finalize(file_name) # Get the file's blob key blob_key = files.blobstore.get_blob_key(file_name) # We're technically done, but lets redirect and display the image return http.HttpResponseRedirect(urlresolvers.reverse(remote_display_image, args=[blob_key])) def remote_display_image(request, blob_key): # Fetch blob by key from blobstore blob_info = blobstore.BlobInfo.get(blob_key) if not blob_info: raise Exception('Blob Key does not exist') blob_file_size = blob_info.size blob_content_type = blob_info.content_type # Attempt to fetch the blob in one or more chunks depending on size and api limits blob_concat = "" start = 0 end = blobstore.MAX_BLOB_FETCH_SIZE - 1 step = blobstore.MAX_BLOB_FETCH_SIZE - 1 while(start < blob_file_size): blob_concat += blobstore.fetch_data(blob_key, start, end) temp_end = end start = temp_end + 1 end = temp_end + step return http.HttpResponse(blob_concat, mimetype=blob_content_type) [/source] Finally, open your app in the browser with the url to a file. For me, my sdk is mapped to the domain garrettclan.org: example: http://garrettclan.org/remote_fetch_image/?image_url=http://blaineyo.digibodies.com/files/2010/01/birthmarks2.jpg You should see the image at the url and it is now in the blobstore. This is an example usage of the new files api. Please note that this is experimental and may change as Appengine progresses. Also, not that the above processes is subject to the urlfetch quota, blobstore quota, and cpu quotas. See more details about the files API here. Also, if interested, the image I used is a painting my collaborative art crew did called "Nuroma Flare", part of the Birthmarks series. The next steps:

1. Part 2: Fetching the Facebook Profile Image and attaching it to a user
2. Part 3: Scale/crop down large uploaded images to a reasonable size before storing in blobstore to save on disk space quotas
3. Part 4: Utilizing existing blobstore features to dynamically generate thumbnails and sized images