Appengine: Transaction Safe Reference Properties with Ancestor Queries

This is a quick article on transactions with "one to many" relationship queries within a single by utilizing entity groups and ancestor queries. If you find it helpful, please feel free to link to it and help spread the google juice. Special thanks to co-worker Robert Kluin for helping me figure this out last week.

My Problem

My initial problem was that I was working inside a transaction and needed to fetch all the entities of a one to many relationship. In my case, I designed my data to be in the same entity group, thus I figured it was transaction safe. However, I was getting the error: BadRequestError: Only ancestor queries are allowed inside transactions.. This had me a bit stumped since I was not actually doing any queries inside of the code. However, Appengine itself was under the hood.

The Issue

With a bit of discussion with my coworkers, I traced the problematic query back to the underlying query behind Appengine's handy named reference property. I had a kind (we'll call it Hand) that had a one-to-many relationship on a kind (we'll call Finger). The issue is that hand.fingers does not run an ancestor query. In my specific case, this did not make sense, since all the fingers were in the same entity group as their parent hand. However, since reference properties do not have to be in the same entity group, the query resembles Finger.all().filter('hand =', hand) which is obviously not a ancestor query.

The solution

The solution is to avoid using the reference property and manually query fingers = Finger.all().ancestor(hand). This will fetch all of the finger entities that have an ancestor (parent) of hand. Note: that this is not the same as a reference property query and that the links between parent/child entities among an entity group have a different purpose than reference properties even though they often coincidentally have an overlap in the data. This solution leverages this coincidence.

Example Code

Note: I didn't actually run this specific code. Use at your own risk. [source=python] from google.appengine.ext import db from base.datastore.util import transaction class Hand(db.Model): is_right = db.BooleanProperty() is_dominant = db.BooleanProperty() class Finger(db.Model): position = db.IntegerProperty(default=0) hand = db.ReferenceProperty(Hand, collection_name='fingers') def trim_nail(self): #... do work pass def trim_my_nails(hand): @transaction def txn(hand): #fingers = hand.fingers # NOT TRANSACTION SAFE fingers = Finger.all().ancestor(hand) # Transaction safe for finger in fingers: finger.trim_nail() #... do whatever transaction friendly tasks return hand trimmed_hand = txn(hand) return hand [/source]

Quick Notes and Pitfalls

* Transactions only allow ancestor queries * Ancestor queries only work when the data is in the same entity group * Given the above, transactions can only query data in the same entity group. * Ancestor queries are far faster than regular queries because of the way that GAE stores their data for entity groups. * Entity groups are recommended to only be "about the size of one user's data" and should not get too big. In the example above, you probably don't want "everyone's hands and their fingers" in the same entity group just to be able to work inside of transactions. Model your workflow better to avoid this. * If you are on high-replication datastore (HRD), then this works even faster. * If you have a long chain of parent relationships and the model in the ancestor query is the parent of the parent, you may get more entities back than you expect. The hand/finger example is a bad analogy in this case. However, if you think of Forums with Sub-Forums and you try to get all the posts with the ancestor some forum entity, you might be getting back all the posts of that forum as well as any of its child forums. This is ideal for some situations and is unexpected behavior for others. Further Reading

• Datastore Indexes and Queries Google App Engine Docs
• Transactions Google App Engine Docs