There are many reasons to update all the entities for a model: including
- Updating a model's schema.
- Adding new fields to exisiting entities.
- Removing obsolete non indexed properties from exisiting entities.
- Adding an index for an existing property that was previously not indexed
Under the new pricing scheme developers pay for datastore writes and reads. To write an entity to the datastore the cost depends on the number of indexes on that entity:
| New Entity Put (per entity, regardless of entity size) | 2 Writes + 2 Writes per indexed property value + 1 Write per composite index value |
| Existing Entity Put (per entity) | 1 Write + 4 Writes per modified indexed property value + 2 Writes per modified composite index value |
Therefore, writing just one entity might result in 20 or more write operations. Now considering the free limit for datastore writes is just 50,000 operations, we can reach the limit of writes in a day by writing just 2,500 entities that require 20 write operations each. If developers try to use map reduce on any data set the daily quota can be exceeded in just one mapreduce step. What used to be a completely free operation now could cost some money.
 |
| Image Source |
Ideally it would be nice to have a method to search for all entities that do not have a given property or are not indexed, but unfortunately that cannot be done in App Engine.
To work around this problem, I have added a property "meta_version" to all of my models. Initially meta_version is set to 1 for all my entities. Now if I need to update my model's schema, over several days, I can run a search for entities with their meta_version set to 1 and bump the version to 2. Since meta_version is indexed, I can run a search for those entities and just update 100 entities at a time. In this way I can stay under the free quota.
For example after adding the field to your model:
class MyModel(db.Model):
# Add this field to all models with a large number of entities.
meta_version = db.StringProperty(default='1')
Add a view which iterates through the entities a few at a time:
def update_my_model(request, num_entities = 100):
entity_list = MyModel.all().filter('meta_version =', '1').fetch(num_entities)
for entity in entity_list:
entity.meta_version = '2'
db.put(entity_list)
Now you can slowly evolve your model's schema or index old unindexed properties while staying under your free quota! Of course the final irony is that in order to add the meta_version field to all entities you have to update all of those entities without using this method.
