Forgot your password?
Using the ZODB
Installing and using the Zope Object Database (ZODB) without Zope or Plone.
Introduction
The Zope Object Database (ZODB) is a way of storing persistent data, just as with a relational database such as MySQL or PostgreSQL. The difference is that the data is not stored in rows or columns in a table, but instead in Python dictionaries, lists, and other data structures. Data objects are serialized and stored in a database file using pickle, a standard Python module. To retrieve data, the database file is opened, and the objects are accessed just as regular dictionaries, lists, and objects are.
The ZODB can be used by directly opening and closing a database file, which is what this document covers. For more enterprise or collaborative uses, the ZODB can be accessed through a server that manages transactions called ZEO.
Installing
First, you will need Python. Preferably at least version 2.4, version 2.5 if possible. Information on installing Python is available here for Windows, Mac, and Linux. Be sure to get easy_install. All you need to get started is Python and easy_install.
Once Python and easy_install are installed, open a command prompt (start, run, 'cmd') and run:
easy_install ZODB3
If you are on Windows, you will notice that a dependency, zope.interface, may not be able to install, giving a message about Visual Studio 2003. An earlier version of this dependency is available and can be installed with the command below. After installing this dependency, rerun easy_install ZODB3 to ensure that all of the dependencies are met and scripts installed.
easy_install zope.interface==3.3.0 easy_install ZODB3
Imports
I will probably expand more on this section in the future, but the imports are pretty straightforward.
# Imports needed to open the DB and interact with it: from ZODB import DB from ZODB.FileStorage import FileStorage from ZODB.PersistentMapping import PersistentMapping import transaction
# Imports needed for Persistent classes: # PersistentDict and PersistentList are ready to use as is. # Classes you create that need to be Persistent (ZODB aware) should inherit from Persistent. from persistent import Persistent from persistent.dict import PersistentDict from persistent.list import PersistentList
The following imports are not strictly needed to interact with the ZODB, but are extremely useful.
# Import needed for the writing function below - see 'def writing_to_zdb()' for more info import time
# Imports useful for datetime stuff from datetime import datetime from datetime import timedelta
Creating and accessing a ZODB
If a class inherits from Persistent, it will automatically become database aware. The only difference between regular dicts/lists and persistent dicts/lists is that you use PersistentDict() and PersistentList() instead of the typical {} and [] to create them. Regular dicts and lists don't register changes, while the PersistentDict and PersistentList do. That makes them cooler.
# To open or create a DB:
storage = FileStorage("/location/zodb_file.fs")
db = DB(storage)
connection = db.open()
root = connection.root()
# To put things in the DB (overly simplistic - see below about the function that writes to the ZODB): # NOTE - the root of the database is a Python dictionary! root['key'] = value # generates a transaction transaction.get().commit() # commits the transaction, like flushing a buffer
# use standard dictionary (and list) methods to remove things: del root['key'] # completely removes key and value, generates a transaction transaction.get().commit() # commits the transaction, like flushing a buffer
# To close the DB: transaction.get().abort() # we definitely want a clean close, so abandon any pending writes connection.close() db.close() storage.close()
Handy functions and their uses
Because opening and closing are done so much, I made functions to make a dict with the storage, db, connection, and root. The dict's name is zdb:
def open_db():
zdb = {}
zdb['storage'] = FileStorage("/location/zodb_file.fs")
zdb['db'] = DB(zdb['storage'])
zdb['connection'] = zdb['db'].open()
zdb['root'] = zdb['connection'].root()
return zdb
def close_db(zdb):
transaction.get().abort()
zdb['connection'].close()
zdb['db'].close()
zdb['storage'].close()
Now a function that reads from the ZODB looks like this:
def only_reading_the_zdb():
zdb = open_db() # this is always done
# Read stuff here
result_value = zdb['root']['key']
close_db(zdb) # this is always done
return result
A function that writes needs to take into account that other processes and threads may be accessing the ZODB. So we do a loop, attempting to write a certain number of pre-determined times and returning an error if we fail enough times.
# defined somewhere - top of the file, for example
max_retry = 10
sleep_delay = 1
def writing_to_zdb():
zdb = open_db()
retry = 0
while retry < max_retry:
try:
# this is where you do your writing, deleting, changing of the DB
zdb['root']['new_dict'] = PersistentDict() #for example, this creates a new {} that is persistent
transaction.get().commit()
except ConflictError:
retry += 1
time.sleep(sleep_delay)
pass
else:
break
else:
close_db(zdb)
return "Error - transaction could not complete" # we retried too many times
close_db(zdb)
return "Data written successfully"
An example application
Here is the structure of the database that we initially came up with for a time clock application. The idea is that a person can clock in and clock out, and we will keep track of when they did so. There is a need to keep track of employees, their attributes, and their in/out times.
When I work with the ZODB, I map out the data structures in Python notation. The whole ZODB is a Python dictionary, so I ask myself "what would it look like if I declared a variable with example data?" In this case, it looks like the following:
{
"Employees" : {
"employee_id_1" : {
"first_name" : "John",
"last_name" : "Doe",
"pay_amount" : 800 # Do pay grade as integers to avoid rounding errors?
},
"employee_id_2" : {
"first_name" : "Jane",
"last_name" : "Dowe",
"pay_amount" : 950
}
},
"Pending" : {
"employee_id_1" : datetime() # start time, user is clocked in
},
"Records" : {
"employee_id_1" : [ # note that these are lists, not dicts
[datetime(), datetime()], # start time, end time
[datetime(), datetime()],
[datetime(), datetime()]
],
"employee_id_2" : [
[datetime(), datetime()],
[datetime(), datetime()],
[datetime(), datetime()]
]
}
}
Here's a function that will write all of our above sample data into a database.
def write_sample_data():
zdb = open_db()
retry = 0
while retry < max_retry:
try:
zdb['root']['Employees'] = PersistentDict()
zdb['root']['Pending'] = PersistentDict()
zdb['root']['Records'] = PersistentDict()
zdb['root']['Employees']['employee_id_1'] = PersistentDict()
zdb['root']['Employees']['employee_id_1']['first_name'] = "John"
zdb['root']['Employees']['employee_id_1']['last_name'] = "Doe"
zdb['root']['Employees']['employee_id_1']['pay_amount'] = 800
zdb['root']['Employees']['employee_id_2'] = PersistentDict()
zdb['root']['Employees']['employee_id_2']['first_name'] = "Jane"
zdb['root']['Employees']['employee_id_2']['last_name'] = "Dowe"
zdb['root']['Employees']['employee_id_2']['pay_amount'] = 950
zdb['root']['Pending']['employee_id_1'] = datetime.strptime("2008-3-2-11-30", "%Y-%m-%d-%H-%M")
zdb['root']['Records']['employee_id_1'] = []
zdb['root']['Records']['employee_id_1'].append( [datetime.strptime("2008-3-2-11-30", "%Y-%m-%d-%H-%M"),
datetime.strptime("2008-3-2-13-30", "%Y-%m-%d-%H-%M")] )
zdb['root']['Records']['employee_id_1'].append( [datetime.strptime("2008-3-3-11-30", "%Y-%m-%d-%H-%M"),
datetime.strptime("2008-3-3-13-30", "%Y-%m-%d-%H-%M")] )
zdb['root']['Records']['employee_id_1'].append( [datetime.strptime("2008-3-4-11-30", "%Y-%m-%d-%H-%M"),
datetime.strptime("2008-3-4-13-30", "%Y-%m-%d-%H-%M")] )
zdb['root']['Records']['employee_id_2'] = []
zdb['root']['Records']['employee_id_2'].append( [datetime.strptime("2008-3-2-11-30", "%Y-%m-%d-%H-%M"),
datetime.strptime("2008-3-2-13-30", "%Y-%m-%d-%H-%M")] )
zdb['root']['Records']['employee_id_2'].append( [datetime.strptime("2008-3-3-11-30", "%Y-%m-%d-%H-%M"),
datetime.strptime("2008-3-3-13-30", "%Y-%m-%d-%H-%M")] )
zdb['root']['Records']['employee_id_2'].append( [datetime.strptime("2008-3-4-11-30", "%Y-%m-%d-%H-%M"),
datetime.strptime("2008-3-4-13-30", "%Y-%m-%d-%H-%M")] )
transaction.get().commit()
except ConflictError:
retry += 1
time.sleep(sleep_delay)
pass
else:
break
else:
close_db(zdb)
return "Error - transaction could not complete" # we retried too many times
close_db(zdb)
return "Data written successfully"
Complete example
Here is a complete example. If you have Python and ZODB3 installed correctly, it should be possible to run the code below. Be sure to run it from the terminal or command prompt. This demonstrates creating or opening a ZODB, clearing out any data that it might contain, then filling it up with example data and iterating over the entries.
from ZODB import DB
from ZODB.FileStorage import FileStorage
from ZODB.PersistentMapping import PersistentMapping
import transaction
from ZODB.POSException import ConflictError
from persistent import Persistent
from persistent.dict import PersistentDict
from persistent.list import PersistentList
import time
from datetime import datetime
from datetime import timedelta
max_retry = 10
sleep_delay = 1
def open_db():
zdb = {}
zdb['storage'] = FileStorage("timeclock.fs")
zdb['db'] = DB(zdb['storage'])
zdb['connection'] = zdb['db'].open()
zdb['root'] = zdb['connection'].root()
return zdb
def close_db(zdb):
transaction.get().abort()
zdb['connection'].close()
zdb['db'].close()
zdb['storage'].close()
def clear_data():
zdb = open_db()
if len(zdb['root'].keys()) > 0:
retry = 0
for k in zdb['root'].keys():
while retry < max_retry:
try:
del zdb['root'][k]
transaction.get().commit()
except ConflictError:
retry += 1
time.sleep(sleep_delay)
pass
else:
break
else:
close_db(zdb)
return "Error - transaction could not complete" # we retried too many times
close_db(zdb)
return "Database cleared"
def get_data():
zdb = open_db()
result = "Employees\n---------"
for employee_id in zdb['root']['Employees'].keys():
result += "\nUser ID: " + employee_id + "\n"
result += "Name: " + zdb['root']['Employees'][employee_id]['first_name'] + " " + zdb['root']['Employees'][employee_id]['last_name'] + "\n"
result += "Pay: " + repr(zdb['root']['Employees'][employee_id]['pay_amount']) + "\n"
if zdb['root']['Pending'].has_key(employee_id):
result += "Currently Clocked In: " + repr(zdb['root']['Pending'][employee_id]) + "\n"
else:
result += "Currently Clocked In: " + "No\n"
result += "Records\n-------"
for record in zdb['root']['Records'][employee_id]:
result += "\nClocked In: " + repr(record[0])
result += "\nClocked Out: " + repr(record[1])
result += "\n"
close_db(zdb)
return result
def write_sample_data():
zdb = open_db()
retry = 0
while retry < max_retry:
try:
zdb['root']['Employees'] = PersistentDict()
zdb['root']['Pending'] = PersistentDict()
zdb['root']['Records'] = PersistentDict()
zdb['root']['Employees']['employee_id_1'] = PersistentDict()
zdb['root']['Employees']['employee_id_1']['first_name'] = "John"
zdb['root']['Employees']['employee_id_1']['last_name'] = "Doe"
zdb['root']['Employees']['employee_id_1']['pay_amount'] = 800
zdb['root']['Employees']['employee_id_2'] = PersistentDict()
zdb['root']['Employees']['employee_id_2']['first_name'] = "Jane"
zdb['root']['Employees']['employee_id_2']['last_name'] = "Dowe"
zdb['root']['Employees']['employee_id_2']['pay_amount'] = 950
zdb['root']['Pending']['employee_id_1'] = datetime.strptime("2008-3-2-11-30", "%Y-%m-%d-%H-%M")
zdb['root']['Records']['employee_id_1'] = []
zdb['root']['Records']['employee_id_1'].append( [datetime.strptime("2008-3-2-11-30", "%Y-%m-%d-%H-%M"),
datetime.strptime("2008-3-2-13-30", "%Y-%m-%d-%H-%M")] )
zdb['root']['Records']['employee_id_1'].append( [datetime.strptime("2008-3-3-11-30", "%Y-%m-%d-%H-%M"),
datetime.strptime("2008-3-3-13-30", "%Y-%m-%d-%H-%M")] )
zdb['root']['Records']['employee_id_1'].append( [datetime.strptime("2008-3-4-11-30", "%Y-%m-%d-%H-%M"),
datetime.strptime("2008-3-4-13-30", "%Y-%m-%d-%H-%M")] )
zdb['root']['Records']['employee_id_2'] = []
zdb['root']['Records']['employee_id_2'].append( [datetime.strptime("2008-3-2-11-30", "%Y-%m-%d-%H-%M"),
datetime.strptime("2008-3-2-13-30", "%Y-%m-%d-%H-%M")] )
zdb['root']['Records']['employee_id_2'].append( [datetime.strptime("2008-3-3-11-30", "%Y-%m-%d-%H-%M"),
datetime.strptime("2008-3-3-13-30", "%Y-%m-%d-%H-%M")] )
zdb['root']['Records']['employee_id_2'].append( [datetime.strptime("2008-3-4-11-30", "%Y-%m-%d-%H-%M"),
datetime.strptime("2008-3-4-13-30", "%Y-%m-%d-%H-%M")] )
transaction.get().commit()
except ConflictError:
retry += 1
time.sleep(sleep_delay)
pass
else:
break
else:
close_db(zdb)
return "Error - transaction could not complete" # we retried too many times
close_db(zdb)
return "Data written successfully"
print clear_data()
print
print write_sample_data()
print
print get_data()
