• ExSite::Store

• Storage Keys
• Storage Lifetime
• Garbage Collection

• Notification Callbacks

• Implementation
• Store Configuration
• Special Methods

ExSite::Store

ExSite::Store is a multi-purpose data store that allows for sharing of complex data structures among different processes. It is used by ExSite for the following purposes:

• database cache
  
• storing system configuration
  
• session management
  

Furthermore, any plug-in module can utilize the Store for persistent storage of data.

The data store exposes itself to the application as a simple hash (%store) which may be used by any code in a read/write fashion. Values written to %store will persist beyond the life of the process, and will be available to other processes to read.

It is assumed that the store will be accessed via wrapper classes that will handle the job of fetching data from original sources, loading the data into the store, and checking the store for the data before returning to the original source. See for example, ExSite::Cache and ExSite::Session. Those classes are responsible for specifying any special timeouts for stored values, otherwise the default expiry timeout is used (typically 1 hour).

The values written to the store can be arbitrarily complex Perl data structures. They are serialized using Storable::freeze before being saved, and are recovered using Storable::thaw. Freezing and thawing of stored values is handled automatically by the store.

Storage Keys

Since the store can keep track of widely disparate data, it is important to use unique keys when reading or writing to it, to avoid grabbing or overwriting unrelated data accidentally. However, the store does not police this in any way, so you can easily clobber your own data if not careful. If in doubt, use a standard prefix on your keys (eg. your plug-in name) to keep them in an identifiable group. For example, the database cache prefixes its store keys with ``cache'': by default, and the session manager prefixes its store keys with ``session:''.

The store records a few internal items under plain text keys that are prefixed with ``_''. Of special importance is the ``_id'' key, which records the ExSite working directory (ie. your CGI-BIN). If this does not match the working directory of the current process, the store connection will be rejected. This is to prevent one site/installation from connecting to another's store.

Storage Lifetime

Every item is placed into the store with an expiry time (which is a Unix timestamp). Items past their expiry time are cleaned out in two cases: (1) if they are fetched after their expiry time, and (2) when garbage collection runs.

The default lifetime of an item placed into the store is defined in the store configuration (see below), and is typically 1 hour. The lifetime can be change/extended by renewing the item:

    (tied %store)->renew($key,$new_expiry_time);

You can store an item and set a custom lifetime in one step, as follows:

    (tied %store)->put($key,$value,$expiry_time);

If you set the expiry time to 0 using either of these methods, the item will persist in the store indefinitely. Use this for system configurations that are not expected to change during normal system use.

Garbage Collection

Purging of expired items occurs automatically after a certain amount of time has passed (default is every 15 minutes). This helps to clear out old or infrequently-used items that could use a refresh, or are simply taking up space.

Because the store will remove expired data on a regular basis, you cannot rely on a stored item being present when you request it. Your code is responsible for obtaining the desired data from alternate sources if this is the case. For this reason, the store is useful primarily as a multipurpose cache to improve performance on costly data-fetch operations, or as a place to save state for a limited term.

If you want to extend the lifetime of something in the store, to prevent it from being expired when it is still needed, use the following method:

    (tied %store)->renew($key,$new_expiry_time);

Notification Callbacks

The store can be configured to callback to another system component when it automatically clears out expired items. This is so that higher-level components that use the store can clean themselves up if necessary when garbage collection has been done.

If you want to be informed when an item is automatically reaped from the store, register your callback with the store:

    (tied %store)->set_callback($my_obj, $notify_method, $regexp);

$my_obj is the object that should receive the notification. $notify_method is the object method that will process the notification. $regexp is a regular expression that is used to match store keys; only matching keys will trigger the notification.

The notification callback method will be passed the key of the expired item. Ie. when an item is expired, the following callback is executed:

    $my_obj->$notify_method($key);

Example: When cached items are expired, the cache requires notification so that it can clean up its internal list of cache keys. It registers the callback using something like:

    (tied %store)->set_callback($share{Cache}, "delnotify", "^cache:");

Now when the store expires any key starting with ``cache:'', it informs the cache of this by calling:

    $share{Cache}->delnotify($key);

Implementation

This version of the ExSite store uses a GDBM database for persistent storage of data. This is significantly faster than a SQL database, and also has the advantage of persisting on disk even through system reboots, so that sessions and other state data are not lost.

The utility script bin/store.pl should be used to generate and initialize the GDBM file. Use this command:

    ../bin/store.pl --reset

(It will not be automatically generated because the webserver process does not normally have the necessary privileges to create the GDBM files in the working directory.) You can re-run this script to reset the store, but that will clear all stored items from the GDBM file. You should re-run this script if you ever move your CGI-BIN to a different directory (because the store will refuse connection when it detects this), or if you have any problems with file corruption.

Note that DBM files tend to grow as new items are added, but do not shrink when old items are removed. This is because the DBM file holds on to its disk space to reuse it for future items. If the DBM file is taking up too much disk space, you can use this command to shrink the file:

    ../bin/store.pl --rebuild

The StoreAdm plug-in supports scheduling of this function at regular intervals.

Store Configuration

The routine Local::store_conf() contains the configuration parameters for the store. (These parameters cannot be stored in %config, because %config is itself read from the store if possible.) The important configuration settings are:

name

This is the name of the store database file, which will be created in CGI-BIN.

lifetime

This is the default lifetime of an item placed into the store, in seconds.

purge_interval

This defines how often garbage collection runs, in seconds. If a connection to the store is made after this many seconds have elapsed since the last purge, then purge() is called when the job disconnects from the store (ie. when the store is untied).

purge_load

This defines the maximum system load under which the store will run garbage collection. If the load exceeds this number, then garbage collection will be deferred until things are quieter.

log

Set this to 1 to enable logging of store operations. The store log is written to the store itself (under the key _log).

loglimit

This is the maximum number of lines of log that will be kept on file. When the log exceeds this length, the oldest lines will be expired to make room for the new lines.

lock

This is the name of the store lockfile. If not provided it will default to the store database name, above, with .lock appended.

retry

When the store needs to write to itself, it requires an exclusive lock. It may not be able to acquire this lock if the system is busy. If that is the case, it waits for a second in the hopes that the business will pass, and tried again. The retry parameter defines the maximum number of retries before it gives up and cancels the operation.

Special Methods

The following special methods can be accessed from the store object itself:

connect()

Opens the GDBM file and validates that it is the correct store.

get($key)

Retrieves a specific item from the store. (Fails if the item has expired.)

getraw($key)

Retrieves a specific item from the store, along with store metadata used to manage the item (such as its expiry time). (This returns the item, even if the item has expired.)

put($key,$value,$expiry)

Places an item ($value) into the store indexed under $key. An optional $expiry time can be provided (Unix timestamp), but the store default will be used if this is not given.

putraw($key,$store_record)

Places an item ($value) into the store indexed under $key. An The item should be formatted as a store record, including fields for the data and expiry time.

delete($key)

Removes the item indexed under $key.

renew($key,$new_expiry_time)

Updates the expiry time of the item indexed under $key.

purge()

Removes all expired items from the store.

reset()

Removes all items from the store.

rebuild()

Reclaims disk space from deleted items, to shrink the size of the store database. Because this generates a new database, Apache will not likely have the necessary privileges to create the file. For that reason, the rebuild command should be issued from cron or manually.

size()

Returns the total size of the frozen data in the store.

summary()

Returns a text string describing the store contents.

is_persistent()

Returns TRUE if the store is connected to persistent storage, and FALSE if not.