bes  Updated for version 3.20.6
BESFileLockingCache Class Reference

Implementation of a caching mechanism for compressed data. More...

#include <BESFileLockingCache.h>

Collaboration diagram for BESFileLockingCache:
Collaboration graph

Public Member Functions

 BESFileLockingCache (const std::string &cache_dir, const std::string &prefix, unsigned long long size)
 Make an instance of FileLockingCache. More...
 
bool cache_enabled () const
 
virtual bool cache_too_big (unsigned long long current_size) const
 look at the cache size; is it too large? Look at the cache size and see if it is too big. More...
 
virtual bool create_and_lock (const std::string &target, int &fd)
 Create a file in the cache and lock it for write access. More...
 
void disable ()
 Disable the cache. More...
 
virtual void dump (std::ostream &strm) const
 dumps information about this object More...
 
void enable ()
 Enable the cache. More...
 
virtual void exclusive_to_shared_lock (int fd)
 Transfer from an exclusive lock to a shared lock. More...
 
const std::string get_cache_directory ()
 
virtual std::string get_cache_file_name (const std::string &src, bool mangle=true)
 
const std::string get_cache_file_prefix ()
 
virtual unsigned long long get_cache_size ()
 Get the cache size. More...
 
virtual bool get_exclusive_lock (const std::string &target, int &fd)
 
virtual bool get_exclusive_lock_nb (const std::string &target, int &fd)
 
virtual bool get_read_lock (const std::string &target, int &fd)
 Get a read-only lock on the file if it exists. More...
 
void initialize (const std::string &cache_dir, const std::string &prefix, unsigned long long size)
 Initialize an instance of FileLockingCache. More...
 
bool is_unlimited () const
 Is this cache allowed to store as much as it wants? More...
 
virtual void lock_cache_read ()
 
virtual void lock_cache_write ()
 
virtual void purge_file (const std::string &file)
 Purge a single file from the cache. More...
 
virtual void unlock_and_close (const std::string &target)
 
virtual void unlock_cache ()
 
virtual void update_and_purge (const std::string &new_file)
 Purge files from the cache. More...
 
virtual unsigned long long update_cache_info (const std::string &target)
 Update the cache info file to include 'target'. More...
 

Static Public Member Functions

static bool dir_exists (const std::string &dir)
 

Friends

class cacheT
 
class FileLockingCacheTest
 

Detailed Description

Implementation of a caching mechanism for compressed data.

This cache uses simple advisory locking found on most modern unix file systems. It was originally designed to hold the decompressed versions of compressed files. Compressed files are uncompressed and stored in a cache where they can be used over and over until removed from the cache. Several processes can share the cache with each reading from files. At the same time, new files can be added and the cache can be purged, without disrupting the existing read operations.

How it works. When a file is added to the cache, the cache is locked - no other processes can add, read or remove files. Once a file has been added, the cache size is examined and, if needed, the cache is purged so that its size is 80% of the maximum size. Then the cache is unlocked. When a process looks to see if a file is already in the cache, the entire cache is locked. If the file is present, a shared read lock is obtained and the cache is unlocked.

Methods: create_and_lock() and get_read_lock() open and lock files; the former creates the file and locks it exclusively iff it does not exist, while the latter obtains a shared lock iff the file already exists. The unlock() methods unlock a file. The lock_cache_info() and unlock_cache_info() are used to control access to the whole cache - with the open + lock and close + unlock operations are performed atomically. Other methods that operate on the cache info file must only be called when the lock has been obtained.

Note
The locking mechanism uses Unix fcntl(2) and so is per process. That means that while getting an exclusive lock in one process will keep other processes from also getting an exclusive lock, it will not prevent other threads in the same process from getting another 'exclusive lock.' We could switch to flock(2) and get thread-safe locking, but we would trade off the ability to work with files on NFS volumes.

Definition at line 85 of file BESFileLockingCache.h.

Constructor & Destructor Documentation

◆ BESFileLockingCache()

BESFileLockingCache::BESFileLockingCache ( const std::string &  cache_dir,
const std::string &  prefix,
unsigned long long  size 
)

Make an instance of FileLockingCache.

Instantiate the FileLockingClass, using the given values for the cache directory, item prefix and size. If the cache directory is the empty string, do not initialize the cache, but instead mark the cache as 'disabled.' Otherwise, attempt to build the FileLockingCache object and throw an exception if the directory, ..., values are not valid.

Note
A side effect of this is that the cache_enabled() property will be set to true or false based on whether the cache should be used or note. This ensures that the evaluation of the various parameters (and the BESKeys that are likely used by specializations) will happen only once when the cache is disabled.
Parameters
cache_dirThe directory into which the cache files will be written.
prefixThe prefix that will be added to each cache file.
sizeThe size of the cache in MBytes
Exceptions
BESInternalErrorIf the cache_dir does not exist or is not writable. size is 0, or if cache dir does not exist.
BESErrorIf the parameters (directory, ...) are invalid.

Definition at line 94 of file BESFileLockingCache.cc.

Member Function Documentation

◆ cache_enabled()

bool BESFileLockingCache::cache_enabled ( ) const
inline
Returns
Is this cache enabled?

Definition at line 198 of file BESFileLockingCache.h.

◆ cache_too_big()

bool BESFileLockingCache::cache_too_big ( unsigned long long  current_size) const
virtual

look at the cache size; is it too large? Look at the cache size and see if it is too big.

Returns
True if the size is too big, false otherwise.

Definition at line 780 of file BESFileLockingCache.cc.

◆ create_and_lock()

bool BESFileLockingCache::create_and_lock ( const std::string &  target,
int &  fd 
)
virtual

Create a file in the cache and lock it for write access.

If the file does not exist, make it, open it for read-write access and get an exclusive lock on it. The locking operation blocks, although that should never happen.

Parameters
targetThe name of the file to make/open/lock
fdValue-result param that holds the file descriptor of the opened file
Returns
True if the operation was successful, false otherwise. This method will return false if the file already existed (the file won't be locked and the descriptor reference is undefined - but likely -1).
Exceptions
BESBESInternalErrororif any error except EEXIST is returned by open(2) or if fcntl(2) returns an error.

Definition at line 599 of file BESFileLockingCache.cc.

◆ dir_exists()

bool BESFileLockingCache::dir_exists ( const std::string &  dir)
static

Does the directory exist?

Note
This is a static method, so it can be called from other static methods like those that build instances of singletons.
Parameters
dirThe pathname to test.
Returns
True if the directory exists, false otherwise

Definition at line 1136 of file BESFileLockingCache.cc.

◆ disable()

void BESFileLockingCache::disable ( )
inline

Disable the cache.

Definition at line 204 of file BESFileLockingCache.h.

◆ dump()

void BESFileLockingCache::dump ( std::ostream &  strm) const
virtual

dumps information about this object

Displays the pointer value of this instance along with information about this cache.

Parameters
strmC++ i/o stream to dump the information to

Implements BESObj.

Definition at line 1151 of file BESFileLockingCache.cc.

◆ enable()

void BESFileLockingCache::enable ( )
inline

Enable the cache.

Definition at line 210 of file BESFileLockingCache.h.

◆ exclusive_to_shared_lock()

void BESFileLockingCache::exclusive_to_shared_lock ( int  fd)
virtual

Transfer from an exclusive lock to a shared lock.

If the file has an exclusive write lock on it, change that to a shared read lock. This is an atomic operation. If the call to fcntl(2) is protected by locking the cache, a dead lock will result given typical use of this class. This method exists to help with the situation where one process has the cache locked and is blocking on a shared read lock for a file that a second process has locked exclusively (for writing). By changing the exclusive lock to a shared lock, the first process can get its shared lock and then release the cache.

Parameters
fdThe file descriptor that is exclusively locked and which, on exit, will have a shared lock.

Definition at line 630 of file BESFileLockingCache.cc.

◆ get_cache_directory()

const std::string BESFileLockingCache::get_cache_directory ( )
inline
Returns
The directory used for the an instance of BESFileLockingCache

Definition at line 188 of file BESFileLockingCache.h.

◆ get_cache_file_name()

string BESFileLockingCache::get_cache_file_name ( const std::string &  src,
bool  mangle = true 
)
virtual

Returns the fully qualified file system path name for the cache file associated with this particular cache resource.

Note
How names are mangled: ALl occurrences of the characters '<', '>', '=', ',', '/', '(', ')', '"', ''', ':', '?', and ' ' with the '#' character.
Parameters
srcThe source name to cache
mangleIf True, assume the name is a file pathname and mangle it. If false, do not mangle the name (assume the caller has sent a suitable string) but do turn the string into a pathname located in the cache directory with the cache prefix. The 'mangle' param is true by default.

Reimplemented in cmr::CmrCache, httpd_catalog::RemoteHttpResourceCache, and BESUncompressCache.

Definition at line 451 of file BESFileLockingCache.cc.

◆ get_cache_file_prefix()

const std::string BESFileLockingCache::get_cache_file_prefix ( )
inline
Returns
The prefix used for items in an instance of BESFileLockingCache

Definition at line 182 of file BESFileLockingCache.h.

◆ get_cache_size()

unsigned long long BESFileLockingCache::get_cache_size ( )
virtual

Get the cache size.

Read the size information from the cache info file and return it. This methods locks the cache.

Returns
The size of the cache.

Definition at line 792 of file BESFileLockingCache.cc.

◆ get_read_lock()

bool BESFileLockingCache::get_read_lock ( const std::string &  target,
int &  fd 
)
virtual

Get a read-only lock on the file if it exists.

Try to get a read-only lock on the file, blocking until we can get it. If the file does not exist, return false.

Note
If this code returns false, that means the file did not exist in the cache at the time of the test. by the time the caller gets the result, the file may have been added to the cache by another process.
Parameters
targetThe path of the cached file
fdA value-result parameter set to the locked cached file. Undefined if the file could not be locked for read access.
Returns
true if the file is in the cache and has been locked, false if the file is/was not in the cache.
Exceptions
Errorif the attempt to get the (shared) lock failed for any reason other than that the file does/did not exist.

Definition at line 544 of file BESFileLockingCache.cc.

◆ initialize()

void BESFileLockingCache::initialize ( const std::string &  cache_dir,
const std::string &  prefix,
unsigned long long  size 
)

Initialize an instance of FileLockingCache.

Initialize and instance of FileLockingCache using the passed values for the cache directory, item prefix and max cache size. This will ignore the value of enable_cache() (but will correctly (re)set it based on the directory, ..., values). This provides a way for clients to re-initialize caches on the fly.

Parameters
cache_dirThe directory into which the cache files will be written.
prefixThe prefix that will be added to each cache file.
sizeThe size of the cache in MBytes
Exceptions
BESInternalErrorIf the cache_dir does not exist or is not writable. size is 0, or if cache dir does not exist.
BESErrorIf the parameters (directory, ...) are invalid.

Definition at line 116 of file BESFileLockingCache.cc.

◆ is_unlimited()

bool BESFileLockingCache::is_unlimited ( ) const
inline

Is this cache allowed to store as much as it wants?

If the size of the cache is zero bytes, then it is allowed to grow with out bounds.

Returns
True if the cache is unlimited in size, false if values will be purged after a preset size is exceeded.

Definition at line 177 of file BESFileLockingCache.h.

◆ lock_cache_read()

void BESFileLockingCache::lock_cache_read ( )
virtual

Get a shared lock on the 'cache info' file.

Definition at line 669 of file BESFileLockingCache.cc.

◆ lock_cache_write()

void BESFileLockingCache::lock_cache_write ( )
virtual

Get an exclusive lock on the 'cache info' file. The 'cache info' file is used to control certain cache actions, ensuring that they are atomic. These include making sure that the create_and_lock() and read_and_lock() operations are atomic as well as the purge and related operations.

Note
This is intended to be used internally only but might be useful in some settings.

Definition at line 654 of file BESFileLockingCache.cc.

◆ purge_file()

void BESFileLockingCache::purge_file ( const std::string &  file)
virtual

Purge a single file from the cache.

Purge a single file from the cache. The file might be old, etc., and need to be removed. Don't use this to shrink the cache when it gets too big, use update_and_purge() instead since that file optimizes accesses to the cache control file for several changes in a row.

Parameters
fileThe name of the file to purge.

Definition at line 1085 of file BESFileLockingCache.cc.

◆ unlock_and_close()

void BESFileLockingCache::unlock_and_close ( const std::string &  target)
virtual

Unlock the named file.

This does not do any name mangling; it just closes and unlocks whatever is named (or throws BESBESInternalErroror if the file cannot be closed). If the file was opened more than once, all descriptors are closed. If you need to close a specific descriptor, use the other version of unlock_and_close().

Note
This method assumes that the file was opened/locked using one of read_and_lock() or create_and_lock(). Those methods record the name/file- descriptor pairs so that the files can be properly closed and locks released.
Parameters
file_nameThe name of the file to unlock.
Exceptions
BESBESInternalErroror

Definition at line 713 of file BESFileLockingCache.cc.

◆ unlock_cache()

void BESFileLockingCache::unlock_cache ( )
virtual

Unlock the cache info file.

Note
This is intended to be used internally only but might be useful in some settings.

Definition at line 686 of file BESFileLockingCache.cc.

◆ update_and_purge()

void BESFileLockingCache::update_and_purge ( const std::string &  new_file)
virtual

Purge files from the cache.

Purge files, oldest to newest, if the current size of the cache exceeds the size of the cache specified in the constructor. This method uses an exclusive lock on the cache for the duration of the purge process.

Note
If the cache size in bytes is zero, calling this method has no affect (the cache is unlimited in size). Other public methods like update_cache_info() and get_cache_size() still work, however.
Parameters
new_fileDo not delete this file. The name of a file this process just added to the cache. Using fcntl(2) locking there is no way this process can detect its own lock, so the shared read lock on the new file won't keep this process from deleting it (but will keep other processes from deleting it).

Definition at line 940 of file BESFileLockingCache.cc.

◆ update_cache_info()

unsigned long long BESFileLockingCache::update_cache_info ( const std::string &  target)
virtual

Update the cache info file to include 'target'.

Add the size of the named file to the total cache size recorded in the cache info file. The cache info file is exclusively locked by this method for its duration. This updates the cache info file and returns the new size.

Parameters
targetThe name of the file
Returns
The new size of the cache

Definition at line 737 of file BESFileLockingCache.cc.


The documentation for this class was generated from the following files: