org.archive.crawler.frontier
Class BdbMultipleWorkQueues

java.lang.Object
  org.archive.crawler.frontier.BdbMultipleWorkQueues

public class BdbMultipleWorkQueues
extends java.lang.Object

A BerkeleyDB-database-backed structure for holding ordered groupings of CrawlURIs. Reading the groupings from specific per-grouping (per-classKey/per-Host) starting points allows this to act as a collection of independent queues.

For how the bdb keys are made, see calculateInsertKey(CrawlURI).

TODO: refactor, improve naming.

Author:

gojomo

Nested Class Summary

class

BdbMultipleWorkQueues.BdbFrontierMarker Marker for remembering a position within the BdbMultipleWorkQueues.

Constructor Summary

BdbMultipleWorkQueues(com.sleepycat.je.Environment env, com.sleepycat.bind.serial.StoredClassCatalog classCatalog, boolean recycle)
Create the multi queue in the given environment.

Method Summary

void	addCap(byte[] origin) Add a dummy 'cap' entry at the given insertion key.
(package private) static com.sleepycat.je.DatabaseEntry	calculateInsertKey(CrawlURI curi) Calculate the insertKey that places a CrawlURI in the desired spot.
(package private) static byte[]	calculateOriginKey(java.lang.String classKey) Calculate the 'origin' key for a virtual queue of items with the given classKey.
void	close() clean up
void	delete(CrawlURI item) Delete the given CrawlURI from persistent store.
long	deleteMatchingFromQueue(java.lang.String match, java.lang.String queue, com.sleepycat.je.DatabaseEntry headKey) Delete all CrawlURIs matching the given expression.
protected void	forAllPendingDo(org.apache.commons.collections.Closure c) Utility method to perform action for all pending CrawlURI instances.
CrawlURI	get(com.sleepycat.je.DatabaseEntry headKey) Get the next nearest item after the given key.
protected com.sleepycat.je.DatabaseEntry	getFirstKey()
java.util.List	getFrom(FrontierMarker m, int maxMatches)
FrontierMarker	getInitialMarker(java.lang.String regexpr) Get a marker for beginning a scan over all contents
protected com.sleepycat.je.OperationStatus	getNextNearestItem(com.sleepycat.je.DatabaseEntry headKey, com.sleepycat.je.DatabaseEntry result)
void	put(CrawlURI curi, boolean overwriteIfPresent) Put the given CrawlURI in at the appropriate place.
(package private) void	sync() Method used by BdbFrontier during checkpointing.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BdbMultipleWorkQueues

public BdbMultipleWorkQueues(com.sleepycat.je.Environment env,
                             com.sleepycat.bind.serial.StoredClassCatalog classCatalog,
                             boolean recycle)
                      throws com.sleepycat.je.DatabaseException

Create the multi queue in the given environment.

Parameters:

env - bdb environment to use

classCatalog - Class catalog to use.

recycle - True if we are to reuse db content if any.

Throws:

com.sleepycat.je.DatabaseException

Method Detail

deleteMatchingFromQueue

public long deleteMatchingFromQueue(java.lang.String match,
                                    java.lang.String queue,
                                    com.sleepycat.je.DatabaseEntry headKey)
                             throws com.sleepycat.je.DatabaseException

Delete all CrawlURIs matching the given expression.

Parameters:

match -

queue -

headKey -

Returns:

count of deleted items

Throws:

com.sleepycat.je.DatabaseException

com.sleepycat.je.DatabaseException

getFrom

public java.util.List getFrom(FrontierMarker m,
                              int maxMatches)
                       throws com.sleepycat.je.DatabaseException

Parameters:

m - marker

maxMatches -

Returns:

list of matches starting from marker position

Throws:

com.sleepycat.je.DatabaseException

getInitialMarker

public FrontierMarker getInitialMarker(java.lang.String regexpr)

Get a marker for beginning a scan over all contents

Parameters:

regexpr -

Returns:

a marker pointing to the first item

getFirstKey

protected com.sleepycat.je.DatabaseEntry getFirstKey()
                                              throws com.sleepycat.je.DatabaseException

Returns:

the key to the first item in the database

Throws:

com.sleepycat.je.DatabaseException

get

public CrawlURI get(com.sleepycat.je.DatabaseEntry headKey)
             throws com.sleepycat.je.DatabaseException

Get the next nearest item after the given key. Relies on external discipline -- we'll look at the queues count of how many items it has -- to avoid asking for something from a range where there are no associated items -- otherwise could get first item of next 'queue' by mistake.

TODO: hold within a queue's range

Parameters:

headKey - Key prefix that demarks the beginning of the range in pendingUrisDB we're interested in.

Returns:

CrawlURI.

Throws:

com.sleepycat.je.DatabaseException

getNextNearestItem

protected com.sleepycat.je.OperationStatus getNextNearestItem(com.sleepycat.je.DatabaseEntry headKey,
                                                              com.sleepycat.je.DatabaseEntry result)
                                                       throws com.sleepycat.je.DatabaseException

Throws:

com.sleepycat.je.DatabaseException

put

public void put(CrawlURI curi,
                boolean overwriteIfPresent)
         throws com.sleepycat.je.DatabaseException

Put the given CrawlURI in at the appropriate place.

Parameters:

curi -

Throws:

com.sleepycat.je.DatabaseException

calculateOriginKey

static byte[] calculateOriginKey(java.lang.String classKey)

Calculate the 'origin' key for a virtual queue of items with the given classKey. This origin key will be a prefix of the keys for all items in the queue.

Parameters:

classKey - String key to derive origin byte key from

Returns:

a byte array key

calculateInsertKey

static com.sleepycat.je.DatabaseEntry calculateInsertKey(CrawlURI curi)

Calculate the insertKey that places a CrawlURI in the desired spot. First bytes are always classKey (usu. host) based -- ensuring grouping by host -- terminated by a zero byte. Then 8 bytes of data ensuring desired ordering within that 'queue' are used. The first byte of these 8 is priority -- allowing 'immediate' and 'soon' items to sort above regular. Next 1 byte is 'cost'. Last 6 bytes are ordinal serial number, ensuring earlier-discovered URIs sort before later. NOTE: Dangers here are: (1) priorities or costs over 2^7 (signed byte comparison) (2) ordinals over 2^48 Package access & static for testing purposes.

Parameters:

curi -

Returns:

a DatabaseEntry key for the CrawlURI

delete

public void delete(CrawlURI item)
            throws com.sleepycat.je.DatabaseException

Delete the given CrawlURI from persistent store. Requires the key under which it was stored be available.

Parameters:

item -

Throws:

com.sleepycat.je.DatabaseException

sync

void sync()

Method used by BdbFrontier during checkpointing.

The backing bdbje database has been marked deferred write so we save on writes to disk. Means no guarantees disk will have whats in memory unless a sync is called (Calling sync on the bdbje Environment is not sufficent).

Package access only because only Frontiers of this package would ever need access.

See Also:

Deferred Write Databases

close

public void close()

clean up

addCap

public void addCap(byte[] origin)

Add a dummy 'cap' entry at the given insertion key. Prevents 'seeks' to queue heads from holding lock on last item of 'preceding' queue. See: http://sourceforge.net/tracker/index.php?func=detail&aid=1262665&group_id=73833&atid=539102

Parameters:

origin - key at which to insert the cap

forAllPendingDo

protected void forAllPendingDo(org.apache.commons.collections.Closure c)
                        throws com.sleepycat.je.DatabaseException

Utility method to perform action for all pending CrawlURI instances.

Parameters:

c - Closure action to perform

Throws:

com.sleepycat.je.DatabaseException