org.archive.crawler.util
Class MemFPMergeUriUniqFilter

java.lang.Object
  extended by org.archive.crawler.util.FPMergeUriUniqFilter
      extended by org.archive.crawler.util.MemFPMergeUriUniqFilter
All Implemented Interfaces:
UriUniqFilter

public class MemFPMergeUriUniqFilter
extends FPMergeUriUniqFilter

Crude all-in-memory FP-merging UriUniqFilter.

Author:
gojomo

Nested Class Summary
 
Nested classes/interfaces inherited from class org.archive.crawler.util.FPMergeUriUniqFilter
FPMergeUriUniqFilter.PendingItem
 
Nested classes/interfaces inherited from interface org.archive.crawler.datamodel.UriUniqFilter
UriUniqFilter.HasUriReceiver
 
Field Summary
protected  it.unimi.dsi.fastutil.longs.LongArrayList allFps
           
protected  it.unimi.dsi.fastutil.longs.LongArrayList newFps
           
 
Fields inherited from class org.archive.crawler.util.FPMergeUriUniqFilter
DEFAULT_MAX_PENDING, FLUSH_DELAY_FACTOR, maxPending, mergeDupAtLast, mergeDuplicateCount, nextFlushAllowableAfter, pendDupAtLast, pendDuplicateCount, pendingSet, profileLog, quickCache, quickDupAtLast, quickDuplicateCount, receiver
 
Constructor Summary
MemFPMergeUriUniqFilter()
           
 
Method Summary
protected  void addNewFp(long currFp)
          Add an FP (which may be an old or new FP) to the new complete list.
protected  it.unimi.dsi.fastutil.longs.LongIterator beginFpMerge()
          Begin merging pending candidates with complete list.
 long count()
           
protected  void finishFpMerge()
          Complete the merge of candidate and previously-known FPs (closing files/iterators as appropriate).
 
Methods inherited from class org.archive.crawler.util.FPMergeUriUniqFilter
add, addForce, addNow, close, createFp, flush, forget, note, pend, pending, profileLog, requestFlush, setDestination, setMaxPending, setProfileLog
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

allFps

protected it.unimi.dsi.fastutil.longs.LongArrayList allFps

newFps

protected it.unimi.dsi.fastutil.longs.LongArrayList newFps
Constructor Detail

MemFPMergeUriUniqFilter

public MemFPMergeUriUniqFilter()
Method Detail

beginFpMerge

protected it.unimi.dsi.fastutil.longs.LongIterator beginFpMerge()
Description copied from class: FPMergeUriUniqFilter
Begin merging pending candidates with complete list. Return an Iterator which will return all previously-known FPs in turn.

Specified by:
beginFpMerge in class FPMergeUriUniqFilter
Returns:
Iterator over all previously-known FPs

addNewFp

protected void addNewFp(long currFp)
Description copied from class: FPMergeUriUniqFilter
Add an FP (which may be an old or new FP) to the new complete list. Should only be called after beginFpMerge() and before finishFpMerge().

Specified by:
addNewFp in class FPMergeUriUniqFilter
Parameters:
currFp - the FP to add

finishFpMerge

protected void finishFpMerge()
Description copied from class: FPMergeUriUniqFilter
Complete the merge of candidate and previously-known FPs (closing files/iterators as appropriate).

Specified by:
finishFpMerge in class FPMergeUriUniqFilter

count

public long count()
Returns:
Count of already seen URIs.


Copyright © 2003-2011 Internet Archive. All Rights Reserved.