org.archive.crawler.util
Class MemFPMergeUriUniqFilter
java.lang.Object
org.archive.crawler.util.FPMergeUriUniqFilter
org.archive.crawler.util.MemFPMergeUriUniqFilter
- All Implemented Interfaces:
- UriUniqFilter
public class MemFPMergeUriUniqFilter
- extends FPMergeUriUniqFilter
Crude all-in-memory FP-merging UriUniqFilter.
- Author:
- gojomo
Field Summary |
protected it.unimi.dsi.fastutil.longs.LongArrayList |
allFps
|
protected it.unimi.dsi.fastutil.longs.LongArrayList |
newFps
|
Fields inherited from class org.archive.crawler.util.FPMergeUriUniqFilter |
DEFAULT_MAX_PENDING, FLUSH_DELAY_FACTOR, maxPending, mergeDupAtLast, mergeDuplicateCount, nextFlushAllowableAfter, pendDupAtLast, pendDuplicateCount, pendingSet, profileLog, quickCache, quickDupAtLast, quickDuplicateCount, receiver |
Method Summary |
protected void |
addNewFp(long currFp)
Add an FP (which may be an old or new FP) to the new complete
list. |
protected it.unimi.dsi.fastutil.longs.LongIterator |
beginFpMerge()
Begin merging pending candidates with complete list. |
long |
count()
|
protected void |
finishFpMerge()
Complete the merge of candidate and previously-known FPs (closing
files/iterators as appropriate). |
Methods inherited from class org.archive.crawler.util.FPMergeUriUniqFilter |
add, addForce, addNow, close, createFp, flush, forget, note, pend, pending, profileLog, requestFlush, setDestination, setMaxPending, setProfileLog |
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
allFps
protected it.unimi.dsi.fastutil.longs.LongArrayList allFps
newFps
protected it.unimi.dsi.fastutil.longs.LongArrayList newFps
MemFPMergeUriUniqFilter
public MemFPMergeUriUniqFilter()
beginFpMerge
protected it.unimi.dsi.fastutil.longs.LongIterator beginFpMerge()
- Description copied from class:
FPMergeUriUniqFilter
- Begin merging pending candidates with complete list. Return an
Iterator which will return all previously-known FPs in turn.
- Specified by:
beginFpMerge
in class FPMergeUriUniqFilter
- Returns:
- Iterator over all previously-known FPs
addNewFp
protected void addNewFp(long currFp)
- Description copied from class:
FPMergeUriUniqFilter
- Add an FP (which may be an old or new FP) to the new complete
list. Should only be called after beginFpMerge() and before
finishFpMerge().
- Specified by:
addNewFp
in class FPMergeUriUniqFilter
- Parameters:
currFp
- the FP to add
finishFpMerge
protected void finishFpMerge()
- Description copied from class:
FPMergeUriUniqFilter
- Complete the merge of candidate and previously-known FPs (closing
files/iterators as appropriate).
- Specified by:
finishFpMerge
in class FPMergeUriUniqFilter
count
public long count()
- Returns:
- Count of already seen URIs.
Copyright © 2003-2011 Internet Archive. All Rights Reserved.