org.archive.crawler.postprocessor
Class LowDiskPauseProcessor
java.lang.Object
javax.management.Attribute
org.archive.crawler.settings.Type
org.archive.crawler.settings.ComplexType
org.archive.crawler.settings.ModuleType
org.archive.crawler.framework.Processor
org.archive.crawler.postprocessor.LowDiskPauseProcessor
- All Implemented Interfaces:
- java.io.Serializable, javax.management.DynamicMBean
public class LowDiskPauseProcessor
- extends Processor
Processor module which uses 'df -k', where available and with
the expected output format (on Linux), to monitor available
disk space and pause the crawl if free space on monitored
filesystems falls below certain thresholds.
- See Also:
- Serialized Form
Method Summary |
protected void |
innerProcess(CrawlURI curi)
Notes a CrawlURI's content size in its running tally. |
Methods inherited from class org.archive.crawler.framework.Processor |
checkForInterrupt, finalTasks, getController, getDecideRule, getDefaultNextProcessor, initialTasks, innerRejectProcess, isContentToProcess, isEnabled, isExpectedMimeType, isHttpTransactionContentToProcess, kickUpdate, process, report, rulesAccept, rulesAccept, setDefaultNextProcessor, spawn |
Methods inherited from class org.archive.crawler.settings.ComplexType |
addElementToDefinition, checkValue, earlyInitialize, getAbsoluteName, getAttribute, getAttribute, getAttribute, getAttributeInfo, getAttributeInfo, getAttributeInfoIterator, getAttributes, getDataContainerRecursive, getDataContainerRecursive, getDefaultValue, getDescription, getElementFromDefinition, getLegalValues, getLocalAttribute, getMBeanInfo, getMBeanInfo, getParent, getPreservedFields, getSettingsHandler, getUncheckedAttribute, getValue, globalSettings, invoke, isInitialized, isOverridden, iterator, removeElementFromDefinition, setAsOrder, setAttribute, setAttribute, setAttributes, setDescription, setPreservedFields, toString, unsetAttribute |
Methods inherited from class org.archive.crawler.settings.Type |
addConstraint, equals, getConstraints, getLegalValueType, isExpertSetting, isOverrideable, isTransient, setExpertSetting, setLegalValueType, setOverrideable, setTransient |
Methods inherited from class javax.management.Attribute |
getName, hashCode |
Methods inherited from class java.lang.Object |
clone, finalize, getClass, notify, notifyAll, wait, wait, wait |
ATTR_MONITOR_MOUNTS
public static final java.lang.String ATTR_MONITOR_MOUNTS
- List of mounts to monitor; should match "Mounted on" column of 'df' output
- See Also:
- Constant Field Values
DEFAULT_MONITOR_MOUNTS
public static final java.lang.String DEFAULT_MONITOR_MOUNTS
- See Also:
- Constant Field Values
ATTR_PAUSE_THRESHOLD
public static final java.lang.String ATTR_PAUSE_THRESHOLD
- Space available level below which a crawl-pause should be triggered.
- See Also:
- Constant Field Values
DEFAULT_PAUSE_THRESHOLD
public static final int DEFAULT_PAUSE_THRESHOLD
- See Also:
- Constant Field Values
ATTR_RECHECK_THRESHOLD
public static final java.lang.String ATTR_RECHECK_THRESHOLD
- Amount of content received between each recheck of free space
- See Also:
- Constant Field Values
DEFAULT_RECHECK_THRESHOLD
public static final int DEFAULT_RECHECK_THRESHOLD
- See Also:
- Constant Field Values
contentSinceCheck
protected int contentSinceCheck
VALID_DF_OUTPUT
public static final java.util.regex.Pattern VALID_DF_OUTPUT
AVAILABLE_EXTRACTOR
public static final java.util.regex.Pattern AVAILABLE_EXTRACTOR
LowDiskPauseProcessor
public LowDiskPauseProcessor(java.lang.String name)
- Parameters:
name
- Name of this writer.
innerProcess
protected void innerProcess(CrawlURI curi)
- Notes a CrawlURI's content size in its running tally. If the
recheck increment of content has passed through since the last
available-space check, checks available space and pauses the
crawl if any monitored mounts are below the configured threshold.
- Overrides:
innerProcess
in class Processor
- Parameters:
curi
- CrawlURI to process.
Copyright © 2003-2011 Internet Archive. All Rights Reserved.