org.archive.crawler.datamodel
Class RobotsDirectives
java.lang.Object
org.archive.crawler.datamodel.RobotsDirectives
- All Implemented Interfaces:
- java.io.Serializable
public class RobotsDirectives
- extends java.lang.Object
- implements java.io.Serializable
Represents the directives that apply to a user-agent (or set of
user-agents)
- See Also:
- Serialized Form
Field Summary |
(package private) java.util.concurrent.ConcurrentSkipListSet<java.lang.String> |
allows
|
(package private) float |
crawlDelay
|
(package private) java.util.concurrent.ConcurrentSkipListSet<java.lang.String> |
disallows
|
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
disallows
java.util.concurrent.ConcurrentSkipListSet<java.lang.String> disallows
allows
java.util.concurrent.ConcurrentSkipListSet<java.lang.String> allows
crawlDelay
float crawlDelay
RobotsDirectives
public RobotsDirectives()
allows
public boolean allows(java.lang.String path)
longestPrefixLength
protected int longestPrefixLength(java.util.concurrent.ConcurrentSkipListSet<java.lang.String> prefixSet,
java.lang.String str)
- Parameters:
prefixSet
- str
-
- Returns:
- length of longest entry in
prefixSet
that prefixes str
, or zero
if no entry prefixes str
addDisallow
public void addDisallow(java.lang.String path)
addAllow
public void addAllow(java.lang.String path)
setCrawlDelay
public void setCrawlDelay(float i)
getCrawlDelay
public float getCrawlDelay()
Copyright © 2003-2011 Internet Archive. All Rights Reserved.