org.archive.crawler.datamodel
Class Robotstxt

java.lang.Object
  org.archive.crawler.datamodel.Robotstxt

All Implemented Interfaces:

java.io.Serializable

public class Robotstxt
extends java.lang.Object
implements java.io.Serializable

Utility class for parsing and representing 'robots.txt' format directives, into a list of named user-agents and map from user-agents to RobotsDirectives.

See Also:

Serialized Form

Field Summary

(package private) java.util.Map<java.lang.String,RobotsDirectives>	agentsToDirectives
(package private) boolean	hasErrors
(package private) static RobotsDirectives	NO_DIRECTIVES
(package private) static long	serialVersionUID
(package private) java.util.LinkedList<java.lang.String>	userAgents

Constructor Summary

Robotstxt(java.io.BufferedReader reader)

Method Summary

boolean	allowsAll() Does this policy effectively allow everything? (No disallows or timing (crawl-delay) directives?)
RobotsDirectives	getDirectivesFor(java.lang.String ua)
java.util.List<java.lang.String>	getUserAgents()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

serialVersionUID

static final long serialVersionUID

See Also:

Constant Field Values

userAgents

java.util.LinkedList<java.lang.String> userAgents

agentsToDirectives

java.util.Map<java.lang.String,RobotsDirectives> agentsToDirectives

hasErrors

boolean hasErrors

NO_DIRECTIVES

static RobotsDirectives NO_DIRECTIVES

Constructor Detail

Robotstxt

public Robotstxt(java.io.BufferedReader reader)
          throws java.io.IOException

Throws:

java.io.IOException

Method Detail

allowsAll

public boolean allowsAll()

Does this policy effectively allow everything? (No disallows or timing (crawl-delay) directives?)

Returns:

getUserAgents

public java.util.List<java.lang.String> getUserAgents()

getDirectivesFor

public RobotsDirectives getDirectivesFor(java.lang.String ua)