org.archive.crawler.fetcher
Class FetchDNS

java.lang.Object
  javax.management.Attribute
      org.archive.crawler.settings.Type
          org.archive.crawler.settings.ComplexType
              org.archive.crawler.settings.ModuleType
                  org.archive.crawler.framework.Processor
                      org.archive.crawler.fetcher.FetchDNS

All Implemented Interfaces:

java.io.Serializable, javax.management.DynamicMBean, CoreAttributeConstants, FetchStatusCodes

public class FetchDNS
extends Processor
implements CoreAttributeConstants, FetchStatusCodes

Processor to resolve 'dns:' URIs. TODO: Refactor to use org.archive.util.DNSJavaUtils.

Author:

multiple

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.archive.crawler.settings.ComplexType

ComplexType.MBeanAttributeInfoIterator

Field Summary

protected java.net.InetAddress

serverInetAddr

Fields inherited from class org.archive.crawler.framework.Processor

ATTR_DECIDE_RULES, ATTR_ENABLED, attrDecideRules

Fields inherited from class org.archive.crawler.settings.ComplexType

definition, definitionMap

Fields inherited from interface org.archive.crawler.datamodel.CoreAttributeConstants

A_ANNOTATIONS, A_CONTENT_DIGEST, A_CONTENT_TYPE, A_CREDENTIAL_AVATARS_KEY, A_DELAY_FACTOR, A_DISTANCE_FROM_SEED, A_DNS_FETCH_TIME, A_DNS_SERVER_IP_LABEL, A_ETAG_HEADER, A_FETCH_BEGAN_TIME, A_FETCH_COMPLETED_TIME, A_FETCH_HISTORY, A_FORCE_RETIRE, A_FTP_CONTROL_CONVERSATION, A_FTP_FETCH_STATUS, A_HERITABLE_KEYS, A_HTML_BASE, A_HTTP_BIND_ADDRESS, A_HTTP_PROXY_HOST, A_HTTP_PROXY_PORT, A_HTTP_TRANSACTION, A_LAST_MODIFIED_HEADER, A_LOCALIZED_ERRORS, A_META_ROBOTS, A_MINIMUM_DELAY, A_MIRROR_PATH, A_PREREQUISITE_URI, A_REFERENCE_LENGTH, A_RETRY_DELAY, A_RRECORD_SET_LABEL, A_RUNTIME_EXCEPTION, A_SOURCE_TAG, A_STATUS, A_WRITTEN_TO_WARC, HEADER_TRUNC, LENGTH_TRUNC, TIMER_TRUNC, TRUNC_SUFFIX

Fields inherited from interface org.archive.crawler.datamodel.FetchStatusCodes

S_BLOCKED_BY_CUSTOM_PROCESSOR, S_BLOCKED_BY_QUOTA, S_BLOCKED_BY_RUNTIME_LIMIT, S_BLOCKED_BY_USER, S_CONNECT_FAILED, S_CONNECT_LOST, S_DEEMED_CHAFF, S_DEEMED_NOT_FOUND, S_DEFERRED, S_DELETED_BY_USER, S_DNS_SUCCESS, S_DOMAIN_PREREQUISITE_FAILURE, S_DOMAIN_UNRESOLVABLE, S_GETBYNAME_SUCCESS, S_OTHER_PREREQUISITE_FAILURE, S_OUT_OF_SCOPE, S_PREREQUISITE_UNSCHEDULABLE_FAILURE, S_PROCESSING_THREAD_KILLED, S_ROBOTS_PRECLUDED, S_ROBOTS_PREREQUISITE_FAILURE, S_RUNTIME_EXCEPTION, S_SERIOUS_ERROR, S_TIMEOUT, S_TOO_MANY_EMBED_HOPS, S_TOO_MANY_LINK_HOPS, S_TOO_MANY_RETRIES, S_UNATTEMPTED, S_UNFETCHABLE_URI, S_UNQUEUEABLE

Constructor Summary

FetchDNS(java.lang.String name)
Create a new instance of FetchDNS.

Method Summary

protected byte[]	getDNSRecord(long fetchStart, org.xbill.DNS.Record[] rrecordSet)
protected org.xbill.DNS.ARecord	getFirstARecord(org.xbill.DNS.Record[] rrecordSet)
protected void	innerProcess(CrawlURI curi) Classes subclassing this one should override this method to perform their custom actions on the CrawlURI.
protected boolean	isQuadAddress(CrawlURI curi, java.lang.String dnsName, CrawlHost targetHost)
protected void	recordDNS(CrawlURI curi, org.xbill.DNS.Record[] rrecordSet)
protected void	setUnresolvable(CrawlURI curi, CrawlHost host)
protected void	storeDNSRecord(CrawlURI curi, java.lang.String dnsName, CrawlHost targetHost, org.xbill.DNS.Record[] rrecordSet)

Methods inherited from class org.archive.crawler.framework.Processor

checkForInterrupt, finalTasks, getController, getDecideRule, getDefaultNextProcessor, initialTasks, innerRejectProcess, isContentToProcess, isEnabled, isExpectedMimeType, isHttpTransactionContentToProcess, kickUpdate, process, report, rulesAccept, rulesAccept, setDefaultNextProcessor, spawn

Methods inherited from class org.archive.crawler.settings.ModuleType

addElement, listUsedFiles

Methods inherited from class org.archive.crawler.settings.ComplexType

addElementToDefinition, checkValue, earlyInitialize, getAbsoluteName, getAttribute, getAttribute, getAttribute, getAttributeInfo, getAttributeInfo, getAttributeInfoIterator, getAttributes, getDataContainerRecursive, getDataContainerRecursive, getDefaultValue, getDescription, getElementFromDefinition, getLegalValues, getLocalAttribute, getMBeanInfo, getMBeanInfo, getParent, getPreservedFields, getSettingsHandler, getUncheckedAttribute, getValue, globalSettings, invoke, isInitialized, isOverridden, iterator, removeElementFromDefinition, setAsOrder, setAttribute, setAttribute, setAttributes, setDescription, setPreservedFields, toString, unsetAttribute

Methods inherited from class org.archive.crawler.settings.Type

addConstraint, equals, getConstraints, getLegalValueType, isExpertSetting, isOverrideable, isTransient, setExpertSetting, setLegalValueType, setOverrideable, setTransient

Methods inherited from class javax.management.Attribute

getName, hashCode

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

serverInetAddr

protected java.net.InetAddress serverInetAddr

Constructor Detail

FetchDNS

public FetchDNS(java.lang.String name)

Create a new instance of FetchDNS.

Parameters:

name - the name of this attribute.

Method Detail

innerProcess

protected void innerProcess(CrawlURI curi)

Description copied from class: Processor

Classes subclassing this one should override this method to perform their custom actions on the CrawlURI.

Overrides:

innerProcess in class Processor

Parameters:

curi - The CrawlURI being processed.

storeDNSRecord

protected void storeDNSRecord(CrawlURI curi,
                              java.lang.String dnsName,
                              CrawlHost targetHost,
                              org.xbill.DNS.Record[] rrecordSet)

isQuadAddress

protected boolean isQuadAddress(CrawlURI curi,
                                java.lang.String dnsName,
                                CrawlHost targetHost)

recordDNS

protected void recordDNS(CrawlURI curi,
                         org.xbill.DNS.Record[] rrecordSet)
                  throws java.io.IOException

Throws:

java.io.IOException

getDNSRecord

protected byte[] getDNSRecord(long fetchStart,
                              org.xbill.DNS.Record[] rrecordSet)
                       throws java.io.IOException

Throws:

java.io.IOException

setUnresolvable

protected void setUnresolvable(CrawlURI curi,
                               CrawlHost host)

getFirstARecord

protected org.xbill.DNS.ARecord getFirstARecord(org.xbill.DNS.Record[] rrecordSet)