Package 

Interface RobotRules

  • 
public interface RobotRules

    This class holds the rules which were parsed from a robots.txt file, and can test paths against those rules.

    • Method Summary

      Modifier and Type Method Description
      abstract Boolean isAllowed(URL url) Returns false if the robots.txt file prohibits us from accessing the given url, or true otherwise.
      abstract Long getExpireTime() Get expire time
      abstract Long getCrawlDelay() Get Crawl-Delay, in milliseconds.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Method Detail

      • isAllowed

         abstract Boolean isAllowed(URL url)

        Returns false if the robots.txt file prohibits us from accessing the given url, or true otherwise.

      • getCrawlDelay

         abstract Long getCrawlDelay()

        Get Crawl-Delay, in milliseconds. This returns -1 if not set.