skwpspace yan pritzker’s home on the web

webcrawler bot detection

  def self.bot_agent_list
    [ "panscient", "larbin", "dummy", "Teoma", "alexa",
      "froogle", "inktomi", "looksmart", "URL_Spider_SQL",
      "Firefly", "NationalDirectory", "Ask Jeeves", "TECNOSEEK",
      "InfoSeek", "WebFindBot", "crawler", "girafobot", "Scooter",
      "Baidu", "bot", "Google", "SiteUptime", "Slurp",
      "WordPress", "ZIBB", "ZyBorg", "msnbot", "check_http",
      "libwww-perl", "lwp-trivial", "wget", "curl", "SimplePie",
      "Python", "Feed", "HTTPClient", "Tumblr", "Spider", "sanszbot"]
  end

Full source at http://pastie.org/191922

robots don’t smell.

but they can sure cause a stink

(cymbal crash)