Dessa borde duga som robotar.
Kod:
$Agent1 = "Googlebot/2.1 (+http://www.google.com/bot.html)";
$Agent2 = "Googlebot/2.1 (+http://www.googlebot.com/bot.html)";
$Agent3 = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)";
$Agent4 = "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)";
$Agent5 = "msnbot/1.0 (+http://search.msn.com/msnbot.htm)";
Och googles media-bot om du kör adsense.
Kod:
Mediapartners-google/2.1 (+http://www.googlebot.com/bot.html)
Utöver det så en robots.txt i stil med
denna.
Thomas