# Author: Tim Ju# Created: 11/25/98 # must use all lowercase # Last Updated: 07/05/00 8/19/02 7/16/2007 8/1/2008 12/30/2008 #-----Examples------------------------------------------------------------------------------ #-------------- ALL robots DENIED access to the whole site #User-agent: * #Disallow: / #-------------- ALL robots DENIED access to folder dallas #User-agent: * #Disallow: /DALLAS/ #-------------- robot webcrawler CAN access whole site #User-agent: webcrawler #Disallow: #-------------- robot webcrawler DENIED access to the whole site #User-agent: webcrawler #Disallow: / #------------- robot bspider can't access file abqaccnt.dat and folderS updates, tmp, & logs #User-agent: bspider #Disallow: /ABQACCNT.DAT #Disallow: /UPDATES/ #Disallow: /tmp #Disallow: /logs #Disallow: /_Forms #------------------------------------------------------------------------------------------- #------------Sub Webs---------- User-agent: * Disallow: /Cstmr/ Disallow: /invMain/ Disallow: /invstr/ Disallow: /Interim/ #------------FOLDERS----------- Disallow: /citrix/ Disallow: /citrix1/ Disallow: /DL_Wstr/ Disallow: /debugx/ Disallow: /receipts/ Disallow: /tsweststar/ Disallow: /TS_PC/ Disallow: /_incl/ Disallow: /maps/ Disallow: /nfuseclients/ Disallow: /nfuseicons/ Disallow: /script/ Disallow: /uploadprogress/ Disallow: /visits/ Disallow: /YearEnd/ Disallow: /java-ie/ Disallow: /java-netscape/ Disallow: /report/ Disallow: /script/ Disallow: /TJ_*/ Disallow: /*test/ Disallow: /UploadProgress/ Disallow: /Upload/ Disallow: /UserInfo/ #------------ALL FOLDERS BEGINNING WITH Z OR z (don't know if case sensitive TJ)----------- Disallow: /z*/ Disallow: /Z*/ #-----Root Files----- Disallow: /browtype.asp Disallow: /global.asa Disallow: /junk.txt