Mike Rynchek
Spyder Trap Online Marketing

Robots.txt Files

• Tells the Search Engine what to/not to crawl (pure data)

- Java Files
- Flash (swf)
- CSS

• Keep certain pages from showing in SERPs
• Add to root level of site

- Google Webmaster account

• Declare xml sitemap

Sample Robots.txt File

Stay tuned to the Spyder Trap Online Marketing Quick Tips, as next week we will discuss xml sitemaps.