In order for search engines to find your content, they need access. The Robots file is a text document that tells the robots which pages are ok and not allowed. It also includes an XML sitemap of all other resources on the domain so you don't have to manually tell them where everything is located!