# robots.txt
#
# This file serves to prevent the crawling and indexing of certain parts
# of the site by web crawlers. By telling web crawlers where to go and not to go on your site,
# you save bandwidth and server resources.
#
# This file will be ignored unless it is at the root of your host:
# Used:    http://example.com/robots.txt
# Ignored: http://example.com/site/robots.txt


User-agent: *
Allow: /
Disallow: /mdat/

Sitemap: https://data.census.gov/sitemap.xml