User-agent: * User-agent: RavenCrawler User-agent: rogerbot User-agent: dotbot User-agent: SemrushBot User-agent: SemrushBot-SA User-agent: PowerMapper User-agent: Swiftbot User-agent: Twitterbot Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Allow: / Disallow: /magazine/ Disallow: /?s=* Sitemap: https://www.newstatesman.com/sitemap.xml Sitemap: https://www.newstatesman.com/news-sitemap.xml User-agent: Yandex Disallow: / User-agent: YandexBot Disallow: /