robots.txt

Web Crawlers

• Googlebot by Google
• Bingbot by Microsoft
• Applebot by Apple
• OpenAI
• Anthrophic