KrawlerBot is an automated web crawler operated by the Krawler project. Its purpose is to discover and analyze publicly accessible .dk, .se, and .de domain websites for research and informational purposes on krawler.app.
If you have questions, concerns, or requests related to KrawlerBot, please contact us at contact@krawler.app.
KrawlerBot makes good faith efforts to respect all robots.txt directives and avoids crawling sensitive endpoints such as login or admin pages. However, due to the decentralized nature of the web, mistakes can happen. If you observe unwanted crawling behavior, we will promptly investigate and resolve the issue.
KrawlerBot adheres to applicable laws including the General Data Protection Regulation (GDPR) within the European Union. No personally identifiable information (PII) is intentionally collected or stored. The crawler limits its data to publicly available metadata such as titles, links, structured data, and server headers.
If you wish to block KrawlerBot from accessing your site, add the following lines to your robots.txt file:
User-agent: KrawlerBot Disallow: /
This will instruct our crawler to avoid indexing any pages on your domain.