The getdomaindata.com crawler

How to identify it

Every request our crawler makes carries this User-Agent:

What it does

• Requests the homepage of publicly reachable sites, over standard web ports (TCP 80 and 443) only.
• Identifies itself on every request with the User-Agent above.
• Honors robots.txt — if your site disallows our crawler (User-agent getdomaindata, or *), we will not request its pages.
• Detects the technologies a site uses; page content is not retained.
• Visits any given site infrequently, at a low, polite request rate.

What it does not do

• No port scanning, vulnerability probing, brute-forcing, or exploitation of any kind.
• No logins, form submissions, or attempts to access non-public areas.
• No connections to anything other than public web servers on ports 80/443.

Opt out

The simplest way to opt out is your robots.txt: disallow the User-agent getdomaindata (or *) and our crawler will not request your pages.

You can also have a domain, hostname, or IP range permanently excluded — email [email protected] with the domains or network ranges you want removed. We add exclusions promptly and they persist across future crawls.

Network operators & abuse reports

If you operate a network and observed traffic from our crawler, please reach out to [email protected]. We honor exclusion requests, maintain a permanent do-not-contact list for reported sensor and honeypot ranges, and are happy to coordinate on attribution or whitelisting.