How can I ignore traffic from bots and crawlers?

Occasionally, your website may be visited by bots and crawlers who are not real people. Some bots, such as Googlebot or Bingbot, scan the content of your website to find information for the search results. Bots from Facebook, enter your website to capture the page title, description, and preview of the page when somebody pastes the link to your website on Facebook. Other bots, visit your site to scan the technical environment.

By default, we exclude traffic coming from the most popular bots and crawlers but you may run into the ones that are still being tracked. In that case, you can deliberately ignore bots and crawlers from tracking.

For a site or app

To ignore traffic from bots and crawlers, follow these steps:

Available from version 16.0.0. For versions below 16.0.0, go to Menu > Administration > Websites & apps > Settings > Exclude browsers.

1. Go to Menu > Administration.
2. Navigate to Sites & Apps.
3. On the left, select the site or app you want to work with.
4. Navigate to Data collection > Filters.
5. In Add crawlers, type a user agent name or full user agent string to define crawlers.
   

   

   Tip: See how to define crawlers.

For the whole account

Available from version 16.0.0. For versions below 16.0.0, go to Menu > Administration > Platform > Global website settings > Exclude browsers.

1. Go to Menu > Administration.
2. Navigate to Settings.
3. On the left, click Global site & app settings.
4. Navigate to Data collection > Filters.
5. In Add crawlers, type a user agent name or full user agent string to define crawlers.
   

   

   Tip: See how to define crawlers.

How to define crawlers

If you observe unusual heavy traffic on your site coming from the same source, it could be a crawler. To find a user agent for that crawler, follow these steps:

1. Go to Menu > Analytics.
2. Navigate to Settings.
3. On the left, click Tracker debugger.
4. Look at the session that you’re sure is a crawler’s session.
5. Click </> to see a raw request.
   

   

6. Find user agent details. Example:

   [
     "user-agent",
     "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15",
   ]