Meet Our Enhanced Crawler

Andrew Levey

On 13th September 2022 • 10 min read

You may have seen the recent news about the enhancements we’ve made to our crawler! These exciting improvements not only make our crawler the fastest on the market today, but also brings Deepcrawl even closer to Google’s own behaviors, and allows us to increase the breadth and flexibility of data we will be able to extract from pages. Let’s take a closer look. 

Increased Crawl Speeds

By using the latest serverless technology, we’ve been able to increase crawl rates to the fastest possible speeds. During our field testing, we achieved speeds of 450 URLs per second for unrendered crawls and 350 URLs per second for rendered crawls. By completing crawls in a faster time, data is made available for processing faster, meaning you get the insights you need as quickly as possible. 

Important Considerations

While the enhancements we’ve made to the crawler mean that it can effectively crawl as fast as possible, there are some important considerations that need to be taken into account. Crawling at too fast a rate means your server might not be able to keep up, and your site will experience performance issues. To avoid this, Deepcrawl sets a maximum crawl speed for your account, which we can update for you to enable faster crawling speeds. However, there are some important considerations to keep in mind:


Closer Alignment to Google Behaviors 

We’re taking full advantage of Google’s industry-leading rendering and parsing engine to align as closely to Google behaviors as possible. This has some implications for customers once the enhanced crawler is enabled:


Data Improvements

We’ve also made data improvements to reflect Google practices and enhance the insights you get from Deepcrawl, including:


Other Improvements

In addition, we’ve made a few other improvements, including:

Once the enhanced crawler is enabled, there are also a few final things to be aware of:


Choose a better way to grow

With tools that will help you realize your website’s true potential, and support to help you get there, growing your enterprise business online has never been so simple.

Book a Demo