Deepcrawl is the most advanced crawler on the market and is an essential part of any search marketer’s toolstack. With all of our advanced features, configurations and reporting, we know that you might need a helping hand getting started with our crawler.
To help ease you into crawling with Deepcrawl, we thought it would be helpful to give you some guidance on the various aspects that you will likely want to get to grips with.
So, what can we help you with?
1. How Can I Crawl a Website?
We’ve written a thorough guide on how to crawl a website with Deepcrawl which is a great place to start if you’re new to setting up crawls. In the rest of this walkthrough you’ll learn how to perform a technical site audit using Deepcrawl to understand your website’s technical health and performance in organic search.
If you’re interested in crawling multiple domains or subdomains then we have a separate guide which shows you how to configure this in Deepcrawl.
2. How Can I Deal With Crawl Errors or Credit Issues?
Are your crawls generating errors, or are you having issues getting a crawl to reach the required level of URLs? Here’s a guide on troubleshooting failed crawls.
3. How Can I Use Crawl Limits to Reduce Credit Usage?
Don’t forget to use crawl limits so you don’t waste crawl budget, especially on large sites or where your site has resources of low value. If you’re having issues with URL parameters or want to rewrite URLs on the fly for lookup services using APIs, we’ve got a guide for that too.
4. How Can I Change the Size and Depth of my Crawls?
Whether your site has several million URLs or only a few thousand, crawling the entirety of a website isn’t always the best plan of attack. Knowing when and how to restrict your crawls to specific parts of a website is a fine art that will help you use Deepcrawl more effectively, as well as using up fewer credits.
Our guide on limiting the size and depth of a crawl is a great introduction on how to crawl more efficiently with Deepcrawl.
Learn to Keep Your Crawls on a Tight Leash
5. How Can I Restrict a Crawl to Certain Pages?
There are some cases where you are only going to be interested in focusing your crawls on a specific area of a site. You might only want to crawl one country version of a large international site, for example.
If you need to learn about how to restrict crawls using URL filtering, page groupings or regex, we’ve got your back.
Give Me a Lesson in Crawl Restriction
6. How Can I Enrich My Crawls Using Regex?
Regex can be a love/hate thing… Once you truly understand it you will love it, especially with the extra power it gives you within Deepcrawl. From identifying thin content, to finding unique data about your products and copy, this post will detail how you can use custom extractions to get more out of your crawls.
Help Me Use Custom Extractions
7. How Can I Crawl a Test Site?
From a full migration to simple updates, code regression and plain bugs impacting SEO performance are still very common. By following our guide to crawling your live and test sites you can analyze and avoid any potential setbacks.
8. How Can I Crawl a Separate Mobile Site?
Separate mobile sites can be slightly more complex to crawl compared to other mobile configurations. If you have user-agent redirects that automatically take you to the correct version of the site, you’ll need to specify a separate mobile user agent in your crawl setup. Our guide on crawling mobile websites gives you all the information you need.
Teach Me How to Crawl a Separate Mobile Site
We hope that you find these guides helpful, but if you have any other questions then check out our FAQs or drop us a message. We’d love to hear from you!