Lesson One: Wave Goodbye To Crawling Issues

Crawling tools such as Screaming Frog and Xenu can be problematic at times.

They should render your work easier and faster, but in reality you really have to be focussed to achieve the results that you are looking for. This is important because the crawling goes to the heart of the entire SEO process.

Know-How

A deeper knowledge of crawling tools, combined with some tricks, could help you have a better understanding of the issues that you are facing daily. Specifically, my advice is to consider some tips that could help the crawling run smoothly, particularly when it comes to analysing a large website, which could be a very tedious process indeed.

Say you are performing the crawling of amazon.com, whose performance has been recently investigated by our team. Consider that 175 million URLs are indexed, and that is only for Google UK.

Making Sense Of Non-Sense

Before performing the crawling, also consider that the strength of certain tools ebbs when SEO experts do not use their own judgment to determine whether or not the output makes any sense at all. Before starting the crawling of a large website, ask yourself which part may be critical not only with regard to technical and on-page, but also with regard to the corporate and capital allocation strategy of the client. Talk to the client about this.

Our Initial Recommendations

  • Make sure you choose the right domain — corporate websites are ofter empty boxes
  • Save your work every time the crawling of up to 100,000 URLs has been performed

Moving on to PC settings, focus is on memory allocation.

The image below shows how much RAM should be allocated to the tool, according to Screaming Frog‘s online guide for Mac OS X.

Screaming Frog's GuideSOURCE: SCREAMING FROG

 

If you want this topic further with our team, please email us at info@hedgingbeta.com

Subscribe
Notify of

0 Comments
Inline Feedbacks
View all comments