Question

Just ran a Screaming Frog crawl on a client site that was developed by someone else. The site is WP and uses the All In One Event Calendar plugin. The plugin is generating 10's of thousands of unique URLs which are being picked up by Screaming Frog. This is making my crawl audit a much tougher task than it should be. Thoughts? Have the developer simply use an embedded calendar instead or some other solution?  

Answer 1

In your Configuration before running the crawl, use exclude feature to basically exclude set of parameters or a URL syntax that this calendar is generating. they should theoretically have a common syntax or directory of sort. so that's what you can do to make your crawl cleaner. Hope this helps!  

Comments

@consolata35 yeah I ended up doing that for our purposes. Was wondering if it would affect other crawlers though - Google specifically

---

Use your standard techniques to block

Answer 2

Edasy as that above