The new crawler is deeply integrated throughout the Dragon Metrics platform in the same way as before, powering tools such as the Site Auditor, Site Explorer, URL X-Ray, and more. But now, the technology behind it runs off of Chromium (the open source foundational core of Chrome, Edge, and other browsers).
This means our crawler will see your site as close to site visitors and search engines like Google and Bing as possible.
Recently, both Google and Bing have done just that — crawling the entire web with the latest version (often referred to as “evergreen”) Chromium. This meant that for the first time, the biggest search engines in the world now view your site’s content essentially like a normal visitor.
That’s exactly what today’s announcement is about. With our new crawler, you can be much more confident that content visible to Google and Bing will be visible in your Dragon Metrics site crawls as well.
Our new crawler has been in development and invite-only beta for some time. While we feel it’s been reasonably stable in our testing, we’re still leaving the beta label on it for now due to the unpredictable nature of the web as we continue to round of edge cases.
As with any beta product, your feedback along the way is vital to its development. Please share any questions, comments, or bugs you may encounter with our Customer Success team to help make our crawler the best it can possibly be.
Want more information on exactly how our JS crawler works? get in touch with our Customer Success team who can provide our work-in-progress documentation.