Today we are introducing a new feature called Spider activity graphs. These allow you to visualize quickly how your spiders are working, and it's a very useful tool for busy projects to find out which spiders are not working as expected.
This new graph shows you a snapshot of your project each day, displaying one dot per spider. The best way to illustrate how it works is to look at a particular example. This a snapshot of a single day in a busy project (each dot is a spider running that day):
That is just one day, from the last 30 days covered by the graph (which you can quickly traverse). Here is how the graph is read:
Once we learn how to read it, we can get a sense of the following things by quickly glancing over the graph:
As you can see, this graph concentrates a lot of information into a single place, which allows to quickly grasp an idea of how the spiders are working and how they compare against each other.
There are other useful features not mentioned here, like being able to track a specific spider (or group of them), changing how metrics are displayed (for example, mapping colors to errors instead of job counts), and playing an automated animation over time. Our customers can already enjoy this new report in the panel, by going to Reports -> Spiders activity.
These are some of the features you can expect from our platform, and it covers both the Scrapy Cloud and Autoscraping services. We believe visualization plays an important role when monitoring and keeping track of many spiders, even if you are lucky enough to have time to dive into detailed reports and lots of numbers (most of us aren't).