Web scraping can look deceptively easy these days. There are numerous open-source libraries/frameworks, visual scraping tools, and data extraction tools that make it very easy to scrape data from a website.

Unbeknownst to many, there is a data revolution happening in finance.
In their never ending search for alpha hedge funds and investment banks are increasingly turning to new alternative sources of data to give them an informational edge over the market.

Throughout the history of the financial markets information has been power.

Over the last couple weeks, GDPR has brought data protection center stage. What was once a fringe concern for most businesses overnight became a burning problem that needed to be solved immediately.

It’s been another standout year for Scrapinghub and the scraping community at large. Together we crawled 79.1 billion pages (nearly double 2016), with over 103 billion scraped records; what a year!

We’re very excited to announce a new look for Zyte!

This is a guest post from the folks over at Intoli, one of the awesome companies providing Scrapy commercial support and longtime Scrapy fans.

It got very easy to do Machine Learning: you install an ML library like scikit-learn or xgboost, choose an estimator, feed it some training data, and get a model that can be used for predictions.

Up until now, your deployment process using Scrapy Cloud has probably been something like this: code and test your spiders locally, commit and push your changes to a GitHub repository, and finally deploy them to Scrapy Cloud using shub deploy.

We started 2016 with an eye on blowing 2015 out of the water. Mission accomplished.

One negative review can cost your business up to 22% of its prospects. This was one of the sobering findings in a study highlighted on Moz last year.

Computers are great at repetitive tasks. They don't get distracted, bored, or tired.