Should you BUY your WEB DATA, or Write Code?

So, as a developer, I'm always an advocate of building stuff yourself. It gives you full control. It's very customizable. You can manage everything, and it's all done by you, so you know exactly what's going on. But, I got to understand that this just isn't feasible and isn't the reality for a lot of businesses out there that don't have their own big development teams or in-house developers, or maybe they just have a few developers working on their core product. So, in this video, I want to talk about the pros and cons and the buy versus build and also propose a third solution which I think will work for people in that use case where they have some in-house developers that don't have a lot of time. So, we'll come to that a little bit later. So, why shouldn't you build your own web scrapers and your own spiders and your own web scraping system? Well, first of all, as I said, if you do that, it's very customizable. You can tailor it to exactly what you need and it will work exactly how you want to. And if you have some developers in place, your initial cost should be very low. Just a little bit of time for them to build up an MVP and get something working for you. But the downsides to this approach are, you know, much bigger, especially when you're not equipped to do so. There's obviously extremely resource inensive. Building a single scraper is pretty easy. Building a 100 is not so much. And then managing them and and keeping up with them with the maintenance is much different. It's a much different piece. and it's much more difficult. So then you got to consider what happens if the websites that you need data from have active bot antibbot measures. There's antibban. There's problems. There's things that are going to get in the way from you getting that data. That's a whole another level of extreme complexity that you're going to have to manage. And then there's what happens when websites change. Maybe some structure changes. Maybe the website completely revamps itself or maybe you need to add more sites. How you add and manage those is going to impact the amount of resource you're going to need to actually effectively manage this web scraping system. And this is outside of things like the extra infrastructure costs that you'll need. Where are you going to host these? Where you going to run these? Assuming that maybe you have this in house already, it's still another layer that your DevOp teams are going to have to manage. Then we look at things like the hidden costs. When you're scraping data at scale, you need IPs to rotate through so you can actually have a chance at getting the amount of data that you need. Now how much that is going to depend on how well you manage that proxy infrastructure and how you do that is going to be dependent on the skill and the talent and the resources available that you have. So again this could be low cost, high cost, medium cost. It's all a bit unknown. And then there's the developer cost. And take it from me, I've been writing web scrapers for a very long time. The initial setup does seem quite easy. But once you start adding and compounding, it can make it very difficult to keep going and keep managing those scrapers. Keeping them alive and well and running is a full-time job at the very minimum for a couple of devs depending on the amount of spiders that you have. And then we can look into the like the indirect costs. So what happens if something goes wrong in your scraping system and then you don't have the data that you needed that you relied on for your quarterly meeting for your, you know, your marketing insights. When that doesn't happen, all of a sudden your whole team's behind and your business starts to lose out. And then there's the other thing to look at, which is verifying the data that you're getting. That data quality and data integrity is extremely important. You could spend all your time and money building a scraping system, but if what comes out the other end is not useful to anybody or is not consistent and reliable or takes someone hours, if not days to clean, you're going to have to account for that into the actual business use case of building your own scrapers. So, let's look at the advantages of buying data as in, and when I say that, I mean buying a service, paying someone to do it for you, like what we do here at Zeit. So there's obviously the benefit is it's fully managed. You come to us with a project scope, scope it out for you, build it, get it done for you, and then you literally just receive the data. The data is obviously going to be on time and it's guaranteed. It goes through all of our QA, all of our uh our checks to make sure that everything is what you want. We agree a schema with you and that then is what you will receive every week or however often you need it to. This allows you to focus much more on your own business, your own core product, and use the data that we can provide you to build that and enhance the service you provide to your customers. So, you don't have to worry about it. You don't have to worry about the maintenance. You don't need to worry about what happens when new antimand technology comes out, when websites change and all of a sudden the data schema that you've been looking at is all over the place. That's all handled for you. And that's just a massive time-saving thing. Now, obviously, this all comes at an extra cost, but what we've got here at Zite is we actually can really minimize that initial setup cost for you. Where it used to be quite expensive and perhaps prohibitive to some people, we have our own AI technology that allows us to build upon specifically already generated templates for common schema types that allows us to in some cases even have zero setup cost for you. So, how do you know which is right for you? Well, I boiled it down to a few core questions that you need to ask yourself when you're starting this web scraping project up is the first one is obviously do you have an in-house development team and what how much free time and resource do they have available to uh put into this project now and maybe you have some extra resource and extra time but you're concerned about scaling up now as I said at the beginning of this video when you start to scale you start to notice a lot more problems that you never noticed before simply because of the amount of requests that's going through it becomes a much more complicated beast. You'll need to educate your developers on this. They'll need to understand how to not only build and maintain but also debug and overcome any of the anti-ban or specific web scraping challenges that we've obviously been encountering and doing for 14 years. What data points you also need is a very important question to ask. If you're looking at very specific data that is, you know, not structured and it's an extremely obscure way, then perhaps having that inhouse might benefit you because the extra effort that uh you go through to get that would be would be rewarded to you because that has to happen anyway. We have set schemas that we use that cover almost all of the standard data types that we can utilize for you and then build upon those which obviously as I said keeps that setup cost down and also then how many websites you need to scrape. If you're just pulling a few pieces of data from a site once a week then perhaps that's going to be manageable for you. When but when it comes to lots of data over multiple sites which is generally what people want because they want to have a full view of what's happening in their industry. this becomes a whole different beast that becomes you know much more difficult to manage. The last two things that I would ask is you know how quickly do you need this solution up and running and then you know how much control do you need. I think it boils down to a lot of like if you can do this inhouse or if you have an in-house development team that are willing to take this on but that's kind of like the most dangerous part because you need to understand the full scope of the project. I think a lot of cases people will start something um and this is not necessarily specific to web scraping but you know it is for me because of the industry that I work in. People will start it thinking that it's going to be easy and you know to get a few bits of data from one site generally is they don't understand you know how to manage it when the website starts to ban you when your IPs get blocked when you start to run into issues whether you need to run and manage a load of browsers to get the data that you need. All of this comes with extra overhead and extra costs and essentially is going to sway your decision either way. Now, at the beginning of this video, I did mention that there was a third approach and this is what I like to refer to as more like a hybrid approach where you utilize our services uh but keep the code as much in-house as possible. So obviously I'm talking about the ZI API and what this will do for you is it will provide you with a reliable API endpoint that you can query with lots of different parameters uh and you can send a URL to and we'll return the data as you need it to either through one of our predefined schemas at a slightly extra cost or just the raw HTML back to you. You'll also have the choice whether you want to render it through a browser. But the benefits of this is that we'll handle all of the hard stuff for you through our API. You don't have to worry about scaling. You won't have to worry about bands or sessions or how to, you know, manage the browsers. All of that's done automatically. What that means is you can have a small development team that are focused on your core product and just query our API for the data that you need and then put that back into your system. And I think this is going to benefit a lot of the smaller to medium-siz businesses that have a very specific data need but with a development team but don't want to go the full data as a service route and want to utilize some of their own team's response some of their own team resources that way. So if you're considering any of these options I hope this has helped you out in some way. Obviously this is our business and this is what we do. So if you are interested in working with us in any way, whether you want to find out more about our Zite API for that hybrid approach or whether you just want the data delivered to you as you need it as you want it, then go ahead and check out the links in the description or Google for Zite and we'll be able to help you from there. So thanks for watching. and I'll see you in the next one.