So I'm looking at starting a little project. It is very data intensive as it involves scraping a lot of websites, downloading a bunch of csv's off SSL sites requiring client side certs, scraping and parsing various kml files and storing all of this data into a database.

There will be some data that is inserted just once, but majority of the data will be scraped frequently. Some change as often as every 5 seconds, majority change every 5 minutes.

I then have a few reports and models that will run based on this data.

My questions are:

1) Any recommendations for webhosts that I can use for this
2) Any recommendations for which framework/language/database that would be ideal for this
3) Anything you think I should know heading in to this

Keep in mind the volume of data isn't small. Initial load will include in excess of half a billion rows of data, with over half a mill rows added daily.

Thanks!

Good Luck EDge


YESSSS

good luck

I'm just a Sr CS student but....the fluff are you trying to do?

Good Luck EDge

EDge, wtf are you trying to o here?

Thats's fluffloads of data man, and it ill eat HD-space...

A projct this big I'd run on my own boxes, set up a machine as database server, another one as webserver.

You really want to rent hosting for this? It ain't gonna be cheap, thats for sure.

When it comes to famework/database it's impossible to say with this little info

Your best bet would be to set up your own server to handle the data load. Either Oracle or SQL should manage it easily.

Now, as for scraping data, that's something outside my area of expertise. I would figure you would have a client machine exposed to the internet to gather the data and pass that over to the Server on your own internal network. So, there should also be some kind of firewall there.

The current setup that is working is with a separate SQL Server box, and an app server. The high frequency data is scraped using c# windows processes and the not as frequent data is scraped using c# console apps launched with windows task scheduler.

I would like to migrate this all to a remote host though and was thinking I could use mysql as the database, and python scripts for the scrapes executed by crontab. I am not all too familiar with any of this technology, but I figure basic data scrapes shouldn't require any real advanced programming in Python. The only thing I will need to become familiar with is db tuning the mysql database since it will have so much data.

Also, not sure how the 5 second scrapes will work with crontab, or if there's a better approach to take for those.

Good Luck EDge

Edge, it would really depend on the data type / content type you are trying to scrape and where it is stored ? You could run php scrappers which will store directly to MySQL. Again it's very little info, with that amount of data and frequency you'd want to have seperate servers running ssd and on the same network for storing data to the MySQL Db.

Also secondly is the data being overwritten or is it adding as new data ?

data will only be inserted. majority of the tables will have 3 columns. Datetime, int, decimal

primary key will be the datetime int combo

I'm looking at Dreamhost and hostgator right now. Either the shared hosting on either provider since they offer unlimited disk space and bandwidth, or the VPS on dreamhost since it allows unlimited bandwidth/diskspace.

Anyone have any experience with either provider, or see any issues with their plans vs what I'm planning to do?

The current servers both have 24 core amd opteron processors with 16gb of ram and raid harddrives. I know performance wise it will be a significant step down with these hosting plans, but I won't be doing nearly as much computing or queries off the webhost.

dreamhost has been pretty damn solid for me. havent heard of hostgator.

Speeds pretty good with dreamhost azz kikr? How is their support team? Do you use them from apps, or plain html sites?

I've never heard of Hostgator myself.

My company uses Dreamhost and we haven't had a problem for 2 years. Used for our website(s), and also as a DNS redirection to our actual game servers (Rackspace).

You can also consider Rackspace for database. They have price plans up to 150 GB (of db space), or alternate mounted drives (1 TB) if you really need that much more db space.

I think Amazon is a lot cheaper though.

So looking over one of the tables for 2012, there are about 270M rows of data using 17GB of space in MS Sql Server. Based on that, I am starting to think I might be able to get away without an "unlimited" plan.

Xin, why is it that you guys are using Rackspace for production instead of dreamhost? Better performance? Reliability? Speed? Or do they support products that dreamhost doesnt?

edge: i haven't had need to deal with their support, so i don't really know on that one.
the speeds that i've encountered are pretty good, but i'm on a shared host, not a private host, so i don't know how that affects things, and the site i have hosted isn't a high throughput site.
i'm running a mysql db with a php website.

Thanks, think I'll just sign up for a dreamhost account.

Anyone have an idea for how I can get a scrape working that checks a site every few seconds for changes? Right now I have windows system processes take care of that, but need an alternative that will work on these linux servers. Having a cronjob that executes every few seconds doesn't seem optimal to me.

dreamhost has an online interface to set up cron jobs. they also allow you to create shell accounts if you prefer command line

beyond that, i don't know much about page scraping

You should probably reconsider using any shared hosting for this kind of thing given the nature of how those sites/servers are managed. They host is apt to to stop you from crawling so make sure to check the TOS before you pay any money. I run into issues putting content management systems onto budget shared plans (something like EE would never go on there, despite falling well under what should be required. If you're going to go with a provider like Dreamhost, Hostgator or Siteground (my hosting recommendation at the moment) then you're likely to need your own VPS, which will start at like $50/month and go up from there. But with that you have the autonomy and resources to actually set up your scraper.

As others said, scraping is incredibly expensive in terms of space, bandwidth and CPU operations and unless you start going to some place like Amazon or Google that allow you to create "use what you need" kind of setups you're likely going to run into major performance issues on any budget host. I've done a lot of scraping in a bootstrap sort of way (research) and I ended up building my own server to avoid having to go down that road with no budget.

In terms of writing the scraper.... that's a broad question :p
But here is some advice:
- Stay away from anything Microsoft
- MySQL will scale fine
- Consider making the system distributed (or at least distributable for scalability)
- I'd offer you language suggestions, but it's more about your comfort. EE does process handling (including running a processes constantly) via PHP but there's no shortage of ways to do it.
- Consider asynchronously loaded content on crawled pages and consider crawling via a headless browser with JS so you get accurate content
- What are you actually trying to crawl?

Good Luck, EDge.

How many sites do you plan to be checking every few seconds? You're going to hit a bottleneck fast...

I was planning on using the shared unlimited hosts, or the dreamhost VPS which is unlimited disk/bandwidth, to begin with just so I can gauge what kind of numbers I'll be looking at. After everything is up and running I should have enough usage data to look for other options that will provide better performance but allow for enough bandwidth/disk space.

In terms of making the system distributed, the current setup has a different console app/windows process for each logically different scrape.

I have worked with PHP, Java, C/C++, C#, VB, in the past but was planning to go with Python due to some libraries available for it which I plan to utilize for some of the modelling. However, if a specific language will be more efficient than another, I'm okay having the scrapes in one language and the models in another.

There will be maybe a dozen pages that are of the high frequency type; but if it does become a significant issue, I can just run something on one of my boxes at home to handle those. I was just hoping to have everything on a hosted server for security and accessibility.

Here's an example of something that I scrape:
www dot spp dot org/XML/LIP-Pricing dot xml

This is not one of the frequently updated sources though, this will only be revised every five minutes. And yes, I did replace . with dot lol. Just don't need this thread showing up on a google search result if a competitor searches the same url.

Good Luck EDge!