This is a discussion on Scaling static content delivery capacity? within the Apache Web Server forums, part of the Web Server and Related Forums category; I have been running a HTTP mirror site for free software on a single server for the past 13 months ...
|
|||||||
| FAQ | Members List | Calendar | Search | Today's Posts | Mark Forums Read |
|
|||
|
I have been running a HTTP mirror site for free software on a single
server for the past 13 months and it has now grown to the stage where I need to scale up, but I can't actually scale this machine any further (vertically). My first instinct was to find another machine of similar specs, get another RAID array and mirror my system entirely, then round-robin DNS them. This is affordable but I'm not convinced it is the most cost-effective solution, nor an efficient use of the resources. It would also become increasingly difficult to manage with the addition of each node. Right now I deliver files directly to the client from direct-attached RAID storage. My file set is large and growing and I have plenty of room to scale there by simply adding more RAID enclosures. What I am yet to settle on is a method of scaling my delivery capacity as my audience expands. I am imagining I would place a cluster of reverse-proxy servers in front of my current server, effectively making it the "origin" server. The proxy servers would each have a fraction of the disk space of my current server, just enough to handle current downloads. Perhaps a couple of small 15KRPM disks in RAID-0. Each new request should be forwarded to the proxy server bearing the least load at the time of the request (using something like mod_backhand for Apache 1.3). The proxy will then fetch the file from the origin server and deliver it to the client. The reason I am holding off on deploying such an architecture is because I am not totally convinced by it. Also, mod_backhand/Wackamole is the only software I have found that would enable this kind of setup, and I'm interested in what other methods are being used out there. I realize that most of big corporations have moved towards CDNs, but how would somewhere like CNET have scaled their delivery capacity before outsourcing it to Kontiki/Akamai? If you can offer me any advice, feedback on my architecture, or topics I should read up on, I would really appreciate it. Cheers -colin |