Scaling static content delivery capacity?

This is a discussion on Scaling static content delivery capacity? within the Apache Web Server forums, part of the Web Server and Related Forums category; I have been running a HTTP mirror site for free software on a single server for the past 13 months ...


Go Back   Usenet Forums > Web Server and Related Forums > Apache Web Server

FAQ Members List Calendar Search Today's Posts Mark Forums Read
  #1 (permalink)  
Old 01-17-2007
colindermott@gmail.com
 
Posts: n/a
Default Scaling static content delivery capacity?

I have been running a HTTP mirror site for free software on a single
server for the past 13 months and it has now grown to the stage
where I need to scale up, but I can't actually scale this machine
any further (vertically).

My first instinct was to find another machine of similar specs, get
another RAID array and mirror my system entirely, then round-robin
DNS them. This is affordable but I'm not convinced it is the most
cost-effective solution, nor an efficient use of the resources. It
would also become increasingly difficult to manage with the addition
of each node.

Right now I deliver files directly to the client from direct-attached
RAID storage. My file set is large and growing and I have plenty of
room
to scale there by simply adding more RAID enclosures. What I am yet to
settle on is a method of scaling my delivery capacity as my audience
expands.

I am imagining I would place a cluster of reverse-proxy servers in
front
of my current server, effectively making it the "origin" server.

The proxy servers would each have a fraction of the disk space of my
current server, just enough to handle current downloads. Perhaps a
couple
of small 15KRPM disks in RAID-0.

Each new request should be forwarded to the proxy server bearing the
least load at the time of the request (using something like
mod_backhand
for Apache 1.3). The proxy will then fetch the file from the origin
server and deliver it to the client.

The reason I am holding off on deploying such an architecture is
because
I am not totally convinced by it. Also, mod_backhand/Wackamole is the
only software I have found that would enable this kind of setup, and
I'm
interested in what other methods are being used out there.

I realize that most of big corporations have moved towards CDNs, but
how
would somewhere like CNET have scaled their delivery capacity before
outsourcing it to Kontiki/Akamai?

If you can offer me any advice, feedback on my architecture, or topics
I
should read up on, I would really appreciate it.

Cheers
-colin

 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are Off
[IMG] code is Off
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT +1. The time now is 10:32 PM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO 3.0.0