RSS growing pains. While I've been buried in other CTO duties, I've seen a lot of useful feedback on my latest column, “RSS Growing Pains.” There's even a healthy discussion going on over at Slashdot, including entertaining (at least to me) debates about InfoWorld IT departments past and present. Rob Malda
noted in his post: “We've seen similiar problems over the years. RSS
(or as it should be called, 'Speedfeed') is such a useful thing, it's
unfortunate that it's ultimately just very stupid.”
First, a quick quote from my column:
Our hourly RSS surge has all the characteristics of a distributed DoS
attack, and although the requests are legitimate and small, the sheer
number of requests in that short time period creates some aggravating
scaling issues. These issues aren’t enough to make me want to abandon
RSS (in fact, I’ll keep pushing it), but its workings can create
operational annoyances. If RSS is going to go from fairly big to
absolutely huge, we’re all going to need to do a little more work on
the plumbing.
I pointed out this quote because the “operational annoyance” (not quite
the level of a “problem”) had less to do with bandwidth (as many folks
who responded wrongly assumed) and more to do with Apache configuration
(it's always worthwhile to re-read the “Apache Performance Tuning” page). I would advise folks out there to read this page
and make sure that your Apache settings support the level of
simultaneous connections that a bunch of newsreaders waking up at the
same time will create. In our case, we had fairly conservative settings
for MaxClients and ServerLimit. Read the Apache docs for more on these two settings and you'll know where I'm headed:
So, basically, the “operational annoyance” was the
thousands of simultaneous connections at the same time that eventually
overwhelmed our MaxClients
settting and caused requests to queue at the top of the hour, slowing
down our response time but not killing our servers our draining our
bandwidth. It's the kind of thing you don't notice until you hit the
wall, and as our RSS traffic grew steadily, we didn't see the wall
approaching. We had a higher MaxClients setting than
recommended anyway, but we raised the number further to something more
respectable. Believe me, though — when RSS gets really popular at your
site, the characteristics of tons of incoming connections all at once
on a regular basis will seem unusual relative to “regular” web traffic.
This is not the kind of news-driven surge that I saw during big
breaking news events during my days at CNN.com (and even Salon.com to a
lesser degree) — it really does look like a DDoS attack, even when you
know it isn't.
In the final analysis, there are a number of ways to deal with RSS
surges, and none of them are rocket science, but no one should assume
they will just take care of themselves and it's clear from what I'm
seeing that more people need to know about the solutions. Not every IT
group will roll with the RSS punches, and the more conservative ones
might just cut RSS off at the knees even if it's a minor hassle on top
of everything else. One of the wonders of weblogs (and the web in
general) though is that some poor soul will face the same annoyance we
had and hopefully will find this post and fix it quickly.