Logo-with-dark-gray
RSS Blog

Hulu go boom…

October 21st, 2008 by Eric Feng

As many of you saw, Hulu was pretty much wiped off the face of the Internet tonight from about 6:45pm – 7:45pm PST. During that period, all *.hulu.com would not resolve and emails sent to any @hulu.com email address bounced.

So what happened? Well there’s been some interesting theories about this that our users tweeted about during our outage including:

  1. DanStasiewski: Hulu appears to be down. Should I get on the roof and wiggle the antenna?
  2. JoeAnderson: Maybe bringing down Hulu.com will finally get me into the Evil League of Evil.
  3. MattMusgrave: With Hulu down, productivity sky rockets.
  4. benfortney: I think hulu said something bad about the chinese gov’t.
  5. (and another one) benfortney: I blame the Republicans for taking out Hulu.
  6. demosthe: Stale DNS cache maybe?

If you guessed #6, then you’d be correct. Around 6:45pm, we accidentally deployed an incorrect DNS record and this change was cached out to public DNS servers causing all *.hulu.com domains (including our email server, SVN depot, the whole kitchen sink) to resolve into the ether.

With the help of our guardian angel (thanks Phil!), we fixed the DNS change but then had to wait helplessly for it to fully propagate which took a bit of time, during which we kind of felt like this:

Our sincere apologies to all the users this outage affected and hope things are back to normal for you. If not, drop me a line and we’ll get you back up and running too.

Thanks again for your patience and understanding.

Eric “Got a lot of explaining to do” Feng ()

Last comment: Oct 24th 2008 4 Comments
  • matthew says:

    There does still seem to be a problem with Hulu defaulting to the slowest streaming bitrates regardless of actual broadband speed. This was not a problem before the crash. Any ideas?

  • Joe says:

    I agree that the site seems to be lagging since it came back up last night (not going elsewhere though, that’s just silly).

  • Mark says:

    Seems that the service is horrible at the moment with its stuttering. Hopefully the DNS problems will get resolved. In the meantime… I’m going elsewhere.

  • Naresh says:

    I forgive you.

    Thanks for the technical explanation! I love this stuff.

*
*