Comments on: Amazon’s S3 Web Service, our #1 cause of failure http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/ Fri, 21 Nov 2008 12:03:31 +0000 http://wordpress.org/?v=2.3.3 By: codediva http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-399 codediva Fri, 25 Jul 2008 02:48:44 +0000 http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-399 I understand your frustration, however, anytime you used hosted services (even those from a trusted provider) you should plan on implementing some redundancy. I know this can be costly for small orgs, but it could have minimized the impact of this occurance. I understand your frustration, however, anytime you used hosted services (even those from a trusted provider) you should plan on implementing some redundancy. I know this can be costly for small orgs, but it could have minimized the impact of this occurance.

]]> By: LusciousPainGeek http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-395 LusciousPainGeek Thu, 24 Jul 2008 08:48:32 +0000 http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-395 Isn't this one of the risks you take in exchange for the convenience of using a centralized system like S3? Unfortunate, but bound to happen and something you need to look into when performing risk assessment. Isn’t this one of the risks you take in exchange for the convenience of using a centralized system like S3? Unfortunate, but bound to happen and something you need to look into when performing risk assessment.

]]> By: Ted Z http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-394 Ted Z Tue, 22 Jul 2008 19:15:35 +0000 http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-394 Look into Akamai, they'll do this correctly (disclaimer: I've worked for them). Or implement their system (quickly deteriorating DNS endpoint lookups) yourself, it's not hard if your scope is limited. You can just have S3 be your primary and when it fails, switch DNS over to your backup. Look into Akamai, they’ll do this correctly (disclaimer: I’ve worked
for them). Or implement their system (quickly deteriorating DNS
endpoint lookups) yourself, it’s not hard if your scope is limited.
You can just have S3 be your primary and when it fails, switch DNS
over to your backup.

]]> By: Ryan Haney http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-393 Ryan Haney Tue, 22 Jul 2008 15:14:43 +0000 http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-393 Have you considered using S3 as a backup provider with a local disk as cache? I can envision software that emulates a hard disk with a caching mechanism that copies to and from S3 in the background, as files are requested / uploaded. I do think though that you haven't load tested your own storage solution. I have a friend that works for Seagate and hard drives fail ALL THE TIME. It's just nature of the beast. Have you considered using S3 as a backup provider with a local disk as cache? I can envision software that emulates a hard disk with a caching mechanism that copies to and from S3 in the background, as files are requested / uploaded.

I do think though that you haven’t load tested your own storage solution. I have a friend that works for Seagate and hard drives fail ALL THE TIME. It’s just nature of the beast.

]]> By: Joe http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-392 Joe Tue, 22 Jul 2008 03:05:39 +0000 http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-392 Get a grip !!! It's not like you're running a stock brokerage system on Wall St ! My experience is that it's not hosting computers that are the reliability issue, but the ISP who provides the connection. Try getting any level of service guarantee is next to impossible. When I build "five nines" systems for clients, we end up putting data centers in multiple cities using multiple carrier connections. It costs a heck of a lot more than what Amazon charge for S3. It's like you paid for a cheap car, so don't whinge about the rattles. Get a grip !!! It’s not like you’re running a stock brokerage system on Wall St !

My experience is that it’s not hosting computers that are the reliability issue, but the ISP who provides the connection. Try getting any level of service guarantee is next to impossible. When I build “five nines” systems for clients, we end up putting data centers in multiple cities using multiple carrier connections. It costs a heck of a lot more than what Amazon charge for S3. It’s like you paid for a cheap car, so don’t whinge about the rattles.

]]> By: Striddy http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-391 Striddy Tue, 22 Jul 2008 02:06:19 +0000 http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-391 Should've been: Also, I don’t get the sense that FaceStat’s load is so heavy as to ... stress your other hosting service to the point of failure much. Should’ve been:
Also, I don’t get the sense that FaceStat’s load is so heavy as to … stress your other hosting service to the point of failure much.

]]> By: Striddy http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-390 Striddy Tue, 22 Jul 2008 02:04:56 +0000 http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-390 I agree that this deep and long of an outage is not acceptable, even for a service with tempered service level. However, your "this months FaceStat downtimes by cause" chart is disingenuous and useless. Foremost because looking at the last weeks of data immediately after a huge outage skews the picture severely. Why not show us since launch? Or several other months' charts for comparison? Also, I don't get the sense that FaceStat's load is so heavy as to > It’s astonishing that serving content off our own boxes can be more reliable than serving content off of Amazon. Hypothetically, yes. Necessarily? No. Are you going to get much more reliable at serving content in the next few years? No. Is going Amazon S3? Probably. 99.99% is a primary design goal, and unless I'm misinformed, they've been hitting that most months. This is obviously a disaster for them, but be real. Or, you know, jump ship and be sure to also write posts about how unreliable or expensive self-hosting is when it eventually bites you in the ass or becomes too costly to sustain. I agree that this deep and long of an outage is not acceptable, even for a service with tempered service level.

However, your “this months FaceStat downtimes by cause” chart is disingenuous and useless. Foremost because looking at the last weeks of data immediately after a huge outage skews the picture severely. Why not show us since launch? Or several other months’ charts for comparison? Also, I don’t get the sense that FaceStat’s load is so heavy as to

> It’s astonishing that serving content off our own boxes can be more reliable than serving content off of Amazon.

Hypothetically, yes. Necessarily? No.

Are you going to get much more reliable at serving content in the next few years? No. Is going Amazon S3? Probably. 99.99% is a primary design goal, and unless I’m misinformed, they’ve been hitting that most months. This is obviously a disaster for them, but be real.

Or, you know, jump ship and be sure to also write posts about how unreliable or expensive self-hosting is when it eventually bites you in the ass or becomes too costly to sustain.

]]> By: Alex Ranger http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-389 Alex Ranger Tue, 22 Jul 2008 01:45:17 +0000 http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-389 Yes. Did you read the Amazon SLA information before you started using their service? S3 going down is the fault of Amazon's engineering team. Your site going down is the site of yours. You signed up and used a service that gives you tiny rebates for downtime rather than true uptime guarantees and you got what you deserved. Quit whining. Yes. Did you read the Amazon SLA information before you started using their service? S3 going down is the fault of Amazon’s engineering team. Your site going down is the site of yours. You signed up and used a service that gives you tiny rebates for downtime rather than true uptime guarantees and you got what you deserved. Quit whining.

]]> By: Danny Thorpe http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-388 Danny Thorpe Mon, 21 Jul 2008 21:55:56 +0000 http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-388 <strong>Amazon S3 Down for 7 Hours; S3 Clients Looking for Exit...</strong> Lukas Biewald lays bare his frustrations with Amazon’s S3 service, particularly after the recent S3 service outtage that left his FaceStat business offline for more than 7 hours recently.  Actually, Lukas has double posted on this issue - he has... Amazon S3 Down for 7 Hours; S3 Clients Looking for Exit…

Lukas Biewald lays bare his frustrations with Amazon’s S3 service, particularly after the recent S3 service outtage that left his FaceStat business offline for more than 7 hours recently.  Actually, Lukas has double posted on this issue - he has…

]]> By: Tim http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-386 Tim Mon, 21 Jul 2008 17:46:24 +0000 http://blog.doloreslabs.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-386 Guys, cloud computing is still what should be called beta. Massively parallel systems are difficult and this is going to take a while to get right. If you put all your data on one of these sites, or pin your business model to theirs, you are asking for trouble. Don't put mission critical applications on new, unproven technology with no failover plan and then start whining that you aren't getting what you wanted. It's new, it's cool, it's going to be the future, but right now it's still going through growing pains. Guys, cloud computing is still what should be called beta. Massively parallel systems are difficult and this is going to take a while to get right. If you put all your data on one of these sites, or pin your business model to theirs, you are asking for trouble. Don’t put mission critical applications on new, unproven technology with no failover plan and then start whining that you aren’t getting what you wanted. It’s new, it’s cool, it’s going to be the future, but right now it’s still going through growing pains.

]]>