Comments on: Amazon’s S3 Web Service, our #1 cause of failure http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/ Thu, 02 Sep 2010 21:25:28 +0000 hourly 1 http://wordpress.org/?v=3.0 By: Святослав Маминов http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-1429 Святослав Маминов Sat, 22 Aug 2009 22:03:21 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-1429 Интересная тема, продолжайте. Иногда нахожу ответы, которые получить самому просто реально не хватает времени. Благодарю! Интересная тема, продолжайте. Иногда нахожу ответы, которые получить самому просто реально не хватает времени. Благодарю!

]]> By: Артем http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-1418 Артем Tue, 18 Aug 2009 23:56:49 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-1418 Вот это да... Куча комментов и практически ни одного по теме :) Вот это да… Куча комментов и практически ни одного по теме :)

]]> By: codediva http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-399 codediva Fri, 25 Jul 2008 02:48:44 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-399 I understand your frustration, however, anytime you used hosted services (even those from a trusted provider) you should plan on implementing some redundancy. I know this can be costly for small orgs, but it could have minimized the impact of this occurance. I understand your frustration, however, anytime you used hosted services (even those from a trusted provider) you should plan on implementing some redundancy. I know this can be costly for small orgs, but it could have minimized the impact of this occurance.

]]> By: LusciousPainGeek http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-395 LusciousPainGeek Thu, 24 Jul 2008 08:48:32 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-395 Isn't this one of the risks you take in exchange for the convenience of using a centralized system like S3? Unfortunate, but bound to happen and something you need to look into when performing risk assessment. Isn’t this one of the risks you take in exchange for the convenience of using a centralized system like S3? Unfortunate, but bound to happen and something you need to look into when performing risk assessment.

]]> By: Ted Z http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-394 Ted Z Tue, 22 Jul 2008 19:15:35 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-394 Look into Akamai, they'll do this correctly (disclaimer: I've worked for them). Or implement their system (quickly deteriorating DNS endpoint lookups) yourself, it's not hard if your scope is limited. You can just have S3 be your primary and when it fails, switch DNS over to your backup. Look into Akamai, they’ll do this correctly (disclaimer: I’ve worked
for them). Or implement their system (quickly deteriorating DNS
endpoint lookups) yourself, it’s not hard if your scope is limited.
You can just have S3 be your primary and when it fails, switch DNS
over to your backup.

]]> By: Ryan Haney http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-393 Ryan Haney Tue, 22 Jul 2008 15:14:43 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-393 Have you considered using S3 as a backup provider with a local disk as cache? I can envision software that emulates a hard disk with a caching mechanism that copies to and from S3 in the background, as files are requested / uploaded. I do think though that you haven't load tested your own storage solution. I have a friend that works for Seagate and hard drives fail ALL THE TIME. It's just nature of the beast. Have you considered using S3 as a backup provider with a local disk as cache? I can envision software that emulates a hard disk with a caching mechanism that copies to and from S3 in the background, as files are requested / uploaded.

I do think though that you haven’t load tested your own storage solution. I have a friend that works for Seagate and hard drives fail ALL THE TIME. It’s just nature of the beast.

]]> By: Joe http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-392 Joe Tue, 22 Jul 2008 03:05:39 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-392 Get a grip !!! It's not like you're running a stock brokerage system on Wall St ! My experience is that it's not hosting computers that are the reliability issue, but the ISP who provides the connection. Try getting any level of service guarantee is next to impossible. When I build "five nines" systems for clients, we end up putting data centers in multiple cities using multiple carrier connections. It costs a heck of a lot more than what Amazon charge for S3. It's like you paid for a cheap car, so don't whinge about the rattles. Get a grip !!! It’s not like you’re running a stock brokerage system on Wall St !

My experience is that it’s not hosting computers that are the reliability issue, but the ISP who provides the connection. Try getting any level of service guarantee is next to impossible. When I build “five nines” systems for clients, we end up putting data centers in multiple cities using multiple carrier connections. It costs a heck of a lot more than what Amazon charge for S3. It’s like you paid for a cheap car, so don’t whinge about the rattles.

]]> By: Striddy http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-391 Striddy Tue, 22 Jul 2008 02:06:19 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-391 Should've been: Also, I don’t get the sense that FaceStat’s load is so heavy as to ... stress your other hosting service to the point of failure much. Should’ve been:
Also, I don’t get the sense that FaceStat’s load is so heavy as to … stress your other hosting service to the point of failure much.

]]> By: Striddy http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-390 Striddy Tue, 22 Jul 2008 02:04:56 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-390 I agree that this deep and long of an outage is not acceptable, even for a service with tempered service level. However, your "this months FaceStat downtimes by cause" chart is disingenuous and useless. Foremost because looking at the last weeks of data immediately after a huge outage skews the picture severely. Why not show us since launch? Or several other months' charts for comparison? Also, I don't get the sense that FaceStat's load is so heavy as to > It’s astonishing that serving content off our own boxes can be more reliable than serving content off of Amazon. Hypothetically, yes. Necessarily? No. Are you going to get much more reliable at serving content in the next few years? No. Is going Amazon S3? Probably. 99.99% is a primary design goal, and unless I'm misinformed, they've been hitting that most months. This is obviously a disaster for them, but be real. Or, you know, jump ship and be sure to also write posts about how unreliable or expensive self-hosting is when it eventually bites you in the ass or becomes too costly to sustain. I agree that this deep and long of an outage is not acceptable, even for a service with tempered service level.

However, your “this months FaceStat downtimes by cause” chart is disingenuous and useless. Foremost because looking at the last weeks of data immediately after a huge outage skews the picture severely. Why not show us since launch? Or several other months’ charts for comparison? Also, I don’t get the sense that FaceStat’s load is so heavy as to

> It’s astonishing that serving content off our own boxes can be more reliable than serving content off of Amazon.

Hypothetically, yes. Necessarily? No.

Are you going to get much more reliable at serving content in the next few years? No. Is going Amazon S3? Probably. 99.99% is a primary design goal, and unless I’m misinformed, they’ve been hitting that most months. This is obviously a disaster for them, but be real.

Or, you know, jump ship and be sure to also write posts about how unreliable or expensive self-hosting is when it eventually bites you in the ass or becomes too costly to sustain.

]]> By: Alex Ranger http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-389 Alex Ranger Tue, 22 Jul 2008 01:45:17 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-389 Yes. Did you read the Amazon SLA information before you started using their service? S3 going down is the fault of Amazon's engineering team. Your site going down is the site of yours. You signed up and used a service that gives you tiny rebates for downtime rather than true uptime guarantees and you got what you deserved. Quit whining. Yes. Did you read the Amazon SLA information before you started using their service? S3 going down is the fault of Amazon’s engineering team. Your site going down is the site of yours. You signed up and used a service that gives you tiny rebates for downtime rather than true uptime guarantees and you got what you deserved. Quit whining.

]]> By: Danny Thorpe http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-388 Danny Thorpe Mon, 21 Jul 2008 21:55:56 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-388 <strong>Amazon S3 Down for 7 Hours; S3 Clients Looking for Exit...</strong> Lukas Biewald lays bare his frustrations with Amazon’s S3 service, particularly after the recent S3 service outtage that left his FaceStat business offline for more than 7 hours recently.  Actually, Lukas has double posted on this issue - he has... Amazon S3 Down for 7 Hours; S3 Clients Looking for Exit…

Lukas Biewald lays bare his frustrations with Amazon’s S3 service, particularly after the recent S3 service outtage that left his FaceStat business offline for more than 7 hours recently.  Actually, Lukas has double posted on this issue – he has…

]]> By: Tim http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-386 Tim Mon, 21 Jul 2008 17:46:24 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-386 Guys, cloud computing is still what should be called beta. Massively parallel systems are difficult and this is going to take a while to get right. If you put all your data on one of these sites, or pin your business model to theirs, you are asking for trouble. Don't put mission critical applications on new, unproven technology with no failover plan and then start whining that you aren't getting what you wanted. It's new, it's cool, it's going to be the future, but right now it's still going through growing pains. Guys, cloud computing is still what should be called beta. Massively parallel systems are difficult and this is going to take a while to get right. If you put all your data on one of these sites, or pin your business model to theirs, you are asking for trouble. Don’t put mission critical applications on new, unproven technology with no failover plan and then start whining that you aren’t getting what you wanted. It’s new, it’s cool, it’s going to be the future, but right now it’s still going through growing pains.

]]> By: jeremy http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-385 jeremy Mon, 21 Jul 2008 16:41:49 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-385 This is something any online business deals with, it's always better to host it yourself but not really cost effective for a business that's just starting. As a web business owner currently experiencing inexplicable down time I feel your pain. If you're not responsible for getting it to work again it does give you an excuse to go outside for a little while. This is something any online business deals with, it’s always better to host it yourself but not really cost effective for a business that’s just starting. As a web business owner currently experiencing inexplicable down time I feel your pain.

If you’re not responsible for getting it to work again it does give you an excuse to go outside for a little while.

]]> By: JD http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-383 JD Mon, 21 Jul 2008 16:35:25 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-383 Well at least Amazon S3 is back up and has not lost any data! Nirvanix, an Amazon S3 competitor, lost half of MediaMax's files and put them out of business: http://www.techcrunch.com/2008/07/10/mediamaxthelinkup-closes-its-doors/#comments What a disaster. Well at least Amazon S3 is back up and has not lost any data!

Nirvanix, an Amazon S3 competitor, lost half of MediaMax’s files and put them out of business:

http://www.techcrunch.com/2008/07/10/mediamaxthelinkup-closes-its-doors/#comments

What a disaster.

]]> By: rob http://blog.crowdflower.com/2008/07/amazons-s3-web-service-our-1-cause-of-failure/#comment-382 rob Mon, 21 Jul 2008 16:02:54 +0000 http://blog.doloreslabs.com/2008/07/amazons-web-services-our-1-cause-of-failure/#comment-382 Did amazon give you any explanation about why the service went down? Do you have any dialog with them at all? I don't think I'd be concerned about 8 hours of downtime due to an equipment failure (when I ran a datacenter the best we could get from IBM was a 4-hour replacement part guarantee, so 8 hours of downtime for a failed RAID controller or something isn't really unreasonable), but I *would* be concerned if the problem was tied to the software that provides the service; say, if another user was able to take the whole system down, or if something about the hosting environment was prone to error. I don't think being angry about the downtime is going to be productive for anyone, but I do think finding out something about the weak points of your hosting provider can give you the power to make good choices for your application. If hosting it yourself is more reliable or at least more controllable for you, and that's important to you and your business, then that's what you have to do. I'm sorry it took a service outage for you to consider that problem, but at least now you know the answer that makes the most sense for your business. Did amazon give you any explanation about why the service went down? Do you have any dialog with them at all? I don’t think I’d be concerned about 8 hours of downtime due to an equipment failure (when I ran a datacenter the best we could get from IBM was a 4-hour replacement part guarantee, so 8 hours of downtime for a failed RAID controller or something isn’t really unreasonable), but I *would* be concerned if the problem was tied to the software that provides the service; say, if another user was able to take the whole system down, or if something about the hosting environment was prone to error. I don’t think being angry about the downtime is going to be productive for anyone, but I do think finding out something about the weak points of your hosting provider can give you the power to make good choices for your application. If hosting it yourself is more reliable or at least more controllable for you, and that’s important to you and your business, then that’s what you have to do. I’m sorry it took a service outage for you to consider that problem, but at least now you know the answer that makes the most sense for your business.

]]>