AWS的验尸报告~

[xatest]

http://aws.amazon.com/message/65648/

AWS这次严重的故障，影响到了Quora和Foursquare等多家客户，不知道有看了官方验尸报告的吗？
我花了2个小时看完，整个事故的过程分析起来惊心动魄啊！

[MarkFull]

这里有个简版。

http://internet.solidot.org/internet/11/04/30/0915200.shtml

[ry_wang]

发现异常为啥不能及时回滚呢?其实我好奇这个

[xatest]

@ry_wang 他们错误的回滚策略造成了雪崩，引起更大范围的错误~

[ry_wang]

@xatest 哦?有详细点的说明么?官方文档看不懂呀

[xatest]

@ry_wang 这还不够详细？5700字了~

[Sidney]

看着有种心惊肉跳的感觉

两点感受
1. 人为的操作容易出现错误。之前碰到过hosting的人员在配置Firewall的时候误操作，Firewall Hang住，导致网站无法访问。
The traffic shift was executed incorrectly and rather than routing the traffic to the other router on the primary network, the traffic was routed onto the lower capacity redundant EBS network.

2. 对回滚操作的后果估计不足，带来连锁雪崩效应。除非对系统各个模块和它们之间的交互有足够的理解，说不定就会踩中地雷。
Incorrect traffic shift was rolled back and network connectivity was restored. This quickly led to a “re-mirroring storm,” where a large number of volumes were effectively “stuck” while the nodes searched the cluster for the storage space it needed for its new replica.