I'm with Hund. It was only monitoring the cached, unauthenticated news page, which was up throughout the downtime. However, it now monitors the authenticated news landing as well.
This was monitoring the unauthenticated news page, which is why it didn't catch it. It now monitors authentication as well. It is not official, and was made by a co-founder years ago.
I'm with Hund. Our hn.hund.io page did not catch this because it was requesting the cached, unauthenticated page. It now monitors authentication as well.
We've banned this account for repeatedly breaking the site guidelines and ignoring our requests to stop.
If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future. They're here: https://news.ycombinator.com/newsguidelines.html.
What a rude thing to post. Hund, don't listen to this entitled nonsense. There is a reason it's called human error. Companies 100x your size and 10000x your revenue like AWS, Microsoft, CloudFlare, CrowdStrike can't figure out how to accurately provide status dashboards. At least you took the time to explain your mistakes. If anything you got another supporter for your honesty.
This was a focus in our after-action review. The nodes responded as healthy to active checks, while silently dropping updates on their replication lag, together this created the impression of a healthy node. The missing bit was verifying the absence of lag updates. (Which we have now.)
You might want to clarify this in the post. To me it reads like you knowingly had degraded infra for days leading up to an incident which might have been preventable had you recovered this instances.
I am a curious and very amateur person, but do you think that if "100%" uptime were your goal, this:
"[Three months prior to the incident] We upgraded our databases to a new minor version that introduced a subtle, undetected fault in the database’s failover system."
could have been prevented if you had stopped upgrading minor versions, i.e. froze on one specific version and not even applied security fixes, instead relying on containing it as a "known" vulnerable database?
The reason I ask is that I heard of ATM's still running windows XP or stuff like that. but if it's not networked could it be that that actually has a bigger uptime than anything you can do on windows 7 or 10?
what I mean is even though it is hilariously out of date to be using windows xp, still, by any measure it's had a billion device-days to expose its failure modes.
when you upgrade to the latest minor version of databases, don't you sacrifice the known bad for an unknown good?
> could have been prevented if you had stopped upgrading minor versions, i.e. froze on one specific version and not even applied security fixes, instead relying on containing it as a "known" vulnerable database?
This is a valid question.
As a database and security expert, I carefully weigh database changes. However, developers and security zealots typically charge ahead "because compliance."
Same user (sorry I guess I didn't enter my password carefully as I can't log in.)
Well I mean they're not exactly on the Internet with an IP address and no firewall, are they? (Or they would have been compromised already.)
Whatever it is, it must be separated off as an "insecure enclave".
So that's why I'm wondering about this technique. You don't just miss out on security updates, you miss performance and architecture improvements, too, if you stop upgrading.
But can that be the path toward 100% uptime? Known bad and out of date configurations, carefully maintained in a brittle known state?
No, it's a fair question. The word "enclave" has a general meaning in English as a state surrounded entirely by another, or metaphorically a zone with some degree of isolation from its surroundings.
So the legit question is, can insecure systems (e.g. ancient mainframes) be wrapped by a security layer (WAF, etc.) to get better uptime than patching an exposed system?
Stripe splits data by kind into different database clusters and by quantity into different shards. Each cluster has many shards, and each shard has multiple redundant nodes.
having a few nodes down is perfectly acceptable. I guess they would have had an alert if the number of down nodes exceeded some threshold.
this case that doesn't sound like it was the issue, it was the lack of promotion of new master due to the bug in the system in terms of shard promotion.
Right, but they didn't recover speedily. To have the cluster in such a state for so long sounds like poor monitoring to me because this can knowingly interfere with an election later.
If I'm not mistaken, Rdio's engineering team helped build Pandora's on-demand service. I've switched to it from Spotify after using Rdio in the past. Its recommendations feel similar to Rdio's, but the interface is a bit lacking. There aren't general recommendations. Instead, you must build a playlist and then you can add a set of recommended songs to the playlist. Of course, there are also radio stations.
> Lightsail uses burstable performance instances that provide a baseline level of CPU performance with the additional ability to burst above the baseline.
The biggest benefit over StatusPage.io is that our platform's design focuses on automation through integrations with third parties. StatusPage.io pages can only be automated through their direct integration with PagerDuty, or by parsing email notifications (from services like New Relic or Pingdom). We have the ability to add countless integrations with third parties for both monitoring and notifications.
Branding is important for companies, so we've chosen to offer complete white-labeling on our single plan. We leverage Let's Encrypt so that status pages can be instantly configured to use a secure custom domain.
These are just a few of the key differences, our features section outlines these and more.
FYI, my first question was also "how is this different than StatusPage.io?". It might be a good idea to have a comparison page or make the differentiation very clear in some other way. Your target audience has most likely heard of StatusPage.io (they are the incumbent in my mind) and will likely ask themselves the same question as soon as they land on your site.
If someone is using StatusPage.io, why should they switch to your service? Or, are you only targeting customers who do not already have a status page solution in place?
reply