Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Fastmail is down (fastmailstatus.com)
49 points by _jcwu on Aug 1, 2018 | hide | past | favorite | 15 comments


Core router issue at NYI - apparently it's not just us, though we're one of their major non-US customer-base customers, so it affects a lot of our users. I spoke to their techs briefly to confirm details, but otherwise am leaving them alone to do their jobs!

It's particularly annoying that it's a partial outage and our main 66.111.4.x range is the most affected (I suspect because it runs through DDoS protection that our more hidden ranges don't) so most our monitoring is running just fine, but customers can't get in. And it would take an hour (thanks "sensible" DNS timeouts) to move the traffic with the knobs we have access to.


And... we appear to be back :) Far out. Stay tuned for a blog post about this later.

No data lost, and the servers themselves have been happily humming along the entire time, so we don't even have filesystem checks to worry about.


> And... we appear to be back :) Far out. Stay tuned for a blog post about this later.

Congrats and thanks, good luck with the search for another upstream or two :)


Thanks for posting this. I live in China and thought Fastmail was finally being blocked by the GFW. Over the past year I've had intermittent issues with connectivity and I've wondered if FM's time in China was up. Are there any HN readers in China that have experienced similar issues with FM?


Their status page says

> We’re experiencing network issues with several providers. Get updates on fastmailstatus.com as the situation progresses.

(including the quirks characters, copied directly).

That implies the outage isn't universal, and some ISPs/locations may not be affected. Is anybody able to access through a particular VPN (or country)? If so, posting that here could help others get access to their inboxes faster.


Yeah, Linode Newark is an unaffected vantage point. Same with Vultr, but Digital Ocean is no go.

Since the MX are spread between NYI/AS11403 and AS201628, I think that all mail delivery should still be fine.

I'm receiving occasional push notifications on my iOS FastMail app, can't actually read them though!


I was able to connect via Melbourne, Asutralia through Mullvad VPN.


I randomly checked few servers using NordVPN and was able to connect via Russia.


Top work Fastmail team!

Good news! Mail access has been restored for users on some major networks. We apologize to those customers who still cannot reach their Inboxes - we're working to restore full access as fast as we can. No mail has been lost during this downtime. https://www.fastmailstatus.com

3:09 AM - 1 Aug 2018

https://twitter.com/FastMail/status/1024598033793601536


Looks like an issue with their upstream, nyi.net:

traceroute to mail.fastmail.com (66.111.4.148), 30 hops max, 60 byte packets

1 192.168.1.1 (192.168.1.1) 2.582 ms 4.760 ms 4.748 ms

2 142.254.XXX.XXX (142.254.XXX.XXX) 14.748 ms 41.677 ms 43.837 ms

3 agg60.vnnzca2401h.socal.rr.com (76.167.27.73) 44.784 ms 47.498 ms 47.498 ms

4 agg11.vnnycajz01r.socal.rr.com (72.129.14.96) 47.489 ms 47.479 ms 47.499 ms

5 agg29.lsancarc01r.socal.rr.com (72.129.13.0) 47.475 ms 47.466 ms 47.452 ms

6 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 47.458 ms 20.444 ms 19.780 ms

7 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 24.220 ms 25.283 ms 25.654 ms

8 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 27.438 ms 29.922 ms 29.924 ms

9 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 32.877 ms 33.666 ms 34.208 ms

10 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 35.399 ms 35.901 ms 36.878 ms

11 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 34.177 ms 34.492 ms 34.462 ms

12 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 31.085 ms 28.907 ms 30.610 ms

13 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 27.499 ms 35.354 ms 36.167 ms

14 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 38.251 ms 39.675 ms 39.303 ms

15 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 36.657 ms 36.652 ms 38.186 ms

16 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 39.633 ms 46.706 ms 45.702 ms

17 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 43.710 ms 34.548 ms 42.508 ms

18 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 42.399 ms 38.360 ms 39.058 ms

19 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 41.678 ms 43.000 ms 44.069 ms

20 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 44.913 ms 45.806 ms 46.209 ms

21 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 51.609 ms 50.484 ms 51.050 ms

22 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 53.087 ms 55.045 ms 39.316 ms

23 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 52.467 ms 52.470 ms 52.450 ms

24 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 53.596 ms 43.254 ms 46.939 ms

25 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 51.514 ms 53.067 ms 54.051 ms

26 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 61.721 ms 61.724 ms 61.715 ms

27 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 54.020 ms 54.011 ms 54.003 ms

28 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 51.628 ms 53.483 ms 53.482 ms

29 agg10.lsancarc01r.socal.rr.com (66.75.161.49) 56.560 ms 57.956 ms 58.392 ms

30 agg10.tustcaft01r.socal.rr.com (66.75.161.48) 57.149 ms 58.921 ms 57.447 ms


Matches up with their current Twitter thread:

We're experiencing a network outage. Stay tuned at https://www.fastmailstatus.com . We're in close communication with our provider trying to restore service.

1:34 AM - 1 Aug 2018

https://twitter.com/FastMail/status/1024573967393157121


I find it a bit concerning that they don't have a backup provider.


Yeah I'd have expected them to have multiple upstreams and an ASN to announce their IPs but apparently they don't. Pretty odd.


They do have AS redundancy, but only for their MX.


Diminishing returns. This is the first time I’ve ever heard of a FastMail outage, so it seems their architecture is commercially reasonable as it relates to outages.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: