It's super easy for you. I tried to set this up but it's such a pain in the backside to tell people how to pull the latest code. Ended up with Gitlab and haven't looked back.
I didn't even realise Gitlab doesn't have auto-deploy yet.. We set up a .gitlab-ci.yml script that deploys the code.
That's really interesting. I've been considering moving back to my own mail server from gmail for a long time but have been worried about the security issues.
This is one more thing pushing me in that direction.
I've been hosting my personal email on a $5/month DigitalOcean server, running Postfix + Dovecot, for almost two years. I think it's reasonably secure. I run updates regularly and trust the distro maintainers to release timely fixes for new vulns. Aside from that, I mostly ignore it because it works. I should probably look at the logs, but eh.
After the typical host hardening stuff (which isn't much work with modern OS defaults), I have configured SPF, DKIM, DMARC policy, opportunistic TLS for server-to-server, and mandatory TLS for IMAP client connections.
From a data privacy perspective, I know that nobody is mining the contents of my mailbox (except the messages I send to Gmail, etc. users!), and my server is not a high-value target for compromise. (Yes, DigitalOcean could snoop on my non-GPG-encrypted messages if they wanted. I guess I could migrate the server back to my own hardware.) I also encourage friends to use GPG, though this is orthogonal to one's choice of email host.
For clients I use Thunderbird on desktop and K-9 Mail on Android. Mobile push via IMAP IDLE works out of the box. (I also run a CalDAV/CardDAV server to sync contacts/calendar/todo across devices, but that is technically separate.)
Overall I'm really happy with the arrangement. The only annoyance was having my messages to Gmail users consistently marked as spam, but after doing everything suggested by mail-tester.com, I think I'm making it through most of the time.
Inability to reliably send messages to gmail is huge. I'm using my own server, I own my domain for years, I asked a lot of people to remove "Spam" label from my messages, I did everything (reverse DNS, DKIM, SPF, checked that Gmail likes it all) I could, yet for new recipients gmail might mark my e-mail as spam. Gmail is worse e-mail provider, I liked it before, but I really don't like it, because it seems to punish people for decentralization. And I don't have this problem with other e-mail providers that I checked, they are happy to receive my e-mails.
Something is definitely missing in your setup, either entirely or just incorrectly setup.
I've been hosting my own email for about 3 years now, in an even worse-for-my-domain-reputation way: by using SRS.
Basically any email that arrives at my domain gets forwarded to my Gmail account.
And because I use SRS, it will re-write the envelope so it's something like hash-original_domain=user@mydomain.
Yet I've never had a problem with my emails getting marked as spam.
Actionable advice:
- setup DMARC and use something like dmarcian[1] so you have a pretty dashboard of your DMARC reports (otherwise you have to find some tool that will read the XML and aggregate the reports).
- use mxtoolbox to verify your IPs aren't blacklisted, you aren't an open relay, etc.
- if you have enough email traffic (not likely if you're using it for personal email only), signup for Google's Postmaster[3]
I don't think vbezhenar necessarily does anything wrong. I've been (and still am?) running into the same issues with Gmail, as well as with Yahoo (but not any other providers). I'm owning domain and IPs for about a decade, using reverse DNS, DKIM, SPF, on no blacklists, also using SRS for forwarding, registered the domain with Google, not bouncing possible spam, older recipients who moved mail out of spam folder: no problem, new recipients: tagged as spam. I'm using my mail server for a few small-volume mailing lists now, with an added comment on the subscription instructions that subscribers should watch out for mails moved to the spam folder; that has been working well enough for over a year now, but I'm not sure whether that has helped alleviate the problem (I should do a test run again, but it's time consuming and worrisome to create new Gmail user accounts for testing as I fear Google linking those accounts to myself might actually contribute to the bad scoring).
I know someone who does his own mail and does less than me (e.g. no SRS), and yet he doesn't have problems and thinks I'm doing something wrong...
My best hypothesis is that Google doesn't like the network that my server is in (Hetzner, various neighbours in the network are listed on a couple blacklists). When I find the time I'll set up a server with a different provider and see if it improves things. Other hypotheses are the software used (buddy uses OpenBSD and its mail server instead of Linux and QPSMTPD/Qmail+patches, perhaps they care about software headers or are doing OS fingerprinting), the fact that I'm using a fallback server on another continent (US), network timings, and plain noise (perhaps they use some kinds of machine learning that have persistent low-level irregularities).
PS. yes I'm not doing DMARC yet, but neither does my buddy.
Shame on the big providers to force this dilemma on us.
I bet that's not the whole story. There was a reason they were lazy in the first place, but I bet there's another reason why they stay lazy. Punishing decentralization is a great way to get more users, after all: sent from gmail? works. Sent from little provider? Doesn't work.
The only way out of this I see right now is the generalization of the Freedom Box. Though even then, one would need to run protests to be able to send email from home. Between ISP wide firewalls, interdictions on home servers (by contract with some ISP), or the blacklisting of all residential IPs (Hotmail), it will take a lot of collective action before we can send mail from home again.
It wouldn't take much for it to be. We basically need 3 things: a usable Freedom Box (some commercial implementation of this idea are starting to pop up), the authorization to send email, and then end of "little provider" blacklisting. The last one is basically under Google's and Microsoft's control. If they put an end to it, we now have only 2 hurdles to overcome, and both are already partly solved.
Since Snowden, people know they are being spied on. The only reason so many of them still use Gmail is because they don't know how not to. Give them a little box that's as usable as Gmail, and they will use it.
While I agree decentralization is hardly a threat to Gmail right now, it could be, and I don't see them taking any step to make it even more threatening.
The pattern -- and volume -- of messages that Gmail would receive from an independent server with legitimate users would look very different than someone running a similar server to send spam.
Perhaps a spammer with a new server could gain Gmail's trust by passing genuine-looking correspondence, but this would stop working as soon as they start to send bulk messages.
Seems to me that tuning Gmail's filters to recognize and trust small-volume email servers would not be difficult or time-consuming.
That could well be. It's hard to know without access to the data, but they could probably do better if they tried.
However, I fear if Google were to try harder, spammers would also jump on that opportunity. It's easier to mimic the patterns of a small legit SMTP server than to pose as one of a few big well established email providers with known IP ranges. They could potentially use millions of compromised PCs, each sending spam in low volumes mixed with non-spam traffic.
I can recommend radicale [1]. It's very simple and doesn't need a lot of resources. It also supports authentication via an IMAP server so you don't have to keep a separate user database.
I also use Radicale. My only annoyance is that when you use its built-in HTTP server, the process eventually stops working if one of your clients has an intermittent connection (e.g. phone with a poor signal).
Can we stop hacking ill-conceived one-offs like this and just start to assume that a monitoring utility (Monit, Supervisor, Munin, Nagios, ServerDensity, Solarwinds to name a few) in some form is the standard for notifying of anomalies and automation of process recovery?
Moreover, systemd has a OnFailure directive that activates other units when the process exits uncleanly. Upstart has a similar directive. With a more classical approach, `respawn' is a utility that can invoke and maintain the running status of a process launched from System V Init, or you could use the regarded inittab solution.
I've seen answers to questions similar to this on StackOverflow respond with scripts that are nothing more than a while(true) loop that checks to see if a PID exists in languages ranging from Bourne shell to Python and NodeJS. This is the wrong way to do it, especially the Node one. Lets take a language intended to be used inside a web browser client and slap it onto a server to be used for process monitoring. I'm sure it does a fine job with it's while loop, but we're hitting copper pipe with a tire iron wondering why our lightbulb isn't turning on here.
The GP's using it for personal use (I think). OP merely suggested a hack to let GP get on with their life as quickly as possible in 2 lines of bash and one crontab entry - and without learning a new process monitoring utility's configuration. Obviously this wouldn't be suitable for a production site for many users.
I have the same experience. It works perfectly for me with ldap authentication (which I also use for my mailserver) over multiple devices (macOS, Windows, Android, Linux).
Started using it after Owncloud/Nextcloud broke their ldap auth.
Not the parent, but I use Nextcloud for this (previously ownCloud). There are carddav and caldav sync applications for Android. I assume other operating systems have them too.
I've found Baïkal quite reliable over what must be a half decade of use. Integrates well with Apple devices (iOS/OS X 10.9), with the one Android I ever used, and via Lightning and Sogo Connector, also reasonably well with Thunderbird.
Thank you for that post. I've been trying some time ago, mostly for fun, to set up an email server. I did get one up and running with a setup similar to what you describe but was spam-filtered by Gmail as you describe. And because of this I kind of gave up.
You can expect a fresh IP from one of the cloud hosting services to be spam filtered out of the box by Google et al these days.
If you’re lucky, the IP in question hasn’t previously been used to spam & by very slowly ramping up your email rate Google will eventually decide that you’re probably OK. Getting this right appears to be something of a black art however :(
I run my own mail server, I can tell you this: Buffer overflows and format string vulns in your imap implementation will be the least of your worries.
It is certainly still valuable work to audit the software used for the mailserver. But the challenges lie elsewhere. The biggest chunks are configuration issues and - if you have users that aren't very security savvy - stolen passwords and subsequent abuse of mail accounts for spam.
This is where having a mail server with a sane configuration file comes handy. OpenSMTPd is simply awesome in that regard. Plus, it too was recently audited and vulnerabilities fixed so it should be secure.
OpenSMTPd + Dovecot makes for an awesome combination for a mail server.
Well, that setup offers a personal "cloud", not just mail. And the focus is more on "independent" than "secure".
I wouldn't worry too much about running many services. They all run as their own unprivileged users. (Of course chroots or even jails/containers would be better.)
To be honest, I don't think it so much of an issue any more. I have a library of parts I use regularly and just design footprints for new parts.
I'm my case, I doubt that Altium would have footprints for many of the parts I want to use anyway (quite often cheap connectors, switches etc). I feel like being comfortable with footprint design is an important part of the design process.
That said, there are many features of Altium that look very attractive. For example the push and shove routing.
An unrelated but interesting point he noted was that Apple are one of the only vendors that provide long term firmware updates (he mentioned 8 years). It's a shame nobody else really does this.
I think I've seen this before in an HPC context. But they've build a firmware distribution called Heads. It boots using coreboot then fires up a Linux kernel from flash.
The kernel is then used as a second stage bootloader. It takes about 2 seconds to get Linux booted from flash.
They can then boot the system OS, optionally using kexec to smoothly transition to the system kernel.
Very neat! Along the way they've also done other important work, like put together a minimal firmware for the Management engine (a second CPU in Intel system with its own OS, and many many issues).
The biggest problem here is same issue that coreboot has. Coreboot support is really limited. I think it down supports Lenovo X220s, but late time I looked not much modern hardware.
> put together a minimal firmware for the Management engine
I thought that management engine CPU was still a black box, and the best anyone has done is neuter the firmware running there by judiciously zeroing bits out.
I encourage other people to use them for their own security. My argument for ad blocking has zero to do with advertisements and everything to do with preventing malware.
I often want to share an article but hate the fact that people who'll click will expose themselves to a lot of crap if they're not running a blocker. Obviously I'd rather not link to such sites at all. But with so much of the news media now filled with ads, it's often hard to avoid. I feel a little better doing it through this.
I was also thinking of showing a before and after of the linked site, with and without an ad blocker. :)
