> How would you write your code today if you knew it would of been your last commit and still in use in 30 years ?
Generally: minimize dependencies. External library or API dependencies? Versions can drift, the system can change out from under you in incompatible ways. That goes for the OS, too, of course. Data dependency that you aren't 100% in control of? Same. All are forms of state, really. Code of the form "take thing, do thing, return thing, halt" (functional, if you like—describes an awful lot of your standard unixy command line tools) is practically eternal if statically compiled, as long as you can execute the binary. Longer, if the code and compiler are available.
This. It doesn't mean go overboard with NIH, but you have to evaluate and select your dependencies judiciously. It's not about developer productivity with these types of products.
Also, make as much of your program configurable as possible so you can tweak things out in the field. For example, if you have a correlation timeout. Make that configurable. But don't go overboard with that either. :)
Another aspect of this is pick dependencies that are well encapsulated (so if you need to change them or update them it's generally easy to).
Of course, this is just a good choice regardless. Still, it shocks me how often people will choose libraries and frameworks that require very opinionated structure on large swathes of code, rather than having well defined minimal touchpoints.
And vendor your dependencies, archiving tested and blessed snapshots in your version control instead of pulling them live from GitHub, NPM, or BinTray.
FWIW I've given up on trying to gauge my own performance. I've plainly been disappointing people when feeling like I'm working about has hard as I could. I've (over, and over, and over) been complimented on both the quantity and quality of my output when I feel like I'm half-assing it, at best. There doesn't seem to be a much relation between how I feel like I'm working and what others perceive.
Yeah, I think the repetitive part is what drives us crazy.
The worst jobs are probably done by prison labor and slaves. 8, I would actually classify as dangerous labor, things like how prisoners are made to peel garlic which corrodes their fingernails and teeth.
> In other words, while I considered what the accountant guy did to be a rather low-effort incomplete job, he did exactly what was expected of him by the bank and what I needed to get my deal done.
The degree to which everything runs on this sort of system is kind of horrifying, once you're exposed to enough things like this for that to sink in.
Relatedly, I guess my contribution to the broader thread would be:
Almost no-one knows what they're doing, let alone is much good at it—so few, in fact, that society and the economy (and everything else) basically run on a massive and super-serious game of playing pretend. Yes, even that big important organization (public or private) where you expect everyone to be pretty damn competent. The difference between them and some normal place is that 5% of their people are impressively good at their jobs, rather than 2%.
This is what becoming an adult has taught me. I'm still baffled that things work as well as they do given the incredible levels of incompetence you find everywhere.
Growing up, I had this idea that adults always know what to do, at least within their specialities, or at least know how to figure out what they don't know. So far from true. The vast majority is just winging it. To the best of their abilities, of course, but it's still all improv.
Right—that significant amounts of labor (=human life) go into this kind of waste that everyone involved knows is a joke, and that the non-joke variety is rarely treated much differently, both contribute to what's scary about it.
I agree that the non-joke variety not getting the serious treatment it deserves is scary, I just don't agree that it happens as often as you were implying.
>The degree to which everything runs on this sort of system is kind of horrifying, once you're exposed to enough things like this for that to sink in.
So? It clearly works.
People want everything to be high quality master craftsman type work but almost nothing is and it almost never matters. People like to think that the auditors are going over the papers with a magnifying glass and that their landscaper is applying exactly the right fertilizer for the soil conditions. In reality the auditors are skimming the papers and the landscaper is just using whatever brand has worked decently in the past.
Most work done most of the time is little more than the minimum and that's all the world needs. You can either look at that as sloppy and low effort or efficient allocation of resources.
It works right up until it doesn't: see 737 Max, mortgage crisis...
Perhaps the best thing that can be said for pro-forma processes is that they usually put people on notice that they may be held accountable for their actions -- though, of course, that is not always the case.
And look at all the planes that don't fall out of the sky and all the financial products that don't cause crisis.
The "even rare failures are unacceptable" line of reasoning is simply too conservative to permit our modern society to exist. The resources we would have to dedicate to preventing these kinds of things (don't forget the 90/10 rule) in every case would be astronomical. Imagine if every bit of software had to be written the way NASA rights their software for human carrying vehicles and if every financial transaction had to be strictly scrutinized. Software would be so expensive that it would be used far less than it is today and all the good that brings would not exist. Credit would be far, far less accessible to just about everyone you could fill a library with books that have been written about why that is bad. And those are just two industries(!!!!).
It's really east to say "But the MAX" or "but the mortgage crisis" but occasional failures like that are simply the price of a reality where normal everyday people can travel thousands of miles in a day and have access to credit.
People don't just "ignore their responsibilities" out of malice. Generally people "ignore their responsibilities" either because they are pressured into it, or because they are lead to believe it is the right thing to do in that situation.
Either is unacceptable, of course, but the boots on the ground are rarely to blame.
Kind of ironically, pulling out 3 anecdotal examples of well-known system failures doesn't make a very high-quality argument that most things being done sloppily most of the time doesn't usually work out fine. The real argument is, that while those 3 things were happening - and the War on Drugs goes back ~100 years - roughly 3 billion other things were also sloppily done, and most of them worked out well enough.
If it didn't work, we would all be starving and dodging lions in a jungle somewhere, instead of writing posts on an internet forum about how a few well-known things were sloppily done, yet didn't really cause that much damage in the great scheme of things.
tpxl and you are using different definitions of "work out fine".
Incarceration and systemic discrimination against an entire class of people does not count as "well enough", in my opinion.
And it's specious reasoning to conclude that if a society made up of untrustworthy actors committing fraud isn't starving and dodging lions, then it's worked out fined. One can go live in a country with low societal trust to see what that's like (Brazil, India, Pakistan, Somalia, etc).
I posit that it's the proportion of trustworthy actors in the system, along with a healthy dose of conveniently timed technological advances as well as luck providing resources at the right time that leads to a prosperous society. There are countless examples in history of a society doing well enough, and every time there is a tipping point where sufficient trust is lost and it starts degrading, or in some cases, collapses.
> tpxl and you are using different definitions of "work out fine".
I suppose we are. I'd argue that mine corresponds to reality in every modern nation that currently exists on the planet. I think they're all doing pretty well indeed overall, compared to the historical record and the current conditions of some of the countries and places that aren't doing so well. Comparing current reality to an imaginary utopia is a whole different ballgame.
There's nothing wrong with recognizing the problems, mistakes, and injustices that we do have now and working to fix them. We just need to keep a little perspective - despite the problems, things are still going pretty well one the whole. Plans that talk of tearing the whole system down to fix a few small problems aren't a good idea, and have historically mostly led to things getting much, much worse.
Change is inevitable. I merely hope to slow the change caused by corruption by doing what I can to prevent it.
The only way to stop or slow corruption is by shining light on it. I support all efforts to increase transparency. Put all those spreadsheets online so people can audit each other. Perhaps it is inevitable still, but it’s the best chance we have.
> The economic crisis of 2008/9 could have easily been prevented.
That's not clear at all. There are systemic changes one might have made that could have reduced the impact at the margins, but ultimately the cause was too much much money chasing too few assets and I have yet to see any plan that would have changed that.
Aldrin gets mentioned in the same breath as Armstrong often enough I bet quite a few people could pull that one out. You'd lose a bunch with "who's the guy who stayed in orbit?" and almost everyone with "name any Apollo 12 astronaut" or even the more-generous "name any Apollo astronaut who at least reached Lunar orbit and wasn't on 11 or 13". Most of the people who get that second one would probably just luckily guess Alan Shepard without actually knowing for sure he was on an Apollo mission, just the first famous space-program-guy who came to mind who wasn't Armstrong or Aldrin (and in fact the Apollo part of the career is not why they know his name).
Never works. I was driving my friend’s car. I had a tire burst on me. I pulled over to the side of the road. Cops showed up in a few minutes and charged me with reckless driving. Made me take three different breathalyzers. I hadn’t been drinking so obviously it showed “well below the legal limit”. But my friend apparently had an empty beer can in the back.
You can never say it wasn’t yours, apparently. In any case, paid the fine. Oh well.
You have to understand, much of he law that pertains to using a motor vehicle in the US is a pretense to legitimize traffic stops, which are then used for some other purpose (mostly busting drivers for more serious crimes, or else extracting fines from out-of-towners/minorities).
What is the point of having traffic stops of this kind? For example, why can't you just have traffic checks as a normal activity? In some countries (Botswana is an interesting example) the police stop you on public holidays and give you flyers and tell you "drive safely".
Our system allows police to selectively apply traffic rules. This means that the people who find them onerous and can fight back against them tend to be avoided, while more vulnerable people are subject to them more often.
You might remember the protests in Ferguson, MO, after Michael Brown was m̶u̶r̶d̶e̶r̶e̶d̶ killed. A federal investigation later revealed the tense police/public relations under which that incident took place: the jurisdiction was essentially using the traffic laws to extract rent, overwhelmingly from black residents. A traffic citation would require attending court (often without access to a vehicle); if a court date was missed, additional fines and an arrest warrant were issued, which of course the person would only become aware of during the next traffic stop. Police were encouraged to increase stops and citations to make up for lowered taxes. It's regressive fiscal policy you see popular among American conservatives because it shifts the tax burden off of people who are then more likely to vote for them, onto people "deserving" of punishment.
The protection from unreasonable search and seizure is taken to mean that the police can't stop your car without a reason. Since the justice system really really wants to be able to stop your car at will, anything at all can be a "reason".
That would violate our constitution. These sorts of traffic stops are either a bug or a feature, depending who you talk to. I feel that in the absence of clearly reckless driving the police should be unable to stop you. I view these sorts of stops as government run amuck, and a police state. Many Americans would agree, which is why we have constitutional protections beyond most countries (in theory at least). The problem is that hinders the police's ability to solve cases, which is where the "hack" comes in.
Yea...US DUI laws are "interesting". I make every effort to not drink and drive, but I definitely don't let anyone in a car I'm driving have an open drink. I've seen way to many 'you just handed your drink to the guy in the back seat' scenes to risk that.
Not everywhere, and it should be nowhere. Which states don't specify that there needs to be alcohol in the container? That seems like a bizarre last, but it certainly wouldn't surprise me.
Hitching in the 80s, I got picked up by this guy who was drinking beer while driving. That worried me some. But I was shocked when he threw the empty out the window. And when I commented on it, he pointed out that littering was the safest option.
If there’s a real downturn we’ll see the big boys cut jobs—maybe a lot. Or even just some regulatory changes (especially reigning in spying and fruits-of-spying monetization) without a broader downturn. Wages downmarket would suffer a ton with all those top-comp folks with (by definition) above-average résumés and work history suddenly looking for a job. Probably upward mobility (into lead/senior/et c.) will suffer a lot. Doesn’t take more supply, lower demand is something we will see again.
I eat mostly vegetarian (I'm not a principled vegetarian so I'll eat meat in some circumstances, just way less than most people) and that's pretty easy, but milk is really cheap & convenient (chemically) for so much cooking, eggs are like a nutrition-on-a-budget cheat code and also chemically useful in lots of cooking, and I love cheese. Going vegan would hurt.
[EDIT] if there's one modification I'd really like to make to my diet, it'd be to add fish once or so per week, ideally the fatty, oily, low-on-the-food-chain sort that're supposed to be so heatlhy. I need some kind of guide for how to work up to enjoying fish when you didn't grow up eating it. I can tolerate larger fish when cooked & seasoned very well, but don't really enjoy it at all, and have no understanding of what even to do with the smaller, healthier sorts that isn't stomach-turning (to me) to even consider. Though for some reason I love sushi and calamari, so, go figure.
An awful lot of server systems can tolerate a hardware failure on their one server every couple years given 1) good backups, 2) "shit's broken" alerts, and 3) reliable push-button re-deploy-from-scratch capability, all of which you should have anyway. Lots of smaller shops trying to run to k8s and The Cloud probably have at least that much downtime (maybe an hour or two a year, on average) due to configuration fuck-ups on their absurd Rube Goldberg deployment processes anyway.
[EDIT] oh and of course The Cloud itself dies from time to time, too. Usually due to configuration fuck-ups on their absurd Rube Goldberg deployment processes :-) I don't think one safely-managed (see above points) server is a ton worse than the kind of cloud use any mid-sized-or-smaller business can afford, outside certain special requirements. Your average CRUD app? Just rent a server from some place with a good reputation, once you have paying customers (just host on a VPS or two until then). All the stuff you need to do to run it safely you should be doing with your cloud shit anyway (testing your backups, testing your re-deploy-from-scratch capability, "shit's broken" alerts) so it's not like it takes more time or expertise. Less, really.
Business services generally need high availability goals, so often that doesn't cut it. And your single server doesn't autoscale to load.
AWS gives you availability zones, which are usually physically distinct datacenters in a region, and multiple regions. Well designed cloud apps failover between them. Very very rarely have we seen an outage across regions in AWS, if ever.
In practice I see a lot of breakage (=downtime), velocity loss, and terrible "bus factor" from complex Cloud setups where they're really not needed—one beefy server and some basic safety steps that are also needed with the Cloud, so aren't any extra work, would do. "Well designed" is not the norm and lots of the companies are heading to the cloud without an expert at the wheel, let alone more than one (see: terrible bus factor)
Businesses always ask for High Availability, but they never agree on what that actually means. IE, does HA mean "Disaster Recovery", in which case rebuilding the system after an incident could qualify? Does it require active-active runtimes? Multiple data centers? Geographic distribution?
And by the way, how much are they willing to spend on their desired level of availability?
I still need a better way to run these conversations, but I'm trying to find a way to bring it back to cost. How much does an hour of downtime really cost you?
Agree - different business functions have different availability goals. An system that computes live risk for a trading desk might have different availability goals from an HR services portal.
I once ran a Linux server on an old IBM PC out of a run-down hotel's closet with a tiny APC battery for 10 years without a reboot. Just because I got away with it doesn't make it a great idea. (It failed because the hard drive died, but for a year and a half nobody noticed)
> An awful lot of server systems can tolerate a hardware failure on their one server every couple years given 1) good backups, 2) "shit's broken" alerts, and 3) reliable push-button re-deploy-from-scratch capability, all of which you should have anyway
Just.... just... no. First of all, nobody's got good backups. Nobody uses tape robots, and whatever alternative they have is poor in comparison, but even if they did have tape, they aren't testing their restores. Second, nobody has good alerts. Most people alert on either nothing or everything, so they end up ignoring all alerts, so they never realize things are failing until everything's dead, and then there goes your data, and also your backups don't work. Third, nobody needs push-button re-deploy-from-scratch unless they're doing that all the time. It's fine to have a runbook which documents individual pieces of automation with a few manual steps in between, and this is way easier, cheaper and faster to set up than complete automation.
> Just.... just... no. First of all, nobody's got good backups. Nobody uses tape robots, and whatever alternative they have is poor in comparison, but even if they did have tape, they aren't testing their restores. Second, nobody has good alerts. Most people alert on either nothing or everything, so they end up ignoring all alerts, so they never realize things are failing until everything's dead, and then there goes your data, and also your backups don't work.
But you should test your backups and set up useful alerts with the cloud, too.
> Third, nobody needs push-button re-deploy-from-scratch unless they're doing that all the time. It's fine to have a runbook which documents individual pieces of automation with a few manual steps in between, and this is way easier, cheaper and faster to set up than complete automation.
Huh. I consider getting at least as close as possible to that, and ideally all the way there, vital to developer onboarding and productivity anyway. So to me it is something you're doing all the time.
[EDIT] more to the point, if you don't have rock-solid redeployment capability, I'm not sure how you have any kind of useful disaster recovery plan at all. Backups aren't very useful if there's nothing to restore to.
[EDIT EDIT] that goes just as much for the cloud—if you aren't confident you can re-deploy from nothing then you're just doing a much more complicated version of pets rather than cattle.
> more to the point, if you don't have rock-solid redeployment capability, I'm not sure how you have any kind of useful disaster recovery plan at all. Backups aren't very useful if there's nothing to restore to.
As Helmuth von Moltke Sr said, "No battle plan survives contact with the enemy." So, let's step through creating the first DR plan and see how it works out.
1) Login to your DR AWS account (because you already created a DR account, right?) using your DR credentials.
2) Apply all IAM roles and policies needed. Ideally this is in Terraform. But somebody has been modifying the prod account's policies by hand and not merging it into Terraform (because reasons), and even though you had governance installed and running on your old accounts flagging it, you didn't make time to commit and test the discrepancy because "not critical, it's only DR". But luckily you had a recurring job dumping all active roles and policies to a versioned write-only S3 bucket in the DR account, so you whip up a script to edit and apply all those to the DR account.
3) You begin building the infrastructure. You take your old Terraform and try to apply it, but you first need to bootstrap the state s3 and dynamodb resources. Once that's done you try to apply again, but you realize you have multiple root modules which all refer to each other's state (because "super-duper-DRY IaC" etc) so you have to apply them in the right sequence. You also have to modify certain values in between, like VPC IDs, subnets, regions and availability zones, etc.
You find odd errors that you didn't expect, and re-learn the manual processes required for new AWS accounts, such as requesting AWS support to allow you to generate certs for your domains with ACM, manually approving the use of marketplace AMIs, and requesting service limit increases that prod depended on (to say nothing of weird things like DirectConnect to your enterprise routers).
Because you made literally everything into Terraform (CloudWatch alerts, Lambda recurring jobs, CloudTrail trails logging to S3 buckets, governance integrations, PrivateLink endpoints, even app deployments into ECS!) all the infrastructure now exists. But nothing is running. It turns out there were tons of whitelisted address ranges needed to connect with various services both internal and external, so now you need to track down all those services whose public and private subnets have changed and modify them, and probably tell the enterprise network team to update some firewalls. You also find your credentials didn't make it over, so you have to track down each of the credentials you used to use and re-generate them. Hope you kept a backed up encrypted key store, and backed up your kms customer key.
All in all, your DR plan turns out to require lots of manual intervention. By re-doing DR over and over again with a fresh account, you finally learn how to automate 90% of it. It takes you several months of coordinating with various teams to do this all, which you pay for with the extra headcount of an experienced cloud admin and a sizeable budget accounting gave you to spend solely on engineering best practices and DR for an event which may never happen.
....Or you write down how it all works and keep backups, and DR will just be three days of everyone running around with their heads cut off. Which is what 99% of people do, because real disaster is pretty rare.
This is kind of what I'm talking about WRT the cloud being more trouble than it's worth if you app sits somewhere in between "trivial enough you can copy-paste some cloud configs then never touch them" on the one end and "so incredibly well-resourced you can hire three or more actual honest-to-god cloud experts to run everything, full time". Unless you have requirements extreme/weird enough that you're both not-well-resourced but also need the cloud to practically get off the ground, in which case, god help you. I think the companies in that middle ground who are "doing cloud" are mostly misguided and burning cash & harming uptime while thinking they're saving and improving them, respectively.
Generally: minimize dependencies. External library or API dependencies? Versions can drift, the system can change out from under you in incompatible ways. That goes for the OS, too, of course. Data dependency that you aren't 100% in control of? Same. All are forms of state, really. Code of the form "take thing, do thing, return thing, halt" (functional, if you like—describes an awful lot of your standard unixy command line tools) is practically eternal if statically compiled, as long as you can execute the binary. Longer, if the code and compiler are available.