>> The way I interpret DKE is that low level people don’t (yet) know what they don’t know.
Exactly, it's like people who think the company CEO is a useless idiot because they don't see him writing React code or whatever their limited definition of "adding value" is.
Another example I see on HN - whenever a hiring thread comes up, people chime in how stupid whiteboard interviews are, and surely the poster would be a super-star at a FAANG if only they weren't gate-kept by these dumb interviews. Isn't it more likely that these super-successful companies use these interviews because they end up hiring the people they want to hire, and it's you, the poster, lacking a clue as to what that looks like?
Basically the heuristic is, if something/someone is successful and it seems stupid to you, start by assuming it's you who likely has something to learn.
Glad you brought up the hiring threads on HN. Recently I've felt so many of these threads have devolved into "these stupid companies using whiteboard hiring are all a bunch of idiots!" that I usually stop commenting.
What's most illuminating about those kinds of comments is they rarely even try to view thing from the hiring company's perspective. Having been in that boat, you'll find:
1. The cost of a mishire is huge. And, even worse than hiring someone who is downright awful is hiring someone who is just plain mediocre, or slow. I mean, I've hired diligent, hard-working, otherwise smart people who could only get stuff out the door at about a third the rate of other devs. Having a performance discussion with those folks can be extremely difficult, because oftentimes there is little they can realistically do to go faster.
2. Experience can have little correlation with proficiency. I've interviewed people who were "senior engineers" and "architects" at major corporation who couldn't code FizBuzz, moreover they could barely code a syntactically correct function in their chosen language.
3. Many companies want to diversify their employee base by going outside just friends/referrals and well known college programs, but to "take a chance" on other folks means they need to get a strong signal during the interview process.
I'm not saying there aren't problems in hiring, but given how there is a giant economic incentive to make hiring as accurate and efficient as possible it's at least valuable to attempt to understand why a company might interview the way it does.
Hiring threads are a microcosm of the classic forum disease: people who actually know what they’re talking about are busy doing it for a living, while the malcontents and loudmouths have all day to complain on the internet. HN is one of the better forums when it comes to avoiding that outcome but still not perfect.
Number 1 wouldn't have to be a problem if we changed the social contract. The literal contract (nearly always) states your employment is at will. You can be let go at any time, for any reason or no reason at all. Employers should be more willing to (1) take a risk on a hire, and (2) let them go if it doesn't work out.
Then, hiring doesn't have to be this grandiose, talk to 20 people over 2 days in 20 30-minute session, but actually the VP has a final call, sort of decisions.
JMTQp8lwXL, LOL. So your proposal is instead of thoroughly interviewing, just give jobs out randomly and then fire people?
That works aside for the fact that it's horrible for everyone:
- The employee: they presumably quit another job to take this one and now they are out on their ass because the employer didn't bother to test for proper fit.
- The team: they invest in training the new person, get to like them, only to have them fired. Meanwhile they are short a person as hiring effort has to be restarted so they lose a ton of time understaffed.
- The company: dealing with all this terrible churn, stopping and starting the hiring effort, is there still severance in this world? How does unemployment work?
This is just the craziest proposal I've ever heard. It's like saying "dating is too onerous, people should just marry randos and not hesitate to get divorced if it turns out the other person brushes their teeth in an annoying way."
I certainly don't think this is the craziest proposal I've ever heard. Netflix, for example, is quite transparent that they have a "hire and fire" culture, and I for one wish that many tech companies would be more willing to have open, honest conversations with their employees and ensure everyone is up to whatever bar they set. I've seen it become incredibly demoralizing for the rest of the team when subpar performers don't get let go, and I've also seen rather disingenuous uses of "layoffs" as an excuse to finally cut people who should have been cut long ago.
Still, there are practicalities that can make "hire fast, fire fast" difficult in the real world. First, many countries outside the US have much more onerous requirements to fire someone. Even in the US, it's quite trivial for any employee to take legal action if they think they've been let go unfairly, so most HR departments will require many months of detailed documentation (e.g. email conversation, performance reviews, PIPs) before someone can be let go.
I think you may be mis-representing what I am saying. You can generally sense if someone could be a possibly good fit in 20 minutes of conversation. The ROI of an additional 500 minutes of conversation doesn't change the error bars. They could be great leetcoders, but not a good colleague. Hopefully the first 20 minutes gave you a sense of their soft skills, but even those can be mis-represented in small bursts that are un-representative of how the individual typically communicates on the job.
You're saying I am saying we should randomly give jobs. I am not. I am saying it should be, at max, a 1 hour phone call. Background checks can verify past employment and job titles. Just talk to them about work matters. Ask a fizzbuzz if you must. No additional data will meaningfully tell you much more anyways.
> You can generally sense if someone could be a possibly good fit in 20 minutes of conversation.
The last position I hired for had 700 applicants. Paper filtered down to 75. Interview screened down to 30. Detailed interviews with 5.
How would you propose this work? 30 people seemed like a possibly good fit after 20 minutes. Should we randomly hire one? Should we hire all and have them fight it out madmax thunder dome style?
I think if the position has low startup time and is assembly line work or something we could hire all 30 and fire the poor performers. Outside of tech support, I’m not aware of any positions like this.
I’ve worked with managers who thought programmers were like this and would hire 10 or 20 at a time to see who stuck. It was very frustrating as an existing employee because they all had to be trained, reviewed, etc. After a year of arguing, I (and every other senior, over about two years) left that company.
I think I need to update my mental models around the tech job market here, and what the interview process is trying to achieve. To take this example further: of the 700 initial applicants, how many would be _adequate_ for the position? You had the opportunity to have quite detailed interviews from the pick of the bunch - is the goal to find the very best of the lot because you can afford to be picky; or there are genuinely only 3-4 in the 700 who could _adequately_ do the job?
My preexisting mental model is that from the 700 applicants, 30 could have filled the position adequately, with perhaps a ~5% gain across them. Each additional step in the interview process yields increasingly diminishing returns but is still worth it because ... you can choose to be picky in this market. The risks associated with a mishire are also presumably greatly reduced with each diminishingly selective step. Or is it that there genuinely exists only 2-3 people in that pool who are a good fit for the role, and you have little choice but to engage in kissing a whole load of frogs to find your prince?
I’m not sure how many could adequately do the job. Generally because the phone screen doesn’t give a good sense of this so the 30 I talked with maybe 5-10 of them could potentially do the job adequately.
We only extended an offer to one person and maybe a second if the first declined. If both of those two declined then we wouldn’t have offered to the other 3 detail interviewees.
I’d certainly like a better way that uses less time. But it does seem that I need to kiss a lot of frogs.
Also, the goal isn’t to get an “adequate” person but to get a really good or great person. I think that a good person can do 5x more than an ok person, maybe even more. So for this position, I’d rather keep looking than just get a body.
Again, if I just needed assembly line workers or warm bodies that could bill in a consultant sweatshop, that’s a different story. But I don’t want to be in such a hiring position.
>I think that a good person can do 5x more than an ok person, maybe even more
This is really surprising to me, and I'm not sure how this can be the case. I would imagine that this would be true of a high level position at the top of their game, able to define their own schedules & goals. For a typical worker with clearly defined short & long term objectives defined by the organization, how does one become 5X more productive? Clearly my heuristics about the tech industry are way out of step with reality.
However, I could see something like this happening in my own field: academia. As a bioinformatics postdoc, the bread & butter work most of my colleagues do is routine, so they can hit their goals fairly predictably. My project, in comparison, is mostly ad-hoc, struggling to parse a novel dataset right on the edge of what is technically possible. My productivity is in the toilet, and I can imagine a different researcher in my position being 5-10x more productive. This is not how I imagined the tech industry working, though.
I used to do a lot of programming and the multiples are really high, especially since bad programmers can have negative productivity (kind of like Hammerstein-Equord’s stupid and industrious quadrant).
I remember a McKinsey study from the 80s or 90s that talked about 10x productivity but I can’t find it so maybe I’m misremembering. I don’t think people who actually try to measure this succeed well. I run away from anyone trying to precisely measure programmer productivity because it usually means some point haired boss will try to optimize on lines of code or something stupid.
So for me, it’s a bit of a hunch but a strong hunch. I’ve worked in shops of 100 devs where a single dev wrote the entire authentication stack that teams couldn’t handle. And I have lots more stories like this.
Nowadays I do “strategy” and I’d say the multiple is more than 10x in that there’s usually some magic mix to a good strategy that can’t be accomplished with giant committees and tons of hours. When it comes to creative tasks, I think the productivity leap from “ok” to “good” and from “good” to “super” is really high. Not every job is like this, but I think these are the most fun and so try to go towards them and away from commodity work.
I have limited experience in academia but have seen authors quickly crank out really useful papers where teams have been working for months. I suspect that’s more about just having good alignment of capability and need instead of some magic or intrinsic productivity power.
Personally my productivity is pretty sucky so I’m in a situation where I don’t meet my criteria but am lucky enough to get to work on cool stuff.
Hmm. The way I see benchwork scientists being 'productive' is an endless slog - your cultures grow at their own schedule, and don't respect the boundaries of the work-life balance. Of course, this isn't true for everybody, but was one of the reasons pushing me away from benchwork towards bioinformatics. Our work also tends to be much more insular, gnawing away for years on our own little piece of the problem. I suppose I'm personally at a fairly "creative" point in my career - I have absolute free reign in my study, and my PI encourages it.
If the additional interviewing time doesn't produce real improvement in outcomes over random hiring, then yes, don't waste the interviewees’ time and time the form is paying you for doing it.
My problem is that I don’t know of a good way to determine what improvement. I think it’s hard to design a study and measure.
And I think there’s a deontological vs. utilitarian aspect as I think if applicants learned I was randomly picking 1 of 30 pretty goods it would discourage high quality applicants.
For me, I think the additional 100 hours interviewing helps and is worth it.
I’d be interested in hearing of hiring managers and orgs who just hire randomly from minimally qualified applicants. I hear from lots of applicants who claim they are minimally qualified that this is a successful strategy. But, naturally, they are a bit biased and not very useful for deciding how to hire people.
In this and your previous post, you greatly under-appreciate the cost and disruption of a bad hire. Not to mention the time lost bringing a new hire up-to-speed on the company's way of doing things. A bad hire can be very expensive and disruptive.
If a few more hours of interviews can avoid a bad hire, I think almost all experienced managers would opt for the extra interviews.
If your work culture allowed you to remove bad hires, then the company would not experience the 'cost and disruption'. The problem isn't the time invested getting employees up to speed: you'll do that with good or bad employees. The problem is, bad employees stay, and often for long periods of time. Have you experienced the bureaucratic nightmare of formally dismissing someone? It's something we all would like to avoid, but clearly today's filtering is ineffective if our organizations are filled with dead weight.
I've uncovered craziness in candidates late in the process when they are no longer guarded about what they're saying. So, for myself, it has indeed produced objectively better results. I expect that most seasoned managers have had similar experiences.
But how whiteboard interviews help (1)? I see them as unconnected and would prefer for that (for example) giving a sample assignment for 2 hours on/off-site
I hope this insight helps you and others because it's key:
What does an off-site sample assignment tell the company? That you can write code when given a spec. That's great, but that's insufficient for what they are hiring for.
What it doesn't tell them:
- Can you elicit clarity on a vaguely stated problem? These are conversations that top engineers have with their business counterparts and other technologists constantly.
- Can you iterate with another person on a solution. The whiteboarding exercise is a dialogue, and while contrived, problems are often solved in joint manner like that.
- How do you do under pressure? Something is going wrong and it's affecting millions of users. Can you carry your weight when the team is fire fighting?
Obviously the whiteboard isn't perfectly correlated with the above, but it's as good an indicator as I can think of. Saying "just let me write something off-line" means people are oblivious to these other key attributes. FAANG devs aren't earning 300K a year for just being able to write code on a spec.
> Isn't it more likely that these super-successful companies use these interviews because they end up hiring the people they want to hire...?
I’m not sure it’s that these companies only hire the people they want to hire. It’s probably that highly successful companies know their interview methods don’t work 100% of the time, but they’re willing to tolerate the false negatives of not hiring a few good people if it means all those they do hire meet some minimum standards. And their current system is the best they know how to do.
We arrogantly post on Hacker News: “Oh look, this interview method sucks! It will miss out on this great candidate!”. Ok then, make your own trillion dollar company with a better interview method that filters out the hordes of bullshit artists while simultaneously never missing out on the best people. Filtering out the bullshit artists is more important than hiring every single productive person. In fact, if you hire too many bullshit artists, all the good people will get fed up and leave anyway.
It's still a frustrating experience for candidates though. If you were to split the candidates into three buckets, with the first bucket being people who would never pass a FAANG interview (can't code their way out of a wet paper bag), people who can pass a FAANG interview but not reliably, and people who can reliably pass FAANG interviews (PhD in computer science or just leetcode grinder), most of the people in the first two buckets would complain about the interview process.
As someone in the second bucket, the process feels rather random (eg. passed the Facebook L6 interview but failed the Google L4 one somehow, passed the onsite at half a dozen other companies), which is mostly because it is. Maybe some interviewer really wanted to see a topological sort algorithm implemented in 20 minutes and there's not enough time in the interview to derive it from first principles, or maybe an interviewer felt grumpy that day, and then that's it, better luck next time. Combined with the policy of not telling candidates what to improve on, it's a pretty frustrating experience.
Yes, there are reasons for the company to behave that way, but it's still unpleasant to be on the receiving end of it.
Although I agree in general, the example you've chosen might not be the best one. This might be one of (numerous) areas in business where success is not measured correctly and the methods used are never really validated.
Do human resources departments actually evaluate their hiring criteria and the tests they use in experimental designs, e.g. by hiring two sufficiently large groups A and B based on different interview methods and checking a year later how the groups performed relative to each other?
Bear in mind that there are plenty of people who believe that they're good at doing something just because they have been doing it for a long time. Without an independent method for confirming such claims they are practically useless.
First, candidates who complain about the whiteboard interviews uniformly complain about it "not being a good representation of how I can write code" which betrays a lack of understanding of the other skills besides writing code that these firms care about. People who understand more can connect the interview type to the other relevant attributes, so it's already clear who's more right.
Second, yes, companies do a shitload of triangulation on recruiting performance including looking many years out.
In theory, I like your idea, but in practice, I think corporations large enough to run experiments like that are not able to objectively evaluate anything they do, at least not officially. It's too political.
I've been in dozens of interviews as an interviewer and worked with at least a dozen people after seeing how they performed in interviews, so that's what I go on.
The big software firm I worked at (not FAANG but we competed with them for talent) absolutely tweaked its interview processes based on long run outcomes.
FAANGS evolve their process over time too.
One clear example - at FAANGS even if the manager likes you, it doesn't matter. You have to be evaluated by a committee of people who're indifferent to your manager, aren't under the gun to fill a slot and are less likely to be swayed by a one-time positive conversation.
Do you think this process came out of nowhere? That's just one example of them settling on something that worked, I am sure having previously tried other less structured approaches and tracking the results.
> Isn't it more likely that these super-successful companies use these interviews because they end up hiring the people they want to hire, and it's you, the poster, lacking a clue as to what that looks like?
No it's more likely that the people staffing FAANG companies absolutely suck at interviewing and vetting.
Exactly, it's like people who think the company CEO is a useless idiot because they don't see him writing React code or whatever their limited definition of "adding value" is.
Another example I see on HN - whenever a hiring thread comes up, people chime in how stupid whiteboard interviews are, and surely the poster would be a super-star at a FAANG if only they weren't gate-kept by these dumb interviews. Isn't it more likely that these super-successful companies use these interviews because they end up hiring the people they want to hire, and it's you, the poster, lacking a clue as to what that looks like?
Basically the heuristic is, if something/someone is successful and it seems stupid to you, start by assuming it's you who likely has something to learn.