Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

While not about AI or the algorithm mentioned, on the subject of little errors that you can't convince anyone are errors....

In 2011, I wanted to copy the reddit ranking algorithm in a project of my own, so I went to source code to look at it... the algorithm in the source code I found wasn't doing anything at all sensible with negative-sum voted posts.

I thought I discovered the error, some terms swapped in the simple equation, the sign for positive/negative was misapplied.

I blogged it, and [posted it to reddit](https://www.reddit.com/r/programming/comments/td4tz/reddits_...), only to have MANY people, including reddit employees, tell me I am definitely definitely wrong, and the algorithm was working as intended. And that I was in fact not the first to notice what I thought I noticed, and point it out, and be told by everyone I was wrong.

OK, I didn't really understand what was going on, I couldn't make sense of the algorithm if it wasn't wrong, but so be it. I updated my blog post to say that people smarter than me said there was no error in the reddit algorithm, all I can say is this variation makes more sense to me.

Then, three years later in 2014, a commit was made to the reddit source code with exactly the correction I (and others before me) had suggested all along. The one that everyone piled on to tell me how dare I have the temerity to suggest reddit source code is wrong.

https://github.com/reddit-archive/reddit/commit/50d35de04b92...

¯\_(ツ)_/¯

Open source means there are lots of eyes that can find bugs, but sometimes they can't convince anyone they've found a bug. (And of course, then reddit close-sourced their code in 2017).

I never did end up using the ranking feature in my own project, that I had wanted to copy from reddit. I didn't end adding "vote" features to the app.



When I was an intern at Yahoo working on OAuth back in 2008 (2007? It was long ago and I'm old) I had the pleasure of implementing an internal tool for generating OAuth 1.0 URLs, which meant encoding a lot of things in query parameters. My tool did not generate URLs which were compatible with Yahoo's implementation (certain parameters effectively should be encoded twice, which my tool did). The implementing engineer insisted my tool was wrong, cited my status as a lowly intern, and even pulled out the OAuth spec and bent over backwards to say how his implementation was correct and I'm clearly reading it wrong. It literally took bringing in Eran Hammer-Lahav to weigh in on the topic to say I was correct, at which point the engineer agreed that of course that was correct. I got zero acknowledgment or apology for the days of ad hominem attacks against me.

I did learn an important lesson that more senior people are not always right, and as someone who's usually more senior than my colleagues now I try to remember it daily.


> It literally took bringing in Eran Hammer-Lahav to weigh in on the topic to say I was correct, at which point the engineer agreed that of course that was correct. I got zero acknowledgment or apology for the days of ad hominem attacks against me.

If it weren’t for the torturous gaslighting, this is borderline hilarious. Appeal-to-authority types have a way of submitting so effortlessly when a grander poobah comes around. Spine made of jelly.


I work at a FAANG and it was absolutely astonishing to find out how often this happens.

You can make a long, impactful career by just being "the guy who adds log statements throughout the codebase and reasons through it", doing this at even a simplistic level has always shown me an astonishing fix to some long-standing issue.

n.b. It also attracts a ton of political fun. People's first order reaction is denial, and it only gets worse from there. Absolutely no one except 1-2 colleagues will see it as "oh we should fix that", and at least one person will make sure your boss' boss' boss is CCd on an email with a nice version of "no he's just insufficiently concerned about {concurrency, memory management, take your pick}" Just wait it out quietly when that happens, do not engage or complain. If nothing happens and you're never asked about it by leadership, but your peers ask, make plans to move onto another team.


A long impactful career, or a career of horrible frustration and alienation as everyone gets mad at you for pointing out their bugs? (or, from their point of view, making trouble insisting that something is a bug which isn't and is causing no problems)


What FAANG have you seen this at?

I've been at big tech companies for most of my career and I've never seen anyone deny the existence of a technical bug. I've seen plenty of teams mark a bug as lower priority and never fix it because other things are higher priority. But denying that the bug exists, especially after a detailed explanation? That doesn't resonate with my experiences.


I've told this story before!

It used to be writing the outputs from the C/C++ preprocessor (.i files) to disk took forever (5+ minutes IIRC) with Microsoft's compilers. I asked one of the lead compiler developers why, and he waved me away saying it was just really complicated. Around that time a bunch of tools existed for GCC that worked with .i files, but none existed in the Microsoft ecosystem likely because writing .i files was so slow.

I was on the compiler test team at the time and we did lots of stuff with .i files, our tests were distributed across a large cluster of test machines (see my post about that https://meanderingthoughts.hashnode.dev/how-microsoft-tested...) so it wasn't a big deal, but it still annoyed me.

One day I decided to find out what was going on, so I loaded up process monitor while outputting a .i file and watched what was happening. Much to my surprise, only 1 byte was being written at a time! No wonder writes were taking forever.

A quick dive into the source code revealed a comment above the file write call that read to the effect

// to work around a bug in windows 98

So anyway I opened a bug against the compiler saying we should probably fix that. :)


But that's not the type of story that's being claimed from the person I responded to.

Of course the lead developer waved you off. You wondered why things took forever, and the lead developer knew it was a complicated system and figured it wasn't worth their time investigating. It happened to be incorrect, but the lead developer wasn't in denial. They just filtered the issue out because they can't afford to go down every rabbit-hole they come across. I'm sure once you found the actual bug, it was later fixed.

The person I was responding to seems to think a large number of people are in denial when a bug is filed against them. That doesn't make sense, and isn't something I see. It'd be as if when you pointed out the actual bug, the lead developer continued to say it wasn't actually a bug (which is of course ridiculous and I bet didn't happen).


You are ascribing an absurdly maximalist viewpoint to me, one that would be obviously wrong at its face.

I know it's not so confusing as to get that sort of interpretation, because of the score on the comment, and comments like the above that explain to you how this happens.

As a result, I don't feel comfortable providing more detail publicly about my situation. That far off the mark tends to indicate an aggressive rather than curious interlocutor.

I am comfortable building on their example. The particulars of the issue are quite similar in a very helpful way.

I did the investigation, did a fix, worked it up to my manager and my managers manager. Elated, we work diligently for a couple weeks to document concisely, 3 page tech doc, briefest code deltas possible, one page + slides withs simple diagrams.

It gets bogged down at managers managers coleads submanager for the platform team implicated. They basically say "reading the single byte at a time means its provably serial and thus has no concurrency bugs.", as indicated in my original comment.


It's 2 anecdotes to your 1. Anecdotes are useless, but you are down by 1. I suggest you call for reinforcements, or make a hasty retreat.


I used to work on a large-ish open source project. When I was bored I used to go bug-picking (not hunting, picking). I'd go and browse the source wherever my intuition told me there was likely a bug and I indeed found a few using with such a "method".


Wow, that must have been frustrating. I just looked at the code and it's just so clearly wrong.


Oh, it was! And there was no way to "prove" it, it's just like "look, the equation this way actually makes a lot of sense and is very clever and produces results that seem reasonable and much respect to whoever designed it... and the equation the way it is in code now... does not. It seems like clearly it got accidentally transposed at some point?"

And the response was just like "We disagree, we think it makes sense the way it is and the product is correct"

That's kind of the end of the argument, there's nothing more one can say!

It didn't help that I came in assuming that of course everyone would see which version was correct (as you just did! although i didn't find it obvious, it took me lots of study to figure out), instead of producing a narrative designed to gently persuade them that. (That's on me -- I think I've learned something about technical communications around bugs and disagreements since then, although I'm still far from perfect).

The real answer, I think was given in one of the reddit thread comments -- the way it's broken for the most part _doesn't matter_ in the usual operations of reddit, it matters only in edge cases, and not very important ones, so really people mostly don't notice and we don't care.

Fair enough, I guess? But they did fix it three years later? I forget how I even found out they had fixed it; I can't at this point find any context for _why_ they fixed it, or who with power finally noticed/agreed it could use fixing why.

(And if it had happened three years after that, it would not have been in public source, and my gloating satisfaction would have been stolen!)


In light of the past couple of months, I guess I should not surprised that the interaction with reddit staff went that way.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: