I definitely agree on the importance of personalized benchmarks for really feeling when, where and how much progress is occurring. The standard benchmarks are important, but it’s hard to really feel what a 5% improvement in X exam means beyond hype. I have a few projects across domains that I’ve been working on since ChatGPT 3 launched and I quickly give them a try on each new model release. Despite popular opinion, I could really tell a huge difference between GPT 4 and 5 , but nothing compared to the current delta between 5.1 and Gemini 3 Pro…
TLDR; I don’t think personal benchmarks should replace the official ones of course, but I think the former are invaluable for building your intuition about the rate of AI progress beyond hype.
Yes, it’s a very narrow-minded perspective that cannot understand the second-order implications of this development beyond their own experience as an experienced developer. For argument, let’s imagine that the quality of software at the top valley firms is just phenomenal (a stretch, as we all know, even as a hypothetical). That is obviously not the case for the quality of software at 99% of firms. One could argue that the dominance of SaaS this past decade is an artifact of the software labor market: any vaguely talented engineer could easily get a ridiculously well-paid position in the valley for a firm that sold software at great margins to all the other firms that were effectively priced out of the market for engineers. I think the most interesting case study of this is actually the gaming industry, since it’s a highly technical engineering domain where margins are quickly eroded by paying the actual market wage for enough engineers to ship a good product, leading to the decline of AAA studios. Carmack’s career trajectory from gaming industry to Meta is paradigmatic of the generational shift, here.
TLDR; in my opinion, the interesting question is less what happens at the top firms or to top engineers than what happens as the rest of the world gains access to engineering skills well above the previous floor at a reasonable price point.
Prompting is not engineering nor a skill let alone a whole engineering skill. Excel has been around democratizing programming for the businesses of any kind and people of any kind and created a lot of value, i believe it's a great product YET it didn't lowered the need of engineering people... the contrary
Indeed. Similarly, I like now having ChatGPT as the absolute lowest bar of service I’m willing to accept. If you aren’t a better therapist, lawyer or programmer than it then why would I think about paying you.
Having chatgpt as your therapist is actually wild.
Its yes-man abilities etc. will not help you and its always helpful to talk to another human being in the loop during therapy so that you know you are actually being sane.
Don't ever compromise money on your mental health.
If you need to, either join offline anonymous support groups or join some good forums/discord servers/reddit about therapy if you can not even afford that. They are good enough and I think personally for me, whats therapatic is to try to understand and help the other person who was struggling just as I once was y'know
But saying to use chatgpt as a therapist is just something that I hate a lot simply because in my opinion, it might actually even be harmful but I am not sure.
I've looked at the data for some of the Russell Group and, coming from a US perspective, I was rather shocked at how reliant even top UK universities are on tuition. Apart from Oxbridge, they mostly don't have anywhere near the cashflow from endowments or alumni donations as the US Ivy League does.
For this reason, one of the most fascinating historical relics to me are the Incan Quipu [0]. Not only because their logic appears to be 'proto-computational' (at the very least a very complex system of encoding numeric and narrative information through sequences of knots that were also used directly for calculations), but since, neither in the form of a valuable material like gold nor obviously a book to be destroyed, a large enough number survived to this day. There are few traces of the past we know exist that might contain everything from astronomical calculations to old social-institutional histories.
They're comparable in that sense to the Heculaneum manuscripts, which researchers have lately made great progress on with deep learning [1]. I hope an equivalent initiative someday starts on the Quipu.
> narrative information through sequences of knots
It's unlikely any complex narratives could be encoded there, though.
Possibly a bit like the e.g. the Mycenaean Linear B script. Which is fully decoded and well understood. But despite having a full fledged writing system they mainly used it for accounting and such. They can tell us about how many goats, sheep and various good they had they had but don't tell as much about the society or history as such.
Heculaneum manuscripts are kind of the opposite from the Quipu in the sense that we have zero issues understanding the actual text/symbols just extracting them from the destroyed scrolls is rather complicated.
Minoan tablets are maybe a bit closer and little progress has been made there (then again we are fully capable of reading the script just have no clue about the language it was written in).
I agree with your point about the Heculaneum, but my understanding is we're far enough with research into the form of the Quipus to know that they aren't simply either a linear script or merely accounting data.
For a long time, it was thought that they indeed contained only the latter, but my certainly non-specialist grasp on the matter is that we now know they were used to encode much more than that. In addition to being used directly to verify calculations [0], they contained "histories, laws and ceremonies, and economic accounts" or, as the Spanish testified at the time: "[W]hatever books can tell of histories and laws and ceremonies and accounts of business is all supplied by the quipus so accurately that the result is astonishing"[1]. My---again crude---understanding is this was through embedding categories, names and relational data in addition to numbers, signaled not least through texture and color [2].
I likely come across as if I'm trying to over-inflate the Incan knots, but really it's just to say they appear to be a rather fascinating in-between of legal-administrative inscriptions, whose discovery transformed understanding of Roman institutions over the last century or so, and the straight-forward manuscripts of the Herculaneum.
[0] My elementary and probably out-of-date recollection: an emissary would come to towns with Quipus containing work orders, which would be validated with the community on the spot.
My immediate observation when I first learned of the Quipu and its use: nodes and edges.
Potentially a graph to be completed by the owner via verbal communication/interpretation as a supplement to the material instrument; a single source of information that could be interpreted differently depending on the societal role and vocation of the owner.
I bought my dad a Mac laptop when I got my first job out of college and he used it for well over a decade. I even later got him a MacBook Air and he kept using the old one for years yet out of habit… I imagine that’s not an uncommon pattern for non-programmers who aren’t gamers.
Comically, though, programming communities really seem to have a statistical over representation of both aphantasics and hyperphantasics. One of these articles comes out every few years and I've witnessed at numerous workplaces how quickly a large portion of the engineers realize they're aphantasic and everyone else is aghast that they can't rotate complete architectural diagrams etc.
That said, it really is binary or not whether you cannot see images at all in your head and there are, in fact, some very real downsides related to episodic memory. As someone who realized I was aphantasic late in life, I think it's pretty important to realize you are if in fact you are---ideally as early in your educational process as possible. For everyone else, it's interesting to realize some people have more vivid imagery than you and some people less, but probably that doesn't change very much about your life.
For me this is the other way round. When I was a student (physics) I had a very, for a lack of a better word, "practical" visualization in my head - what I needed to understand what I was studying. There was a lot of maths too, visualized.
Today, 30 years later, I have vivid representations of calligraphy or art, especially when I fall asleep. I fall asleep within at worst minutes so I cannot really take full pleasure of watching these ilages and during the day I am too surrounded by sources of sound, images etc. to meaningfully repeat the exercise.
The _absence_ of visual imagery is binary: you cannot see images at all or, to whatever extent, you can. Those who do have any mental imagery at all, however, fall on a scale. There are numerous studies of certain real downsides to aphantasia, notably tied to episodic memory, which don't seem to be present in those simply with diminished visual imagery.
> I think it's pretty important to realize you are if in fact you are---ideally as early in your educational process as possible.
Is that because it’s hereditary or instead something that was missing in early childhood? Like as a toddler you were never given one of those games where you fit shapes into different sized holes for example?
The question of origin is still pretty unclear. There seems to be a tension between things that are more developmental (if you have mental imagery, for example, you seem to be able to get better or worse) and those that are likely genetic (research does suggest a connection between aphantasia and autism spectrum etc).
As someone said below, I suggested figuring it out early is best because of a lot of things that just work differently, especially in learning. There seems to be a real selection bias that most people who learned they were aphantasic reading a New Yorker article, say, by definition figured out how to make it work somewhere along the line. Aphantasia isn’t at all a learning disability in a real sense, but you definitely have to approach things differently.
I think it’s because you can find supports to help you learn.
I’ve been teaching math for almost 18 years at this point, and only a couple years ago learned that I lean towards aphantasia. Back in high school, geometry was HARD. Calc 3 was HARD. It was presented as visualize and imagine, and I tried my best.
It just turns out other people could do that, and the fuzzy thing thing (or, more commonly, the ‘bulleted list of information’ that make up my imagination) was not “normal.”
If I’d known this (and my teachers were in a position to also know this), then maybe we’d spend more time with external visual models (what Geogebra now does for us, for example) to help me out.
Now that I teach future high school math teachers, it’s definitely something I talk about to normalize “not everyone can see in their mind.”
Do you have any advice for an experienced engineer who is considering changing careers to teaching high school math? I hear horror stories about teaching kids nowadays, with most having smartphones in class and AI use being rampant. Do you think there’s truth to that, or is it overblown?
There is really a fundamental difference as many studies now have shown---and I can attest from personal experience. Honestly, if you have to ask the question there's a pretty high chance you are: everyone at some level believes that their own inner experience generalizes to the rest of humanity, but it's those with aphantasia who thereby believe that everyone else's description is just a manner of speaking ("they, like me, surely don't really think in pictures").
I find the typical thought experiment of "picture an apple" less illustrative than something like "picture the face of a co-worker you see every day but aren't friends with and tell me the color of their eyes." In the apple case, everyone has a "concept" of apple and an experience of "thinking about an apple"---the difference is really in what you can deduce from that thinking and how, if that make sense. Are you reasoning on the basis of an image or from more or less linguistic facts ("apples are red therefore..." etc)?
The main difference that's more than an "implementation" detail of how you think, so to speak, but really a limit concerns what's called "episodic memory." People with aphantasia rather singularly cannot re-experience the emotions of past experiences. There are a lot of studies on this and I can look up the references if you're interested.
When I was really trying to make sense of my own aphantasia, I found https://www.hurlburt.faculty.unlv.edu/codebook.html to be one of the most fascinating resources: it's essentially a catalog of all the different modalities of inner experience a large study found. Probably there are critiques of his methodology etc, but regardless it's an invaluable aid for trying to figure out how exactly you think.
No, certainly not. I was trying to pose a thought experiment that draws one's attention to the how of their thinking more than "think of an apple." Even if you can't figure out the person's eye color, did you bring to mind a blurred workplace image that just didn't have enough detail in the right place? For an aphantasic, especially if you don't even know this person's name, it's really a sort of experience of an empty thought in the way that thinking about an apple isn't.
Weak mental imagery and no visual imagery are distinct.
The connection of aphantasia to strongly deficient autobiographical memory (SDAM) is well-attested now. You can find numerous clinical studies on the matter.
Honestly, I wouldn't be surprised if OpenAI has done the math and determined that even releasing frontier quality models wouldn't put much of a dent in either their B2B or B2C businesses. Or, rather, that any such dent would be vastly overshadowed by the value of fending off potential competitors.
I haven't looked too much into Deepseek's actual business, but at least Mistral seemed to be positioning themselves as a professional services shop to integrate their own open-weight models, compliant with EU regulations etc, at a huge premium. Any firm that has the SOA open model could do the same and cannibalize OpenAI's B2B business---perhaps even eventually pivoting into B2C---especially if regulations, downtime or security issues make firms more cloud-skeptical with respect to AI. As long as OpenAI can establish and hold the lead for best open-weight/on-premise model, it will be hard for anyone to justify premium pricing so as to generate sufficient cash flow from training their own models.
I can even imagine OpenAI eventually deciding that B2C is so much more valuable to them than B2B that it's worth completely sinking the latter market...
TLDR; I don’t think personal benchmarks should replace the official ones of course, but I think the former are invaluable for building your intuition about the rate of AI progress beyond hype.