Its kind of crazy that the knee jerk reaction to failing to one shot your prompt is to abandon the whole thing because you think the tool sucks. It very well might, but it could also be user error or a number of other things. There wouldn't be a good nights sleep in sight if I knew an LLM was running rampant all over production code in an effort to "scale it".
There’s always a trade off in terms of alternative approaches. So I don’t think it’s “crazy” that if one fails you switch to a different one. Sure, sometimes persistence can pay off, but not always.
Like if I go to a restaurant for the first time and the item I order is bad, could I go back and try something else? Perhaps, but I could also go somewhere else.
I'm okay with writing developer docs in the form of agent instructions, those are useful for humans too. If they start to get oddly specific or sound mental, then it's obviously the tool at fault.
The OMSCS degree you get is equivalent to the in person one, so there is no way to make the distinction in an interview. I actually don’t see how people see that an experience like this brings no value, given the rigor of the assignments. One certainly would come out with a better knowledge of how things work, develop a better work ethic, and hopefully make some network connections on the way…
The whole point is, if an LLM can easily complete rigoruous assignments and all the student has to do is add a little bit of personalization to the output, then has that student really learned anything? Can they evem come up with a plan to do such tasks without the LLM, even if it takes a lot longer without it?
Educational certifications in the era of LLMs are going to be increasingly meaningless without proof-of-work, and that's going to mean in-class work without access to computational aids, if you really want to evaluate a person's skill level. This of course is the coding interview rationalization - CS students have been gaming auto-graded courses created by CS professors for some decades, and now that's easier than ever.
There is absolutely no way you’re passing OMSCS tests if you’re winging it on the other assignments, and the tests usually account for over 50% of the grade. Certifications you’re right about but there are ways to test knowledge without asking for code snippets.
> there is no way to make the distinction in an interview
Just ask?
Some online degrees state that they're equivalent, but interviewers may still have their own opinions. I would discourage anyone from failing to mention the online nature of a degree in their CV. You're really not doing yourself a favor. A rigorous online degree is something to be proud of. I see people with PhD's proudly announcing their online course certificates on LinkedIn. However, 'discovering' that an education was of a different nature than one had assumed based on the presented materials may raise questions.
This just reeks of you being insecure and thinking online education is of lower quality than in person education. Are you also pining for everyone to go back to the office? The degree GT gives you is literally the same thing as the in person degree. If GT does not make the distinction, why would I???
That says nothing other than that the interviewers have a narrow mind and/or are ignorant. OMSCS is a very well known program, and it's their problem if they don't know it.
This is very debatable. The courses look like they were recorded in the 90s.
The DB course particularly sticks out. My undergrad's DB course was fathoms harder than this. This is what you'd expect a highschooler should be able to learn through a tutorial not a university course.
If it doesn't talk about systems calls like mmap, locking and the design of the buffer pool manager, it's not a university Database course it's a SQL and ER modelling tutorial.
Respectfully, I think you should do more research.
The OMSCS program is well known and well respected in the tech industry. It's a masters degree from the currently 8th ranked computer science school in the U.S.
The university make no distinction between students who take the courses online, vs in person. I.e., the diploma's are identical.
I’ve taken graduate-level courses in databases, including one on DBMS implementations and another on large-scale distributed systems, and I also spent two summers at Google working on Cloud SQL and Spanner. Database research goes further than DBMS implementation research. There is a lot of research on schemas, data representation, logic, type systems, and more. It’s just like how programming language research goes beyond compilers research.
I don't think watching the lectures is the hurdle that anyone at OMSCS is trying to jump. The program has a pretty low graduation rate, and the tests are known to be fairly difficult, which essentially requires the student to do work outside of class or go to the resources available through GT to understand the material. I can look up the highest quality lectures on any subject on YouTube, it doesn't mean I will understand any of it without the proper legwork.
FWIW I meant the diploma is identical, the actual experience will obviously vary. Some people will get better outcomes online, some will get better outcomes in person.
Is this a common thing to have at university? I'm from one of top universities in Poland; our database courses never included anything more than basic SQL where cursors were the absolute end. Even at Masters.
Do not worry, I do not work with databases in professional life as my main aspect. But I was not given a comprehensive education, and not even once there was a focus on anything more in depth. I came out without even knowing how databases work inside.
Naturally, I know what I could do - read a good book or go through open source projects, like Sqlite. But that knowledge was not was my uni gave me...
I am jealous of American/Canadian unis in this aspect.
OMSCS student here. You are absolutely right that the DB course is one of the weaker offerings. There is a newer Database System Implementation course, which is based on Andy Pavlo's excellent undergrad course (which is also available online), but only the first half or so of that course is covered, which is disappointing for a graduate course. In terms of the larger program, however, the two database courses are outliers and most courses are of much higher quality and definitely not undergrad level.
Hey — head TA of DSI here and want to correct some misconceptions.
DSI (6422) is taught by Andy Pavlo’s first PhD student who help to create the CMU course and a rather famous DB person. It is the same contents as the on-campus course (and were actually working to deepen/increase the depth of coverage). It’s designed to bridge between DB Theory and reading Postgres or MySql source code when it comes to DB designs and trade-offs — and covers topics like r-tries which I don’t think is covered elsewhere + a series of 12 seminal DB papers. As in any other grad-level class, you get out as much as you put in — and it’s super rare to have access to a DB researcher like Joy or hear his takes on DB development as a student at scale.
If anything, the feedback we’ve gotten from both on campus undergrad and MS students is that the OMSCS lectures + improvements are making their session more rigorous.
We actually launched a new class (CS 6422) that addresses exactly this and taught by Andy Pavlo’s first PhD student :) OMSCS db classes reviews are outdated IMO
I get the sentiment though. Happiness is a mix of the right hormones firing, so the question is: how does intelligence affect different types of hormones, if at all. Given how sensitive our hormones are, it would be difficult to control only for “intelligence”.
By that logic, "How does loved one dying affect different type of hormones, if at all. Given how sensitive our hormones are, it would be difficult to control only for 'loss of a loved one'".
If you have depression or another condition affecting your affect and your emotions, sure. Otherwise it's pretty obvious to anyone that concepts on orders of magnitude higher levels than hormones being correlated with happiness, or if you prefer, those concepts having a significant effect on the overall action of those hormones.
Loss of a loved one is a very specific event, intelligence is a very broad idea that isn't even well defined. I have no idea what this means: "concepts on orders of magnitude higher levels than hormones". I guess your intelligence is just much higher than mine.
Knowing almost nothing about memory and the brain, I don't know if I agree with "Every memory your brain has ever produced is still there".
Memories seem to be constructed by a group of neurons together, and it seems clear that neurodegeneration is a thing, whether by trauma or due to aging. When pathways degenerate, maybe you have a partial memory that you brain can help fill the gaps with(and often incorrectly), but that does not make it the original memory.
"If there’s someone out there who just sits on the beach all day and survives by eating coconuts"
This seems like a dire situation for most humans and probably isn't the status quo for human operation. Its perfectly fine to sit on the beach, but eventually a light should go on that would say "lets go build some tools", "lets go see if I can catch a fish", "lets build a hut". Which would all be fine and admirable things to spend your time doing.
I would guess a lot of intrinsic motivation is driven a lot by hormones and things like dopamine release when you do something that is really interesting or exciting. I play tennis and absolutely love it, I will never make a dime from doing it, and that's totally okay with me.
Things that are built with money are often done so for scale. Successful things that are built with money often also have people who have some interest in the thing they are building.
Thanks for helping to refine the thinking. I guess the other side of the coin that would be make the paradox interesting is that, in the long term, it has to seem that most of the stuff built with money (but without intrinsically motivated managers) lose out to the stuff built on pure passion. After discounting for a heap of survival bias.
Yeah I guess it really depends on what the measure for success is. As with all things, the answer is probably that products that work really well or are beloved are often a combination of money and passion. Unfortunately, once the product/company reaches a certain threshold, it seems to get bought out and there is only money left, and thus it becomes crap.
If LLMs still produce code that is eventually compiled down to a very low level...that would mean it can be checked and verified, the process just has additional steps.
JavaScript has a ton of behavior that is very uncertain at times and I'm sure many JS developers would agree that trusting what you're standing on is at times difficult. There is also a large percentage of developers that don't mathematically verify their code, so the verification is kind of moot in those cases, hence bugs.
The current world of LLM code generation lacks the verification you are looking for, however I am guessing that these tools will soon emerge in the market. For now, building as incrementally as possible and having good tests seems to be a decent path forward.
There are 4 important components to describing a compiler. The source language, the target language, and the meaning (semantics in compiler-speak) of both those languages.
We call a C->asm compiler "correct" if the meaning of every valid C program turns into an assembly program with equivalent meaning.
The reason LLMs don't work like other compilers is not that they're non-deterministic, it's that the source language is ambiguous.
LLMs can never be "correct" compilers, because there's no definite meaning assigned to english. Even if english had precise meaning, LLMs will never be able to accurately turn any arbitary english description into a C program.
Imagine how painful development would be if compilers produced incorrect assembly for 1% of all inputs.
English does have precise meaning, if constructed to be precise, the issue is that LLMs do not assign meaning in the way humans assign meaning. Humans assign English meaning to code every day just fine, and sometimes it does result in bugs as well.
The LLM in this loop is the equivalent of a human, which also has ambiguous source language if we’re going by your theory of English being ambiguous. So it sounds like you’re saying that if a human produces a C program, it is not verifiable and testable because the human used an ambiguous source language?
I guess for some reason people thought I meant that the compiler would be LLM > machine code, where actually I meant the compiler would still be whatever language the LLM produces down to machine code. Its just that the language the LLM produces can be checked through things like TDD or a human, etc...
I would probably agree! I came off sounding as if there is no human in the loop. What I meant is that input is still the programming language that is produced and output is the result. Not that the LLM is the initial input. A human in the loop can clean the code produced or create tests that check for an end result(or intermediate results as well).
I understand that an input to an LLM will create a different result in many cases, making the output not deterministic, but that doesn’t mean we can’t use probability to arrive to results eventually.
I mean the things _producing_ the code can be checked and verified, meaning the code generated is guaranteed to be correct. You're talking about verifying the code _produced_. That's the big difference.
Would be curious as to how you check and verify LLMs? And how you get guaranteed correct code?
Verifying code produced is a much simpler task for some code because I, as a human, can look at a generated snippet and reason about it and determine if it is what I want. I can also create tests to say “does this code have this effect on some variable” and then proceed to run the test.
Sooo you launched https://sibylline.dev/, which looks like a bunch of AI slop, then spun up a bunch of GitHub repos, seeded them with more AI slop, and tout that you're shipping 500,000 lines of code?
reply