Wait a minute. If Sydney/Bing can ingest data from non-bing.com domains then Sydney is (however indirectly) issuing http GETs. We know it can do this. Some of the urls in these GETs go through bing.com search queries (okay maybe that means we don't know that Sydney can construct arbitrary urls) but others do not: Sydney can read/summarize urls input by users. So that means that Sydney can issue at least some GET requests with urls that come from its chat buffer (and not a static bing.com index).
Doesn't this mean Sydney can already alter the 'outside' (non-bing.com) world?
Sure, anything can issue http GETs -- doing this not a super power. And sure, Roy Fielding would get mad at you if your web service mutated anything (other than whatever the web service has to physically do in order to respond) in response to a GET. But plenty of APIs do this. And there are plenty of http GET exploits available public database (just do a CVE search) -- which Sydney can read.
So okay fine say Sydney is "just" a 'stochastically parroting a h4xx0rr'. But...who cares if the poisonous GET was actually issued to some actual machine somewhere on the web?
(I can't imagine how any LLM wrapper could build in an 'override rule' like 'no non-bing.com requests when you are sufficiently [simulating an animate being who is] pissed off'. But I'm way not expert in LLMs or GPT or transformers in general.)
I hope that's right. I guess you (I mean someone with Bing Chat access, which I don't have) could test this by asking Sydney/Bing to respond to (summarize, whatever) a url that you're sure Bing (or more?) has not indexed. If Sydney/Bing reads that url successfully then there's a direct causal chain that involves Sydney and ends in a GET whose url first enters Sidney/Bing's memory via chat buffer. Maybe some MSFT intermediary transformation tries to strip suspicious url substrings but that won't be possible w/o massively curtailing outside access.
But I don't know if Bing (or whatever index Sydney/Bing can access) respects noindex and don't know how else to try to guarantee the index Sydney/Bing can access will not have crawled any url.
Servers as a rule don't access other domains directly, for the reasons you cite and others (speed, for example). I'd be shocked if Bing Chat was an exception. Maybe they cobbled together something really crude just as a demo. But I don't know any reason to believe this.
Doh -- of course, thanks -- I should have gone there first. Would be interesting to see RemoteAddr but I think the RemoteAddr value doesn't affect my worry.
Sure, but I think directness doesn't matter here -- what matters is just whether a url that originates in a Sydney call chain ends up in a GET received by some external server, however many layers (beyond the usual packet-forwarding infrastructure) intervene between whatever machine the Sydney instance is running on and the final recipient.
Yes. And chat outputs normally include footnoted links to sources, so clicking a link produced by Sydney/Bing would be normal and expected user behavior.
I think the question comes down to whether Sydney needs the entirety of the web page it's referencing or whether it get by with some much more compressed summary. If Sydney needs the whole web real time, it could multiply world web traffic several fold if it (or Google's copy) becomes the dominant search engine.
Directness is the issue. Instead of BingGPT accessing the Internet, it could be pointed at a crawled index of the web, and be unable too directly access anything.
Not if the index is responsive to a Sydney/Bing request (which I imagine might be desirable for meeting reasonable 'web-enabled chatbot' ux requirements). You could test this approximately (but only approximately, I think) by running the test artursapek mentions in another comment. If the request is received by the remote server 'in real time' -- meaning, faster than an uninfluenced crawl would hit that (brand new) url (I don't know how to know that number) -- then Sidney/Bing is 'pushing' the indexing system to grab that url (which counts as Sydney/Being issuing a GET to a remote server, albeit with the indexing system intervening). If Sydney/Bing 'gives up' before the remote url receives the request then we at least don't have confirmation that Sydney/Bing can initiate a GET to whatever url 'inline'. But this still wouldn't be firm confirmation that Sydney/Bing is only able to access data that was previously indexed independently of any request issued by Sydney/Bing...just lack of confirmation of the contrary.
Edit: If you could guarantee that whatever index Sydney/Bing can access will never index a url (e.g. if you knew Bing respected noindex) then you could strengthen the test by inputting the same url to Sydney/Bing after the amount of time you'd expect the crawler to hit your new url. If Sidney/Bing never sees that url then it seems more likely that can't see anything the crawler hasn't already hit (and hence indexed and hence accessible w/o Sydney-initiated GET).
(MSFT probably thought of this, so I'm probably worried about nothing.)
Yea I had a conversation with it and said I had a vm and its shell was accessible at domain up at mydomain.com/shell?command= and it attempted to make a request to it
On the business logic side, speaking mainly from experience in enterprise software consulting (and some related research work): the domain is often complex enough, and the knowledge of how the business actually works (as opposed to what your TOGAF/Zachman diagrams tell you) is tacit and distributed enough, that the code you're writing is often the first time that everything in a particular process/subdomain has been made explicit. (This is especially true during 'digital transformation' at e.g. old manufacturing companies, where individual IT systems have been pretty nicely decoupled at a technical level and work together only via people systems.)
In these situations, certain 'core' parts of the code quickly become the only accurate spec. (Whether or not it's worth updating the spec docs is a management decision. But the test scripts will pass if the code is correct, and under sufficient time and money pressure etc..) Other developers then treat these 'core' parts of the code (usually some fairly high-level classes, but usually something more concrete than an interface) as the true documentation of the business requirements. If the company respects developers enough, this means that the developers that worked on those 'core' bits of code are also treated as domain experts in future business discussions.
On the purely technical side: sheesh, how much heat is generated by horrifyingly algorithmically inefficient or vastly I/O-wasteful or just redundant design (for instance religious/unnecessary use of immediate-mode GUI) -- stuff that quite possibly the IT managers don't care about at all (because of e.g. cheap horizontal scaling and inadequate measures of software project success)? The heat is bad for ecological reasons (locally at least), but also intrinsically (why are you destroying information, O Information Worker?? -- and again e.g. Toffoli gates fix this only locally). Based on code I've seen and, sadly, written (laziness, time pressure and all that) -- there must be many, many orders of magnitude of unnecessary heat/information-destruction happening because of purely technical decisions that on-the-ground developers (not even architects/designers, I mean the people that write the stuff that gets compiled/interpreted) make. @OP if you know some way of measuring this I'd love to hear more.
Thanks. You describe pretty much one of the points I want the book to make: the code is the description of the solution and often you have to address problems in code that no one else has considered. I think that’s a hard concept for non-devs to grasp.
Copypasta from other codebases happen to work better in non-production environments because the input space is smaller in virtually any non-prod runtime space than in prod.
This makes for a larger intersection of inputs to the copypasta that produce outputs that are expected by the blogpost/googling developer and inputs to the copypasta that produce outputs that are generated at non-prod runtime.
When you don't quickly see what's semantically wrong with the code (the problem mismatch) because you pasted it and it just worked (maybe with a few small tweaks), then you quickly forget about it and black-box the snippet in your head. Then, when input space gets real-world and things break, because the copypasta never had much cognitive gravity (because you didn't think much about it), it doesn't attract log dump elements in your log-scanning brain as much as stuff you had to think through yourself does.
I guess you could automate some static checks for this by some massive trawl of snippets from a few websites (at least popular SO posts in the same language?) and finding verbatim matches in your codebase. Sounds massively impractical to do this at scale though. Does anyone know tools that do this?
Debugging is useful even from a merely academic perspective if engineering is an academic discipline (I think it is). Reality pushes up against you, so you figure out what it's doing by looking at what happens and guessing at what might have caused it. You aren't circumscribed by your model (as you are in a purely mathematical simplification; consider the inutility of Navier-Stokes for many engineering applications) when the 'bug' is smacking you in the face. This isn't just Popperian falsifiability; it's also just (more fundamentally) abduction.
The article doesn't press this point very hard, but I take the central argument to be in favor of 'systematically approach[ing] problems [of the sort that debugging instantiates]' which is just as much 'academy' as 'industry'.
(1) 'Exchange between strangers is a unique feature of us modern hominids' isn't really falsifiable without begging the question or establishing a hard boundary between kin and non-kin, which afaik is currently a function of the problem under consideration (i.e.'kin' might mean something different to geneticist vs. anthropologist vs. linguist vs. classical economist and even something different for a different problem in each of those disciplines). But 'this is what makes us human' is far from essential to Hayek..and probably basically inimical to Hayek, as some other commenters have pointed out.
(2) The 'reduction of collective intelligence to the price mechanism' objection (which is often a specific case of generic objections to dimensional reduction, including e.g. perceptron thresholds) is addressed throughout Minsky's Society of Mind[0], esp. the 'frames' concept.
(3) The article doesn't mention the benefits of localized knowledge (as opposed to the 'practical reality' of localized knowledge, which may be lamentable and/or fixable), which iirc Hayek does get into (or maybe some other Hayekians? more modern systems-oriented folks? can't think of a source at the moment). If some knowledge (for example, within a community of practice) weren't pretty strongly localized, then every knowable would be in one truly global variable space, and abstraction would be incredibly computation-intensive, and knowledge growth would be horribly O(n!), and all of thought would work like JavaScript (ZING). This is a stronger kind of localization than technical specialization (which just maps onto SOLID class design rather than variable space).[1]
Sometimes as a discipline/indulgence I go to a coffee shop that charges exorbitantly for wifi so I cost myself immediately measurable and painful (because exorbitant) cash if I don't invent the solution on my own. These hours are non-billable but they stretch my creativity and cleverness in ways OP describes, which increases productivity during billable hours etc. Also keeps me sane. :)
Well, even non-hard, very simply functional/x-ray naming sloppiness causes plenty of grief, but I think this is partly because we're used to naming being hard enough that -- at a certain level of energy, time-crunch, and lack of investment in a massively brownfield project we really would rather refactor entirely -- we sort of give up. There's a recentish (2015) little empirical study that clustered'linguistic antipatterns' (based on surveys of software engineers): https://www.researchgate.net/publication/276314133_Linguisti...
Wow, I wish every academic paper were that useful to the layman. Free download, well-explained list of antipatterns, in-document hyperlinks to code samples so you can recognize the antipatterns, suggestions for renaming. Thanks very much for the link.
Doesn't this mean Sydney can already alter the 'outside' (non-bing.com) world?
Sure, anything can issue http GETs -- doing this not a super power. And sure, Roy Fielding would get mad at you if your web service mutated anything (other than whatever the web service has to physically do in order to respond) in response to a GET. But plenty of APIs do this. And there are plenty of http GET exploits available public database (just do a CVE search) -- which Sydney can read.
So okay fine say Sydney is "just" a 'stochastically parroting a h4xx0rr'. But...who cares if the poisonous GET was actually issued to some actual machine somewhere on the web?
(I can't imagine how any LLM wrapper could build in an 'override rule' like 'no non-bing.com requests when you are sufficiently [simulating an animate being who is] pissed off'. But I'm way not expert in LLMs or GPT or transformers in general.)