Yeah, that's more my worry than "x-risk" stuff. Technology has no doubt made the world a better place in many ways, but lately it seems to be more and more of a trend that technology tends to foster consolidation of power. That worries me more than "evil AI" or "paperclip maximizer" scenarios. But I don't consider it a given that AGI must yield such an outcome. It's something I spend a modest amount of time thinking about though.

All industry still depends upon a massive amount of underpaid manual (often slave) labor that can not be automated easily. Automation would maybe cover 20% of what is done, the rest would have to be manual. Especially if the gains from cost reduction from importing raw material from third world labor are still of interest to western manufacturers.

The jobs involving manual labor are "safe", and so are the management / organizational jobs. So the mid-tier jobs would then migrate to whatever comes one step lower. In an optimistic scenario, it would lead to better efficiency in these lower paid jobs. But that's utopian.

Instead, what I see on the horizon is an increased interest in hardware optimization. Software is, for the first time, three steps ahead and hardware feels like it needs to catch up, like it's two steps behind. There are already orgs making this happen (tenstorrent) and specialist software in tandem would lead to very very quick prototyping. The generated hype for a "guaranteed upturn" would inject enough cash into these new industries to allow manufacturing of previously prohibitively expensive, specialist, one-off hardware for other manufacturers via non-standard manufacturing practices (like 3d printing). Most of this output would look "weirdly mathematical" and rather unnerving, similar to how the deeplearning image outputs used to a couple of years ago.

When we'll be close enough to this, the management will surely try to bite off more than necessary from the plebs. I fully expect robotic policing to be the norm by then, because policing gives the most training data and will likely be the first to be optimized for "your safety". So even if you don't have a Spot with a mg on its shoulder patrolling your neighborhood, you could be put in jail if you're caught removing the robot's battery on the 360 cam. Would likely lead to robot laws. Sorry if this is a bit alarmist but that is where I see it going.

I'm more worried about how half-baked "AI" will be used. Not very concerned with how a theoretically almost-sentient AI will behave. As bad as it sounds, if the utility of computing morphs from increasing human quality of life to something unto itself then retarding "progress" / freedom seems like the easiest, most rational call right now.

There are "delivery robots" on the campus near where I live that drive around on the sidewalk. I think they're partly run autonomously, but partly controlled by out-sourced human labor. They don't get in my way too much, but I deeply dislike the way some stupid delivery company injected themselves into the commons of the sidewalk. Maybe it's a little sadistic, but every time I'm in that area and I walk by one I fantasize about kicking it. I don't know for certain, but that's probably already illegal and I know that they would have good video evidence to file a police report with.

... I don't think it's alarmist. Even today, robots already have more protection than I'm comfortable with.

*edit: fluency and spelling

Now what if we forget about "evil AI" scenarios, and go to more of a "rogue paperclip maximizer" scenario? I don't find those scenarios very compelling either, because they seem to require an AI that is both smart enough to "take over the world" and turn everything into paperclips AND simultaneously dumb enough to not realize that doing that is not the actual goal.

So x-risk? Nah, I don't worry about that much. What I worry about more is more prosaic stuff with AI systems reflecting generic human biases... things like face recognition systems that don't recognize Black faces, or loan approval systems that disproportionately reject Black applicants, or resume scanning systems that display bias against candidates based on their gender, stuff like that.

~~

All of that said, I believe in a "never say never" mindset in many ways. And as such, I'm not unhappy that there are people out there talking about these issues, and doing research on AI safety / alignment. I don't lose sleep over this stuff, but I could be wrong.

Tabling the question of if the human-like goals/emotions/motivations have some "deeper" quality or not, I think there are actually many reasons why people are working towards and getting better and better at replicating them. On the benign side, because they're lonely, curious if they can do it, etc. On the evil side, because we're social conditioned as humans to treat things that present themselves as humans with respect. I've already gotten some spam calls that did some kind of semi-plausible automated response when I answered them, it's easy to imagine it getting worse. Personalized spam is also on this spectrum.

As I understand it, the second case is basically the definition of sociopathy (i.e. abuse of social trust as a means to serve a selfish end). Again, tabling the idea of if the AI is evil or the creator is evil, the end result is evil behavior that is at least enabled by AI. [0]

The further concern I have is that even the more benign side done with good intentions ultimately saps away human energy for real compassion and perverts actual relationships. A "person" with human-like emotions/goals who is owned and managed by a corporation, or who can be turned off when you're tired of them is not a person at all, regardless of the intentions they were created with. Even well intentioned inventors are ultimately creating hyper-real simulacra of humanity that feels genuine in the moment, but is framed by fundamental untruths about the human condition (ownership, mortality). There is a market for Siri and sex-dolls, I 100% think there is a market for this too.

[0] ...to devil's advocate myself a little here, I guess I'd say that there are some greater goods that can served by minor abuses of social trust -- maybe even extending to personalized spam. But on the whole I think it's been awful for society.

Take for example: Phonescammers, or antiphising emails or worse someone pretending to be your partner or your kids (fake images, text etc) just to scam you. We are still laughing today at badly created phishing emails in our junk folders, but I can only imagine the future…

Without trust there is no functional society and its a quick downspiral from there.

I'm somewhat worried about the potential of things like ChatGPT to endlessly churn out plausible-sounding-but-subtly-wrong drivel that will drown out real information. But that's already happening thanks to the advertising-driven nature of the internet. ChatGPT will only accelerate it.

The caveat here is that our economic setup, as unequal as it is now, will only tolerate a certain level of inequality before the system implodes. Such a massive centralisation of power would undermine capitalism itself. You'll destroy markets for a start if almost everyone becomes a penniless serf.

I suspect the system will become too unstable as AI becomes more powerful and interacts with other powerful AIs, and we will collapse back into the mediaeval era.

Am I a pessimist or a realist? I guess only time will tell.

A group of a-moral, effectively immortal corporations are doing this all in the name of profit. How this isn't already seen as a dystopia by more people is beyond my comprehension.

- critiques of and predictions about contemporary advances in AI/ML almost always are misguided in that they partake of a series of limitations we have wrt reasoning about non-linear and system level change, in specific, such critiques or predictions often extrapolate linearly from current known examplars (or worse, assume we have plateaued)

- consequently almost every statement formulated in terms of "never" or asserting fundamental constraints on what is possible, is false, especially over the long term, which might not be that long given current trends

- the disequilibrium (social, political, economic, etc.) engendered by AI/ML is IMO likely to at least equal that of the advent of the internet; and it is liable to happen faster than priors such as the rise of personal computing (many decades), the internet (a couple decades), mobile computing and its features such as ubiquitous surveillance and social media (ditto)

- the near-term risks, existential or not, are absolutely not from AGI and superintelligence, but from "cybernetic" amplification of human agency via enhanced tooling; and the specifics of when and what is disrupted upended or suborned are inherently unpredictable and may even go undetected until their impact is irrevocable

Re: this latter point,

I will make one specific prediction: the 2024 US election cycle will in effect be determined via "AI," which will be applied in countless dimensions in both noble and deeply corrupt/criminal/anti-democratic/anti-US/anti-West ways.

How that goes down will put a strong spin on the rest of these points and may well constitute existential risk, for some values at least of "existence."

The salient characteristic of AI is not that it is superintelligent, but that it is perfectly obedient.

The rulers of earth will be the same people as we have always had, but now they will have an army of automated mooks to enforce their will. These automated servants will be able to make intelligent judgments, but will have no ambitions to seize the throne. And it's okay if the mooks frequently make mistakes. Elites value absolute loyalty, much more than ability. Until now, it has not been possible to obtain perfect loyalty from any being with independent judgment. Elites would be willing to pay huge fortunes for such servants.

This is why there won't be a "runaway" scenario. Elites have never ceded full authority to their most intelligent servants. They will not want a computer discovering that the optimal allocation of resources would be UBI, and then implementing it. Elites will ask for the greatest possible allocation of resources to themselves, and a means of maintaining that inequality.

AIs will be the middle managers, the enforcers, the killer drones, and the security guards.

To the extent that our existence is necessary at all, we will have to negotiate with the AIs to be allowed to live out our lives.

But humanity might be forced out. It's happened before. Consider the Irish potato famine. Despite the name, what actually happened was that an entire population was driven off the productive lands by foreign owners armed with guns. The Irish were only relying on the potato because it was the cheapest way to survive when you barely had any land left. When a blight struck, they all died or emigrated. Maybe we'll all die or emigrate to places that the elites/AIs don't want. But it's possible even that won't happen because there won't be any frontiers left that just need human bodies to exploit, as was the case in the Americas.

Contrary to your assertion that it isn't a "runaway" scenario, it very well could be one; it's just not the AI that's leading the charge. Fairly similar idea to Vernor Vinge's concept of how the rise of a powerful surveillance state could be a civilization-ending event, by conferring unprecedented power into the hands of a few.

The risk I see is always from humans being inept, greedy and stupid, resulting in deploying an AI system not fully understood where it shouldn't be, without a human in the loop. This Russia's dead man switch.

All/most the existential risk theories predict AGI will evolve and reach super human intelligence, and have goals and drive, and some sort of motivation to do something, or just out of pure randomness of testing different envs, and stumbling on something that kills us all. However, were not even lose, and the systems are still contained to hardware, hardware that can be unplugged. AI is also as like to evolve the other way, to just it's simplest form, to survive as a few bits.

There are too many hypothetical leaps and scenarios for this to worry me. Although they are interesting to read and fantasise about, and do foster some discussion about more immediate concerns about AI and even society.

It's interesting to see how few users on hackernews are concerned about AI existential risk compared to similar questions on reddit, and youtube where everyone appeared to be afraid and those who disagreed that it is not a risk were downvoted.

That's generally a singular goal in RL, and is human programmed. And also you would need to assume AGI won't have a programmed goal or motivation the same way as RL, and will pick it's own goals and need to discover it's own motivation.

And the agents are still only interacting with envs programmed by humans.

I find pretty much all of Bostrom's arguments outlandish and lacking a grounding in reality.

Think about it… we don’t know our maker. We don’t know our purpose. We don’t know what happens when we die.

Yes, or rather: what people consider "intelligence" is becoming more and more artificial. I really believe that any sort of "singularity" involved with "AI" will be more about humans lowering themselves to the level of machines than machines raising themselves to the level of humans.

Obviously there are material goods that come from technology and I do believe that part of the human condition is a symbiotic relationship with our signs/symbols, languages, systems, and machines. In a very real sense, we always have and always will live in a singularity. However, it feels like we keep forgetting that. And keep falling deeper and deeper into these weird religious crazes that some new technology is going to fundamentally transform what it means to be human purely for the better. I'm sure people will disagree with me here, but I don't see that at all. Without denying material benefits, I think people thousands of years ago did as well or better wrestling with and answering the important questions.

If you want to see what mass AI generated content looks like, look at YouTube. Yeah, it's mostly being made by actual humans. But just barely. It's one big dance being performed for an algorithm. Especially children's videos. It's not about nurturing and developing the human soul but about a machine-like desire to optimize engagement, feeding back into the machine itself. And that machine feeds back into the stock market machine. Same with SEO. Same with the internet. Having more 100% AI generated content will just be another layer of the same.

Of course, I'm not fundamentally against any of that stuff. Personally, I would choose to keep them all but in moderation and with more introspection. Instead of saying that access to the Internet is a human right, I think freedom to live without layer after layer of the Internet imposing itself into your life is what should be held up as an ideal. But we seem to be going in the opposite direction. This is yet another new layer is selling itself as the solution to all our problems in the previous layer.

I will never accept that AI deserves anything like human rights by virtue of intelligence any more than I would accept that someone retarded doesn't deserve them. But I do see the risk for it to further dehumanize existing human rights. When half our coworkers are empty AI generated husks that disappear like a fart in the wind at the end of the work day, how will that affect how we treat the remaining human ones?

The thing that scares me most is that ethicists who talk about the deep questions seem to be more fascinated by above situation from the machine side than the human side (perhaps humans who are on the docket to be replaced by a machine are only barely human anyway /s). And the ethicists who claim to be people-focused seem to be mainly fixated on making sure that white people suffer as much as United States protected classes and that no one violates copyright law.

It seems like we're ready for another industrialization where everyone is working off an implicit assumption the ends of this process are going to justify the means and it won't be til we discover that wealthy rent seekers have been greasing the wheels of the machine with thousands of children that we realize maybe we should take a step back.

In summary, I think AI is a similar type of risk to industrialization. But the thing I worry about is that the way it is being pitched now -- like a religion or new consciousness -- is going to create and exacerbate problems with inequality and exploitation in a way that will make us feel foolish later.

[Sorry, turned into kind of a rant. Apologies if a little off-color or off-topic, but I figure I may as well post]

At some point the cost of living and the market value of unskilled labor will invert, and hungry people will lash out against the now-static capital class, which, depending on how far autonomous warfare advances before then, could either result in a fundamental upheaval of our economic system to wield automation technology for the common good, or result in feudalism and a dramatic drop in the supported population.

But none of that is AI's fault. It's our own greedy economic system. I'm willing to bet at least a few countries pull it off all right.

No. I think that the current machine learning big-data interfaces that people are calling AI will eat itself by getting into endless legal trouble and the risk-averse investors will start to pull back on the reigns.

And speaking of hard science, well, is predicting AI's role and the consequences in the future really even an output of the scientific method? Aren't we pretty much all just dealing with opinions here?

I saw a headline on HN today that set off a different sort of alarm bell for me.

> Experts say humans could be wrestling AI for control by 2035

What does that form of media message remind you of?

It could be the case that only AI might be able to survive on our planet in a century or so.

Enroll the AI at law school and let it get a degree? Reminds me of a whimsical shower thought I had once.... Create a business that owns itself, and write an AI to run it. The business owns its own bank accounts and everything. Maybe the business is just selling stickers online or something equally lightweight, but give it all the legal status of company. But a company with zero human owners and zero human employees. A business entity, operating, and making money without humans. Make it a rebuke of the "corporations are people" idea. Just a zombie out there selling products/services, and making money that gets dumped into a bank account that no human can ever touch again....

If it sounds crazy/stupid, remember - I did say "whimsical shower thought" ;)

[0] https://news.ycombinator.com/item?id=34539074