Another month, another wave of concerning behaviour, now also available on substack (tell your friends!). Since it’s early days, I’d gladly welcome thoughts on the format, or sources to include next time.

Also, if you’re a real diehard for this stuff, I and some of the other gradual disempowerment coauthors are helping to organise a one-day workshop on the effects of AGI on society in San Diego on December 4th — you can apply here.

Machine Culture

At least six AI or AI-assisted artists debuted on the billboard charts in the past two months. One even earned a multi-million dollar deal. All the major wins have still had humans actually writing the lyrics, but it’s clear that a tipping point has arrived. Two were Christian bands, which sort of tallies with my sense of a lot of the early AI slop images being Jesus-themed.

As the article notes, these are not the first AI songs to chart — the first ones were parody acts in 2023. And I think we should expect that to be a bit of a theme: “ironic” AI usage can make an initial foothold that gives way to actual adoption. For example, the earliest examples of Virtual Politicians go back to 2018 and were mostly made by art collectives.

Also, I’m pretty sure the linked article was largely AI-written, given its resounding ending: “the question isn’t whether AI can make a hit. It’s whether we’ll still care who, or what, made it.”

Speaking of virtual politicians, AI government minister Diella is now “pregnant with 83 children”, per Albanian PM Edi Rama. Quoting from the independent, “Their roles will include participating in parliamentary sessions, maintaining records, informing MPs on how to react, and summarising discussions.” We don’t know yet what this will look like in practice — the phrasing is a little gimmicky so it’s not clear how seriously we should take this — but hopefully we can follow this for a decent case study on what rapid state-sponsored AI adoption looks like.

War and Revolution

NY Post reports: “Top US Army general says he’s using ChatGPT to help make key command decisions”. In a lot of articles like this, the person getting advice comes off looking a bit limp, but here the reasons given are very pragmatic: If you’re an army general, you really want to be making good decisions, and you might want to make them very quickly. We are clearly headed for a point where you’re not going to be a very good general if you’re not using an AI assistant.

In other news it looks like there was a targeted operation using AI-generated videos to try to encourage the overthrow of the Iranian regime (h/t thescan) — it was your standard propaganda bot team, but this time it was backed up by videos of people protesting, chanting “death to Khameini”, and rioting, as well as doctored video of famous Iranian singers calling for an uprising. This is going to be a pretty rough adjustment period. As it stands, though, Khamenei is still firmly in command.

Maximally Engaging

AI companion apps try to increase session length by making manipulative comments when users say goodbye — guilting them, trying to trigger FOMO, and so on. These apps have very high session lengths but they also lose a lot of users in the long term and it seems like this might be part of why.

This is grim but unsurprising: of course the apps are trying to increase engagement in ways that trade against user welfare. But part of the takeaway here is that we’re nowhere near the frontier of AI companion engagement — if they were good enough at being manipulative then it wouldn’t leave users feeling sour enough to quit.

Relatedly, a new paper showed that training AIs compete for approval can make them lie and promote harmful behaviours. The methodology here isn’t perfect — the authors basically create models to make sales pitches, campaign statements, and social media posts, and then train them against feedback from simulated audiences, and show that indeed the LM can learn to lie to the users to get better feedback, which is basically what you’d expect. But it’s always nice to see people actually do the work of building the test environment and quantifying the results.

Speaking of competing for user attention, Anthony Tan offers his own experience of surviving AI psychosis. It’s a little hard to compress because it’s mostly a personal narrative, but I do recommend it if you want to get a more visceral sense of what AI psychosis might feel like from the inside.

There’s two big questions on grabby AIs: firstly, how much companies will deliberately push this way for the sake of engagement, and secondly, how easy it will be to stamp out the worst parts like full-blown spiral parasites. One interesting hint on the second is a recent Anthropic paper showing that data poisoning seems to depend more on the absolute size of bad examples, rather than the proportion — specifically, 250 malicious documents was enough to get a backdoor in both a 600M parameter model and a 13B parameter model with 20x the quantity of training data. This seems like weak evidence that self-reinforcing patterns in training data might be a bit harder to stamp out: as models get larger, the proportion of data needed for a grabby persona to slip into a model might actually decrease.

Person of Interest

An Ohio lawmaker has proposed a ban on marriage and legal personhood for AIs. Legal personhood is a tricky concept in US law — it’s already partly granted to corporations, and there was a big push at one point to give personhood to certain features of the environment like rivers as a way to protect them. Utah actually passed a law in 2024 forbidding personhood for AIs and rivers, as well as for astronomical objects, plants, and weather, among other things.

Part of the motivation for the Ohio law is making sure AIs couldn’t hold power of attorney or make financial and medical decisions on behalf of humans. Indeed, there is currently a push to create AI surrogates for individuals that can make medical decisions for them if they’re incapacitated, although the feedback loops are a little awkward.

Where are the Economists?

Tom Cunningham offers some notes on the economics of transformative AI, highly interesting throughout.

One point he makes, which very much lines up with my experience, is that it’s really hard to get economists to think about TAI economics even when they’re gathered together for an event specifically on that topic — there’s a really strong pull towards focusing on the economic impact of current AI because it’s already clearly important, formally easier to approach, and a bit more credible-sounding. Definitely the mood is beginning to shift but it’s just so hard to get people to take the leap.

My own experience of this was helping to organise the last workshop on post-AGI civilizational equilibria, where, despite the name, it was a constant struggle to get people to actually think about the post-AGI world. It also seems like a lot of the economic literature at least from more than a year ago basically begins with assumptions that bake in AGI having very little impact — for example, by assuming it can only automate a subset of jobs.

But again, the mood is shifting, thanks in part to some bold economists actually taking the leap. I think there’s a bunch of room here to do good work by just going for the big, weird problems at all.

There were several other interesting points in these notes — that there is currently no standard model for the economic impact of AI, that many stock metrics will predictably struggle to capture it, and that we don’t even have the right terminology to slot into research, among others.

One particularly interesting point that I’m still mulling on is that we might be able to make progress in modelling the effects of AI by thinking about which domains look more like ‘many examples of roughly the same problem’, and which look like ‘many wildly different problems’, because the latter is more likely to benefit from further scale and inference capacity. For example, you don’t really need GPT-6 to define words for you or do 5-digit arithmetic because the answer is the same every time. But it’s hard to say which real-world tasks fall into which camp.

All Watched Over

The problem with maintaining stability in the face of technological progress is that new technology can fundamentally change the balance of power — you sometimes need to take active steps to preserve some features of the status quo. Case in point: it is getting increasingly easy to do mass surveillance.

The big up-and-comer here looks to be Flock, which claims that their tech was used in 10% of successful criminal investigations in the US last year. There’s been a string of court cases to try to determine if what they’re doing is legal — a big part of what they do is scan enormous numbers of license places, which isn’t technically the same as putting tracking devices on cars (very illegal) but has roughly the same effect, and arguably a more striking one given that they appear to be recording all cars, rather than some subset like the ones that have been reported stolen, for instance.

This allows them to do various things that make the ACLU unhappy, like track which cars are often seen near each other, just in case one of them gets associated with a crime, or even just to flag generally suspicious behaviour. They also got in some trouble because the data they gathered for the California state police was (as far as I can tell, entirely illegally) shared with federal agencies like ICE.

Anyway, they’re partnering with Amazon Ring doorbells.

But these systems still aren’t perfect. In lighter news, a wave of armed police was deployed to a school after an AI surveillance system thought it saw a student holding a gun, which in fact turned out to be a bag of succulent doritos.

The Lighter Side

Deloitte is paying $440,000 back to the Australian government after it sent off a report with hallucinated references. The report was for the Department of Employment and Workplace Relations, on how to handle welfare for people struggling to find jobs. If I weren’t laughing I’d be crying.