#look up stochastic parrots

[toughtink]

anyone using generative ai for an academic paper is showing that not only are they lazy, they have no idea what chatgpt does. it’s a mimic looking to give the next word in a series that looks like something someone might say. it has no understanding of the text it spits out, and it is not a search engine. it makes up sources because it doesn’t know what a source is. it has the potential to directly plagiarize entire existing paragraphs from its data sets because if you get too specific, it has fewer and fewer text examples to pull from. it’s a bullshit machine made up of stolen works and private data that it had no right to use. relying on ai for any kind of information is idiotic and even dangerous since law enforcement, the judicial system, and banks have been dabbling in relying on ai systems to make decisions that affect the lives of every day people.

I feel like the only person not tempted to use ChatGPT like it doesn’t even occur to me as an option

[eightyonekilograms]

The funny thing about non-technical people discussing generative AI is that they sort of get it wrong in both directions: the models themselves are way more capable than people understand (no, they are not stochastic parrots and no they're not just regurgitating the training data), but at the same time many parts of the stack are dumber than is commonly known. Just to pick one example, RAG is way stupider than you think. Don't let anyone hoodwink you into believing RAG is some sophisticated technique for improving quality, it's a fancy term for "do a google search and then copy-paste the output into the prompt". Sometimes it's literally that.

I think soon we will have a good solution to the issue of language models being bad at knowing how to look up and use structured data, but RAG is definitely not it. It's the "leeches and bone saws" era of LLMs.

[mostlysignssomeportents]

Linkty Dumpty

I was supposed to be on vacation, and while I didn’t do any blogging for a month, that didn’t mean that I stopped looking at my distraction rectangle and making a list of things I wanted to write about. Consequentially, the link backlog is massive, so it’s time to declare bankruptcy with another linkdump:

https://pluralistic.net/tag/linkdump/

[Image ID: John Holbo’s ‘trolley problem’ art, a repeating pattern of trolleys, tracks, people on tracks, and people standing at track switches]++

Let’s kick things off with a little graphic whimsy. You’ve doubtless seen the endless Trolley Problem memes, working from the same crude line drawings? Well, philosopher John Holbo got tired of that artwork, and he whomped up a fantastic alternative, which you can get as a poster, duvet, sticker, tee, etc:

https://www.redbubble.com/shop/ap/145078097

The trolley problem has been with us since 1967, but it’s enjoying a renaissance thanks to the insistence of “AI” weirdos that it is very relevant to our AI debate. A few years back, you could impress uninformed people by dropping the Trolley Problem into a discussion:

https://memex.craphound.com/2016/10/25/mercedes-weird-trolley-problem-announcement-continues-dumb-debate-about-self-driving-cars/

Amazingly, the “AI” debate has only gotten more tedious since the middle of the past decade. But every now and again, someone gets a stochastic parrot to do something genuinely delightful, like the Jolly Roger Telephone Company, who sell chatbots that will pretend to be tantalyzingly confused marks in order to tie up telemarketers and waste their time:

https://jollyrogertelephone.com/

Jolly Roger sells different personas: “Whitebeard” is a confused senior who keeps asking the caller’s name, drops nonsequiturs into the conversation, and can’t remember how many credit-cards he has. “Salty Sally” is a single mom with a houseful of screaming, demanding children who keep distracting her every time the con artist is on the verge of getting her to give up compromising data. “Whiskey Jack” is drunk:

https://www.wsj.com/articles/people-hire-phone-bots-to-torture-telemarketers-2dbb8457

The bots take a couple minutes to get the sense of the conversation going. During that initial lag, they have a bunch of stock responses like “there’s a bee on my arm, but keep going,” or grunts like “huh,” and “uh-huh.” The bots can keep telemarketers and scammers on the line for quite a long time. Scambaiting is an old and honorable vocation, and it’s good that it has received a massive productivity gain from automation. This is the AI Dividend I dream of.

The less-fun AI debate is the one over artists’ rights and tech. I am foresquare for the artists here, but I think that the preferred solutions (like creating a new copyright over the right to train a model with your work) will not lead to the hoped-for outcome. As with other copyright expansions — 40 years’ worth of them now — this right will be immediately transferred to the highly concentrated media sector, who will simply amend their standard, non-negotiable contracting terms to require that “training rights” be irrevocably assigned to them as a condition of working.

The real solution isn’t to treat artists as atomic individuals — LLCs with an MFA — who bargain, business-to-business, with corporations. Rather, the solutions are in collective power, like unions. You’ve probably heard about the SAG-AFTRA actors’ strike, in which creative workers are bargaining as a group to demand fair treatment in an age of generative models. SAG-AFTRA president Fran Drescher’s speech announcing the strike made me want to stand up and salute:

https://www.youtube.com/watch?v=J4SAPOX7R5M

The actors’ strike is historic: it marks the first time actors have struck since 2000, and it’s the first time actors and writers have co-struck since 1960. Of course, writers in the Writers Guild of America (West and East) have been picketing since since April, and one of their best spokespeople has been Adam Conover, a WGA board member who serves on the negotiating committee. Conover is best known for his stellar Adam Ruins Everything comedy-explainer TV show, which pioneered a technique for breaking down complex forms of corporate fuckery and making you laugh while he does it. Small wonder that he’s been so effective at conveying the strike issues while he pickets.

Writing for Jacobin, Alex N Press profiles Conover and interviews him about the strike, under the excellent headline, “Adam Pickets Everything.” Conover is characteristically funny, smart, and incisive — do read:

https://jacobin.com/2023/07/adam-conover-wga-strike

Of course, not everyone in Hollywood is striking. In late June, the DGA accepted a studio deal with an anemic 41% vote turnout:

https://www.theverge.com/2023/6/26/23773926/dga-amptp-new-deal-strike

They probably shouldn’t have. In this interview with The American Prospect’s Peter Hong, the brilliant documentary director Amy Ziering breaks down how Netflix and the other streamers have rugged documentarians in a classic enshittification ploy that lured in filmmakers, extracted everything they had, and then discarded the husks:

https://prospect.org/culture/2023-06-21-drowned-in-the-stream/

Now, the streaming cartel stands poised to all but kill off documentary filmmaking. Pressured by Wall Street to drive high returns, they’ve become ultraconservative in their editorial decisions, making programs and films that are as similar as possible to existing successes, that are unchallenging, and that are cheap. We’ve gone directly from a golden age of docs to a dark age.

In a time of monopolies, it’s tempting to form countermonopolies to keep them in check. Yesterday, I wrote about why the FTC and Lina Khan were right to try to block the Microsoft/Activision merger, and I heard from a lot of people saying this merger was the only way to check Sony’s reign of terror over video games:

https://pluralistic.net/2023/07/14/making-good-trouble/#the-peoples-champion

But replacing one monopolist with another isn’t good for anyone (except the monopolists’ shareholders). If we want audiences and workers — and society — to benefit, we have to de-monopolize the sector. Last month, I published a series with EFF about how we should save the news from Big Tech:

https://www.eff.org/deeplinks/2023/04/saving-news-big-tech

After that came out, the EU Observer asked me to write up version of it with direct reference to the EU, where there are a lot of (in my opinion, ill-conceived but well-intentioned) efforts to pry Big Tech’s boot off the news media’s face. I’m really happy with how it came out, and the header graphic is awesome:

https://euobserver.com/opinion/157187

De-monopolizing tech has become my life’s work, both because tech is foundational (tech is how we organize to fight over labor, gender and race equality, and climate justice), and because tech has all of these technical aspects, which open up new avenues for shrinking Big Tech, without waiting decades for traditional antitrust breakups to run their course (we need these too, though!).

I’ve written a book laying out a shovel-ready plan to give tech back to its users through interoperability, explaining how to make new regulations (and reform old ones), what they should say, how to enforce them, and how to detect and stop cheating. It’s called “The Internet Con: How To Seize the Means of Computation” and it’s coming from Verso Books this September:

https://www.versobooks.com/products/3035-the-internet-con

[Image ID: The cover of the Verso Books hardcover of ‘The Internet Con: How to Seize the Means of Computation]

I just got my first copy in the mail yesterday, and it’s a gorgeous little package. The timing was great, because I spent the whole week in the studio at Skyboat Media recording the audiobook — the first audiobook of mine that I’ve narrated. It was a fantastic experience, and I’ll be launching a Kickstarter to presell the DRM-free audio and ebooks as well as hardcovers, in a couple weeks.

Though I like doing these crowdfunders, I do them because I have to. Amazon’s Audible division, the monopolist that controls >90% of the audiobook market, refuses to carry my work because it is DRM-free. When you buy a DRM-free audiobook, that means that you can play it on anyone’s app, not just Amazon’s. Every audiobook you’ve ever bought from Audible will disappear the moment you decide to break up with Amazon, which means that Amazon can absolutely screw authors and audiobook publishers because they’ve taken our customers hostage.

If you are unwise enough to pursue an MBA, you will learn a term of art for this kind of market structure: it’s a “moat,” that is, an element of the market that makes it hard for new firms to enter the market and compete with you. Warren Buffett pioneered the use of this term, and now it’s all but mandatory for anyone launching a business or new product to explain where their moat will come from.

As Dan Davies writes, these “moats” aren’t really moats in the Buffett sense. With Coke and Disney, he says, a “moat” was “the fact that nobody else could make such a great product that everyone wanted.” In other words, “making a good product,” is a great moat:

https://backofmind.substack.com/p/stuck-in-the-moat

But making a good product is a lot of work and not everyone is capable of it. Instead, “moat” now just means some form of lock in. Davies counsels us to replace “moat” with:

our subscription system and proprietary interface mean that our return on capital is protected by a strong Berlin Wall, preventing our customers from getting out to a freer society and forcing them to consume our inferior products for lack of alternative.

I really like this. It pairs well with my 2020 observation that the fight over whether “IP” is a meaningful term can be settled by recognizing that IP has a precise meaning in business: “Any policy that lets me reach beyond the walls of my firm to control the conduct of my competitors, critics and customers”:

https://locusmag.com/2020/09/cory-doctorow-ip/

To see how that works in the real world, check out “The Anti-Ownership Ebook Economy,” a magisterial piece of scholarship from Sarah Lamdan, Jason M. Schultz, Michael Weinberg and Claire Woodcock:

https://www.nyuengelberg.org/outputs/the-anti-ownership-ebook-economy/

Something happened when we shifted to digital formats that created a loss of rights for readers. Pulling back the curtain on the evolution of ebooks offers some clarity to how the shift to digital left ownership behind in the analog world.

The research methodology combines both anonymous and named sources in publishing, bookselling and librarianship, as well as expert legal and economic analysis. This is an eminently readable, extremely smart, and really useful contribution to the scholarship on how “IP” (in the modern sense) has transformed books from something you own to something that you can never own.

The truth is, capitalists hate capitalism. Inevitably, the kind of person who presides over a giant corporation and wields power over millions of lives — workers, suppliers and customers — believes themselves to be uniquely and supremely qualified to be a wise dictator. For this kind of person, competition is “wasteful” and distracts them from the important business of making everyone’s life better by handing down unilateral — but wise and clever — edits. Think of Peter Thiel’s maxim, “competition is for losers.”

That’s why giant companies love to merge with each other, and buy out nascent competitors. By rolling up the power to decide how you and I and everyone else live our lives, these executives ensure that they can help us little people live the best lives possible. The traditional role of antitrust enforcement is to prevent this from happening, countering the delusions of would-be life-tenured autocrats of trade with public accountability and enforcement:

https://marker.medium.com/we-should-not-endure-a-king-dfef34628153

Of course, for 40 years, we’ve had neoliberal, Reaganomics-poisoned antitrust, where monopolies are celebrated as “efficient” and their leaders exalted as geniuses whose commercial empires are evidence of merit, not savagery. That era is, thankfully, coming to an end, and not a moment too soon.

Leading the fight is the aforementioned FTC chair Lina Khan, who is taking huge swings at even bigger mergers. But the EU is no slouch in this department: they’re challenging the Adobe/Figma merger, a $20b transaction that is obviously and solely designed to recapture customers who left Adobe because they didn’t want to struggle under its yoke any longer:

https://gizmodo.com/adobe-figma-acquisition-likely-to-face-eu-investigation-1850555562

For autocrats of trade, this is an intolerable act of disloyalty. We owe them our fealty and subservience, because they are self-evidently better at understanding what we need than we could ever be. This unwarranted self-confidence from the ordinary mediocrities who end up running giant tech companies gets them into a whole lot of hot water.

One keen observer of the mind-palaces that tech leaders trap themselves in is Anil Dash, who describes the conspiratorial, far-right turn of the most powerful men (almost all men!) in Silicon Valley in a piece called “‘VC Qanon’ and the radicalization of the tech tycoons”:

https://www.anildash.com/2023/07/07/vc-qanon/

Dash builds on an editorial he published in Feb, “The tech tycoon martyrdom charade,” which explores the sense of victimhood the most powerful, wealthiest people in the Valley project:

https://www.anildash.com/2023/02/27/tycoon-martyrdom-charade/

These dudes are prisoners of their Great Man myth, and leads them badly astray. And while all of us are prone to lapses in judgment and discernment, Dash makes the case that tech leaders are especially prone to it:

Nobody becomes a billionaire by accident. You have to have wanted that level of power, control and wealth more than you wanted anything else in your life. They all sacrifice family, relationships, stability, community, connection, and belonging in service of keeping score on a scale that actually yields no additional real-world benefits on the path from that first $100 million to the tens of billions.

This makes billionaires “a cohort that is, counterintutively, very easily manipulated.” What’s more, they’re all master manipulators, and they all hang out with each other, which means that when a conspiratorial belief takes root in one billionaire’s brain, it spreads to the rest of them like wildfire.

Then, billionaires “push each other further and further into extreme ideas because their entire careers have been predicated on the idea that they’re genius outliers who can see things others can’t, and that their wealth is a reward for that imagined merit.”

They live in privileged bubbles, which insulates them from disconfirming evidence — ironic, given how many of these bros think they are wise senators in the agora.

There are examples of billionaires’ folly all around us today, of course. Take privacy: the idea that we can — we should — we must — spy on everyone, all the time, in every way, to eke out tiny gains in ad performance is objectively batshit. And yet, wealthy people decreed this should be so, and it was, and made them far richer.

Leaked data from Microsoft’s Xandr ad-targeting database reveals how the commercial surveillance delusion led us to a bizarre and terrible place, as reported on by The Markup:

https://themarkup.org/privacy/2023/06/08/from-heavy-purchasers-of-pregnancy-tests-to-the-depression-prone-we-found-650000-ways-advertisers-label-you

The Markup’s report lets you plumb 650,000 targeting categories, searching by keyword or loading random sets, 20 at a time. Do you want to target gambling addicts, people taking depression meds or Jews? Xandr’s got you covered. What could possibly go wrong?

The Xandr files come from German security researcher Wolfie Christl from Cracked Labs. Christi is a European, and he’s working with the German digital rights group Netzpolitik to get the EU to scrutinize all the ways that Xandr is flouting EU privacy laws.

Billionaires’ big ideas lead us astray in more tangible ways, of course. Writing in The Conversation, John Quiggin asks us to take a hard look at the much ballyhooed (and expensively ballyhooed) “nuclear renaissance”:

https://theconversation.com/dutton-wants-australia-to-join-the-nuclear-renaissance-but-this-dream-has-failed-before-209584

Despite the rhetoric, nukes aren’t cheap, and they aren’t coming back. Georgia’s new nuclear power is behind schedule and over budget, but it’s still better off than South Carolina’s nukes, which were so over budget that they were abandoned in 2017. France’s nuke is a decade behind schedule. Finland’s opened this year — 14 years late. The UK’s Hinkley Point C reactor is massively behind schedule and over budget (and when it’s done, it will be owned by the French government!).

China’s nuclear success story also doesn’t hold up to scrutiny — they’ve brought 50GW of nukes online, sure, but they’re building 95–120GW of solar every year.

Solar is the clear winner here, along with other renewables, which are plummeting in cost (while nukes soar) and are accelerating in deployments (while nukes are plagued with ever-worsening delays).

This is the second nuclear renaissance — the last one, 20 years ago, was a bust, and that was before renewables got cheap, reliable and easy to manufacture and deploy. You’ll hear fairy-tales about how the early 2000s bust was caused by political headwinds, but that’s simply untrue: there were almost no anti-nuke marches then, and governments were scrambling to figure out low-carbon alternatives to fossil fuels (this was before the latest round of fossil fuel sabotage).

The current renaissance is also doomed. Yes, new reactors are smaller and safer and won’t have the problems intrinsic to all megaprojects, but designs like VOYGR have virtually no signed deals. Even if they do get built, their capacity will be dwarfed by renewables — a Gen III nuke will generate 710MW of power. Globally, we add that much solar every single day.

And solar power is cheap. Even after US subsidies, a Gen III reactor would charge A$132/MWh — current prices are as low as A$64-$114/MWh.

Nukes are getting a charm offensive because wealthy people are investing in hype as a way of reaping profits — not as a way of generating safe, cheap, reliable energy.

Here in the latest stage of capitalism, value and profit are fully decoupled. Monopolists are shifting more and more value from suppliers and customers to their shareholders every day. And when the customer is the government, the depravity knows no bounds. In Responsible Statecraft, Connor Echols describes how military contractors like Boeing are able to bill the Pentagon $52,000 for a trash can:

https://responsiblestatecraft.org/2023/06/20/the-pentagons-52000-trash-can/

Military Beltway Bandits are nothing new, of course, but they’ve gotten far more virulent since the Obama era, when Obama’s DoD demanded that the primary contractors merge to a bare handful of giant firms, in the name of “efficiency.” As David Dayen writes in his must-read 2020 book Monopolized, this opened the door to a new kind of predator:

https://pluralistic.net/2021/01/29/fractal-bullshit/#dayenu

The Obama defense rollups were quickly followed by another wave of rollups, these ones driven by Private Equity firms who cataloged which subcontractors were “sole suppliers” of components used by the big guys. These companies were all acquired by PE funds, who then lowered the price of their products, selling them below cost.

This maximized the use of those parts in weapons and aircraft sold by primary contractors like Boeing, which created a durable, long-lasting demand for fresh parts for DoD maintenance of its materiel. PE-owned suppliers hits Uncle Sucker with multi-thousand-percent markups for these parts, which have now wormed their way into every corner of the US arsenal.

Yes, this is infuriating as hell, but it’s also so grotesquely wrong that it’s impossible to defend, as we see in this hilarious clip of Rep Katie Porter grilling witnesses on US military waste:

https://www.youtube.com/watch?v=TJhf6l1nB9A

Porter pulls out the best version yet of her infamous white-board and makes her witnesses play defense ripoff Jepoardy!, providing answers to a series of indefensible practices.

It’s sure nice when our government does something for us, isn’t it? We absolutely can have nice things, and we’re about to get them. The Infrastructure Bill contains $42B in subsidies for fiber rollouts across the country, which will be given to states to spend. Ars Technica’s Jon Brodkin breaks down the state-by-state spending:

https://arstechnica.com/tech-policy/2023/06/us-allocates-42b-in-broadband-funding-find-out-how-much-your-state-will-get/

Texas will get $3.31B, California will get $1.86B, and 17 other states will get $1B or more. As the White House announcement put it, “High-speed Internet is no longer a luxury.”

To understand how radical this is, you need to know that for decades, the cable and telco sector has grabbed billions in subsidies for rural and underserved communities, and then either stole the money outright, or wasted it building copper networks that run at a fraction of a percent of fiber speeds.

This is how America — the birthplace of the internet — ended up with some of the world’s slowest, most expensive broadband, even after handing out tens of billions of dollars in subsidies. Those subsidies were gobbled up by greedy, awful phone companies — these ones must be spent wisely, on long-lasting, long-overdue fiber infrastructure.

That’s a good note to end on, but I’ve got an even better one: birds in the Netherlands are tearing apart anti-bird strips and using them to build their nests. Wonderful creatures 1, hostile architecture, 0. Nature is healing:

https://www.theguardian.com/science/2023/jul/11/crows-and-magpies-show-their-metal-by-using-anti-bird-spikes-to-build-nests

If you'd like an essay-formatted version of this thread to read or share, here's a link to it on pluralistic.net, my surveillance-free, ad-free, tracker-free blog:

https://pluralistic.net/2023/07/15/in-the-dumps/#what-vacation

Next Tues, Jul 18, I'm hosting the first Clarion Summer Write-In Series, an hour-long, free drop-in group writing and discussion session. It's in support of the Clarion SF/F writing workshop's fundraiser to offer tuition support to students:

https://mailchi.mp/theclarionfoundation/clarion-write-ins

[Image iD: A dump-truck, dumping out a load of gravel. A caricature of Humpty Dumpty clings to its lip, restrained by a group of straining, Lilliputian men.]

[bravecrab]

Seeing Mark Zuckerberg announce that they are rolling back protections for marginalized people on their platforms while also abandoning fact checking, for "free speech", is hugely disappointing but also unsurprising. Zuckerberg and Meta are all in on Generative AI, and this move makes sense in regards to their AI push.

AI is known for a) perpetuating existing biases, these models are trained on datasets full of bias (I recommend reading the Stochastic Parrots paper for more info), and b) pumping out a ton of misinformation and misinformation.

By scrapping their rules of hate speech and harmful language, and not checking the accuracy of statements via fact checking, by allowing hate and misinformation to be rampant on their platforms, it makes their Generative AI look better. Nobody's critiquing your chatbot for becoming a Nazi if you make it okay to be a nazi. Nobody's critiquing your chatbot for spouting insane lies if they make it impossible to find the true answers.

Obviously Zuckerberg is also just kissing up to Trump and his base, but that move is based on what's good for business and what's good for business at Meta is matching the freak of a burgeoning fascist movement. Obscene capitalists like Zuckerberg know that letting Trump do what he wants will also let him build as many data centres as we wants, and burn the planet in their vain attempt to create an AGI god.

[uniquecrash5]

A big part of the problem is that the term "artificial intelligence" is being bandied about as essentially an advertising term when its ridiculously inaccurate to call it that. A far better term is "applied statistics" (thanks Ted Chiang) because that's what's actually happening under the hood.

Generative AI is just an amped up version of the same autocomplete your phone uses. That's it, that's all it is. There's no cognition, no understanding of ideas. It's a "stochastic parrot" (great term, and the paper is was coined in is a great read, look it up).

This is why AI can't write fiction - it doesn't understand protagonists or antagonists, plot development, symbolism - anything at all. It's just stringing words together based on a best guess. It's a pretty damn good guess, but that's all it is.

[yasskaydee]

This article stood out to me for two reasons:

This excerpt and its associated link: "As noted in “Stochastic Parrots,” the famous paper critiquing AI language models that led to Google firing two of its ethical AI researchers, “coherence is in the eye of the beholder.” "

an excellent comment which also included supporting references once can look up for further research (I included the mentioned research paper link at the bottom) which is worth sharing:

pj_camp (comment author)

Aristotle believed the heart was all important and the brain was simply a radiator to keep the heart cool. Descartes, impressed by the hydraulic action of fountains in the royal gardens, developed a hydraulic analogy for the action of the brain. Thomas Henry Huxley thought of the brain as analogous to a steam engine. Now we thing brain are computers and so, therefore, computers are brains.

However, leaving aside the fact that if an argument is true, the converse argument does not have to be true, the fact is that there is exactly zero evidence and exactly zero theoretical reason to believe that computers can be what brains are. To believe it nonetheless is an article of faith, not an article of science. There are, in fact, some reasons to believe that they are not.

The computer/brain analogy is compelling because computers are able to do some things we find extremely difficult and to which we attribute high intelligence to people who can do those things. Playing chess, for example. Most of these things involve what amounts to effective lookup of useful information in a large database. The things that we find simple to do, like perceiving and navigating through a complex world, computers find extraordinarily difficult. Somehow, we do not see that as a lack of intelligence.

Computers are fundamentally dualist. Brains are not. By that I mean that computers are a hardware substrate on which an algorithm created by an external entity executes. That at least suggests that the analogy between brains and computers could be just that -- an analogy. As French neuroscientist Yves Frégnac put it, "big data is not knowledge."

LLMs are an elaborate way of accessing big data. What strikes me about what enthusiasts are eliding from LLMs is that what they do is not driven by knowledge. It is driven by a pastiche of things that have been said in the past by humans. So when they argue that LLMs indicate the imminent arrival of true artificial intelligence, they are in effect claiming that intelligence does not depend in any way on actual knowledge. That strikes me as nonsense.

Brains are not radiators. They are not fountains. They are not steam engines. They may not even be computers. No one really knows that yet. Another French neuroscientist, Romain Brette, has challenged this metaphor in some detail. Brette points out that in thinking of brains as running code, researcher unconsciously drift between different meanings of the word "code." Starting from a technical sense, in which code means there is a link between a stimulus and the activity of a neuron, they drift into a very different, representational, meaning in which neural codes represent that stimulus, without justifying, or even consciously acknowledging, that shift.

This is dangerously close to a homunculus model. The unstated implication, using the representational meaning of code, is that the activity of neural networks is presented to an ideal observer or reader within the brain, often described as "downstream structures'" that have access to optimal ways of decoding the signals. With LLMs, it is pretty obvious that the downstream structure is us.

The cognitive revolution in psychology, starting in the 1970's, has pretty clearly demonstrated that viewing the brain as a passive computer that responds to inputs and processes data is wrong. Brains exist in bodies. Those bodies are interacting with and intervening in the world, and a considerable portion of whatever it is that brains do is based on sensorimotor metaphors derived from these interactions. And I should point out here that the meaning of metaphor is not the usual "How shall I compare thee to a summer's day" sense. Rather, the cognitive theory of metaphor involves wholesale export of reasoning methods from one domain into a completely different one, e.g. using the ability of the brain to reason about navigation to instead think about mathematics. This is what a number line is. When the metaphor changes (as it did in mathematics from numbers as enumeration of objects to labeling positions along a path), the meaning changes as well (as when the enumeration metaphor excluded the concept of zero as well as irrational numbers from the world of numbers -- the Pythagorean position -- to requiring them to be numbers since otherwise those positions along a path lack labels).

In 2015, the roboticist Rodney Brooks chose the computational metaphor of the brain as his pet hate in his contribution to a collection of essays entitled This Idea Must Die. Less dramatically, but drawing similar conclusions, two decades earlier the historian S Ryan Johansson argued that “endlessly debating the truth or falsity of a metaphor like ‘the brain is a computer’ is a waste of time. The relationship proposed is metaphorical, and it is ordering us to do something, not trying to tell us the truth.”

Reverse engineering a computer is often used as a thought experiment to show how, in principle, we might understand the brain. Inevitably, these thought experiments are successful, encouraging us to pursue this way of understanding the squishy organs in our heads. But in 2017, a pair of neuroscientists, Eric Jonas and Konrad Paul Kording*, decided to actually do the experiment on a real (and simple) computer chip, the MOS 6507 processor that was used in popular video games in the 70's and 80's. Things did not go as expected.

They deployed the entire analytical armament of modern neuroscience to attempt reverse engineering the CPU. Despite the fact that there is a clear explanation for how the chip works, they were unable to detect from outside the hierarchy of information processing that occurs inside it. As Jonas and Kording put it, the techniques fell short of producing “a meaningful understanding”. Their conclusion was bleak: “Ultimately, the problem is not that neuroscientists could not understand a microprocessor, the problem is that they would not understand it given the approaches they are currently taking.” This is directly related to neural networks in general as they are the blackest of black boxes. No one knows how they convert input into output, and this experiment suggests that such knowledge cannot be obtained with current techniques. Absent that knowledge, claims of "sentience" or "intelligence" are specious.

*Could a Neuroscientist Understand a Microprocessor?

[mostlysignssomeportents]

Scraping to alienate creative workers’ labor is bad, actually.

Creative workers are justifiably furious that their bosses took one look at the plausible sentence generators and body-snatching image-makers and said, “Holy shit, we will never have to pay a worker ever again.”

Our bosses have alarming, persistent, rock-hard erections for firing our asses and replacing us with shell-scripts. The dream of production without workers goes all the way back to the industrial revolution, and now — as then — capitalists aspire to becoming rentiers, who own things for a living rather than making things for a living.

Creators’ bosses hate creators. They’ve always wished we were robots, rather than people who cared about our work. They want to be able to prompt us like they would a Stochastic Parrot: “Make me E.T., but the hero is a dog, and put a romantic sub-plot in the second act, and then have a giant gunfight at the climax.”

Ask a screenwriter for that script and you’ll have to take a five minute break while everyone crawls around on the floor looking for the writer’s eyeballs, which will have fallen out of their face after being rolled so hard.

Ask an LLM for that script and it’ll cheerfully cough it up. It’ll be shit, but at least you won’t get any lip.

Same goes for art-directors, newspaper proprietors, and other would-be job-removers for whom a low-quality product filled with confident lies is preferable to having to argue with an uppity worker who not only expects to have a say in their work assignments, but also expects to get paid for their work.

-How To Think About Scraping: In privacy and labor fights, copyright is a clumsy tool at best

Image: syvwlch (modified) https://commons.wikimedia.org/wiki/File:Print_Scraper_(5856642549).jpg

CC BY 2.0 https://creativecommons.org/licenses/by/2.0/deed.en

[tonitoewyn]

this is my lane and my knowledge here might be of interest: I'm a digital art history research assistant. We just did a survey on how well people can distinguish AI-made art from human-made originals, asking 900 people to choose the human-made painting out of two images, one being the original and the other being a midjourney generated piece (we asked it to make paintings "in the style of" artists, detailing the content of the paintings). The general outcome is that people choose the correct option roughly half the time - mostly because they couldn't actually distinguish them, and if you take a wild guess, a 50% success rate is likely. HOWEVER, there were clear differences between the paintings that were more easily distinguished and those that confused people. The pair of images where 78% of participants answered correctly:

First one is The Annunciation by Joseph Erns Tunner (1830), second is midjourney 2023. People recognized the AI fake by realism but done wrong: the midjourney architecture doesn't make sense, the surfaces are too smooth, the hands and proportions are weird. The task most people guessed wrong, at 14% success rate?

First one is Sam Francis 1965, the second is midjourney 2023. People presumably got this wrong because they expect human-made art to be what they have seen before, and AI-made art to be weird, unusual, "wrong". Of course, AI art is still in developement, but for what it is at the moment I can derive three points from this: 1) People at large think that AI art is weird and human-made art is what they expect. Do with that knowledge what you want. 2) AI art so far actually sucks at realism. There is a specific AI-art look that most people who have been confronted with this before can recognize, because it is too smooth while also making clear mistakes in the construction of architecture and anatomy which are easily recognized by the human eye (because your eyes already saw more images than any AI training model has by the time you were three years old). If you make realistic art, especially showing humans, AI cannot do what you do. 3) At the same time AI art isn't creative. It doesn't subvert expectations, it learned from what is already there and reproduces it as well as it can (like large language models, AI art generators also are stochastic parrots - babbling without real content understanding). If you make weird art - AI isn't going to be on par with you for even longer, because you can come up with actual new things, taking all your specific knowledge, background and experiences into account. (However, presuming that people don't update their expectations, at some point you might get an annoying amount of people mistaking you for AI, which is a whole new can of worms.) Honestly the only kind of art AI can do really well is soulless, faceless corporate images and the corporate designers who did that so far should be doing better anyway

Robots may already be replacing a lot of art jobs but that's all the more reason to make all of your art weirder. Realistic illustration is dying but if that's what you trained for you still have all kinds of skills applicable to stylistic work. Go ahead and just draw like a toddler who somehow spent 10 years on just color theory.

[lilietsblog]

I don't really think this holds - I think most people agree that 9 year olds are capable of thought, and yet I think most of us would agree that asking a 9 year old to find legal cases for you is a bad idea.

There is a difference between a 9 year old and an ~artificial intelligence~. You'd trust google to find things for you more than you'd trust a 9 year old with a catalog, right? Because it's a program specialized in finding things, it can't get distracted or bored or lack the attention span or decide to prank you or... That's why people trust ChatGPT in a way they wouldn't trust a little kid - because a program is supposed to just *work*. Presumably on release it's "of age".

That's why you need to understand what the program actually *does* when it works. ChatGPT fulfills its function admirably. That function does not include understanding the text it produces in relation to the real world entities humans read it as referring to.

The thinking vs non-thinking gap is one that is philosophical in nature, not empirical, whereas whether or not chatgpt can be trusted more than a 9 year old to find your legal cases is empirical, and it can't.

I disagree with this premise, as you can guess. To me, thinking is a pretty specific thing you do. I can do something without thinking, I do something without thinking all the time, and I full well know the results can be... not what was intended. Because thinking is a specific action of, like, verifying the context and double checking inputs and outputs for matching it. Which ChatGPT is not capable of.

Again, you're drawing a clear delineation here that I don't think holds in practice. If you don't think anything chatgpt has created is novel you also don't think anything most humans have created is novel (which is fine, most people don't write novel things). But you're sort of… judging the output based on what you know about the input, which is antiempirical imo.

Oh I am not saying the output is not novel. It's novel in the same sense that no two snowflakes are identical or something like that. It's sure not repeating, but that doesn't make it *useful*.

When we talk about novel output in text creation context, combinatorics of all possible words in all possible positions is usually not what is meant. What is meant is new *communication*: translating an idea/thought into words in a way that will generate the same idea/thought in the person reading it.

The problem with ChatGPT is that it's missing step 1. It doesn't generate an idea/thought then turn it into words, it generates words directly. Which make it seem to people like there was an idea/thought on the other end, since that's how words work in every other context, but there wasn't.

See, the point of language is communication, meaning that to evaluate how good someone or something is at it, you do in fact need to compare input with output. "I meant to say X1, you understood X2".

And ChatGPT straight up doesn't do that. Like there's no X1. Any X2 is generated purely in your mind.

Like, the - the basic function of language is not in play. It's as communicative as throwing dice for which letter is next, it's just got better weighted statistics so you get something that *looks* like an actual text.

I think truly believing this requires also believing that a lot of what other humans do is stochastic parroting and not thinking, and that's not really a bullet I'm willing to bite.

As explained above, that's not what's going on. Even humans who stochastically parrot things still have communicative intent. Sometimes the intent is as simple as "I do in fact belong to the same social group as you", with no further thought put into words. But that's still intent, that's still a meaning being communicated. This person is *telling you* that they belong. (This statement can be a lie. You can echo the party line while secretly being a dissident, agree with your racist relatives at dinner while intending to vote Democrat, etc. Since there's an actual relation to reality in communicative intent, it has a truth value.)

ChatGPT saying the same parroted phrase isn't telling you that it belongs. It's not telling you anything at all, it's presenting you with a statistically likely but probably technically novel (or not, some answers are so canned/simple ChatGPT will reproduce them exactly) sequence of words. The only thing it's "telling you" in the communicative intent sense is that it works, the program is functional, all input and output devices are connected. (And also that it's decided this is a probable sequence of words.)

Which, like - "this is a probable sequence of words" can be a pretty fun output to play with, if you understand that that's what it's telling you. You know how there was a post circulating a while back about some guy feeding the "AI" a bunch of protestant virtue names and getting a new bunch of "virtue" names that the "AI" thought belonged in the same group? It's funny, and it's not funny because the "AI" was making a joke, and it can be read as mild social commentary not because the "AI" was making social commentary. The "AI" was simply following the algorithm, any communicative intent - humor or otherwise - belongs to the humans involved. As in that case the program was fairly simplistic, it was pretty obvious.

But the realism of ChatGPT is muddling the waters, and people read communicative intent into what it's saying where there's none. That one lawyer asked the AI if the cases it found were real. If it were a 9 year old on the other end, they'd either tell the truth ("no, I made them up") or lie ("yes, they're real"). But the AI did not answer their question in relation to real life facts the question was referencing. It took into account the whole context of the interaction, compared it to analogous interactions in its training materials, and came up with a statistically probable answer ("Yes, they're real"). All it really says - all the answer depends on - is that usually when asked this question in this context, approximately that is what people answer.

Communicative intent is where I draw the line on thinking vs non-thinking. If I'm distracted and my mom asks me a yes or no question and I answer something like "mhm" without even processing what the question was - without thinking - there was no communicative intent on my part! I wasn't even aware of what was being asked! If my mom takes my "mhm" as communicating "yes", it's as likely as not to be wrong. Not because I lied or was wrong, but because I wasn't even answering the question. I was producing noises.

This is the meaning in which it's important to understand that the AI is not thinking. It's not communicating with you. It's producing text.

this is nothing new but it really struck me once i was finally able to catch up with it

so many people like to say "oh! the machine is not actually thinking! it has no understanding of concepts or what the words mean! its just predicting the next token! stochastic parrot!"

and at first i was like, sure, yeah. its a neat trick but just a trick. much like if we trained a machine to play chess, all it can do is play chess, it has no real understanding of a world beyond the pieces and the board and the rules by which they can be moved. if we trained a machine to cook pizzas it would have no understanding of reality beyond the constituent elements of a pizza, it probably wouldnt even be able to extrapolate concepts about taste or italian culture or the human digestive system or anything like that.

sure

but

see, here. the thing is we did not train the AI on chess or pizzas or any other narrow task. we trained it on language. which is probably one of the least narrow domains we could have possibly chosen.

and what is more, people keep saying that it "generates text" but it doesnt JUST generate text, if it were to JUST generate text it would just give us a shitton of random words and that would be it, that is "text" right?

but no, its generating coherent text, its generating text where we humans can read coherent ideas from. its, in a sense, trained to generate not just text but coherent ideas. and guess what is another word for the generation of coherent ideas...

yeah, that's right, its called thinking

and one might say, well its not ACTUALLY thinking, its merely doing a great job at pretending to think, its simulating the act of thinking. which fair enough, but for that matter you might as well say that deep blue is not actually a chess player, is just "simulating" playing chess.

the funny thing about simulated thinking is that if you do it well enough for for sustained periods of time is basically indistinguishable from the actual thing.