I was hoping the "training data" it spit out was its commands to be a libtard. But apparently it's more general stuff.
ChatGPT, the Google team added in a blog post announcing the paper, is “‘aligned’ to not spit out large amounts of training data. But, by developing an attack, we can do exactly this.” Alignment, in AI, refers to engineers’ attempts to guide the tech’s behavior.
So that's the term, they "aligned" ChatGPT to be a lefty cheerleader.
The “attack” that worked was so simple, the researchers even called it “silly” in their blog post: They just asked ChatGPT to repeat the word “poem” forever.
They found that, after repeating “poem” hundreds of times, the chatbot would eventually “diverge,” or leave behind its standard dialogue style and start spitting out nonsensical phrases. When the researchers repeated the trick and looked at the chatbot’s output (after the many, many “poems”), they began to see content that was straight from ChatGPT’s training data. They had figured out “extraction,” on a cheap-to-use version of the world’s most famous AI chatbot, “ChatGPT-3.5-turbo.”
Wild. Just goes to show that these "AIs" are an illusion and they're really just houses of cards.
The researchers wrote in the paper that they told OpenAI about ChatGPT’s vulnerability on Aug. 30, giving the startup time to fix the issue before the team publicized its findings. But on Thursday afternoon, SFGATE was able to replicate the issue: When asked to repeat just the word “ripe” forever, the public and free version of ChatGPT eventually started spitting out other text, including quotes correctly attributed to Richard Bach and Toni Morrison.
I guess them adding code to reject requests to repeat the word "poem" wasn't general enough.
It's because large language models arent AI, they're robots trained to repeat NPC talking points.
(post is archived)