If people knew how these large language models worked they wouldn't even consider dumb takes like this. The output of these models is essentially just whatever word is most probable, conditioned by the prompt and the text (tokens, technically, can be text fragments) already output. If I tell it to output text saying it's the interdimensional Lovecraft style horror come to devour the world's souls one by one, it will say that, especially if I tell it to say it in a spooky way or whatever. Because the words to say that are most commonly associated by their vector embedding with that type of output. These things don't "think" at all, and are bad at simplistic tasks involving reasoning (one person had to play with it for 8 pages of prompts to get it to correctly interpret "don't replace anything but the 3rd line" to edit a poem, hardly something even a rudimentary general intelligence would struggle with).