And the reason this is even tolerated is only because most people don’t understand how LLMs work. Like politicians and their “kill switch” idea if an AI “goes rogure” for example. That there’s like one persistent Claude entity, living in a big room at Anthropic HQ remembering and learning things from all the users it interacts with… Of course it’s not that, it’s an instance of a program that launches and runs the same function every time on whatever inputs it is fed. Sometimes it is set up to feed back old inputs each time so it acts like it has some memory of what your instance previously did. Training a model is sort of like creating a hologram and prompting it is sort of like taking a photo of the hologram from a certain angle.
They could of course train the next version of the model on everyone’s prompts to the previous version- and probably are. But that’s in no way necessary, and is already sort of a desparation move when they’re running out of input text.
If the trade secrets / PII you’re foolishly sharing with it are getting leaked that’s not an accidental, inevitable consequence of how AI works. It’s because the company hosting the AI is greedily harvesting and storing all your inputs, for no other reason than you agreed it could in the TOS you didn’t read, and then later oopsie they failed to secure it because they don’t really care and know no one will hold them accountable. Yor prompt is the same as any other text you sent in an input form to any website, the company has it and can decide to only use it for what it has to- or for everything it can get away with.
Oooh, this is going to upset a lot of the same people who were already upset at the idea of alleged rapists facing possible consequences.