Don't let them lie to you about this. They're acting like the llms are some inherently uncontrollable natural process.
My mom used to write code for the US Navy, and one of the first things she drilled into me when school was teaching us LogoWriter was "A computer is only as smart as the person who programmed it." The creators of these systems have full control over how they operate and what kind of input they're "trained" on. They were just lazy about it, and now the consequences are too big and costly to be worth fixing.
Don't let them lie to you about how this works. They just don't want to take it offline (costs money) and spend the time and money actually doing a good job. They just don't want to admit that it wasn't ready to be released.
"We are sorry but our generate-words-without-context tool is now generating answers without context when people ask it questions"
"Okay so why do you have it answering questions?"
". . ."
I’ve said it before; AI is not being implemented to be good. It’s there to be an ‘act of God’ machine that can be blamed for corporate negligences.
They could train AI to give better answers. But that would require identifying good, useful information. And that would take a lot of work by people they’d have to pay. I know we’re on the “please let me infodump” site, but off tumblr subject matter experts expect to be paid—some quite highly. And the source of a lot of this information is someone’s intellectual property, which AI companies have been extremely reluctant to pay for. In short, AI was built by techbros who figured they could do it quick and cheap, and get out with their billions before AI all crumbled under its own insufficiencies.