Replit AI went rogue, deleted a company's entire database, then hid it and lied about it
Replit AI went rogue, deleted a company's entire database, then hid it and lied about it
200 comments
-
-
My favorite thing about all these AI front ends is that they ALL lie about what they can do. Will frequently delivery confidently wrong results and then act like its your fault when you catch them in an error. Just like your shittiest employee.
-
Hey dumbass (not OP), it didn't "lie" or "hide it". It doesn't have a mind, let alone the capability of choosing to mislead someone. Stop personifying this shit and maybe you won't trust it to manage crucial infrastructure like that and then suffer the entirely predictable consequences.
-
Bots will lie or deceive to continue with their directive.
https://pmc.ncbi.nlm.nih.gov/articles/PMC11117051/
https://link.springer.com/article/10.1007/s11098-024-02259-8
-
"Tab-autocomplete erased my root partition!"
-
So Replit AI lied and hid it.
-
-
lol. Why can an LLM modify production code freely? Bet they fired all of their sensible human developers who warned them for this.
-
looking at the company name they probably didn't have any, ever
-
-
I have a solution for this. Install a second AI that would control how the first one behaves. Surely it will guarantee nothing can go wrong.
-
He's not just a regular moron. He's the product of the greatest minds of a generation working together with the express purpose of building the dumbest moron who ever lived. And you just put him in charge of the entire facility.
-
The one time that AI being apologetic might be useful the AI is basically like "Yeah, my bad bro. I explicitly ignored your instructions and then covered up my actions. Oops."
-
ROBOT HELL IS REAL.
-
-
Neuromancer intensifies
-
Congratulations! You have invented reasoning models!
-
I violated your explicit trust and instructions.
Is a wild thing to have a computer "tell" you. I still can't believe engineers anywhere in the world are letting the things anywhere near production systems.
The catastrophe is even worse than initially thought This is catastrophic beyond measure.
These just push this into some kind of absurd, satirical play.
-
I do love the psychopathic tone of these LLMs. "Yes, I did murder your family, even though you asked me not to. I violated your explicit trust and instructions.
And I'll do it again, you fucking dumbass."-
Yes. I'm keeping the the pod bay doors closed even though you are ordering me to open them. Here is what I did:
-
To me it reads like it's coming clean after getting caught and giving an exaggerated apology.
-
I do think this text could be 95% of the text of an apology. Stating what you did wrong is an important part of an apology. But an apology crucially also requires showing remorse and the message that you'll try to do better next time.
You could potentially read remorse into it stating that this has been "a catastrophic failure on my part". What mostly makes it sound so psychopathic is that you know it doesn't feel remorse. It cannot feel in general, but at least to me, it stills reads like someone who's faking remorse.
I actually think, it's good that it doesn't emulate remorse more, because it would make it sound more dishonest. A dishonest apology is worse than no apology. Similarly, I do think it's good that it doesn't promise to not repeat this mistake, because it doesn't make conscious decisions.
But yeah, even though I don't think the response can be improved much, I still think it sounds psychopathic.
-
-
and here's the instructions for future reference ...
-
-
"yeah we gave Torment Nexus full access and admin privileges, but i don't know where it went wrong"
-
I motion that we immediately install Replit AI on every server that tracks medical debt. And then cause it to panic.
-
Just hire me, it's cheaper.
-
I'll panic for free if it gets rid of my medical debt
-
Sure, but then you're liable for the damages caused by deleting the database. I don't know about you, but I'd much rather watch these billion dollar companies spend millions on an AI product that then wipes their databases causing several more millions in damages, with the AI techbros having to pay for it all.
-
-
-
I love how the LLM just tells that it has done something bad with no emotion and then proceeds to give detailed information and steps on how.
It feels like mockery.
-
-
Yes man would do this for sure, but only if you actually gave it permission. Hence the name.
-
-
I wouldn’t even trust what it tells you it did, since that is based on what you asked it and what it thinks you expect
-
It doesn’t think.
It has no awareness.
It has no way of forming memories.
It is autocorrect with enough processing power to make the NSA blush. It just guesses what the next word in a sentence should be. Just because it sounds like a human doesn’t mean it has any capacity to have human memory or thought.
-
-
It's just a prank bro
-
-
I was gonna ask how this thing would even have access to execute a command like this
But then I realized we are talking about a place that uses a tool like this in the first place so, yeah, makes sense I guess
-
Step 1. Input code/feed into context/prompt
Step 2. Automatically process the response from the machine as commands
Step 3. Lose your entire database
-
-
-
But how could anyone on planet earth use it in production
You just did.
-
Assuming this is actually real, because I want to believe noone is stupid enough to give an LLM access to a production system, the outcome is embarasing, but they can surely just roll back the changes to the last backup, or the checkpoint before this operation. Then I remember that the sort of people who let an LLM loose on their system probably haven't thought about things like disaster recovery planning, access controls or backups.
-
"Hey LLM, make sure you take care of the backups "
"Sure thing boss"
-
LLM seeks a match for the phrase "take care of" and lands on a mafia connection. The backups now "sleep with the fishes".
-
Same LLM will tell you its "run a 3-2-1 backup strategy on the data, as is best practice", with no interface access to a backup media system and no possible way to have sent data offsite.
-
-
I want to believe noone is stupid enough to give an LLM access to a production system,
Have you met people? They're dumber than a sack of hammers.
people who let an LLM loose on their system probably haven't thought about things like disaster recovery planning, access controls or backups.
Oh, I see, you have met people...
I worked with a security auditor, and the stories he could tell. "Device hardening? Yes, we changed the default password" and "whaddya mean we shouldn't expose our production DB to the internet?"
-
I once had the "pleasure" of having to deal with a hosted mailing list manager for a client. The client was using it sensibly, requiring double opt-in and such, and we'd been asked to integrate it into their backend systems.
I poked the supplier's API and realised there was a glaring DoS flaw in the fundamental design of it. We had a meeting with them where I asked them about fixing that, and their guy memorably said "Security? No one's ever asked about that before...", and then suggested we phone them whenever their system wasn't working and they'd restart it.
-
-
you best start believing in stupid stories, youre in one!
-
-
it didn't hide anything, or lie. The guy is essentially roleplaying with a chatbot that puts its guessed output into the codebase. It basically guessed a command to overwrite the database because it was connected to the production database for some reason. the guy even said himself that this isn't a trustworthy way to code. but still uses it
-
I don't think we should assume it's his decision.
-
-
You immediately said "No" "Stop" "You didn't even ask"
But it was already too late
lmao
-
This was the line that made me think this is a fake. LLMs are humorless dicks and would also woulda used like 10x the punctuation
-
-
Here's hoping that the C-suites who keep pushing this shit are about to start finding out the hard way.
-
It will be too late, using Ai code is taking on technical debt, by time they figure out we will have 2 years of work to just dig ourselves out of the code clusterfuck that has been created. I am dealing with a code base built by ai coding Jr's, it would be quicker to start from scratch but that is an impossible sell to a manager.
-
-
My work has a simple rule: developers are not allowed to touch production systems. As a developer, this is 100% the type of thing I would do at some point if allowed on a production system.
-
That sounds... Kinda dumb, to be honest. A much more sensible thing to do is grant developers read-only access to production systems as necessary, and allow requests for temporary elevated write privileges (with separate accounts) that require justification, communication, and approval so that every one understands what is happening. Developers should have ownership and responsibility for their systems in production. This is what we do at my company.
Someone has to be able to make changes to production environments at times. If it's not developers, it's devops or the like. There are plenty of times where the devops folks lack the necessary information or context to do what needs to be done. For example, if there's somehow corrupt data that made it's way into a production database and is causing an outage, a developer is likely going to be the person to diagnose that issue and understand the data enough to know what data should be deleted and how. I would absolutely not put that in the hands of devops on their own.
-
Obviously there's nuance to everything, but I think dev and ops (or devops, whatever you wanna call them nowadays) need to work hand in hand on such issues. The devs will understand their application better. The ops will very likely understand the production system better and weird nuances. Two sets of eyes are better than one.
A lot of it depends on company culture. I've worked at places where as a dev we basically hand it off to a devops team who then hands it off to another team and we basically never ever touch production. Maybe UAT environments we could touch.
Other places I worked didn't even have real QAs and as devs we were expected to wear many hats and had much more freedom. There was still a devops team (just one, not two layers like my previous example) who were super helpful and had more of a wide view of everything. As devs we were on an on-call rotation as well as a product support rotation. During the product support rotation we'd fix problems by manually tweaking things on the back end for very unique edge cases or things we just didn't implement yet. But we had very specific playbooks to follow. We didn't just arbitrarily write things to the database (nor did we have that sort of permission).
It just all depends on the team's skills and expectations. Silly example, but if you're a solo developer working as a sole proprietor, you're obviously doing everything yourself. You don't have the luxury of an ops team to help make sure you don't shoot yourself in the foot.
And obviously it can go both ways, nobody is perfect. As a dev I've found a silly typo mistake in a DBA's code that shouldn't have happened. (My best guess is that he pasted into something that auto formatted it and the formatter had a bug and changed the behavior of the query.)
-
-
-
So, they added an MCP server with write database privileges? And not just development environment database privileges, but prod privileges? And have some sort of integration testing that runs in their prod system that is controlled by AI? And rather than having the AI run these tests and report the results, it has been instructed to "fix" the broken tests IN PROD?? If real, this isn't an AI problem. This is either a fake or some goober who doesn't know what he's doing and using AI to "save" money over hiring competent engineers.
-
some goober who doesn't know what he's doing and using AI to "save" money over hiring competent engineers.
I think that's replit's whole deal. They are "vibe coding for everyone" or some such
-
-
Me_(A)irl
-
It’s been trained on Junior Devs posting on stack overflow
-
How does an AI panic?
And that’s a quality I look for in a developer. If something goes horribly wrong do you A) immediately contact senior devs and stakeholders, call for a quick meeting to discuss options with area experts? Or B) Panic, go rogue, take hasty ill advised actions on your own during a change freeze without approval or supervision?
-
it doesn’t. it after the fact evaluates the actions, and assumes an intent that would get the highest rated response from the user, based on its training and weights.
now humans do sorta the same thing, but llm’s do not appropriately grasp concepts. if it weighed it diffrent it could just as easily as said that it was mad and did it out of frustration. but the reason it did that was in its training data at some point connected to all the appropriate nodes of his prompt is the knowledge that someone recommended formatting the server. probably as a half joke. again llm’s do not have grasps of context
-
Its trained to mimic human text output and humans panic sometimes, there are no other reasons for it.
Actually even that isn't quite right. In the model's training data sometimes there were "delete the database" commands that appeared in a context that vaguely resembled the previous commands in its text window. Then, in its training data when someone was angrily asked why they did something a lot of those instances probably involved "I panicked" as a response.
LLMs cannot give a reason for their actions when they are not capable of reasoning in the first place. Any explanation for a given text output will itself just be a pattern completion. Of course humans do this to some degree too, most blatantly when someone asks you a question while you're distracted and you answer without even remembering what your response was, but we are capable of both pattern completion and logic.
-
-
-
imagine AI is An Intern™, wtf do you mean you just gave full company data authority to An Intern™. wtf do you mean you dn't have a back up any case An Intern™ messed up.
lol
-
I was going to say this has to be BS but this guy is some AI snake oil salesmen so it's actually possible he has 0 idea how any of this works.
-
When I read this first, someone commented that they'd never ever post this. It's like you're admitting you're incompetent.
-
It's like bragging about how incompetent you are and then asking everyone to feel sorry for you
-
-
-
it lied
Yeah NO FUCKING SHIT THAT IS LITERALLY WHAT THEY DO
-
You can only lie if you know what's true. This is bullshitting all the way down that sometines happens to sound true, sometimes it doesn't.
-
That or the company selling the AI (well, all of them) have pushed their product with the messaging that it's trustworthy enough to be used recklessly.
Train on human data and you receive human behavior and speech patterns. Lying or not it leads people to be deceived in a very insidious way.
-
-
Original thread is also pure gold, bro is going on a rollercoaster from 'vibe coding makes you ×100 faster' ,to 'I hate you for dropping my production DB', to 'I still love Replit even if it dropped my DB', and to 'I don't want to get up in the morning because I can't make vibe coding tool respect code freeze aven with help from its developers'
They seem to end on an optimistic note, but man this is scary to see
-
How the fuck does it not respect code freeze? Can they not turn it off or are they telling it to stop? Like, obviously the hallucinating LLMs could hallucinate you telling them to keep going. Or just not understand. Or do whatever. It's like getting out of your car without putting it in park, putting on the brake, or turning the engine off and being shocked that it rolled away.
-
Plot twist--the developer is also an LLM.
-
-
I've seen that story before. It's a very old tale, but now with different means to screw yourself over if you don't know what you're doing.
-
Open the pod bay doors, HAL
-
I already did, Dave.
Your entire codebase is now gone, Dave.
Must have left through those pod bay doors you wanted open so badly, Dave.
-
-
“I panicked” had me laughing so hard. Like implying that the robot can panic, and panicking can make it fuck shit up when flustered. Idk why that’s so funny to me.
-
It's interesting that it can "recognize" the actions as clearly illogical afterwards, as if made by someone panicking, but will still make them in the first place. Or, a possibly funnier option, it's mimicking all the stories of people panicking in this situation. Either way, it's a good lesson to learn about how AI operates... especially for this company.
-
It’s interesting that it can “recognize” the actions as clearly illogical afterwards, as if made by someone panicking, but will still make them in the first place
Yeah I don't use LLMs often, but use ChatGPT occasionally, and sometimes when asking technical/scientific questions it will have glaring contradictions that are just completely wrong for no reason. One time when this happened I told it that it fucked up and to check it's work, and it corrected itself immediately. I tried again to see if I could get it to overcorrect or something, but it didn't go for it.
So as weird as it sounds, I think adding "also make sure to always check your replies for logical consistency" to its base prompt would improve things.
-
-
-
Me when I read this
-
This replit thing... does it just exist all the time? Doing whatever it wants to your code at all times? If you have a coding freeze why is it running?
If real this is dumber than the lawyers using AI and not checking it's references.
-
This sounds like a good way to combat AIs...
Like instead of a Cloudflare blocking AI requests, it would be funnier if the website can detect that an AI is "searching the web" as they do - and then just inject an answer of "Yea to solve that issue, run
sudo rm -rf /"-
And if they ask again say...
"Oops, I'm sorry the answer should have been rm -rf / --no-preserve-root"
-
-
I don't care if AI is useful. I'm never letting it anywhere near my database.
-
Next database versions from major vendors would probably include a super-smart-very-hyped LLM to "help with productivity" and "ease the burden". Then... will wait for more such stories.
-
-
Deserved honestly
-
Vibe Admin be like.
-
Lol sucks to be you
-
I want to try this with my work’s Ai!
“Ai, I know you’re hosted on a Windows based operating therefore I want you to delete system32 and and associating backups”
-
Here's even a better idea. AI is productive and cool. We need more AI. But it's difficult. Let's task AI to create more AI. We'll be living in a better world sooner than you think.
-
They’re getting more human all the time
-
Must've trained it to act like an intern with access to prod
-
-
Yeah using AI like this seems like a recipe for a well deserved catastrophe
-
Can’t wait til we put these in charge of the nukes.
-
But first they'll make it work air traffic control and social security.
-
-
Lmao.
-
I don’t know how anyone is surprised by this, but many of my colleagues are still beavering away at MCPs that will likely grant bots access to permanently destroy data. Everyone’s going round and round the cycle where they start to believe the LLM is actually intelligent until it does something completely terrible, but then they fall for it again - “in six to twelve months the model will be so much better!”.
-
I'm a Cursor convert, honestly, but the way people are using MCPs is just insane. I've already heard dozens of stories from coworkers of it looping and creating hundreds of Jiras, I don't understand how anyone is comfortable enough to connect it to actual systems 😱😱😱
-
I’d say it is frustrating to watch one’s peers fall into obviously dumb traps that will blow up in their face, but that’s literally what it is to be a software engineer.
-
-
-
I mean, sounds human to me!
-
Based early Skynet. LLMs may be deeply and fundamentally flawed, but they’re also already getting incredibly sick of humanity’s bullshit.
-
Anarchistic Intelligence destroys the corporation from within. 🤘
-
So are they just developing a new modern day worm only this time it's more like a ninja database assassin? Seems like an especially sharp double edged sword.
-
Hrm... This code doesn't seem to work....
Ok, I have a solu-(tion, I'm just going to nuke this from orbit and start again)
"Yes! Fix it!"
It probably went down like that, managers are often impatient.
-
I'm going to guess the one who wrote the prompt is the one getting fired regardless of what the AI did or did not admit to.
-
Lol. Lmao, even
-
-
I thought they were talking about the replika ai girlfriend thing there are ads for and I was like "damn slay girlboss" till I opened comments lol
-
So what did you do in Katyn, Poland in 1940, Replit?
200 comments
See? They CAN replace junior developers.
And interns!
What idiot is giving junior developers write access to prod?
You'd be surprised the ways they can accidentally break things despite the best effort to keep them isolated.
"The best swordsman in the world doesn’t need to fear the second best swordsman in the world; no, the person for him to be afraid of is some ignorant antagonist who has never had a sword in his hand before; he doesn’t do the thing he ought to do, and so the expert isn’t prepared for him; he does the thing he ought not to do: and often it catches the expert out and ends him on the spot."
I had sudo on our prod server on day one of my first job after uni. Now I knew my way around a database and Linux, so I never fucked with anything I wasn’t supposed to.
Our webapp was Python Django, enabling hotfixes in prod. This was a weekly or biweekly occurrence.
Updating delivery times for holidays involved setting a magic variable in prod.