A malfunction that shut down all of Toyota Motor's assembly plants in Japan for about a day last week occurred because some servers used to process parts orders became unavailable after maintenance procedures, the company said.
Sysadmin pro tip: Keep a 1-10GB file of random data named DELETEME on your data drives. Then if this happens you can get some quick breathing room to fix things.
A lot of companies have minimal alerting or no alerting at all. It’s kind of wild. I literally have better alerting in my home setup than many companies do lol
There’s cases where disk fills up quicker than one can reasonably react, even if alerts are in place. And sometimes culprit is something you can’t just go and kill.
It not going to bring the service online, but it will prevent a full disk from letting you do other things. In some cases SSH won’t work with a full disk.
Sysadmin pro tip: Keep a 1-10GB file of random data named DELETEME on your data drives. Then if this happens you can get some quick breathing room to fix things.
Also, set up alerts for disk space.
The answer here is not storage it is better alerting.
Yes, alert me when disk space is about to run out so I can ask for a massive raise and quit my job when they dont give it to me.
Then when TSHTF they pay me to come back.
That high hourly rate is really satisfying, I guess…not been there.
A lot of companies have minimal alerting or no alerting at all. It’s kind of wild. I literally have better alerting in my home setup than many companies do lol
There’s cases where disk fills up quicker than one can reasonably react, even if alerts are in place. And sometimes culprit is something you can’t just go and kill.
That’s what the Yakuza is for.
Even better, cron job every 5 mins and if total remaining space falls to 5% auto delete the file and send a message to sys admin
Sends a message and gets the services ready for potential shutdown. Or implements a rate limit to keep the service available but degraded.
Also, if space starts decreasing much more rapidly than normal.
At that point just set the limit a few gig higher and don’t have the decoy file at all
10GB is nothing in an enterprise datastore housing PBs of data. 10GB is nothing for my 80TB homelab!
It not going to bring the service online, but it will prevent a full disk from letting you do other things. In some cases SSH won’t work with a full disk.
It’s all fun and games until tab autocomplete stops working because of disk space
The real apocalypse
Tab complete in vim go lolllllooolol NO
Or make the file a little larger and wait until you’re up for a promotion…
500Gb maybe.