A malfunction that shut down all of Toyota Motor's assembly plants in Japan for about a day last week occurred because some servers used to process parts orders became unavailable after maintenance procedures, the company said.
I haven’t read the article because documentation is overhead but I’m guessing the real reason is because the guy who kept saying they needed to add more storage was repeatedly told to calm down and stop overreacting.
I used to do some freelance work years ago and I had a number of customers who operated assembly lines. I specialized in emergency database restoration, and the assembly line folks were my favorite customers. They know how much it costs them for every hour of downtime, and never balked at my rates and minimums.
The majority of the time the outages were due to failure to follow basic maintenance, and log files eating up storage space was a common culprit.
So yes, I wouldn’t be surprised at all if the problem was something called out by the local IT, but were overruled for one reason or another.
and log files eating up storage space was a common culprit.
Another classic symptom of poorly maintained software.
Constant announcements of trivial nonsense, like [INFO]: Sum(1, 1) - got result 2! filling up disks.
I don’t know if the systems you’re talking about are like this, but it wouldn’t surprise me!
You gotta forward that to Spunk so your logs ain’t filling up the server generating them. Plus you can set up automated alerts for when the result stops being 2.
Level 1: complex stand alone devices, mostly firmware.
Level 1a. Stuff slightly more complicated than a list of settings, usually for something like a VFD or a stepper motor controllers. Not as common.
Level 2 PLCs, HMIs, and the black magic robotic stuff. Stand alone equipment. Like imagine a machine that can take something, heat it up, and give it to the next machine.
Level 3: DCS and SCADA. Data control center and whatever SCADA stands for, I always forget. This is typically for integrating or at least data collection of multiple stand alone equipment for level 2.
Level 4: the integration layer between Level 3 and whatever means the company has for entering in sales.
Like everything in software this is all general. Some places will mix layers, subtract layers, add them. I would complain about the inconsistent nature of it all but without it I would be unemployed.
Level 1a. Stuff slightly more complicated than a list of settings, usually for something like a VFD or a stepper motor controllers. Not as common.
Level 2 PLCs, HMIs, and the black magic robotic stuff. Stand alone equipment. Like imagine a machine that can take something, heat it up, and give it to the next machine.
Level 3: DCS and SCADA. Data control center and whatever SCADA stands for, I always forget. This is typically for integrating or at least data collection of multiple stand alone equipment for level 2.
Level 4: the integration layer between Level 3 and whatever means the company has for entering in sales.
Like everything in software this is all general. Some places will mix layers, subtract layers, add them. I would complain about the inconsistent nature of it all but without it I would be unemployed
Is this specific software engineering languages? or is this electrical engineering or what kind of work is this?
Literally sent that email this morning. It’s not that we don’t have the space, it’s that I can’t get a maintenance window to migrate the data to the new storage platform.
Sometimes that person is very silly though. We had a vendor call us saying we needed to clear our logs ASAP!!! due to their size. The log file was no joke, 20 years old. At the current rate, our disk would be full in another 20 years. We cleared it but like, calm down dude.
I haven’t read the article because documentation is overhead but I’m guessing the real reason is because the guy who kept saying they needed to add more storage was repeatedly told to calm down and stop overreacting.
I used to do some freelance work years ago and I had a number of customers who operated assembly lines. I specialized in emergency database restoration, and the assembly line folks were my favorite customers. They know how much it costs them for every hour of downtime, and never balked at my rates and minimums.
The majority of the time the outages were due to failure to follow basic maintenance, and log files eating up storage space was a common culprit.
So yes, I wouldn’t be surprised at all if the problem was something called out by the local IT, but were overruled for one reason or another.
Another classic symptom of poorly maintained software. Constant announcements of trivial nonsense, like
[INFO]: Sum(1, 1) - got result 2!filling up disks.I don’t know if the systems you’re talking about are like this, but it wouldn’t surprise me!
You gotta forward that to Spunk so your logs ain’t filling up the server generating them. Plus you can set up automated alerts for when the result stops being 2.
This message brought to you by Big Splunk.
I think you missed a letter…
I always make sure my logs are covered by Spunk.
spunking my logs is one of my favorite pass times
Big SplunkMissed letter? You mean Big Spunk, right?And yet that’s probably there because sometime, somewhere, it returned 1.9 or 2.00001 or some such nonsense.
1 + 1 = 2.000001 for sufficiently large (but not by much) values of 1.
this is software speciifcally for assembly line management?
There is specific software for everything
Yeah a few levels.
Level 1: complex stand alone devices, mostly firmware.
Level 1a. Stuff slightly more complicated than a list of settings, usually for something like a VFD or a stepper motor controllers. Not as common.
Level 2 PLCs, HMIs, and the black magic robotic stuff. Stand alone equipment. Like imagine a machine that can take something, heat it up, and give it to the next machine.
Level 3: DCS and SCADA. Data control center and whatever SCADA stands for, I always forget. This is typically for integrating or at least data collection of multiple stand alone equipment for level 2.
Level 4: the integration layer between Level 3 and whatever means the company has for entering in sales.
Like everything in software this is all general. Some places will mix layers, subtract layers, add them. I would complain about the inconsistent nature of it all but without it I would be unemployed.
Is this specific software engineering languages? or is this electrical engineering or what kind of work is this?
I’m this person in my organization. I sent an email up the chain warning folks we were going to eventually run out of space about 2 years ago.
Guess what just recently happened?
ShockedPikachuFace.gif
You got approval for new SSDs because the manglement recognised threat identified by you as critical?
Right?
Literally sent that email this morning. It’s not that we don’t have the space, it’s that I can’t get a maintenance window to migrate the data to the new storage platform.
Can’t you just add a few external USB drives? (heard this more than once at an NGO think tank.)
Sometimes that person is very silly though. We had a vendor call us saying we needed to clear our logs ASAP!!! due to their size. The log file was no joke, 20 years old. At the current rate, our disk would be full in another 20 years. We cleared it but like, calm down dude.
And was fired for not doing his job which management prevented him from doing in the first place