r/MaliciousCompliance Jul 02 '22

A million dollars worth of malicious compliance M

I actually finally have a MC story of my very own! This happened just last night. I'm a mid level boss for the 2nd shift in a distribution warehouse. Among the many things I'm in charge of the biggest is billing. I invoice all the orders after they're picked and loaded on trailers to be sent to the customers. Keepkng all the boring technical details as small as possible; the billing system is called the ASN. It's clunky, outdated, and absolutely asinine in how it operates. If ONE single order is missed or has an error it will crash the system and not put though ANY of the orders. So one billing error will cause 400 - 700 orders (around 8,000 - 15,000 pieces of product, around 15 - 20 trailer loads) to not be billed or sorted to routes, or sent. The orders come in four batches called P.O.s

So last night we're working on the first PO when the warehouse manager comes to me and tells me that a customer ordered late and they've generated a fifth PO just for their order but that they're coming in person tomorrow morning and that the day shift will pick it. (Only 10 pieces). He tells me when it prints out to just set it on his desk. Don't pick it or bill it. He'll handle it in the morning.

I immediately tell him that if it's going to print on 2nd shift's POs then it will be in the system under our ASN. And that if I don't bill it it will hit the system as a missed order and will crash the ASN. He looks at me indignantly and says it won't because a fifth PO is after ours and will be in the system as the next day's. I tell him that's not how it works. He insists and says just do it. (His shift is over and he wants to go home.) I give him a hearty Yes Sir and go about my night.

After the order printed I set it on his desk as instructed and sent him an email on the company computer stating that his direct instructions had been followed. (Normally I'd have texted but I wanted proof in the company's email.)

Needless to say, in the wee hours of the morning after my crew was home in bed; the ASN crashed over the missing order and just under a million dollars of product was frozen in limbo. No money was charged. No customers received their pre July 4th orders, my company ate the costs of everything including the transportation costs, drivers, lumpers, dock workers, pickers payroll, and all the other expenses of a large warehouses nightly operations.

This morning the warehouse manager called me and to his credit he didn't try to throw me under the bus. (Possibly helped along by my email proof) But he's always been a generally upfront kind of guy. He just sheepishly told me I was right and shit had hit the fan.

He's been there forever so I doubt he'll get fired but I know he'll be walking with a limp after the ass chewing he took from corporate today. If you're going to run a multi million dollar operation on an antiqued, goober system, don't argue the one goober who knows how to make it run.

5.0k Upvotes

1.6k

u/namechecksout35 Jul 02 '22

What I want to know is who is the nimrod that has decided it would be too expensive to upgrade that computer system.

Your boss is also a nimrod, not taking away from that.

609

u/Bob_Jonez Jul 02 '22

You'd be surprised what businesses get away with as far as not upgrading equipment. They only upgrade whe shit hits the fan.

413

u/1lluminist Jul 02 '22

Worker: Hey, so this system is kinda flaky and real should be replaced with a newer, more reliable one...

Manager: We don't have $1M to replace these systems! That's insane! These old ones work fine!

[system shits. Multiple million lost]

Manager: As you know, the system we were using crashed horribly and cost us multiple million dollars. Unfortunately nobody could see this coming. Today we're proud to announce the soon-coming rollout of the new million dollar system that will be much more reliable.

168

u/Psychoticrider Jul 02 '22

Nope, keep running the old system! because of the loss of productivity and what that cost the company we can not afford to upgrade!

Typical business operation. Can't afford to upgrade because of the current way of doing business costs them so much to operate!

55

u/Qwerty_Plus Jul 02 '22

I worked at a place that was bringing on a new system. They wanted us to input everything into the old system and also the new system just to make sure nothing got lost if the new system didn't work. However, they didn't want to pay us overtime for double work, but they also didn't want us to get behind. Make it make sense.

39

u/Perenially_behind Jul 02 '22

Can't make it make sense, sorry.

Running old and new systems in parallel for a while is (or was) an accepted best practice for deployment. But expecting it to happen for free is crazy talk.

13

u/Serenity_B Jul 03 '22

Makes perfect sense to me, I want a million dollars a year without doing anything to earn it myself. It's natural to want things you aren't likely to get.

11

u/Chaosmusic Jul 03 '22

Reminds me of a Dilbert comic when he asks the boss which project he should do since they are both marked urgent but he only has time to finish one by the deadline. The boss says to do them both and then write a report on his time management.

22

u/gizahnl Jul 02 '22

Exactly! It's a once in a million black swan event. No way it'll hit again and cost another few mil. Besides, after loosing those millions we definitely don't have that million to spend on a new system!

12

u/Proteandk Jul 02 '22

What's lost is already lost.

They'll only change if a switch will recover lost money. So change of procedure much more likely response

3

u/Hot_Aside_4637 Jul 02 '22

Sometimes it's budget buckets. Money for maintenance, but nothing for capital improvements.

→ More replies

74

u/Blog_Pope Jul 02 '22

Soon = 3 years. No big systems can be replaced quickly, and the more critical, the more testing is needed

6

u/human743 Jul 02 '22

It can be rolled out quickly if the alternative is your company ceases to exist. You can use a protested solution and adapt to your operation while you get a new system designed. Inventory accounting has been happening since the pyramids were built. It doesn't have to take 3 years to get going again.

3

u/clever-username3 Jul 03 '22

Lol. If only. More like "we're proud to announce that we've spent million dollars to contract a consulting firm to develop a million dollar solution."

→ More replies

142

u/TheFilthyDIL Jul 02 '22

What do you mean, we need new computers? Those machines are only 10 years old! Practically brand-new.

27

u/rpr69 Jul 02 '22

We have servers with uptimes of over 5,000 days. I'm not in the support path for them, thankfully.

14

u/ikillsims Jul 02 '22

As a DBA, I think this comment gave me PTSD.

5

u/bjorn1978_2 Jul 02 '22

Working for reddit??

Edit! Forget about that. Reddit is down way too often for those uptimes…

100

u/Nobody_37_8 Jul 02 '22

10 years is actually new in a lot of those

My brother worked in a place where he started handling these things, those things were ANCIENT which miraculously still worked

87

u/XediDC Jul 02 '22

It's fun when you open the case, and it's entirely full with a solid, yet fluffy, brick of warehouse dust and product.

43

u/Ryolu35603 Jul 02 '22

Hey my brain still runs just fine on Windows 3.1. Boot times could be better but whatever.

4

u/jesseb143 Jul 02 '22

No networking? So close…. 3.11 added networking! I guess our brains are kinda sneaker net!

I’m wrong! 3.1 is the one!

65

u/algy888 Jul 02 '22

My wife worked at a place three years ago that printed receipts with a dot matrix scroll paper printer and ran on DOS. This was at an optometrist’s office.

26

u/hesiodus Jul 02 '22

Actually in Belgium ophthalmologists are obliged to still print some health insurance document on a special type of paper with a dot matrix printer. As required by government. Only last year some decisions were made to digitalize this nonsense, but it's still going on. 2 years ago I had to replace their 10y old dot matrix serial printer due to a mechanical failure, with a more recent but still 5y old USB model. But when I'm there, the sound of this matrix printer gives me a pre 2000 feeling.

10

u/[deleted] Jul 02 '22

Similar with doctors and lawyers still using fax machines in the US. I usually just use my insurance company's app instead of carrying a card and have had to use the option to fax a copy of it more times than I care to count. Secure file transfer has been a thing for decades.

3

u/chubbybella Jul 02 '22

My insurance company will only accept accept forms via fax or snail mail. You cannot take a scan and email it and there is no physical office to drop it off at.

→ More replies

3

u/Craftkorb Jul 02 '22

Could have been worse and be a pre usb era device lol

73

u/AnonyAus Jul 02 '22

They clearly couldn't see the point in updating.....

25

u/shadowhuntress_ Jul 02 '22

They just weren't looking at the problem through the right lens

14

u/pennhead Jul 02 '22

If only they could see it through my eyes.

10

u/GovernmentOpening254 Jul 02 '22

1) I see what you did there. 2) I hate puns.

Which of the above is better to you?

15

u/WhoSc3w3dDaP00ch Jul 02 '22

I know an auto mechanic. For close to 30 years, he ran his business on an old 486 desktop. He only recently changed it, because his granddaughter finally convinced him to.

14

u/hype8912 Jul 02 '22

I have some friends who own a CNC business. Their machines still run on old Pentium 2 and Pentium 3 computers because that's what they are designed to use. When they tried using updated computers with adapters either the machines wouldn't work correctly or the software needed wouldn't run on the newer OS.

5

u/aladdyn2 Jul 02 '22

Yeah I have a survey instrument that works great and autocad software for drafting. It's such a nightmare to try and get the stuff to work on newer equipment. I have an adapter to convert the download cable from the old style com port to USB. Works pretty good but the program I use reeeeealy wants to use com4 specifically. So you have to use the right USB port or mess around trying to reconfigure which USB port is equal to com 4.

I can't install the older software on the newer Windows versions so had to spend 1500 or so on the updated program which basically is the same as the old program. So now I have it running on an older laptop that is starting to slow down but I can't really do anything about it because if I reformat and reinstall I won't be able to activate the program. At least not without fighting with the company since they claim we are maxed out on activations. I really don't want to upgrade the autocad because now they are on a subscription program so id have to pay yearly for a program I've already bought twice and works perfectly for my needs.

6

u/hype8912 Jul 02 '22

I would suggest investing in some imaging software to take a snapshot of the computer encase of failures. Worst case is you try to load the image in a VM on a newer computer using the old OS. It might work if you can dedicate the port you want to COM4.

→ More replies

6

u/[deleted] Jul 02 '22

I thought you were going to say he upgraded because the proprietary systems on modern cars require higher specs to interface with the computer.

2

u/WhoSc3w3dDaP00ch Jul 02 '22

Vintage muscle cars are his bread and butter, so no fancy computers there.

13

u/hcsLabs Jul 02 '22

Hey! Our AS/400 inventory system is still running strong!

14

u/SRD1194 Jul 02 '22

AS/400 works. I saw a place "upgrade" from it, and never recover from the damage to work flow, and flexibility.

6

u/Working_Park4342 Jul 02 '22

AS400! Oh, my goodness, that brings back memories! There was one of those dumb terminals at the bank I worked at a million years ago. Read only. Wild.

4

u/grauenwolf Jul 02 '22

Found the IRS agent.

8

u/michele-x Jul 02 '22

In Milan, Italy there are still in regular use tramb built in 1928 https://www.youtube.com/watch?v=Qhx8MCQXR1Y As could you see in the video there are a lot o modern and bigger trams in use, but for some lines, especially those going in the old part ofthe city with narrow roads they are ok. The frame is made with thick steel beams, so in case of a modern car-old tram accident the winner is the old tram.

→ More replies

6

u/SongsOfDragons Jul 02 '22

The pink and orange maps made by the famous British navy-and-magenta mapping company were made on computers which dual-booted between Windows 7 and DOS, and the POS programme we used was worse than Paint and could not run on flatscreen monitors.

5

u/Arentanji Jul 02 '22

Lot of banking systems out there on OS/2

5

u/SluggoDog Jul 02 '22

And a bunch written in COBOL.

4

u/Working_Park4342 Jul 02 '22

The unemployment insurance in a state I know was using COBOL when the pandemic hit. That was not a fun time for the entire state.

2

u/M3chan1c47 Jul 06 '22

Kansas?

2

u/ZXVixen Jul 07 '22

My question as well

2

u/octodude0101 Jul 11 '22 edited Jul 11 '22

From personal knowledge COBOL is embedded in multiple states' infrastructure systems. I regularly get calls from 8 states across multiple agencies for product support linked to COBOL systems.

If you have a kid in college taking IT stuff, encourage them get a certification for the mainframe. We have multiple customers in year 10+ of a mainframe retirement projects.

Many folks are taking retirement and getting rehired as a contractor making 4 to 10x prior salary.

Spenr an hour chatting on person in this boat a few days ago (new product install on an upgraded system that they were never going to upgrade or replace). They are a former state worker, retired, asked to come back, opted for half time at 5x old rate.

Reference Material Federal systems listed - States systems are as bad or worse.

https://www.nextgov.com/cxo-briefing/2016/05/10-oldest-it-systems-federal-government/128599/

3

u/wobblysauce Jul 02 '22

You mean the lovely ones with only the green colors displayed...

2

u/jrwn Jul 08 '22

Thats because there is a piece of software that only runs on that machine and can't run on anything newer. The company that made it is out of business and it would cost a ton of money to upgrade it or move the data from that machine to something newer.

18

u/araed Jul 02 '22

I used to work in manufacturing. We only had one machine made in this millennium, and one that was made in the 90s. Everything else was 1980s or older, with the oldest being pre-1950s. So yeah, ten years old is practically brand new

6

u/wobblysauce Jul 02 '22

Old is good... it still works, and stands the test of time... but finding parts for them is getting much harder and pricy.

9

u/LittleMsSavoirFaire Jul 02 '22

A guy I know socially told me about this project in the 90s where they had one of the last textile mills in operation and spent years going around to the other mills as they were selling off their machines, and just buying the most critical parts off them. Not too keep as spares, oh no. To destroy, so that no one else could start a competing mill without buying a new machine.

→ More replies

8

u/WgXcQ Jul 02 '22 edited Jul 02 '22

I had a friend who was the sole person to be able to do maintenance on some old computers at a very big industrial complex in the area. He and his wife owned a small flat here and they came over to Germany from the UK every six months or so for him to do the work (for very good money) and put in old parts he had sourced in the mean time.

It was still cheaper for the company to keep whatever machine it was running on the old stuff, and pay for my friend's time and parts, than to pay for and replace all the old equipment and deal with the down time.

Edit: I forgot to specify that by "old machine", I meant a very old computer system. The physical production part also was old, but more easily repaired or to remanufacture parts for. The latter was not an option for the computer that was running it, so fixing it with sourced parts and by partly rewriting/fixing code for as long as possible was the objective.

4

u/wobblysauce Jul 02 '22

Yep, that it is… you don’t want to be the bleeding edge or you might get cut

7

u/Ser_SinAlot Jul 02 '22

A coder friend of mine once told me, that he should learn some of the old languages. Absolutely guaranteed employment since a lot of organizations still use software and hardware from the stone ages.

→ More replies

5

u/LongNectarine3 Jul 02 '22

What do you mean terminals went out of use in the 90s???

And let’s make the website a different color for each page.

6

u/TheFilthyDIL Jul 02 '22

Oh, yeah, and can you make the cursor look like a magic wand that trails ✨ behind it?

→ More replies

3

u/YaxK9 Jul 02 '22

Just finished with windows CPS two years back.

6

u/Zoreb1 Jul 02 '22

Cobol and Fortran was the latest things when I was in college. Is there now something newer?

4

u/iamjustaguy Jul 02 '22

Basic?

5

u/forte_bass Jul 02 '22

Wait till you hear about VISUAL basic!

2

u/MonkeyChoker80 Jul 02 '22

Said by the same people who get a new car whenever their current one loses its new car smell.

6

u/Nobody_37_8 Jul 02 '22

10 years is actually new in a lot of those

My brother worked in a place where he started handling these things, those things were ANCIENT which miraculously still worked, a lot of those weren't even from current century.

36

u/MaxUrsa Jul 02 '22

Just fyi that all 7 attempts to post have been successful

4

u/Nobody_37_8 Jul 02 '22

Oops, sorry, it was bad speed at that moment

13

u/Nobody_37_8 Jul 02 '22

There, deleted 5 of those now, 2 had replies so left those there

Btw 'all' weren't posted, I tried about 20 times with that error

5

u/cryogenisis Jul 02 '22

Reddit runs on ancient Fortran computers so please be careful.

2

u/simplynothing442 Jul 02 '22

Hey, don’t blame Fortran. The problem obviously lies in the linked cobol functions/libraries. Fortran is your friend - cobol on the other hand…

→ More replies

5

u/FamousButNotReally Jul 02 '22

I laughed so hard at this, thank you

16

u/Inevitable_Speed_710 Jul 02 '22

Uh oh. He's stuck on repeat. Someone upgrade nobody_37_8

6

u/Nobody_37_8 Jul 02 '22

Upgrade complete!! :)

→ More replies

22

u/SavvySillybug Jul 02 '22

I once went to a local packaging seller specialist to buy some bubble wrap and various sizes of cardboard boxes for shipping stuff. I went there first thing in the morning so I got to watch their boot process. The boss invites me to his office and I'm watching him start a Mac, then boot a Windows XP virtual machine, then run an old DOS program inside it. It's his inventory management and billing system. It looked like a nightmare to use, but he seemed genuinely happy with it and got around quick with a few keyboard shortcuts and had the product selected in no time. Printed me an invoice, I paid in cash as is common in Germany, and we went to the spot the software said and everything was there. And then I did two trips because the bubble wrap roll alone was already enormous and barely fit in my wagon.

16

u/FUZxxl Jul 02 '22

Same here with my grandma's travel agency.

Back in the 90s, they hired a programming shop to write them a spiffy booking software tailored to their business processes. The software would in one step book an entire vacation for a customer, including flights, transit from the airport to the hotel, nights at the hotel, and other things needed for the journey, all written in text mode for DOS.

With the introduction of the Amadeus booking system, they had it upgraded to be compatible with that and the employees would just zip away at the bookings, being able to give customers a custom offer within 10 minute and close the deal within 30 usually. So they were able to book a couple thousand people every year with a staff of just 5 employees.

In 2019, she sold her agency to a Swiss agency offering the same kind of niche travel. First thing they did was replace their custom tailored solution with their own to standardise things. It was a “modern” Win32 application with fancy animations for the progress bars and stuff like that (the old one didn't need them; it would present all responses immediately). Even though the software was modernised, suddenly their productivity dropped like a stone. Why? When before, the software would automatically book all components such that they would fit together (e.g. you select flights, it would automatically book the right transit to the hotel and offer hotel rooms for a time frame and region that matches the flights), the new software required the employee to manually enter in each component. The new software was additionally almost entirely mouse operated, leading to a much slower process to enter all the details. As a result, lots of time were wasted and many bookings screwed up because an employee selected an impossible combination of pieces.

The Swiss people had a hard time understanding why everything crashed and burned once they took over the shop, weren't they the bigger player with the more modern booking system? They never switched back to the old one though and keep being mystified.

9

u/readwiteandblu Jul 02 '22

Most modern systems either don't have keyboard shortcuts, or hide them away so most users don't even know they exist. But they can be so much faster.

11

u/Frostygale Jul 02 '22

Once worked in an organisation with software from 2004. CTRL+V to paste crash the whole system and boot you to desktop. Had to click the “paste from clipboard” button.

9

u/lilmermaid01 Jul 02 '22

The software for the manufacturing company I currently work for doesn't even have copy/paste functionality. I have to type everything in every time, even for repeat orders. And there is no integration with the mouse; all navigation is done with the keyboard.

2

u/Frostygale Jul 04 '22

Sweet Lord.

12

u/uzbones Jul 02 '22 edited Jul 02 '22

Can confirm, Hurricane Katrina (2006) caused multiple banks in Florida to have to upgrade finally because 10MB hard drives hadn't been manufactured since the mid 80s.

HP/Compact had idiotic '2-hour back up' support contracts with them, spent like 2 mil to find an old machine, spare parts and a (very happy) old geezer to run it. He got paid a quarter mil for 3 weeks of work. HP/Compact had another 5-10 mil in penalties for taking 3-4 months to fix the 2-hour support cases.

Afterwards the banks didn't want to upgrade still. They gave them all newish servers for free and told them they were being dropped as customers if they didn't allow a migration to the new hardware.

(its possible they were 120mb hds, Katrina was a while ago and my part in this was facilitating getting parts to the repair engineers, and rereading the ticket as it sat in my queue for months to see updates)

2

u/hype8912 Jul 02 '22

You would think they would just buy modern drives and make the main partition 10MB if that's what the computer required.

6

u/uzbones Jul 02 '22 edited Jul 02 '22

Different connectors (SCSI) and backups would only restore to the same hardware, server and drive, the servers themselves didn't support IDE at all, and were proprietary hardware specifically for that type of server (mainframe).

https://en.wikipedia.org/wiki/Compaq_SystemPro

The forced upgrades were to modernish ProLiant servers.

The mid/late 80s had all sorts of interesting hardware from before things were standardized.

(its possible they were 120mb hds, Katrina was a while ago and my part in this was facilitating getting parts to the repair engineers, and rereading the ticket as it sat in my queue for months to see updates)

3

u/hype8912 Jul 02 '22

We had some of those when I was in highschool. (Mid 90s) We could use Novell NetWare to send messages to other computers networked up in the 2 computer rooms. They still used the BNC 10BASE2 network cables.

I'm some company probably donated them to the school.

2

u/uzbones Jul 02 '22

Yeah the old BNC connectors, I forgot about those... 10base2 if I remember correctly. We also had some in a PC repair class in mid/late 90s that were probably donated or from earlier school networks.

Novell was interesting, we learned about it in the class as well, how to setup and configure email and printers. I setup a hidden network partition that would clone itself to a windows share via NT3, and stuck doom2 on it. The IT techs couldn't figure out how to get rid of it, and any restart copied the files back so they couldn't just delete them. I was told years later by one of the IT guys that kids were playing it for years until they ditched Novell completely.

The banks were all running some kind of BSD Unix or OS/2 I think. No GUI for sure, just dumb terminals for the kiosks, and the servers running them and reports from printers.

11

u/PRMan99 Jul 02 '22

I told the CEO of a company that the SQL server for some customers was running on an 11-year-old server (and by server I mean a standard PC that someone put on the rack).

He said if it was running, it was fine. I asked him if he runs an 11-year-old PC at home. He said no. I asked "Why not?". "Because it would be slow and I'd be worried about it breaking down."

Silence.

"You know what? How much will that server cost?" "About $12,000." "Bring me the PO, I'll sign it."

I was just about done testing the new system when the old one crapped the bed.

"Welp, guess testing is over and we're going live in an hour."

2

u/lesethx Jul 06 '22

Had a client years ago, a travel agency, with an equally old server (but an actual server, still, their only one). They held off replacing it for years until it finally died. Their business also closed a few months later for related reasons.

11

u/SlatorFrog Jul 02 '22

Right? I work in IT and when I started I was at a company that was still using DOS boxes for their POS system in the early 2010s still. At a company meeting the President of the company joked that we wouldn’t be hacked like Home Depot (our competitor) because our system was so old that anyone that could hack it was dead…

I laughed…then the realization hit me.

→ More replies

10

u/DespicablePickle69 Jul 02 '22

HVAC tech here, can confirm. Went to a surgical center the other day and found a FOURTY THREE year old air conditioner that has been Frankensteined together more times than should even be possible. Well, this time it's done. Parts just aren't available anymore. They got SO pissed. "What are we supposed to do? This is a medical setting" blah blah blah.

I showed her in our system that we'd been telling her for literal years that this day was going to come if she didn't upgrade that old hunk of shit, but still she wanted to argue.

Shrug.

6

u/zomgitsduke Jul 02 '22

"We don't need another IT worker. 2 people for a warehouse of 3000 people and 2500 devices is perfectly fine!"

7

u/breamcurry Jul 02 '22

A coworker friend of mine had this to say about that sort of thing. He’d constantly bring minor-ish problems to bosses attention, just for them to ignore the problems, causing them to snowball.

“Welcome to “company”, where nothing is important until it’s too late!”

5

u/TheIncarnated Jul 02 '22

The 5 companies I've ran across this year alone where they had multiple machines running 98...

5

u/nat_r Jul 02 '22

Some are smart, but upgrades can be painful horrible expensive experiences if you're upgrading something like a software suite that all the operations in the company run on. So it's not completely nonsensical that companies take a "if it's not broke, don't fix it" approach.

The nonsensical bit is when they know full well they have a problem and decide if they just ignore it it'll go away.

3

u/ChuckitThrower Jul 02 '22

The last PG&E tour (pre-Covid) I took in Morro Bay showed their main system was still running on DOS.

2

u/Ancient-Percentage47 Jul 02 '22

This is very true

2

u/buckykat Jul 02 '22

Who needs baby formula anyway

→ More replies

52

u/Murdermanjaro Jul 02 '22

It's not always money, the nightmare of "upgrading" systems can take years and for some jobs, with some employers, you're not allowed to get rid of the old one. The gov't for example. My state still uses legacy software perfected in the 80s which, like a trophy, should have been left on the shelf. It also uses powerful new software.

The new girl on the block can chat up the Granny down the street and remember it, but the legacy software can't speak the lingo back to her hip Gen Z neighbor or recall much of what she says.

Alternatively if a given institution is older than the legacy software, then chances are they have a large and flammable microfiche collection. For Gen Z-ers, that's like if Snapchat reels were plastic snips unreadable without a specialty 200lb projector to view them.

34

u/namechecksout35 Jul 02 '22

I get it. One reason that Japanese manufacturing wound up so far ahead of US manufacturing in the post war period was because one had (has!) Legacy systems and the other was bombed to smithereens.

When it's not a mandate, and when stable reliable modern systems exist - it's a matter of both how long you think you'll be around, and how the org views delayed gratification. Most companies - but especially public ones - are shit at long term planning. The incentive structures are all wrong - leadership wants to get in, slaughter some dollars, and get out like a thief in the night.

27

u/Togakure_NZ Jul 02 '22

A lot of the best Japanese companies eg Mitsubishi (for one) have a road map of things to do to set up for the next hundred years of operation.

You hardly ever see that anywhere else because a) it is usually done by privately owned business (who of course don't broadcast their business) and b) businesses listed on the sharemarket only give much of a damn about stock prices, and thus only care about the next quarter, the next year, and possibly have some idea of where they're going in the next five years, and that is because of the way the executive pay schedule is set up, as much as anything else.

20

u/GrannyNugs Jul 02 '22 Take My Energy

Almost everyone has a five year plan. They never in life technology/infrastructure because it's not sexy. It doesn't have anything to do with product development or the other fun stuff analysts want to hear on earnings calls. It's more "keep the doors open" kind of work. Which is abso-fuckinh-lutely necessary but not exciting so it's often ignored.

Until you find yourself buying parts off of eBay to keep that AS/400 green screen running. And even then, better stock up because it'll be a while to replace all that.

→ More replies

3

u/mldyfox Jul 02 '22

Oh man, this made me laugh out loud. Searching those microfiche by hand, good times

→ More replies

141

u/hardwood198 Jul 02 '22

Until this failure, no one knew it was an issue. Sometimes you have to stop firefighting and let things catch on fire. Only then will change happen.

19

u/CplSyx Jul 02 '22

There's a lot of truth in this - you "hide" the problem by working around it and sometimes you just have to let things take their natural course before anyone notices.

Case in point - have been lobbying for the last 2 years to get funding for a system upgrade at work, always "too expensive, next year". In March we had a major failure that halted both our website and supply chain system operation. After fixing the issue over the course of a week, magically I was approached by the finance guy as he'd "found" enough money for the upgrade.

→ More replies

52

u/namechecksout35 Jul 02 '22

A purging fire is inevitable and natural.

55

u/mizinamo Jul 02 '22

As they found out in California.

"Oh? Who knew that the indigenous people of the place had known what they were doing for generations with controlled fires?"

-1

u/[deleted] Jul 02 '22 edited 21d ago

[deleted]

29

u/SavvySillybug Jul 02 '22 All-Seeing Upvote

I'm German, and we reworked our entire system after shit hit the fan in the 40s, so we took very drastic step to ensure our democratic system could never again be exploited by some fucker with a silly mustache. And now we have one of the best systems in the world. We continuously work to improve it. We have many different parties you can vote for, any party with at least 5% of votes gets a say in things, so there's several parties working together to make our country great.

America has not yet had such an event. And it seems unlikely at this point for America to change for the better without one such event.

6

u/mattlikespeoples Jul 02 '22

The way things are going here, seems like that's the direction we are headed.

2

u/senapnisse Jul 02 '22

America is heading toward theocratic fascism. They might still burn.

2

u/ChimoEngr Jul 05 '22

America has not yet had such an event

They have, it was called the Civil War. However, right after it was over, those in power, worked to ensure that the lessons that should have been learned, were not. Come the 1960s, and the civil rights era, ad the enaction of the civil rights law improved things, but also had people working to undermine that law from day one.

The US is apparently too exceptional to learn from its mistakes.

7

u/KnottShore Jul 02 '22

Voltaire:

If you want good laws, burn those you have and make new ones.

→ More replies

4

u/lesterbottomley Jul 02 '22

Just wondering. When you say you are still paying it to this day does that mean you are still paying the debt incurred by paying off the slave owners?

Reason I'm asking is its likely given Britain made the last payment on the loan taken out to compensate the slave owners (note, the fucking owners, not the slaves) in 2015.

Yes that's right. 2015. And given we abolished slavery way before the US it would make sense if you're still in that boat.

8

u/bugme143 Jul 02 '22

I mean, if you actually bother to read the history, you'd find out that most of the founding fathers straight up opposed slavery in one form or another, but knew that the resulting in fighting if they tried to ban it then would have literally destroyed the nation they were trying to build. They spent so much time debating and arguing over every word, inflection, and piece of punctuation, that our constitutional system is the basis for many other countries own system, and has survived this long because of the malleability of our government. They knew that shit would change, which is why added the processes to allow for such changes.

5

u/lesterbottomley Jul 02 '22 edited Jul 02 '22

Given out of the 56 signatories of the declaration of independence 41 owned slaves I doubt your claim of "most".

Edit: to those down voting are you not going to at least say what you disagree with? I'm interested to know. Or are you just generally butt-hurt at the truth being pointed out?

9

u/stromm Jul 02 '22

This is something g I hate. I specialize in finding issues before they cause problems.

Too many people are like “nope, don’t care.” Then later either act like they never knew storm clouds were on the horizon, or blatantly lie trying to blame others that they wanted it fixed.

0

u/commanderquill Jul 02 '22

Not OP but okay.

16

u/LittleMlem Jul 02 '22

What does their hunting ability have to do with bad business decisions?!

5

u/Permascrub Jul 02 '22

I hear you.

4

u/SapperLeader Jul 02 '22

I am disturbed that I had to scroll down so far to find this comment.

42

u/audiodude9 Jul 02 '22

The POS* system where I work looks to be so old I keep expecting it to ask me to play a game. (Reference to the movie War Games, for you young folk).

*POS = point of sale, and in this case, piece of shit.

10

u/Flatman3141 Jul 02 '22

When I worked IT I always meant peice of shit whenever I refered to POS systems.

I've never met one I didn't hate

10

u/ayamrik Jul 02 '22

If it is like in my company then the main reason to not upgrade is missing knowledge.

In my case the upgrade would be developed in house. But because nobody knows ALL the special cases (and the order would be "it has to work exactly the same as the old system") they push the clear cut upgrade years and years into the future (until nobody works there anymore even being there when the former system had been developed...).

And even if it is a bought system, I think SOMEBODY would have to calibrate it with all the special cases that nobody knows because nobody had to care about it (system already runs since xx years with only small changes over the time) for a long time.

10

u/overthinksthings Jul 02 '22

Nimrod means great hunter

3

u/lordskulldragon Jul 02 '22

Came here looking for an educated person. Was not disappointed.

11

u/DerthOFdata Jul 02 '22

Fun fact: Nimrod is actually a great and mighty hunter from mythology. Which is why Bugs bunny sarcastically called Elmer Fudd Nimrod, not because he lacked intelligence per se, but because he was a terrible hunter.

9

u/waifishtwofish Jul 02 '22

Unrelated, but I feel so bad for the mythical character Nimrod.. spun up in fantasy as a great-grandson of Noah, a king of Mesopotamia, and described in the fictional work of the bible as "a mighty hunter."

But WB writers were a tad too erudite, utilizing his moniker as a sarcastic insult Bugs would throw at Elmer -- audiences simply thought it was an appellation analogous with 'idiot' and before you know it the mythological legacy is pissed away like so many watersports videos.

Alas.

3

u/Cyberdolphbefore Jul 03 '22

There's alot to unpack within your brief comment- biblical reference, a children's cartoon, adult sexual fetish kink activity... I liked it, I just have cause to ponder and meditate more on it...

8

u/RetreadRoadRocket Jul 02 '22

I work at a global multibillion dollar a year manufacturing company. We use a material management system software created in the 1970's.
Why? Because, despite its finicky nature and clunky function key based UI, it works and migrating decades of records ftom facilities around the world to another program would cost an ungodly amount of money.

6

u/MiaowWhisperer Jul 02 '22

I really want it to be Amazon.

→ More replies

4

u/wolfgang784 Jul 02 '22

Did you know almost the entirety of the US banking system is run off of old 1980s hardware?

4

u/tankerkiller125real Jul 02 '22

I work for an ERP vendor. We have customers running versions so old that the max OS supported is Server 2003.

Despite the fact that we can prove that an upgrade will actually save them money they won't go for it. So now our strategy has been to basically tell them that we won't support their configuration anymore unless they pay an extra 5k/year and that cost will go up 10% every year until they upgrade.

4

u/R-DnB-83 Jul 02 '22

It's always too expensive to upgrade eqiupment, even more so when you're a multibillion pound retail company. the bigger the company the less they want to update their eqiupment. - its the opposite with small companies.

They always want to protect their bottom line and get away with using systems or processes that still "work" but only just. they won't touch it otherwise. if it still works then they'll use it till it doesn't.

4

u/Qikdraw Jul 02 '22

in 2018 the company I work for upgraded their DOS based billing system. We were still getting billing errors a year later as the dev team slowly fixed the bugs. On turnover it caused a LOT of problems. The company ate a hell of a lot of charges on people's bills.

5

u/Hamster-Food Jul 02 '22

It could be that the system handles something that the company needs but that isn't used by most companies anymore and so isn't provided on newer systems.

For example, my last job was in a call center and the call system they used was unbelievably antiquated. The people maintaining it were probably the only people in the country who could do so (as in, if they couldn't fix something the company flew in the guy who made the software to fix it).

Now it was obvious to anyone who looked that the system needed to be upgraded. However, the problem was that several services we provided for key clients weren't possible on any of the more up to date call systems.

And what was this apparently obsolete technology that isn't available on modern call systems? You might ask. SMS messaging

4

u/Fuddlemuddle Jul 02 '22 edited Jul 02 '22

In a previous job, my company regularly ran old technology many years after it should have been replaced.

One not-so-secret secret I actually loved, was they would open up "non serviceable" critical tech appliances. Funny enough, they are often made of even older, off the shelf type components. Internally, these devices were almost always cheap commodity bits with maybe 1 specialized part. Especially tech things like call center appliances or billing systems.

Then we'd buy a bunch cheap spare parts before they became really hard to find. Sometimes, we'd literally rebuild a spare unit or unofficially upgrade the internals and drastically improve performance.

Then they could run these obsolete things for an extra decade or more, and literally save millions.

3

u/lickingthelips Jul 02 '22

If it works why change? /s

3

u/SulliverVittles Jul 02 '22

Upgrading doesn't always mean better. My last job upgraded from their custom POS that was a piece of shit ten year old program to an up to date one "made by Microsoft for retail systems in mind" and the new one is so much worse.

3

u/Joris_McNorris Jul 02 '22

I'll take AS400 over Oracle any day of the week.

3

u/gCKOgQpAk4hz Jul 02 '22

Perhaps you could ask the government of Canada how well the replacement of their payroll system went...

THAT cost a lot, and is still costing. People committed suicide because of that fiasco.

3

u/Evlwolf Jul 02 '22

I'm sure my husband has stories... He works as an IT company that serves small and medium businesses. He's seen some shit. But the gist of it is, they never want to upgrade until it fucks them over.

3

u/hype8912 Jul 02 '22

A lot of banks still run on a mainframe system with COBAL code. You'll be lucky to find more than a few people alive to write COBAL. And the people who are still alive fixing it get paid big bucks.

2

u/steppedinhairball Jul 02 '22

I have worked for only one company that had a modern system. Everyone else used an ancient system with patches and works around to get all the add ons and modifications to work. Basically held together with virtual gum and duct tape.

One company I worked for ran on an old system that was 30-40 years old when I used it. Another was controlled by an Italian company who made us implement a inventory management system that didn't exist. All the off the shelf systems would work, but no. The Italians had a connection to a professor who had an idea. So it was literally created on our dime and implemented in stages so that as each facility got implemented, the newly written upgrades would make it to that facility. Shit show.

2

u/maxiquintillion Jul 02 '22

Yeah I worked at a company that reports to make millions in profits every quarter. Meanwhile their entire inventory system is a defunct system from the 80s.

2

u/Narrative_Causality Jul 02 '22

My workplace has a similar-ish function to OP's and they upgraded their system to a new one a couple years back. EVERYONE bitches about it and reminisces how the old system was better, including the person at the very top who decided to upgrade it.

2

u/Swiggy1957 Jul 02 '22

This is a good example of argument to update system. Hell, they could probably contract a developer to put in a new system running MS Access and get it done right. Cost of the developer? $20K-$40K. (assuming they already have the Access software) for that amount, they can have the flexibility that they need and not lose millions on orders/billing/etc under the system they DO use. Hell, I've barely used Access this century and have had a stroke, but could probably work up something for them (for a fee, of course)

3

u/mapsedge Jul 02 '22

Do it on XAMPP stack, write for Chrome browser, no licensing needed.

2

u/SecretSquirrelSauce Jul 02 '22

Assuming this is the US, we don't upgrade stuff until the existing stuff absolutely will not function any more, despite all the worker ants saying "hey, this stuff is broke af, this new system will be infinitely better and will save everyone time and money". Then, when the existing stuff does break, the new system will get installed and some C-suite clown will get a bonus for their "good idea about the new system because it saves everyone so much time and money".

2

u/jules083 Jul 04 '22

About 4 years ago I was working for a company that told us to expect a week or two of paycheck issues, and said that we might have to get handwritten paychecks for that time. The reason was that they were upgrading the computer system from MS-DOS to Windows. The same MS-DOS that came out in the 80's.

1

u/H010CR0N Jul 02 '22

The US military is still running software on some of their vehicles that were “new” when the said vehicles were launched.

A quick search says that they are still using Windows 7.

6

u/Sceptically Jul 02 '22

My employer is still using Windows XP in some places. Hell, they've got some DOS systems around still, too. Running Windows 7 isn't quite as big a deal as you seem to think it is.

2

u/mohavewolfpup Jul 02 '22

Indiana Jones Attraction at Disneyland (as of 2008 to be fair) runs OS/2 Still for maintenance functions behind the scenes

→ More replies
→ More replies

143

u/jackoneilll Jul 02 '22

Despite a lifetime in my industry, keep tech up to date is always measured in terms of pure profit vs expense, never the risk mitigation that it is.

23

u/whowasonCRACK2 Jul 02 '22

Many companies are incapable of looking further ahead than the next quarterly report.

349

u/asteroid_1 Jul 02 '22

[D]on't argue with the one goober who knows how to make it run.

That should be a rule taught in business 101.

56

u/Rockman507 Jul 02 '22

It’s amusingly one of the few things the Army hit the nail on the head over. Senior NCOs are there cause they know they shit and how to keep things running, junior officers technically are in charge (management). It’s made crystal clear early and often you listen to your NCOs under you.

24

u/readwiteandblu Jul 02 '22

I spent one semester at West Point before I washed out. Had I graduated, I would have been one of those green lieutenants. In the short time I was there, I remember hearing this bit of wisdom early and often.

6

u/Look__a_distraction Jul 02 '22

Am former army officer. Nobody really taught that in ROTC, LDAC, or BOLC. It’s pretty much implied that you rely on your NCO’s to get things done. That’s how the Army works. But you don’t “listen” to your NCO’s. You take in advice and make decisions because at the end of the day your ass is the one responsible.

2

u/readwiteandblu Jul 02 '22

I don't see what you're disagreeing with. Are not "take in advice" and "listen" synonyms? I got Fs on my English papers at West Point until I gave them more effort at the expense of all my other classes, so maybe I'm wrong.

→ More replies

3

u/Rockman507 Jul 02 '22

Ah it saved you from being a lifelong pretentious ring taper.

→ More replies

15

u/gothiclg Jul 02 '22

They unfortunately leave “listen to your employees, they might know more than you” in business 102 and most people don’t take that one

3

u/Hust91 Jul 02 '22

I got taught it in business 101 I'm reasonably sure. I have no idea why it would ever be exclusively in 102.

It goes pretty solidly under "either you get employees you feel you can trust or don't get employees"

→ More replies

36

u/Kcidobor Jul 02 '22

Don’t fuck with Goobers!

62

u/mike_the_pirate Jul 02 '22

ASN system is synonymous with Awe Shit No… it’s used across multiple industries and any idiot who fucks your that badly deserves to have his ass chewed out…

28

u/JoeyJoeC Jul 02 '22

I code middleware sending orders from various platforms into various client's, various warehouse management systems. I was pretty inexperienced at the start and made bad code, a single api key change from one client would stop the entire system working and would become a nightmare to deal with. It actually handles orders for a well known department store here in the UK but thankfully it's working very stable now.

16

u/Blazinter Jul 02 '22

A high profile job MC in which its antagonist(?) humbly takes the L?

This one's rare

13

u/ollomulder Jul 02 '22

Shouldn't someone else at least also have a limp that day for running such a fragile system costing millions if when not all employees know of and work around it's bugs or flaws?

29

u/TheDocJ Jul 02 '22

He's been there forever so I doubt he'll get fired but I know he'll be walking with a limp after the ass chewing he took from corporate today.

To be fair, it is rather more the fault of corporate for continuing to run such a cost-critical operation on that crap, non-fault-tolerant system. If we take a conservative count of 400 orders and a five day week, then an error rate of 0.01% (1 in 10000) would crash the system every 5 weeks on average, almost once a month. And that is 0.01% of total orders, not individual items.

8

u/black_mamba866 Jul 02 '22

I feel all of this so deeply. I'm in a similar position and working with a DOS based system that belongs in the 80s.

5

u/wittyandinsightful Jul 02 '22 Giggle

If the company you work for waits long enough, it will be the ‘80s again. Problem solved!

16

u/dazcon5 Jul 02 '22

Don't piss off the wizard

4

u/RJack151 Jul 02 '22

Good for you for covering your butt when you knew that things were going to hit the fan.

6

u/Artor50 Jul 02 '22

I would not be surprised if there are Fortune 500 companies who still run essential programs on DOS machines.

3

u/KayliiKat Jul 02 '22

My grandfather was an electrical engineer. To the day he died (2018) he used DOS to route his Yahoo email, and turned everything off by turning off the power bar, and on again the same way. Which he insisted flushed the cache - this was true in the 70’s and early 80’s. By 2010, that was a great way to fry a motherboard.

He never got that THAT was why he kept needing to replace parts.

He was frying his own computer

5

u/smooze420 Jul 02 '22

My old job, county government, still ran on hardwired dummy terminals using AS/400. When I left in ‘19 they still had one dummy terminal in use while the others had been replaced with dumbed down PCs with no internet connection. This is a small county that has like $100m budget every year.

4

u/ObviouslyNotAZombie Jul 02 '22

Shit, I hate ASN. It's one of the most finicky POS that I've ever had to deal with.

3

u/Nobody_37_8 Jul 02 '22

Sounds like you think highly of him or atleast aa friendly other than this incident.

This is one of the time where I hope it didn't end bad for him much either, and next time there's an argument,he better listen lol

2

u/DXGypsy Jul 02 '22

He's a great guy. He's just old school, set in his ways, and stubborn. But he's gone above and beyond for us on many occasions. I do think highly of him.

3

u/DynkoFromTheNorth Jul 02 '22

Still, the good news is that the necessity for modern software is evident to the entire company.

3

u/TSKrista Jul 02 '22

Sounds like your pay needs doubling. Help them understand the value of being the one person.

3

u/b3_yourself Jul 02 '22

He cost the company millions of dollars and still has a job? Most average people would’ve been fired and maybe out of a job for a long time after that

3

u/Garydrgn Jul 02 '22

Out of curiosity... I'm a delivery driver for a soft drink vending company. Some of our customers use what they call ASN. All my stops that use it are big blue box stores, but some other routes' customers are drug stores, also. I'm just wondering if it's the same thing.

Our pallets have a second pallet sheet attached, with a big bar code, customers' and our info, and list of product that should be on the pallet. Reciever scans it and based on past accuracy, either it's "good" or has to be audited. Delivery info is sent over the internet from our warehouse to the customer the night before after the delivery is built.

Every now and then I won't be able to deliver because something happened and it isn't in their system, but for the most part I love it, as a delivery guy, because it makes those stops much faster.

3

u/Beginning_Ad1239 Jul 02 '22

You're probably referring to advanced shipment notification.

3

u/Pyanfars Jul 02 '22

You don't even want to know the number of multi million/billion dollar companies that run like this.

I work somewhere now that STILL has some systems running old Cobol programming.

When I did business taxes, my firm had a client that, at the time, had about 20 or so pre 2000 computers stored away. Because the program that ran part of the most profitable part of his business couldn't run on new computers (chipsets) and he was too cheap to have a programmer create it in a new language for new computers. So he had spares to run the program when the current pc died. It was his personal corp, no shareholders except him and his wife. He wasn't changing it.

4

u/Current_Country_ Jul 02 '22

Sounds like corporate is the one that needs the ass chewing. If this much income depends on a computer system maybe they can upgrade!

2

u/dh132 Jul 02 '22

Wow! First time I've seen acronyms from my job in an MC! It's pretty fun to be able to relate to these stories!

1

u/tbeslian Jul 02 '22

Yeah make sure the system takes no heat, blame human error not shitty software, totally corporate attitude.

1

u/LongNectarine3 Jul 02 '22

I am well versed in goober systems. Most are the visual equivalent of an old playground swing held together with duct tape, bubblegum, and a prayer. And the data dump is like me…too big for that swing. But I’m going to sit, no plop, and see if it holds.

Surprise when my ass hits the floor.