Our product was used by police forces to input data about people that are arrested and what they are charged with. It would also store digital mugshots and fingerprints, and electronically submit the fingerprints to the FBI. While testing, we would routinely use our own fingerprints for fake bookings that got inserted into the test database. Except for the time that I “temporarily” switched the test machines over to the production database and forgot to switch them back…
Cleaning up our production database was easy, but it took a court order signed by the superintendent of the Boston Police Department to remove my colleague’s fingerprints from the FBI database — she had booked herself under the name “Elroy Jetson”.
I was given a GPS system that was used on ships and given the task of getting a program to interface with it and collect co-ordinates. We didn’t have a manual for the device, but once I got it powered on, I found a big help button. I thought that might at least get us started with how to use it.
After pressing the button, it beeped a couple of times, and then the screen started flashing: Sending S.O.S. signal
Gosh damn – I unplugged the power cable, hoping the thing would turn off, but it must have had an emergency battery inside, because it carried on going, and there was no stopping it.
I waited, very anxiously, expecting a Sea King helicopter to appear outside the office at any moment, wondering how I was going to explain what had happened.
Fortunately – either because I was indoors and the signal didn’t get through or because the receiver of the signal realized an S.O.S. originating 100 miles inland probably wasn’t a real shipping incident – no sea king turned up. Phew.
I had just imported a bunch of old data into a new system, had taken about 5 hours and was due to go live two hours later, this was around 4 in the morning.
For some reason I tried to delete something:
DELETE from important_table; where id=4
Yeah, I didn’t notice the semicolon either. And no, there was no safety net.
I shipped my administrator password for an FTP site inside an open source project I was working on.
I worked for a large bank and added an error message to a piece of code that the application should never have been able to reach (theoretically)…
One Monday morning the unthinkable happened. The error message was proudly displayed on over 10,000 monitors across 1800 branches, and would return when you dismiss the message.
The message read: “If you can see this message the system is all F**KED UP and we might as well go home. Have a nice day.”
Thank goodness this happened before source control systems were implemented at the bank.
I was logged into my new dedicated server box and configuring some firewall rules over ssh. The first thing I did was set it to not accept any connections from anyone. Then I saved it to test that before going through and adding the various ports I wanted to allow.
Needless to say, the first rule worked…
When I set up a new VMware instance at the evening and went home keeping it running. What I didn’t noticed was that I used the IP of the Nameserver as IP of the VMware instance. Suddenly all hosts in our building tried to connect to the VMware for DNS lookups.
Our whole network was practically down.
Since this was a VMware our Admins were not able to track down the MAC address. So they had to plug off every single computer in our office (~500) until the problem was gone.
At the next day I found a letter on my desk: “Who dares to switch this computer on will die”.
Near the end of the dot com bubble, my company was doing research on a sector of the market. We had been given a database with company names in that sector, and we weren’t sure if many of them were still viable companies. I wrote a quick and dirty app which looped through the database and tried the URLs to see if it got a valid response… the assumption being that a 404 would be a failed company. The app used the IE browser COM component and actually displayed the pages while it processed. I split the database into three sections and set it to run on three machines beginning at the close of business and running overnight. My cube was extremely proximate to the CEO and CFO.
Upon arrival the next day, I discovered that the database was not at all accurate. Apparently it was open to the public for update, and numerous spammers and porn companies had inserted records and URLs of their own. This, in itself, was not terrible. What was terrible is that many of the pages when loaded, spawned pop-up windows of extremely explicit details and while the program moved on to the next page, the pop-up windows were orphaned and visible for all to see.
I had some ‘splaining to do.
I rewrote a whole module that was working perfectly but that looked “messy” to me. I had managed to convince my boss that it was the Right Thing To Do, and the rewrite took me 3 weeks.
I still remember the pearls of sweat running from my armpits as my boss, looking over my shoulder, was commenting on bug after bug in my new shiny super-clean module…
I’m no longer “rewriting from scratch” without a really good reason.
Ooh! Embarrassing confessional time. The first one that comes to mind was shortly after I switched to OS X from years on Windows and had basically forgotten anything I knew about unix.
I was working on a personal project and decided I was at a point where I should backup my stuff. So I opened up the command line:
Oh man! It zipped every file individually! Right, I have to tar them first. Okay
Wait! Why is my directory empty?! Oh no….
Yeah, I also forgot that gz doesn’t copy and zip, it zips in-place.
I got lucky, though, and still had most of the files open in my editor.
This was what convinced me to finally install a version control system on my home machine and use it for my own stuff.