By now everyone has probably heard something about the Fairpoint fiasco but I'm going to run the entire thing down for you anyway, from start to finish. I woke up this morning feeling ill so I made the decision to stay home and work today. One of my morning rituals is to scan the email archive for any suspicious messages, like from the students, or things like bounce errors. You know, just doing my part to make sure everything is running like it's supposed to. I was scrolling down through the emails, sipping on my chicken soup..and yes..it's okay to have chicken soup at 8 am. I came across a bounce message from Verizon, not a common occurance so I took a closer look. A teacher had sent a message to the easton rec dept. and the mail server sent a message back saying that the message was refuse by the mail server due to abuse. How odd. I know our mail server isn't blacklisted for spam anywhere. I noticed something a bit funky about this message as well..apparently Verizon's mail server had attempted to forward the message on to a fairpoint server which then rejected the message from Verizon's server, saying it was spam. Great..a server battle royale! In the mean time, no one's receiving their email. I called up MSLN to see if they had any other reports but the 1st tier tech guy I spoke to hadn't heard anything, so he escalated my issue and sent it to the upper tier. A few minutes later I receive a call from Doretta, the "2nd in command" so to speak, telling me that she had a similar issue over the weekend and it seems to be a problem with Fairpoint. Little did either of us know, how big the problem really was. A friend of mine runs an ISP here and his entire mail system was killed. Apparently Fairpoint, in their switch over..had taken control of my friend's ISP's IP pool..and didn't properly forward DNS to his servers..so basically, Fairpoint highjacked his network. Poor guy was up for 20 hours straight the last I had talked to him. The problems have hit thousands upon thousands of people in Maine, Vermont and NH. You can read about the issues here.
I feel bad for everyone that this screw up has had an effect on. This is the perfect example of what NOT to do when you head into the IT business. It gives everyone an excellent idea of what can happen with improper planning, lack of experience and lack of knowledge in a certain area. I spent most of my day fielding phone calls and handing out free tech support on DSLreports.com because I felt bad for all the poor folks that had been left stranded with no support and no options. You couldn't even call Fairpoint to talk to tech support for the majority of the day..you just got a busy signal. The scariest part of the entire situation is, they even messed up the 911 system. 911! Hasn't anyone over at Fairpoint heard of TESTING things before putting them into action? I don't know who is responsible for everything that has gone down, but someone had BETTER be fired for this. There is no excuse. I leave you with this. Funny..yet frightening. Good night all.
0 comments:
Post a Comment