Page 1 of 3 123 LastLast
Results 1 to 10 of 21

Thread: We are back up!

  1. #1

    Exclamation We are back up!

    Apologies for the outage just now, our hosts had a large network issue! We should be getting a full report in due course!

  2. #2
    Pheww... I thought I had broken the forum from overusing it....

  3. #3
    Productivity surged at my desk. Can you do it again?

  4. #4
    Thank god, I was starting to worry I might have to do some work.

  5. #5
    From our hosts...

    Yesterday afternoon, Wednesday 6th March at 14:34 GMT, we experienced
    a partial network outage which affected approximately 30% of our
    customers on our IPv4 network. Our IPv6 network was unaffected.

    What went wrong was an automated configuration error on an internal
    router which resulted in complex internal routing issues and some
    network unavailability. Our monitoring systems alerted us to this
    immediately, and we then started to work on resolving the problem.

    In accordance with our procedures, we attempted a manual failover of
    the affected routers, unfortunately due to the nature of the problem
    this did not resolve the configuration error. The next step was for us
    to roll back the configuration on the affected routers. After this was
    completed at 14:55 GMT we estimate that normal service was restored
    for approximately 80% of affected customers. This also enabled us to
    trace the source of the configuration error, so we were able to
    manually remove all the erroneous additional configuration and return
    all service to normal.

    Normal service for all customers was resumed at 15:35 GMT.

    Having determined exactly what went wrong with the automated
    configuration we are already testing a patch to our systems to ensure
    that this never happens again. We are also updating our rollback
    procedure to include some further tests.

    We should also add that as part of our planned core network upgrades
    over the coming months we will be able to remove a lot of the
    complexity in our internal network. By removing this complexity we are
    aiming to greatly reduce the chances of this type of configuration

    I would like to apologise profusely for the impact that this outage
    may have had on your business and ability to use Memset
    services. Although our rollback procedures worked well in resolving
    this issue we know there is always room for improvement. As such we
    will be reviewing this outage with two objectives. Firstly to speed up
    the time it takes to recover from this type of complex routing issue.
    Additionally we aim to investigate ways in which we can reduce the
    chances of errors like this in our automatic internal router

    Again, please accept our sincere apologies.


    Alex Coke-Smyth
    Operations Manager

  6. #6
    Clear as mud

  7. #7
    Translated: they had a problem, it affected some people, they eventually fixed it, they are trying to make sure it doesn't happen again, oh and they are sorry it happened in the first place....... Simples

  8. #8

  9. #9
    Ah, that explains it then!

  10. #10
    A hypothetical explanation:-

    Over brained network configuration "expert" wants a salary increase - leaves a bug in system - lets it fall over - bosses panic - "expert" rides in on his white charger & fixes it in fairly short time - JOB DONE. He has a lever next review time. Simples!!

    Seen it done when I worked in the wonderful world of I.T. Not seriously suggesting it has happened here - but it could have!!


Similar Threads

  1. They're back!!
    By RED-DOT in forum Deer Stalking General
    Replies: 4
    Last Post: 26-02-2012, 14:08
  2. Welcome back!
    By RED-DOT in forum Deer Stalking General
    Replies: 7
    Last Post: 20-11-2011, 19:41
  3. Always back it up!
    By finnbear270 in forum Deer Stalking General
    Replies: 2
    Last Post: 20-01-2011, 21:50
  4. This one gets its own back
    By Mannlicher_Stu in forum Deer Stalking General
    Replies: 8
    Last Post: 26-04-2010, 14:25
  5. Out the back
    By The Ghillie in forum Photos
    Replies: 5
    Last Post: 01-04-2010, 19:54

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts