• Cloudflare outage

    From Oli@21:3/102 to All on Thu Apr 16 12:53:19 2020
    Yesterday when I tried to update the nameserver records for my domains, I couldn't login to Cloudflare. Once in a while every big cloud provider screws up, here is another silly story:

    April 16, 2020 1:28AM

    Starting at 1531 UTC and lasting until 1952 UTC, the Cloudflare Dashboard and API were unavailable because of the disconnection of multiple, redundant fibre connections from one of our two core data centers.

    This outage was not caused by a DDoS attack, or related to traffic increases caused by the COVID-19 crisis. Nor was it caused by any malfunction of software
    or hardware, or any misconfiguration.

    What happened?

    As part of planned maintenance at one of our core data centers, we instructed technicians to remove all the equipment in one of our cabinets. That cabinet contained old inactive equipment we were going to retire and had no active traffic or data on any of the servers in the cabinet. The cabinet also contained a patch panel (switchboard of cables) providing all external connectivity to other Cloudflare data centers. Over the space of three minutes,
    the technician decommissioning our unused hardware also disconnected the cables
    in this patch panel.

    This data center houses Cloudflare’s main control plane and database and as such, when we lost connectivity, the Dashboard and API became unavailable immediately. [...]

    Full story at: https://blog.cloudflare.com/cloudflare-dashboard-and-api-outage-on-april-15-2020/

    ---
    * Origin: (21:3/102)
  • From Vk3jed@21:1/109 to Oli on Thu Apr 16 23:42:00 2020
    On 04-16-20 08:53, Oli wrote to All <=-

    This data center houses Cloudflare's main control plane and database
    and as such, when we lost connectivity, the Dashboard and API became unavailable immediately. [...]

    Oops! :D


    ... ACRONYM: Abbreviated Coded Rendition Of Name Yielding Meaning
    === MultiMail/Win v0.51
    --- SBBSecho 3.10-Linux
    * Origin: Freeway BBS Bendigo,Australia freeway.apana.org.au (21:1/109)
  • From poindexter FORTRAN@21:4/122 to Oli on Thu Apr 16 13:39:00 2020
    Oli wrote to All <=-

    This outage was not caused by a DDoS attack, or related to traffic increases caused by the COVID-19 crisis. Nor was it caused by any malfunction of software or hardware, or any misconfiguration.

    It's nice to see a company taking responsibility for an operational failure.

    At a company I worked for, a human error caused the endpoint security system on a third of our desktops to stop talking to the internet. It was purely a failure of oversight. Management retained the person responsible,
    acknowledged that people are human, increased the reach of our change management system and implemented a buddy system where people wouldn't be expected to make changes after hours (after working a full day, I might
    add).

    That sort of leadership and responsibility is what keeps employees and customers.


    --- MultiMail/DOS v0.52
    * Origin: http://realitycheckbbs.org (21:4/122)
  • From ryan@21:1/168 to poindexter FORTRAN on Thu Apr 16 18:52:51 2020
    It's nice to see a company taking responsibility for an operational failure.

    Indeed. And as it turns out, totally unrelated to this, guess who's going to martyr himself over an issue that happened over the past 24 hours at work? *raises hand* lol

    --- Mystic BBS v1.12 A46 2020/04/13 (Linux/64)
    * Origin: monterey bbs (21:1/168)
  • From Alpha@21:4/158 to ryan on Fri Apr 17 03:46:05 2020
    Indeed. And as it turns out, totally unrelated to this, guess who's
    going to martyr himself over an issue that happened over the past 24
    hours at work? *raises hand* lol

    That sounds awful! Good luck and hope it works out. Sometimes martyrdom ain't all it's cracked up to be LOL

    |14▐ |15Alpha
    |14▄▌ |07Card & Claw BBS
    |06▐ |08cardandclaw.com:8888

    --- Mystic BBS v1.12 A46 2020/04/13 (Linux/64)
    * Origin: Card & Claw BBS (21:4/158)
  • From Weatherman@21:1/132 to Poindexter Fortran on Fri Apr 17 00:04:45 2020

    It's nice to see a company taking responsibility for an operational failure.
    At a company I worked for, a human error caused the endpoint security system on a third of our desktops to stop talking to the internet. It was purely a failure of oversight. Management retained the person
    responsible, acknowledged that people are human, increased the reach of
    our change management system and implemented a buddy system where people wouldn't be expected to make changes after hours (after working a full
    day, I might add).

    That sort of leadership and responsibility is what keeps employees and customers.

    Agreed, throwing people under the bus is no way to run an organization or foster creativity. Stuff happens, you learn, and move on.

    - Mark
       
    --- WWIVToss v.1.52
    * Origin: http://www.weather-station.org * Bel Air, MD -USA (21:1/132.0)
  • From Vk3jed@21:1/109 to poindexter FORTRAN on Fri Apr 17 13:37:00 2020
    On 04-16-20 09:39, poindexter FORTRAN wrote to Oli <=-

    Oli wrote to All <=-

    This outage was not caused by a DDoS attack, or related to traffic increases caused by the COVID-19 crisis. Nor was it caused by any malfunction of software or hardware, or any misconfiguration.

    It's nice to see a company taking responsibility for an operational failure.

    My best cockup was to take much of Australia Post's Melbourne network offline. :D It was actually the result of a communication breakdown by two of us in our office that caused the problem. At the time, I worked for an IT training provider that was in the same building as Australia Post. Eveny now and then, Australia Post would hire a room, and as we were in the same building, get it connected to their LAN. There was an Ethernet cable running between floors to achieve this. However, to do this required making a connection at one part of our floor and disconnecting the network for the room in another. Anyway, due to the communication breakdown, the connection was made to Australia Post, but our LAN wasn't disconnected from the room. And of course, this was late on a Friday afternoon. :)

    On the following Monday morning, we had an IT person from Australia Post down looking for a rogue DHCP server. Apparently, our DHCP server, despite being an old clunker, was handing out IPs faster than theirs, when clients tried to renew leases, with the result that we took much of their network offline. Oops! :D And furthermore, the layout of our network, along with key systems all having static IP assignments, rather than static DHCP leases meant the impact on our network was insignificant.

    Anyway, that was the only time it happened. From then, there were more precautions taken on both ends, until we upgraded to a managed switch some time after, which both made this type of mistake much harder to do, as well as allowing much more flexibility in LAN to room assignments - we no longer were restricted to a couple of specially prepared rooms, but the managed switch allowed us to put any port in in the company on any network, as needed. We even had a couple of "dark" ports that we could used to interconnect multiple rooms independently of any network, then add a server brought in by a client, all on a safely isolated LAN.


    ... For certain people, after fifty, litigation takes the place of sex.
    === MultiMail/Win v0.51
    --- SBBSecho 3.10-Linux
    * Origin: Freeway BBS Bendigo,Australia freeway.apana.org.au (21:1/109)
  • From Vk3jed@21:1/109 to ryan on Fri Apr 17 13:37:00 2020
    On 04-16-20 14:52, ryan wrote to poindexter FORTRAN <=-

    It's nice to see a company taking responsibility for an operational failure.

    Indeed. And as it turns out, totally unrelated to this, guess who's
    going to martyr himself over an issue that happened over the past 24
    hours at work? *raises hand* lol

    Haha as you can see in my previous message, been there, done that. :D


    ... Famous last words: "I think the dragon's asleep "
    === MultiMail/Win v0.51
    --- SBBSecho 3.10-Linux
    * Origin: Freeway BBS Bendigo,Australia freeway.apana.org.au (21:1/109)
  • From marlow@21:3/103 to poindexter FORTRAN on Fri Apr 17 07:14:13 2020
    On 16 Apr 2020, poindexter FORTRAN said the following...

    Oli wrote to All <=-

    This outage was not caused by a DDoS attack, or related to traffic increases caused by the COVID-19 crisis. Nor was it caused by any malfunction of software or hardware, or any misconfiguration.

    It's nice to see a company taking responsibility for an operational failure.


    It is rare to see these days. But even in my own business, we operate a non-bullshit policy and open approach as much as possible.

    Some plebs won't touch us (as an ISP), because they deem, we have too many problems. I don't mind that. Cause they'd be p*** in the arse customers
    anyway.

    The ones that know better, can see, what we're doing and appreciating the openess .... tend to stay customers for decades. And that is worth it.

    Cloudflare .. at least .. seems to have their priorities right. And that is a good thing.

    -M

    --- Mystic BBS v1.12 A45 2020/02/18 (Linux/32)
    * Origin: The Alien Mindbenders BBS (21:3/103)
  • From ryan@21:1/168 to Alpha on Fri Apr 17 04:32:14 2020
    That sounds awful! Good luck and hope it works out. Sometimes martyrdom ain't all it's cracked up to be LOL

    Yeah, true, but I also prefer to own up to my mistakes. I think I just prefer ripping the band-aid off when avoidable things happen and it's my fault. I'm happy to say it's pretty infrequent and I'll have no issues absorbing this
    and taking in stride. I'm still mad it happened in the first place :P

    --- Mystic BBS v1.12 A46 2020/04/13 (Linux/64)
    * Origin: monterey bbs (21:1/168)
  • From ryan@21:1/168 to Vk3jed on Fri Apr 17 04:33:29 2020
    Haha as you can see in my previous message, been there, done that. :D

    I used to serve in the military, and after every mission we'd have a
    hot-wash. This was to identify what every person on the mission screwed up,
    and it was nice because it taught you to let your guard down and not pretend
    to be infallible. I learned at lot during this time, not least of which was
    the value of humility and integrity.

    --- Mystic BBS v1.12 A46 2020/04/13 (Linux/64)
    * Origin: monterey bbs (21:1/168)
  • From Vk3jed@21:1/109 to ryan on Sat Apr 18 01:36:00 2020
    On 04-17-20 00:33, ryan wrote to Vk3jed <=-

    Haha as you can see in my previous message, been there, done that. :D

    I used to serve in the military, and after every mission we'd have a hot-wash. This was to identify what every person on the mission screwed up, and it was nice because it taught you to let your guard down and
    not pretend to be infallible. I learned at lot during this time, not
    least of which was the value of humility and integrity.

    Ues, recognising we're human and owning our screwups is definitely a good thing. :)


    ... I'm not gonna lie to you. Heaven is a nice place. -Satan
    === MultiMail/Win v0.51
    --- SBBSecho 3.10-Linux
    * Origin: Freeway BBS Bendigo,Australia freeway.apana.org.au (21:1/109)
  • From poindexter FORTRAN@21:4/122 to ryan on Fri Apr 17 11:20:00 2020
    ryan wrote to Vk3jed <=-

    Haha as you can see in my previous message, been there, done that. :D

    I used to serve in the military, and after every mission we'd have a hot-wash. This was to identify what every person on the mission screwed up, and it was nice because it taught you to let your guard down and
    not pretend to be infallible.


    "The only mistakes are the ones you repeat..."

    --me



    ... Magnify the most difficult details
    --- MultiMail/DOS v0.52
    * Origin: http://realitycheckbbs.org (21:4/122)
  • From Alpha@21:4/158 to ryan on Fri Apr 17 19:19:04 2020
    Yeah, true, but I also prefer to own up to my mistakes. I think I just prefer ripping the band-aid off when avoidable things happen and it's my fault. I'm happy to say it's pretty infrequent and I'll have no issues absorbing this and taking in stride. I'm still mad it happened in the first place :P

    Yes, big difference in owning up to our own mistakes vs. taking the blame because a system just needs to point fingers and assign blame! For me,
    mistakes are pretty much how I've learned every significant thing in my professional life, including my recent years in launching, running--and then losing--my own company. Painful, but there was no one else i COULD blame, either.

    Weirdly enough, I landed in a big tech company that actually is run with compassion and humanity, and a strict culture of no-blame-games (so far). We had a bug get through our smokescreen to production this week that really put this idea to the test, but the entire team attacked it, solved in 24 hours and the CPO just asked: "So what did we learn from this?" Nice, as in a waaay previous startup job I literally had a cake thrown at my head by an SVP
    when a feature didn't release on time. And I was in the Design org, not a PM (who usually took the brunt of blame). LOL

    |14▐ |15Alpha
    |14▄▌ |07Card & Claw BBS
    |06▐ |08cardandclaw.com:8888

    --- Mystic BBS v1.12 A46 2020/04/13 (Linux/64)
    * Origin: Card & Claw BBS (21:4/158)
  • From djatropine@21:1/121 to ryan on Sat Apr 18 18:00:38 2020


    On Apr 17th 1:33 am ryan said...



    I used to serve in the military, and after every mission we'd have a hot-wash. This was to identify what every person on the mission screwed up, and it was nice because it taught you to let your guard down and not pretend to be infallible. I learned at lot during this time, not least of which was the value of humility and integrity.
    --- Mystic BBS v1.12 A46 2020/04/13 (Linux/64)
    * Origin: monterey bbs (21:1/168)


    * Origin: monterey bbs (21:1/168)


    While I have never served in the military, my experience with military folks, is : Thanks to two people who were stationed at the naval base in a city near me, I was able to experience Daft Punk in 1996.











    --- ENiGMA 1/2 v0.0.11-beta (linux; x64; 12.13.1)
    * Origin: Xibalba -+- xibalba.l33t.codes:44510 (21:1/121)
  • From Ogg@21:4/106.21 to djatropine on Sat Apr 18 23:41:00 2020
    Hello djatropine!

    ** 18.04.20 - 14:00, djatropine wrote to ryan:

    While I have never served in the military, my experience with military
    folks, is : Thanks to two people who were stationed at the naval base in a
    city near me, I was able to experience Daft Punk in 1996.

    Never heard of Daft Funk. But according to WikiPedia they are quite a success story primarily because of exclusivity contracts of various
    products, appearances, signature helmets, etc. ..tapping into a market
    that was ripe for the unusual. The mystique of the helmets helped for
    sure. Sounds like they certainly had a lot of fun crafting unique and unusual sounds in the studio.



    --- OpenXP 5.0.43
    * Origin: [} Pointy McPointFace (21:4/106.21)
  • From ryan@21:1/168 to djatropine on Sat Apr 18 22:16:27 2020
    While I have never served in the military, my experience with military folks, is : Thanks to two people who were stationed at the naval base
    in a city near me, I was able to experience Daft Punk in 1996.

    I feel like I need to know the back story here hehe

    --- Mystic BBS v1.12 A46 2020/04/13 (Linux/64)
    * Origin: monterey bbs (21:1/168)
  • From djatropine@21:1/157 to Ogg on Sun Apr 19 07:56:01 2020

    On 18 Apr 2020, Ogg said the following...

    Hello djatropine!

    ** 18.04.20 - 14:00, djatropine wrote to ryan:

    While I have never served in the military, my experience with military
    folks, is : Thanks to two people who were stationed at the naval base
    city near me, I was able to experience Daft Punk in 1996.

    Never heard of Daft Funk. But according to WikiPedia they are quite a success story primarily because of exclusivity contracts of various products, appearances, signature helmets, etc. ..tapping into a market that was ripe for the unusual. The mystique of the helmets helped for sure. Sounds like they certainly had a lot of fun crafting unique and unusual sounds in the studio.



    --- OpenXP 5.0.43
    * Origin: [} Pointy McPointFace (21:4/106.21)


    It was a event in Wisconsin. Didn't have the masks then .

    Experiencing Daft Punk in 1996 ( A Ah I miss the days when experiemcing a DJ who took the time to master a craft would not only "match beats" yet it was
    a magical experience when the dj would ride a mix for anywhere fromn 4, 8 to sometimes 32 bars. (The modern mediums ... :D

    --- Mystic BBS v1.12 A46 2020/03/12 (Windows/32)
    * Origin: * Shadowscope BBS * (21:1/157)
  • From Avon@21:1/101 to djatropine on Mon Apr 20 00:41:41 2020
    On 19 Apr 2020 at 03:56a, djatropine pondered and said...

    Experiencing Daft Punk in 1996 ( A Ah I miss the days when experiemcing
    a DJ who took the time to master a craft would not only "match beats"

    I'm a big fan of their Tron Legacy work... I listen to it quite often. :)

    --- Mystic BBS v1.12 A46 2020/04/13 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)
  • From djatropine@21:1/121 to Avon on Sun Apr 19 11:26:20 2020


    On Apr 19th 2:42 am Avon said...
    I'm a big fan of their Tron Legacy work... I listen to it quite often. :)


    I'm a fan of *HOMEWORK* and the tune Television rules a Nation &*
    TeCHNOLOGIC


    --- ENiGMA 1/2 v0.0.11-beta (linux; x64; 12.13.1)
    * Origin: Xibalba -+- xibalba.l33t.codes:44510 (21:1/121)
  • From Joacim Melin@21:2/130 to Oli on Mon Apr 20 01:10:47 2020
    Yesterday when I tried to update the nameserver records for my
    domains, I couldn't login to Cloudflare. Once in a while every big
    cloud provider screws up, here is another silly story:

    April 16, 2020 1:28AM

    Starting at 1531 UTC and lasting until 1952 UTC, the Cloudflare
    Dashboard and API were unavailable because of the disconnection of multiple, redundant fibre connections from one of our two core data centers.

    This outage was not caused by a DDoS attack, or related to traffic increases caused by the COVID-19 crisis. Nor was it caused by any malfunction of software or hardware, or any misconfiguration.

    What happened?

    As part of planned maintenance at one of our core data centers, we instructed technicians to remove all the equipment in one of our
    cabinets. That cabinet contained old inactive equipment we were going
    to retire and had no active traffic or data on any of the servers in
    the cabinet. The cabinet also contained a patch panel (switchboard of cables) providing all external connectivity to other Cloudflare data centers. Over the space of three minutes, the technician
    decommissioning our unused hardware also disconnected the cables in
    this patch panel.

    This data center houses Cloudflare?s main control plane and database
    and as such, when we lost connectivity, the Dashboard and API became unavailable immediately. [...]

    Full story at:

    https://blog.cloudflare.com/cloudflare-dashboard-and-api-outage-on-april-1 >5-2020/

    It's a disgrace and also very telling how how voulnable their infrastructure is. I've been hosting some 20 domains there for the past year or so also use them as a registrar and I have since this happened moved 16 of 20 domains from Cloudflare to my own domain servers instead.


    --- NiKom v2.5.0
    * Origin: Delta City (deltacity.se, Vallentuna, Sweden) (21:2/130.0)
  • From Oli@21:3/102 to Joacim Melin on Mon Apr 20 02:13:05 2020
    Joacim wrote (2020-04-19):

    The cabinet also contained a patch panel (switchboard of
    cables) providing all external connectivity to other Cloudflare data
    centers. Over the space of three minutes, the technician
    decommissioning our unused hardware also disconnected the cables in
    this patch panel.

    It's a disgrace and also very telling how how voulnable their infrastructure is. I've been hosting some 20 domains there for the past year or so also use them as a registrar and I have since this happened moved 16 of 20 domains from Cloudflare to my own domain servers instead.

    And the infrastructure you use for your own nameservers is not vulnerable?

    ---
    * Origin: (21:3/102)
  • From Joacim Melin@21:2/130 to Oli on Mon Apr 20 20:40:59 2020
    Joacim wrote (2020-04-19):

    The cabinet also contained a patch panel (switchboard of
    cables) providing all external connectivity to other Cloudflare data
    centers. Over the space of three minutes, the technician
    decommissioning our unused hardware also disconnected the cables in
    this patch panel.

    It's a disgrace and also very telling how how voulnable their
    infrastructure is. I've been hosting some 20 domains there for the past
    year or so also use them as a registrar and I have since this happened
    moved 16 of 20 domains from Cloudflare to my own domain servers instead.

    And the infrastructure you use for your own nameservers is not
    vulnerable?

    I have name servers at home via two different internet connections and a third running on a VPS on the other side of the country. I'd say I'm pretty well covered.


    --- NiKom v2.5.0
    * Origin: Delta City (deltacity.se, Vallentuna, Sweden) (21:2/130.0)
  • From Avon@21:1/101 to djatropine on Wed Apr 22 15:35:19 2020
    On 19 Apr 2020 at 07:26a, djatropine pondered and said...

    I'm a fan of *HOMEWORK* and the tune Television rules a Nation &* TeCHNOLOGIC

    Cool, I'll be sure to check those out. Thanks :)

    --- Mystic BBS v1.12 A46 2020/04/20 (Windows/32)
    * Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)