On 04-16-20 08:53, Oli wrote to All <=-
This data center houses Cloudflare's main control plane and database
and as such, when we lost connectivity, the Dashboard and API became unavailable immediately. [...]
Oli wrote to All <=-
This outage was not caused by a DDoS attack, or related to traffic increases caused by the COVID-19 crisis. Nor was it caused by any malfunction of software or hardware, or any misconfiguration.
It's nice to see a company taking responsibility for an operational failure.
Indeed. And as it turns out, totally unrelated to this, guess who's
going to martyr himself over an issue that happened over the past 24
hours at work? *raises hand* lol
It's nice to see a company taking responsibility for an operational failure.
At a company I worked for, a human error caused the endpoint security system on a third of our desktops to stop talking to the internet. It was purely a failure of oversight. Management retained the person
responsible, acknowledged that people are human, increased the reach of
our change management system and implemented a buddy system where people wouldn't be expected to make changes after hours (after working a full
day, I might add).
That sort of leadership and responsibility is what keeps employees and customers.
On 04-16-20 09:39, poindexter FORTRAN wrote to Oli <=-
Oli wrote to All <=-
This outage was not caused by a DDoS attack, or related to traffic increases caused by the COVID-19 crisis. Nor was it caused by any malfunction of software or hardware, or any misconfiguration.
It's nice to see a company taking responsibility for an operational failure.
On 04-16-20 14:52, ryan wrote to poindexter FORTRAN <=-
It's nice to see a company taking responsibility for an operational failure.
Indeed. And as it turns out, totally unrelated to this, guess who's
going to martyr himself over an issue that happened over the past 24
hours at work? *raises hand* lol
Oli wrote to All <=-
This outage was not caused by a DDoS attack, or related to traffic increases caused by the COVID-19 crisis. Nor was it caused by any malfunction of software or hardware, or any misconfiguration.
It's nice to see a company taking responsibility for an operational failure.
That sounds awful! Good luck and hope it works out. Sometimes martyrdom ain't all it's cracked up to be LOL
Haha as you can see in my previous message, been there, done that. :D
On 04-17-20 00:33, ryan wrote to Vk3jed <=-
Haha as you can see in my previous message, been there, done that. :D
I used to serve in the military, and after every mission we'd have a hot-wash. This was to identify what every person on the mission screwed up, and it was nice because it taught you to let your guard down and
not pretend to be infallible. I learned at lot during this time, not
least of which was the value of humility and integrity.
ryan wrote to Vk3jed <=-
Haha as you can see in my previous message, been there, done that. :D
I used to serve in the military, and after every mission we'd have a hot-wash. This was to identify what every person on the mission screwed up, and it was nice because it taught you to let your guard down and
not pretend to be infallible.
Yeah, true, but I also prefer to own up to my mistakes. I think I just prefer ripping the band-aid off when avoidable things happen and it's my fault. I'm happy to say it's pretty infrequent and I'll have no issues absorbing this and taking in stride. I'm still mad it happened in the first place :P
I used to serve in the military, and after every mission we'd have a hot-wash. This was to identify what every person on the mission screwed up, and it was nice because it taught you to let your guard down and not pretend to be infallible. I learned at lot during this time, not least of which was the value of humility and integrity.
--- Mystic BBS v1.12 A46 2020/04/13 (Linux/64)
* Origin: monterey bbs (21:1/168)
* Origin: monterey bbs (21:1/168)
While I have never served in the military, my experience with military
folks, is : Thanks to two people who were stationed at the naval base in a
city near me, I was able to experience Daft Punk in 1996.
While I have never served in the military, my experience with military folks, is : Thanks to two people who were stationed at the naval base
in a city near me, I was able to experience Daft Punk in 1996.
Hello djatropine!
** 18.04.20 - 14:00, djatropine wrote to ryan:
While I have never served in the military, my experience with military
folks, is : Thanks to two people who were stationed at the naval base
city near me, I was able to experience Daft Punk in 1996.
Never heard of Daft Funk. But according to WikiPedia they are quite a success story primarily because of exclusivity contracts of various products, appearances, signature helmets, etc. ..tapping into a market that was ripe for the unusual. The mystique of the helmets helped for sure. Sounds like they certainly had a lot of fun crafting unique and unusual sounds in the studio.
--- OpenXP 5.0.43
* Origin: [} Pointy McPointFace (21:4/106.21)
Experiencing Daft Punk in 1996 ( A Ah I miss the days when experiemcing
a DJ who took the time to master a craft would not only "match beats"
I'm a big fan of their Tron Legacy work... I listen to it quite often. :)
Yesterday when I tried to update the nameserver records for my
domains, I couldn't login to Cloudflare. Once in a while every big
cloud provider screws up, here is another silly story:
April 16, 2020 1:28AM
Starting at 1531 UTC and lasting until 1952 UTC, the Cloudflare
Dashboard and API were unavailable because of the disconnection of multiple, redundant fibre connections from one of our two core data centers.
This outage was not caused by a DDoS attack, or related to traffic increases caused by the COVID-19 crisis. Nor was it caused by any malfunction of software or hardware, or any misconfiguration.
What happened?
As part of planned maintenance at one of our core data centers, we instructed technicians to remove all the equipment in one of our
cabinets. That cabinet contained old inactive equipment we were going
to retire and had no active traffic or data on any of the servers in
the cabinet. The cabinet also contained a patch panel (switchboard of cables) providing all external connectivity to other Cloudflare data centers. Over the space of three minutes, the technician
decommissioning our unused hardware also disconnected the cables in
this patch panel.
This data center houses Cloudflare?s main control plane and database
and as such, when we lost connectivity, the Dashboard and API became unavailable immediately. [...]
Full story at:
The cabinet also contained a patch panel (switchboard of
cables) providing all external connectivity to other Cloudflare data
centers. Over the space of three minutes, the technician
decommissioning our unused hardware also disconnected the cables in
this patch panel.
It's a disgrace and also very telling how how voulnable their infrastructure is. I've been hosting some 20 domains there for the past year or so also use them as a registrar and I have since this happened moved 16 of 20 domains from Cloudflare to my own domain servers instead.
Joacim wrote (2020-04-19):
The cabinet also contained a patch panel (switchboard of
cables) providing all external connectivity to other Cloudflare data
centers. Over the space of three minutes, the technician
decommissioning our unused hardware also disconnected the cables in
this patch panel.
It's a disgrace and also very telling how how voulnable their
infrastructure is. I've been hosting some 20 domains there for the past
year or so also use them as a registrar and I have since this happened
moved 16 of 20 domains from Cloudflare to my own domain servers instead.
And the infrastructure you use for your own nameservers is not
vulnerable?
I'm a fan of *HOMEWORK* and the tune Television rules a Nation &* TeCHNOLOGIC
Sysop: | echicken |
---|---|
Location: | Toronto, Ontario |
Users: | 2,224 |
Nodes: | 6 (0 / 6) |
Uptime: | 06:33:59 |
Calls: | 14,143 |
Files: | 295 |
Messages: | 551,239 |