Kardol is company specialized hosting ERP solutions.
Area
Hosting provider
Perimeter
MCO Datacenter
The starting point & challenges
At the end of 2022, Kardol decided to deploy a new datacenter with new network equipment based on NVIDIA Cumulus technology. But things weren’t going to run smoothly without a few issues and challenges along the way.
Luckily, Kardol had the good idea of calling in the IT Lumberjacks.
The goal & the plan
At first glance, the plan looks quite simple:
Deploy a full dual-site EVPN-VXLAN infrastructure based on NVIDIA Cumulus technology.
Deploy a dedicated storage network in parallel.
Nothing impossible. Just an audit to define the requirements, constraints, challenges, key steps, and timeline.
We sharpen our axes — and off we go!
Build
Except for one tiny little problem…
Due to mandatory design requirements, we had to go with NVIDIA Cumulus 5.0, released… at the end of 2022.
For a deployment that was due to start in early 2023.
Which meant using one of the very first versions, with little to no feedback available.
Basically, a live blind test!
Off we went with non-regression testing in every possible direction!
At the same time, our IT Lumberjacks had to tackle the challenges of inter-site connectivity: building the management network, finding the right DWDM optics, and debugging remote console access.
A lot to keep the IT Lumberjacks busy.
And surprisingly, things are actually going pretty well.
Run
After commissioning, the IT Lumberjacks handled operations and maintenance during business hours. Everything ran smoothly…until February 2024.
Cumulus operation
Then, on February 22, 2024, at 5:00 a.m., the entire network began going down, equipment by equipment.
The IT Lumberjacks jumped out of bed in their checked socks, mobilizing and joining forces for an intense troubleshooting session. By 11:00 a.m., all services were back up and running.
The outage was caused by a series of bugs affecting Cumulus Linux 5.0, the version running on the network at the time.
Mountain Men conducted a thorough incident analysis and identified the root cause: a combination of an SNMPD bug and a WATCHFRR bug.
The full report was submitted to NVIDIA for validation and remediation.
Meanwhile, to prevent another outage, the team decided to urgently upgrade to Cumulus Linux 5.8 and modify the low-level Cumulus configuration, without waiting for NVIDIA’s feedback.
NVIDIA later confirmed the IT Lumberjacks’ analysis and validated the proposed fixes.
The incident demonstrated the IT Lumberjacks’ efficiency and responsiveness under pressure.
Following the incident, Kardol decided to entrust the IT Lumberjacks with full 24/7 IT outsourcing.
