Multi-tenant observability platform operated since December 2024, consolidating heterogeneous infrastructure telemetry across 12+ clients.
02. Data Ops
SystemFrame
Multi-tenant observability platform for 12+ clients and 390+ endpoints, with GitOps deploy and an alert pipeline integrated with GLPI and ServiceNow.
K3s / Helm ArgoCD Prometheus
-60% MTTR / 50MIN -> 20MIN
-
01 Context
The platform monitors heterogeneous infrastructure across 12+ clients: Aruba and Meraki equipment, plus Zabbix and GLPI already installed in each client's fleet. There was no consolidated telemetry -- each client was an island, and diagnosing an incident could take a whole day. Alerts reached the team but didn't turn into a ticket on their own.
-
02 Action
I took ownership of the platform layer with K3s, Helm, ArgoCD, and Ansible, sustaining continuous GitOps deployment. I wrote 5 custom exporters for Aruba, Meraki, Zabbix, and GLPI, consolidating 390+ endpoints into a single format. On top of that I built the alert pipeline connecting Alertmanager to GLPI and ServiceNow, running 296 rules, plus analysis tools built on the Prometheus and VictoriaMetrics APIs distributed across 2 availability zones.
-
03 Result
MTTR dropped from 50 to 20 minutes (-60%), and incident diagnosis time went from 1 day to 2 hours. The GitOps deploy pipeline accumulated more than 3,500 CI/CD runs, averaging around 8 deploys per business day.
-
04 Lessons learned
A custom exporter is maintenance debt: each one becomes proprietary code that has to track vendor firmware and API changes. It was worth it because unified-format telemetry was what made the 296 rules possible -- without it, each client would need its own set of alerts. And an alert that doesn't automatically become a ticket doesn't reduce MTTR: the gain came from the GLPI and ServiceNow integration, not from detecting issues faster.
Architecture
Each component carries the decision made there and what it cost.
Drag to explore the diagram.
entryservicedataexternalasync
Click a component to see the decision made there and what it cost.
View all decisions as text
- 5 custom exporters
- Write proprietary exporters for each source, instead of adopting a single off-the-shelf agent or accepting the native formats.
- Trade-off: Each exporter is code that has to track vendor firmware and API changes. It was what enabled unified-format telemetry across heterogeneous fleets -- and without it, the 296 rules wouldn't exist.
- Prometheus / VictoriaMetrics
- Keep VictoriaMetrics alongside Prometheus and distribute across 2 availability zones.
- Trade-off: Two metrics systems to operate and understand instead of one. It buys longer retention and survival if one zone goes down.
- Alertmanager
- Concentrate the 296 rules in Alertmanager and handle deduplication there, before any external integration.
- Trade-off: The rules become a large, centralized artifact that requires discipline to keep from turning into a mess. It prevents each integration from reimplementing its own silencing logic.
- Analysis tools
- Build query tools on top of the Prometheus and VictoriaMetrics APIs instead of relying solely on visual dashboards.
- Trade-off: More internal code to maintain, outside the product's core path. It cut incident diagnosis time from a day to two hours, because investigating no longer depended on building a dashboard on the fly.
- ArgoCD
- GitOps deployment with ArgoCD and Helm, with desired state versioned instead of applied by hand.
- Trade-off: Every change now requires a commit and sync, which is slower for a one-off hotfix. In exchange, onboarding a new client became Helm values, and cluster state stopped depending on who applied what.
- K3s + Helm
- K3s instead of full Kubernetes to sustain the multi-tenant fleet.
- Trade-off: Fewer components out of the box, so part of what a managed cluster delivers ready-made had to be built. It paid off through lower per-client resource consumption and simpler operations.