diff --git a/documentation/INCIDENTS.md b/documentation/INCIDENTS.md index 35b5bb8..736a211 100644 --- a/documentation/INCIDENTS.md +++ b/documentation/INCIDENTS.md @@ -1167,3 +1167,39 @@ Therefore we've installed super old and shitty transcievers (> 10 years old, >70 **impact**: * L2 interruption (<= 10 seconds) for all tents + + +027 2022.11.21 02:00 | (maintenance) attach volume to `eae-adp-jump01` for prometheus +------------------------------------------------------------------------------------- + +**problem**: +After installing a prometheus stack onto `eae-adp-jump01` (`8389a18`) the `/var/` parition filled up after a few days. +Limiting the size of the TSDB did not resolve this issues (maybe i've misconifigured the limit). + +**solution**: +attach 20GB block device onto vm and mount it as `/var/prometheus`: +``` +eae-adp-jump01# rcctl stop prometheus +eae-adp-jump01# rm -r /var/prometheus/* +eae-adp-jump01# sysctl hw.disknames +eae-adp-jump01# fdisk -iy sd1 +eae-adp-jump01# disklabel -E sd1 +> a a +> +> * +> q +eae-adp-jump01# newfs sd1a +eae-adp-jump01# diff -Naur /etc/fstab.20221121 /etc/fstab +--- /etc/fstab.20221121 Sun Jun 26 23:00:39 2022 ++++ /etc/fstab Mon Nov 21 02:01:03 2022 +@@ -8,3 +8,4 @@ + e1c3571d54635852.j /usr/obj ffs rw,nodev,nosuid 1 2 + e1c3571d54635852.i /usr/src ffs rw,nodev,nosuid 1 2 + e1c3571d54635852.e /var ffs rw,nodev,nosuid 1 2 ++a0469c9f38992e1d.a /var/prometheus ffs rw,nodev,nosuid 1 2 +eae-adp-jump01# mount /var/prometheus +eae-adp-jump01# chown _prometheus:_prometheus /var/prometheus +eae-adp-jump01# rcctl start prometheus +eae-adp-jump01# syspatch +eae-adp-jump01# reboot +```