add incident 027: eae-adp-jump01 increase /var/prometheus to 20GB

documentation/INCIDENTS.md

@@ -1167,3 +1167,39 @@ Therefore we've installed super old and shitty transcievers (> 10 years old, >70
 
 **impact**:
 * L2 interruption (<= 10 seconds) for all tents
+
+
+027 2022.11.21 02:00 | (maintenance) attach volume to `eae-adp-jump01` for prometheus
+-------------------------------------------------------------------------------------
+
+**problem**:
+After installing a prometheus stack onto `eae-adp-jump01` (`8389a18`) the `/var/` parition filled up after a few days.
+Limiting the size of the TSDB did not resolve this issues (maybe i've misconifigured the limit).
+
+**solution**:
+attach 20GB block device onto vm and mount it as `/var/prometheus`:
+```
+eae-adp-jump01# rcctl stop prometheus
+eae-adp-jump01# rm -r /var/prometheus/*
+eae-adp-jump01# sysctl hw.disknames
+eae-adp-jump01# fdisk -iy sd1
+eae-adp-jump01# disklabel -E sd1
+> a a
+>
+> *
+> q
+eae-adp-jump01# newfs sd1a
+eae-adp-jump01# diff -Naur /etc/fstab.20221121 /etc/fstab
+--- /etc/fstab.20221121	Sun Jun 26 23:00:39 2022
++++ /etc/fstab	Mon Nov 21 02:01:03 2022
+@@ -8,3 +8,4 @@
+ e1c3571d54635852.j /usr/obj ffs rw,nodev,nosuid 1 2
+ e1c3571d54635852.i /usr/src ffs rw,nodev,nosuid 1 2
+ e1c3571d54635852.e /var ffs rw,nodev,nosuid 1 2
++a0469c9f38992e1d.a /var/prometheus ffs rw,nodev,nosuid 1 2
+eae-adp-jump01# mount /var/prometheus
+eae-adp-jump01# chown _prometheus:_prometheus /var/prometheus
+eae-adp-jump01# rcctl start prometheus
+eae-adp-jump01# syspatch
+eae-adp-jump01# reboot
+```