add incident 027: eae-adp-jump01 increase /var/prometheus to 20GB
This commit is contained in:
parent
35f48f1bad
commit
34e4fbf000
|
@ -1167,3 +1167,39 @@ Therefore we've installed super old and shitty transcievers (> 10 years old, >70
|
||||||
|
|
||||||
**impact**:
|
**impact**:
|
||||||
* L2 interruption (<= 10 seconds) for all tents
|
* L2 interruption (<= 10 seconds) for all tents
|
||||||
|
|
||||||
|
|
||||||
|
027 2022.11.21 02:00 | (maintenance) attach volume to `eae-adp-jump01` for prometheus
|
||||||
|
-------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
**problem**:
|
||||||
|
After installing a prometheus stack onto `eae-adp-jump01` (`8389a18`) the `/var/` parition filled up after a few days.
|
||||||
|
Limiting the size of the TSDB did not resolve this issues (maybe i've misconifigured the limit).
|
||||||
|
|
||||||
|
**solution**:
|
||||||
|
attach 20GB block device onto vm and mount it as `/var/prometheus`:
|
||||||
|
```
|
||||||
|
eae-adp-jump01# rcctl stop prometheus
|
||||||
|
eae-adp-jump01# rm -r /var/prometheus/*
|
||||||
|
eae-adp-jump01# sysctl hw.disknames
|
||||||
|
eae-adp-jump01# fdisk -iy sd1
|
||||||
|
eae-adp-jump01# disklabel -E sd1
|
||||||
|
> a a
|
||||||
|
>
|
||||||
|
> *
|
||||||
|
> q
|
||||||
|
eae-adp-jump01# newfs sd1a
|
||||||
|
eae-adp-jump01# diff -Naur /etc/fstab.20221121 /etc/fstab
|
||||||
|
--- /etc/fstab.20221121 Sun Jun 26 23:00:39 2022
|
||||||
|
+++ /etc/fstab Mon Nov 21 02:01:03 2022
|
||||||
|
@@ -8,3 +8,4 @@
|
||||||
|
e1c3571d54635852.j /usr/obj ffs rw,nodev,nosuid 1 2
|
||||||
|
e1c3571d54635852.i /usr/src ffs rw,nodev,nosuid 1 2
|
||||||
|
e1c3571d54635852.e /var ffs rw,nodev,nosuid 1 2
|
||||||
|
+a0469c9f38992e1d.a /var/prometheus ffs rw,nodev,nosuid 1 2
|
||||||
|
eae-adp-jump01# mount /var/prometheus
|
||||||
|
eae-adp-jump01# chown _prometheus:_prometheus /var/prometheus
|
||||||
|
eae-adp-jump01# rcctl start prometheus
|
||||||
|
eae-adp-jump01# syspatch
|
||||||
|
eae-adp-jump01# reboot
|
||||||
|
```
|
||||||
|
|
Reference in New Issue