add incident 027: eae-adp-jump01 increase /var/prometheus to 20GB
This commit is contained in:
parent
35f48f1bad
commit
34e4fbf000
|
@ -1167,3 +1167,39 @@ Therefore we've installed super old and shitty transcievers (> 10 years old, >70
|
|||
|
||||
**impact**:
|
||||
* L2 interruption (<= 10 seconds) for all tents
|
||||
|
||||
|
||||
027 2022.11.21 02:00 | (maintenance) attach volume to `eae-adp-jump01` for prometheus
|
||||
-------------------------------------------------------------------------------------
|
||||
|
||||
**problem**:
|
||||
After installing a prometheus stack onto `eae-adp-jump01` (`8389a18`) the `/var/` parition filled up after a few days.
|
||||
Limiting the size of the TSDB did not resolve this issues (maybe i've misconifigured the limit).
|
||||
|
||||
**solution**:
|
||||
attach 20GB block device onto vm and mount it as `/var/prometheus`:
|
||||
```
|
||||
eae-adp-jump01# rcctl stop prometheus
|
||||
eae-adp-jump01# rm -r /var/prometheus/*
|
||||
eae-adp-jump01# sysctl hw.disknames
|
||||
eae-adp-jump01# fdisk -iy sd1
|
||||
eae-adp-jump01# disklabel -E sd1
|
||||
> a a
|
||||
>
|
||||
> *
|
||||
> q
|
||||
eae-adp-jump01# newfs sd1a
|
||||
eae-adp-jump01# diff -Naur /etc/fstab.20221121 /etc/fstab
|
||||
--- /etc/fstab.20221121 Sun Jun 26 23:00:39 2022
|
||||
+++ /etc/fstab Mon Nov 21 02:01:03 2022
|
||||
@@ -8,3 +8,4 @@
|
||||
e1c3571d54635852.j /usr/obj ffs rw,nodev,nosuid 1 2
|
||||
e1c3571d54635852.i /usr/src ffs rw,nodev,nosuid 1 2
|
||||
e1c3571d54635852.e /var ffs rw,nodev,nosuid 1 2
|
||||
+a0469c9f38992e1d.a /var/prometheus ffs rw,nodev,nosuid 1 2
|
||||
eae-adp-jump01# mount /var/prometheus
|
||||
eae-adp-jump01# chown _prometheus:_prometheus /var/prometheus
|
||||
eae-adp-jump01# rcctl start prometheus
|
||||
eae-adp-jump01# syspatch
|
||||
eae-adp-jump01# reboot
|
||||
```
|
||||
|
|
Reference in New Issue