incident 017: add another icmp probe `mon-e2e-wan01`

ansible-inventory

@@ -29,6 +29,7 @@ eae-adp-jump01 ip=162.55.53.85 monitoring_ip=10.84.254.0 ansible_python_interpre
 [container]
 monitoring01      ip=10.84.1.51  cpus=2 disk=50 memory=1024 net='{"net0":"name=eth0,ip=10.84.1.51/24,gw=10.84.1.1,bridge=vmbr0"}'
 mon-e2e-clients01 ip=10.84.7.30  cpus=1 disk=10 memory=256  net='{"net0":"name=eth0,ip=dhcp,bridge=vmbr1"}'
+mon-e2e-wan01     ip=192.168.0.3 cpus=1 disk=10 memory=256  net='{"net0":"name=eth0,ip=dhcp,bridge=vmbr3"}'
 
 [container:vars]
 ostemplate=local:vztmpl/debian-11-standard_11.3-1_amd64.tar.zst

documentation/INCIDENTS.md

@@ -847,7 +847,7 @@ the facility management reached out and reported that
 * they need to switch to the public wifi instead of using the backoffice wifi
 * according to the security the wifi is unuseable at around 12:00
 
-**thougts**:
+**thoughts**:
 To help diagnose the problem I expanded the monitoring with end to end monitors.
 They are pinging two sites in the internet.
 One is using the normal wan (like the backoffice) and one is using the vpn (like the public wifi).
@@ -856,4 +856,20 @@ See commit `f011562` for details.
 I am noticing that sometimes the icmp probes fail, which is odd.
 But at the same time the node exporter on `eae-adp-jump01` is reachable.
 
-I will implement another icmp probe that directly uses the `gigacube` without traversing `gw-core01` before.
+I implemented another icmp probe that directly uses the `gigacube` without traversing `gw-core01` before.
+
+While checking the gigacube for the size of the dhcp pool I have noticed that `Signal Strength` is only `-91 dBm`.
+That seems way to little. Maybe this is the cause for the wan instability ?
+
+**changes in reponse**:
+* set password for `gigacube-2001` (see `pass`)
+	* the gui force me to set a password
+	* reused the password from the old gigacube that was set by the facility management
+	* will write the changed pw onto the current gigacube next time i'll visit
+	* probably going to create a new secure password beforehand :)
+* configures to static ip bindings on the gigacube
+	* for `gw-core01`:     `192.168.0.2`
+	* for `mon-e2e-wan01`: `192.168.0.3`
+* added `WAN/gigacube` vlan onto proxmox port via `gw-core01`
+* added `vmbr3` on proxmox which is used for the `WAN/gigacube` vlan
+* created `mon-e2e-wan01` that lives inside the gigacube network and probes the same stuff as `mon-e2e-clients01`

playbook_provision_monitoring.yml

@@ -22,6 +22,7 @@
 - name: provision blackbox exporters
   hosts:
     - mon-e2e-clients01
+    - mon-e2e-wan01
     - monitoring01
   tasks:
     - name: install blackbox exporter

templates/prometheus.yml

@@ -33,6 +33,7 @@ scrape_configs:
     static_configs:
       - targets:
         - {{ hostvars['mon-e2e-clients01']['ip'] }}:9115
+        - {{ hostvars['mon-e2e-wan01']['ip'] }}:9115
         - {{ hostvars['monitoring01']['ip'] }}:9115
 
   - job_name: 'e2e_clients_v4'
@@ -71,3 +72,21 @@ scrape_configs:
         target_label: instance
       - target_label: __address__
         replacement: {{ hostvars['monitoring01']['ip'] }}:9115
+
+  - job_name: 'e2e_wan_v4'
+    metrics_path: /probe
+    params:
+      module: [icmp_v4]
+    static_configs:
+      - targets:
+        - freifunk-leipzig.de
+        - harald.brainpeach.de
+        - 195.201.165.118       # freifunk-leipzig.de without dns query
+        - 88.198.195.242        # harald.brainpeach.de without dns query
+    relabel_configs:
+      - source_labels: [__address__]
+        target_label: __param_target
+      - source_labels: [__param_target]
+        target_label: instance
+      - target_label: __address__
+        replacement: {{ hostvars['mon-e2e-wan01']['ip'] }}:9115