incidents: add maintenance incident 024 about fw upgrades on aps

only issue a sysupgrade if the desired garet version and profile
specified in the inventory does not match the firmware running on the
device.

Because we do not put the firmware files into this git, simply
clone the garet repo, checkout the desired commit, build the needed
profile and then put the firmware file into
  firmware/garet_${profile}-${version}.bin

ansible-inventory

@@ -10,6 +10,8 @@ ap-8f39 ip=10.84.1.37 location=tent-5          channel_2g=1  channel_5g=157
 
 [accesspoints:vars]
 ansible_remote_tmp=/tmp
+garet_profile=aruba-ap-105_21.02
+garet_release=845a6ba
 
 [switches]
 sw-access01 ip=10.84.1.11 

documentation/INCIDENTS.md

@@ -1084,3 +1084,18 @@ recharged mullvad account
 **extended monitoring**:
 This is not the first time the public wifi selectivly stopped working because something was wrong with the vpn.
 To be proactively notified when this happens again I've created a alarm that should trigger when every end to end test from the public wifi/client network stops working (`ec917a2`).
+
+
+024 2022.10.24 ~ 01:00 | (maintenance) upgrade accesspoints to OpenWrt 21.02.5
+------------------------------------------------------------------------------
+
+upgrade firmware on all accesspoints to the latest old stable version using the new `playbook_sysupgrade` (since `8d79518`).
+
+garet release: `845a6ba`
+garet profile: `aruba-ap-105_21.02`
+
+**timetable**:
+* 2022.10.24 00:16 - 00:20: `ap-c5d1` (office container)
+* 2022.10.24 00:27 - 00:32: `ap-ac7c` (social work)
+* 2022.10.24 00:39 - 00:44: `ap-0b99`, `ap-1a38`, `ap-2bbf`, `ap-8f39`, `ap-8f42`, `ap-c495` (tents)
+=> downtime of the accesspoints in the specified timeframe

playbook_sysupgrade.yml

@@ -1,19 +1,30 @@
 ---
 - name: upgrade firmware on openwrt device(s)
-  gather_facts: no
   hosts: accesspoints
+  vars:
+    running_garet_profile: "{{ ansible_local['garet']['profile'] | default() }}"
+    running_garet_release: "{{ ansible_local['garet']['release'] | default() }}"
+    firmware_file: "firmware/garet_{{ garet_profile }}-{{ garet_release }}.bin"
+  gather_facts: no
   tasks:
-    - name: upload new firmware
-      copy:
-        src: "{{ firmware_file }}"
-        dest: "/tmp/{{ firmware_file | basename }}"
+    - name: gather firmware version
+      setup:
+        filter: ansible_local
 
-    - name: issue sysupgrade command
-      command:
-        cmd: "sysupgrade /tmp/{{ firmware_file | basename }}"
-      ignore_errors: yes
+    - name: handle update
+      block:
+        - name: upload new firmware
+          copy:
+            src: "{{ firmware_file }}"
+            dest: "/tmp/{{ firmware_file | basename }}"
 
-    - name: wait till device is back online
-      wait_for_connection:
-        delay: 10
-        timeout: 600
+        - name: issue sysupgrade command
+          shell:
+            cmd: "sysupgrade /tmp/{{ firmware_file | basename }} && exit 0"
+          ignore_unreachable: yes
+
+        - name: wait till device is back online
+          wait_for_connection:
+            delay: 10
+            timeout: 600
+      when: garet_profile != running_garet_profile or garet_release != running_garet_release