eae-am-deutschen-platz/documentation/INCIDENTS.md

Collection of all incidents
===========================

2022.06.30 17:00 - 18:00 | power issues on gw-core01
----------------------------------------------------

**issue**:

The protective cap of the power strip yanked the power supply of `gw-core01` out of its socket.
Therefore `gw-core01` had no power.

**solution**:

Tape back the protective cap of the power strip and reinsert the power supply

**impact**:

No internet access for 2 hours


2022.06.30 12:00 - ??:?? | wifi issues in Tent 5
------------------------------------------------

**issue**:

A resident reported slow internet speeds. He resides in tent 5. I do not have more information.
But it seems that the ap in tent 5 `ap-ac7c` is very slow and hangs/freezes a lot via ssh.

good ap:
```
user@freifunk-admin:~$ time ssh ap-2bbf uptime
 01:04:43 up 1 day,  6:21,  load average: 0.01, 0.02, 0.00

real	0m1.438s
user	0m0.071s
sys	0m0.011s
user@freifunk-admin:~$ time ssh ap-2bbf uptime
 01:17:49 up 1 day,  6:34,  load average: 0.00, 0.01, 0.00

real	0m1.924s
user	0m0.070s
sys	0m0.010s
```

bad ap:
```
user@freifunk-admin:~$ time ssh ap-ac7c uptime
 01:05:00 up 1 day,  6:33,  load average: 0.01, 0.08, 0.03

real	0m29.526s
user	0m0.070s
sys	0m0.014s
user@freifunk-admin:~$ time ssh ap-ac7c uptime
 01:06:22 up 1 day,  6:34,  load average: 0.00, 0.06, 0.03

real	1m15.379s
user	0m0.081s
sys	0m0.015s
user@freifunk-admin:~$
```

a reboot (2022.07.01 03:09:10 - 03:10:29) did not solve the slugish ssh connections.

**solution**:

shortterm: poweroff ap so clients can hopefully roam to a non-broken ap (done)
```
root@sw-access02:~# uci show poe | grep lan2
poe.@port[1].name='lan2'
root@sw-access02:~# uci set poe.@port[1].enable=0
root@sw-access02:~# uci commit poe
root@sw-access02:~# /etc/init.d/poe restart
root@sw-access02:~# ubus call poe info
{
	"firmware": "v22.4",
	"mcu": "ST Micro ST32F100 Microcontroller",
	"budget": 77.000000,
	"consumption": 0.000000,
	"ports": {
		"lan1": {
			"priority": 0,
			"mode": "PoE+",
			"status": "Searching"
		},
		"lan3": {
			"priority": 0,
			"mode": "PoE+",
			"status": "Searching"
		},
		"lan4": {
			"priority": 0,
			"mode": "PoE+",
			"status": "Searching"
		},
		"lan5": {
			"priority": 0,
			"mode": "PoE+",
			"status": "Searching"
		},
		"lan6": {
			"priority": 0,
			"mode": "PoE+",
			"status": "Searching"
		},
		"lan7": {
			"priority": 0,
			"mode": "PoE+",
			"status": "Disabled"
		},
		"lan8": {
			"priority": 0,
			"mode": "PoE+",
			"status": "Disabled"
		}
	}
}
root@sw-access02:~#
```
longterm: replace the broken ap

**impact**:

**2022.06.31 12:00 - 2022.07.01 03:30**:
(very) slow wifi for clients connected to `ap-ac7c`


**2022.07.01 03:30 - ??**:
bad signal strength for clients in tent 5

**notes**:

While restarting `poe` on `sw-access02` all poe ports dropped power for a few seconds....
Therefore `ap-2bbf` also rebooted (whopsi):
```
user@freifunk-admin:~$ ssh sw-access02 ubus call poe info
{
	"firmware": "v22.4",
	"mcu": "ST Micro ST32F100 Microcontroller",
	"budget": 77.000000,
	"consumption": 4.600000,
	"ports": {
		"lan1": {
			"priority": 0,
			"mode": "PoE+",
			"status": "Searching"
		},
		"lan3": {
			"priority": 0,
			"mode": "PoE+",
			"status": "Delivering power",
			"consumption": 4.600000
		},
		"lan4": {
			"priority": 0,
			"mode": "PoE+",
			"status": "Searching"
		},
		"lan5": {
			"priority": 0,
			"mode": "PoE+",
			"status": "Searching"
		},
		"lan6": {
			"priority": 0,
			"mode": "PoE+",
			"status": "Searching"
		},
		"lan7": {
			"priority": 0,
			"mode": "PoE+",
			"status": "Searching"
		},
		"lan8": {
			"priority": 0,
			"mode": "PoE+",
			"status": "Searching"
		}
	}
}
user@freifunk-admin:~$ ssh ap-2bbf uptime
 01:34:18 up 6 min,  load average: 0.02, 0.07, 0.04
user@freifunk-admin:~$
```
Next time i'll only `reload` the `poe` service