Freifunk-Leipzig
/
eae-am-deutschen-platz
13
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

incidents: add 042 and 043

This commit is contained in:
Gregor Michels 2023-03-22 23:07:40 +01:00
parent e5a0e2352d
commit d7206111fa
1 changed files with 92 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

92
documentation/INCIDENTS.md
View File

@ -1474,3 +1474,95 @@ Tue Feb 28 08:44:16 2023 daemon.warn dnsmasq[1]: Maximum number of concurrent DN
* changed upstream dns to `9.9.9.9` (quad9) and `1.1.1.1` (cloudflare)
see `a236643` for details
042 2023.03.12 18:00 - 2023.03.22 19:30 (RGS) | `ap-1374` (`kitchen-og`) down
-----------------------------------------------------------------------------
`ap-1374` is (mostly) down since 2023.03.12 18:00.
Neither the ethernet link nor the poe is coming up.
```
user@freifunk-admin:~$ date && ssh sax-rgs-sw-access02
Wed 15 Mar 2023 12:07:55 AM CET
[...]
sax-rgs-sw-access02# show logging buffered
Log messages in buffer
[...]
5;Feb 17 2000 05:37:36;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to down
4;Feb 17 2000 05:37:37;%TRUNK-4-INFO: Power-Over-Ethernet on gi0/7 Powered Down!
4;Feb 17 2000 05:37:48;%TRUNK-4-INFO: Power-Over-Ethernet on gi0/7: Detected Standard PD, Delivering power!
5;Feb 17 2000 05:37:54;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to up
5;Feb 17 2000 05:38:26;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to down
5;Feb 17 2000 05:38:28;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to up
5;Feb 17 2000 05:38:32;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to down
5;Feb 17 2000 05:38:35;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to up
5;Feb 17 2000 05:38:38;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to down
5;Feb 17 2000 05:38:59;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to up
5;Feb 20 2000 10:02:32;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/6, changed state to down
5;Feb 20 2000 10:02:35;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/6, changed state to up
5;Feb 24 2000 22:50:15;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to down
5;Feb 24 2000 22:50:15;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to up
5;Feb 24 2000 22:50:15;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to down
4;Feb 24 2000 22:50:18;%TRUNK-4-INFO: Power-Over-Ethernet on gi0/7 Powered Down!
5;Feb 25 2000 13:57:06;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/6, changed state to down
5;Feb 25 2000 13:57:09;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/6, changed state to up
4;Feb 26 2000 21:52:17;%TRUNK-4-INFO: Power-Over-Ethernet on gi0/7: Detected Standard PD, Delivering power!
5;Feb 26 2000 21:52:22;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to up
5;Feb 26 2000 21:52:54;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to down
5;Feb 26 2000 21:52:57;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to up
5;Feb 26 2000 21:53:01;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to down
5;Feb 26 2000 21:53:03;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to up
5;Feb 26 2000 21:53:06;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to down
5;Feb 26 2000 21:53:26;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to up
5;Feb 27 2000 00:31:56;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to down
4;Feb 27 2000 00:31:57;%TRUNK-4-INFO: Power-Over-Ethernet on gi0/7 Powered Down!
5;Feb 27 2000 05:24:48;%AAA-5-LOGIN: New ssh connection for user admin, source 10.86.254.0 ACCEPTED
6;Feb 27 2000 05:25:00;%AAA-6-INFO: User 'admin' enter privileged mode from ssh with level '15' success
sax-rgs-sw-access02# show clock
2000-02-27 05:43:37 Coordinated(UTC+0)
```
**needed fix**:
* check keystone modules on site
* also check module for `0/6` (there are some `ifInErrors`)
**additional work - set correct time on switches (done)**:
```
sax-rgs-sw-access0X> enable
sax-rgs-sw-access0X# configure terminal
sax-rgs-sw-access0X(config)# clock timezone CET +1
sax-rgs-sw-access0X(config)# clock set 00:26:15 mar 15 2023
sax-rgs-sw-access0X(config)# clock source ntp
sax-rgs-sw-access0X(config)# ntp server pool.ntp.org
sax-rgs-sw-access0X(config)# exit
sax-rgs-sw-access0X(config)# write
```
**disable port till fix is there - done 16.03.2023 00:40**:
```
sax-rgs-sw-access02> enable
sax-rgs-sw-access02# configure terminal
sax-rgs-sw-access02(config-if-GigabitEthernet0/7)# no poe enable
sax-rgs-sw-access02(config-if-GigabitEthernet0/7)# exit
sax-rgs-sw-access02(config)# exit
sax-rgs-sw-access02# write
```
**actual fix - done 22.03.2023**:
* reterminate keystone modules for both links (`GigabitEthernet0/6` and `GigabitEthernet0/7`)
* reenable poe on `GigabitEthernet0/7`
* test by
* resetting link counters on `sax-rgs-sw-access02`
* ` iperf3` from ap to core gateway (bidirectional)
* looking at the counters again
043 2023.03.20 01:30 | (maintenance) update eae-adp-jump01
----------------------------------------------------------
```
syspatch
pkg_add -uU
reboot
```