diff --git a/documentation/INCIDENTS.md b/documentation/INCIDENTS.md index 5dd2552..cbb57f2 100644 --- a/documentation/INCIDENTS.md +++ b/documentation/INCIDENTS.md @@ -1474,3 +1474,95 @@ Tue Feb 28 08:44:16 2023 daemon.warn dnsmasq[1]: Maximum number of concurrent DN * changed upstream dns to `9.9.9.9` (quad9) and `1.1.1.1` (cloudflare) see `a236643` for details + + +042 2023.03.12 18:00 - 2023.03.22 19:30 (RGS) | `ap-1374` (`kitchen-og`) down +----------------------------------------------------------------------------- + +`ap-1374` is (mostly) down since 2023.03.12 18:00. +Neither the ethernet link nor the poe is coming up. +``` +user@freifunk-admin:~$ date && ssh sax-rgs-sw-access02 +Wed 15 Mar 2023 12:07:55 AM CET +[...] +sax-rgs-sw-access02# show logging buffered + +Log messages in buffer +[...] +5;Feb 17 2000 05:37:36;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to down +4;Feb 17 2000 05:37:37;%TRUNK-4-INFO: Power-Over-Ethernet on gi0/7 Powered Down! +4;Feb 17 2000 05:37:48;%TRUNK-4-INFO: Power-Over-Ethernet on gi0/7: Detected Standard PD, Delivering power! +5;Feb 17 2000 05:37:54;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to up +5;Feb 17 2000 05:38:26;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to down +5;Feb 17 2000 05:38:28;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to up +5;Feb 17 2000 05:38:32;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to down +5;Feb 17 2000 05:38:35;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to up +5;Feb 17 2000 05:38:38;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to down +5;Feb 17 2000 05:38:59;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to up +5;Feb 20 2000 10:02:32;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/6, changed state to down +5;Feb 20 2000 10:02:35;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/6, changed state to up +5;Feb 24 2000 22:50:15;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to down +5;Feb 24 2000 22:50:15;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to up +5;Feb 24 2000 22:50:15;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to down +4;Feb 24 2000 22:50:18;%TRUNK-4-INFO: Power-Over-Ethernet on gi0/7 Powered Down! +5;Feb 25 2000 13:57:06;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/6, changed state to down +5;Feb 25 2000 13:57:09;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/6, changed state to up +4;Feb 26 2000 21:52:17;%TRUNK-4-INFO: Power-Over-Ethernet on gi0/7: Detected Standard PD, Delivering power! +5;Feb 26 2000 21:52:22;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to up +5;Feb 26 2000 21:52:54;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to down +5;Feb 26 2000 21:52:57;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to up +5;Feb 26 2000 21:53:01;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to down +5;Feb 26 2000 21:53:03;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to up +5;Feb 26 2000 21:53:06;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to down +5;Feb 26 2000 21:53:26;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to up +5;Feb 27 2000 00:31:56;%LINEPROTO-5-UPDOWN: Line protocol on GigabitEthernet0/7, changed state to down +4;Feb 27 2000 00:31:57;%TRUNK-4-INFO: Power-Over-Ethernet on gi0/7 Powered Down! +5;Feb 27 2000 05:24:48;%AAA-5-LOGIN: New ssh connection for user admin, source 10.86.254.0 ACCEPTED +6;Feb 27 2000 05:25:00;%AAA-6-INFO: User 'admin' enter privileged mode from ssh with level '15' success +sax-rgs-sw-access02# show clock +2000-02-27 05:43:37 Coordinated(UTC+0) +``` + +**needed fix**: +* check keystone modules on site +* also check module for `0/6` (there are some `ifInErrors`) + +**additional work - set correct time on switches (done)**: +``` +sax-rgs-sw-access0X> enable +sax-rgs-sw-access0X# configure terminal +sax-rgs-sw-access0X(config)# clock timezone CET +1 +sax-rgs-sw-access0X(config)# clock set 00:26:15 mar 15 2023 +sax-rgs-sw-access0X(config)# clock source ntp +sax-rgs-sw-access0X(config)# ntp server pool.ntp.org +sax-rgs-sw-access0X(config)# exit +sax-rgs-sw-access0X(config)# write +``` + +**disable port till fix is there - done 16.03.2023 00:40**: +``` +sax-rgs-sw-access02> enable +sax-rgs-sw-access02# configure terminal +sax-rgs-sw-access02(config-if-GigabitEthernet0/7)# no poe enable +sax-rgs-sw-access02(config-if-GigabitEthernet0/7)# exit +sax-rgs-sw-access02(config)# exit +sax-rgs-sw-access02# write +``` + +**actual fix - done 22.03.2023**: +* reterminate keystone modules for both links (`GigabitEthernet0/6` and `GigabitEthernet0/7`) +* reenable poe on `GigabitEthernet0/7` +* test by + * resetting link counters on `sax-rgs-sw-access02` + * ` iperf3` from ap to core gateway (bidirectional) + * looking at the counters again + + +043 2023.03.20 01:30 | (maintenance) update eae-adp-jump01 +---------------------------------------------------------- + +``` +syspatch +pkg_add -uU +reboot +```