LAP/WLC MIC or SSC lifetime expiration causes DTLS failure
CSCuq19142
Description
Symptom:
Wireless Access Points fail to connect to the Wireless LAN Controller.
Symptom 1 (where the AP's certificate has expired):
At the time of the join failure, the WLC's msglog may show messages similar to
the following:
Jul 10 16:13:52.443 spam_lrad.c:6164 LWAPP-3-PAYLOAD_ERR: Join request does not contain valid certificate in certificate payload - AP 00:11:22:33:44:55
Symptom 2 (where the WLC's manufacturing installed certificate has expired):
Once the WLC's MIC expires, the currently joined AP CAPWAP sessions will remain established.
However, once an AP needs to reestablish the CAPWAP connection, it will fail.
The AP logger will show messages similar to the following:
*Oct 29 18:01:56.107: %PKI-3-CERTIFICATE_INVALID_EXPIRED: Certificate chain validation has failed.
The certificate (SN: 7E3446C40000000CBD95) has expired. Validity period ended on 14:38:08 UTC Oct
26 2021Peer certificate verification failed 001A
*Oct 29 18:01:56.107: DTLS_CLIENT_ERROR: ../capwap/base_capwap/capwap/base_capwap_wtp_dtls.c:496
Certificate verified failed!
*Oct 29 18:01:56.107: %DTLS-5-SEND_ALERT: Send FATAL : Bad certificate Alert to 192.168.10.10:5246
*Oct 29 18:01:56.107: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 192.168.10.10:5246
On the WLC side, you will only see a message like this:
*osapiBsnTimer: Oct 29 11:05:04.571: #DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:2962 Failed to complete DTLS handshake with peer 192.168.202.8
Conditions:
For MIC expiration: this symptom will occur after 10 years of the device manufacturing date.
For SSC expiration: this symptom will occur after Jan. 1, 2020
The oldest APs (1120, 1130, 1230, 1310 series) with MICs were manufactured in July 2005,
so those APs will be unable to join AireOS controllers starting in July 2015.
This problem also affects WLCs approximately 10 years after manufacturing date.
To determine when the AP's MIC was created, run this command on the WLC to find the SN:
(Cisco Controller) >show ap inventory all
Inventory for lap1130-sw3-9
NAME: "Cisco AP" , DESCR: "Cisco Wireless Access Point"
PID: AIR-LAP1131AG-E-K9, VID: V01, SN: FCZ1128Q0PE
NAME: "Dot11Radio0" , DESCR: "802.11G Radio"
PID: UNKNOWN, VID: , SN: GAM112706LC
NAME: "Dot11Radio1" , DESCR: "802.11A Radio"
PID: UNKNOWN, VID: , SN: ALP112706LC
The AP chassis SN is in the first section of the output, for example: PID: AIR-LAP1131AG-E-K9, VID: V01, SN: FCZ1128Q0PE
The serial number format is: "LLLYYWWSSSS"; where "YY" is the year of manufacture and "WW" is the week of manufacture. The date code can be found in the 4 middle digits of the serial number.
Manufacturing Year Codes:
01 = 1997 06 = 2002 11 = 2007 16 = 2012
02 = 1998 07 = 2003 12 = 2008 17 = 2013
03 = 1999 08 = 2004 13 = 2009 18 = 2014
04 = 2000 09 = 2005 14 = 2010
05 = 2001 10 = 2006 15 = 2011
Manufacturing Week Codes:
1-5 : January 15-18 : April 28-31 : July 41-44 : October
6-9 : February 19-22 : May 32-35 : August 45-48 : November
10-14 : March 23-27 : June 36-40 : September 49-52 : December
Example: SN FCZ1128Q0PE has year code 11, meaning it was manufactured in 2007. The week code is 12, meaning it was manufactured in March.
The SN can also be found using Prime Infrastructure Reporting to find SNs for all of the APs.
Workaround:
Workaround 1: Generate and use Locally Significant Certificates for authentication
between the affected WLC(s) and APs.
Workaround 2: Configure the WLC/APs to ignore MIC expiration.
Code with fix is available on CCO for 7.0, 7.4, 8.x
For 7.6, you may contact TAC for escalation code, although it is recommended to move to 8.0 for future support
Recovery for APs in a failed scenario:
NOTE: this workaround should be used only in order to allow APs with expired certificates to join the WLC for long enough to upgrade the software.
If the certificates have expired, disable NTP, then change the WLC clock time to a recent earlier time when the certificates were still valid. If you set the clock back too far, newer APs may not be able to join. Once the software has been upgraded, and the affected APs have joined, the WLC clock should be reset to the valid time.
Solution:
Cisco has released AireOS 7.0.252.0, and will release rebuilds for 7.4 in April 2015 and 8.0 in June 2015.
These rebuilds will implement a new CLI command to disable on the WLC
the lifetime validity checks for MICs and SSCs. By default, the command will be disabled, i.e. APs with expired MICs and SSCs will not be able to join.
After upgrading to the new rebuild, use the new command to disable the
lifetime validity check, allowing APs with MICs or SSCs older than 10 years to
join.
Further Problem Description:
This bug CSCuq19142 does not fix the problem for certain 4400 series WLCs
manufactured in 2005, whose MICs expire as early as March, 2015. For such WLCs, track the bug CSCuu02970.
The command implemented via this bug CSCuq19142, to disable expiration
checking, is "config ap lifetime-check {mic|ssc} enable". Because the effect
of this command is logically opposite to the plain sense of this command,
later AireOS releases implement a different command to disable expiration
checking (config ap cert-expiry-ignore {mic|ssc} enable)
For more information see:
Field Notice 63942 - http://www.cisco.com/c/en/us/support/docs/field-notices/639/fn63942.html
https://community.cisco.com/t5/wireless-mobility-documents/lightweight-ap-fail-to-create-capwap-lwapp-connection-due-to/ta-p/3155111
Important note: For 7.6 escalation code, this command is not sync to the peer in a High Availability pair, it may be needed to be configured in each of the controllers for HA while they are active
The command syncs correctly in 7.4, and 8.x available CCO codes
Important note: AP-COS APs (i.e. 802.11ac Wave 2 / 802.11ax APs such as 1800/2800/3800/9100 series) cannot ignore certificate expiration prior to 8.5. See CSCvb93909.
Important note: IOS APs (i.e. 802.11n / 802.11ac Wave 1 APs), which were manufactured with SHA-2 certificates, cannot ignore WLC certificate expiration prior to 8.5.160.0. See CSCvs22835.