“At 8:00 AM on a Monday morning, 40 enterprise customer portals went down simultaneously with ‘NET::ERR_CERT_DATE_INVALID’. A single wildcard SSL certificate had quietly expired over the weekend because renewal emails were sent to a former employee’s inbox.”

In June 2016, during my tenure as Technical Lead at Wipro, we experienced the single most embarrassing application outage of the year.

Our team managed the application delivery layer for a major retail client.

The infrastructure was anchored by high-availability F5 BIG-IP LTM (Local Traffic Manager) 5000-series load balancers terminating SSL/TLS encryption for 40 corporate web portals.

On Monday morning at 8:00 AM, our NOC helpdesk was overwhelmed by thousands of tickets.

Every single corporate web portal—customer checkouts, supplier portals, executive dashboards—was completely offline.

Browsers blocked incoming users with bright red security warnings: NET::ERR_CERT_DATE_INVALID.

---

The Shared Wildcard Trap

When we logged into the F5 BIG-IP LTM tmsh console, the root cause was painfully simple:

A primary wildcard SSL certificate (*.clientdomain.com) had reached its expiration timestamp at 11:59 PM on Sunday night.

Because our architecture team had configured a single shared ClientSSL profile referencing this wildcard certificate and bound it across 40 separate Virtual Servers (VIPs), all 40 production applications died at the exact same microsecond.

# F5 BIG-IP tmsh Command Output inspecting Certificate Expiry
tmsh list sys crypto cert wildcard_2016.crt expiration
# Output: Expiration: Jun 19 23:59:59 2016 GMT (EXPIRED 8 HOURS AGO!)

The underlying failure was human process:

• Certificate renewal dates were tracked manually inside an Excel spreadsheet (SSL_Certificate_Tracker_v2.xlsx).
• The system administrator who created the spreadsheet had left the company six months earlier.
• When the commercial Certificate Authority (CA) sent automated 30-day and 7-day expiration warning emails, they were delivered to the former employee’s disabled email address.

Nobody checked the spreadsheet. Nobody received the warning emails.

---

The Mess: The Self-Signed Certificate Disaster

Panicked by calls from executive leadership, a junior engineer attempted an emergency workaround.

He generated a self-signed wildcard certificate on the F5 LTM (sys crypto cert self-signed) and bound it to the ClientSSL profile to get the VIPs back online.

It made the outage worse.

When users refreshed their browsers, instead of a simple expiration warning, browsers threw a severe security alert: NET::ERR_CERT_AUTHORITY_INVALID (Untrusted Certificate Authority).

# Browser Security Escalation

- **Expired Certificate Warning:** "The certificate has expired." (User can click proceed)
- **Self-Signed Certificate Alert:** "WARNING: Attacker may be stealing your passwords!" (Browser HARD BLOCKS connection!)

Corporate security scanners flagged the self-signed certificate as a active Man-In-The-Middle attack, and regional security software blocked the client’s domains globally.

Self-signed certificates cannot be used as emergency patches for public enterprise VIPs.

We had to wait four agonizing hours for DigiCert to execute emergency domain validation and issue a signed production wildcard certificate.

Four hours of downtime cost the client over $180,000 USD in lost e-commerce transactions.

---

The Solution: Automated Python Expiration Scanning & REST API Audit

We permanently banned manual Excel spreadsheet certificate tracking and built an automated Python PKI Expiration Inspector.

The Python script runs on a weekly cron job inside our management environment. It reads an inventory of 200+ public and internal hostnames, executes a TLS handshake, extracts the X.509 notAfter expiration timestamp, and calculates days remaining.

#!/usr/bin/env python3
# Automated TLS Certificate Expiration Inspector
import socket
import ssl
import datetime
import requests

TARGET_HOSTS = ["app.clientdomain.com", "checkout.clientdomain.com", "portal.clientdomain.com"]
SLACK_WEBHOOK_URL = "https://hooks.slack.com/services/T00/B00/X00"

def audit_certificate(hostname, port=443):
    context = ssl.create_default_context()
    try:
        with socket.create_connection((hostname, port), timeout=5) as sock:
            with context.wrap_socket(sock, server_hostname=hostname) as ssock:
                cert = ssock.getpeercert()
                # Extract X.509 Expiration Date
                expiry_str = cert['notAfter']
                expiry_date = datetime.datetime.strptime(expiry_str, '%b %d %H:%M:%S %Y %Z')
                days_left = (expiry_date - datetime.datetime.utcnow()).days

                # Trigger Escalation Alerts based on SLA thresholds
                if days_left <= 30:
                    alert_msg = f"⚠️ CRITICAL: Certificate for '{hostname}' expires in {days_left} days! (Expiry: {expiry_date})"
                    requests.post(SLACK_WEBHOOK_URL, json={"text": alert_msg})
                print(f"Host: {hostname:<30} | Days Remaining: {days_left}")
    except Exception as e:
        print(f"ERROR: Failed to inspect {hostname}: {str(e)}")

for host in TARGET_HOSTS:
    audit_certificate(host)

F5 BIG-IP iControl REST API Integration

To prevent un-tracked SSL certificates from existing on our load balancers, we integrated a script that queries the F5 BIG-IP iControl REST API directly, auditing every installed SSL profile in hardware memory:

# Querying F5 LTM SSL Certificate Expiry via iControl REST API
curl -k -u admin:secret https://f5-ltm01.corp.local/mgmt/tm/sys/crypto/cert | jq '.items[] | {name: .name, expiration: .expirationString}'

The script automatically generates PagerDuty tickets at 60, 30, 14, and 7 days prior to expiration, escalating directly to the active NOC on-call engineer.

---

The Impact

• Zero Expired Certificate Outages: Eliminated 100% of certificate expiration outages across 200+ enterprise web VIPs.
• Automated Audit Pipeline: Replaced manual spreadsheets with continuous Python OpenSSL and F5 REST API automated monitoring.
• 60-Day Renewal SLA: Established a mandatory policy requiring all production TLS certificates to be renewed 30 days before expiration.

Key Takeaway

Automate Public Key Infrastructure (PKI) Monitoring Programmatically.

Never rely on human memory, individual sysadmin email accounts, or manual Excel spreadsheets for TLS certificate renewal tracking. Build automated Python OpenSSL scanners and query load balancer REST APIs programmatically to audit certificate expiration dates weekly, triggering multi-tier PagerDuty alerts at 60, 30, and 14 days before expiry.

---

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin