CVE-2022-21698
📋 TL;DR
CVE-2022-21698 is a denial-of-service vulnerability in Prometheus client_golang's promhttp package where HTTP servers using certain instrumentation middleware are vulnerable to memory exhaustion attacks. Attackers can send requests with non-standard HTTP methods to create unbounded metric cardinality, causing memory exhaustion and service disruption. Affected systems are those using client_golang instrumentation with promhttp middleware that includes method labels.
💻 Affected Systems
- Prometheus client_golang
📦 What is this software?
Client Golang by Prometheus
Extra Packages For Enterprise Linux by Fedoraproject
View all CVEs affecting Extra Packages For Enterprise Linux →
Extra Packages For Enterprise Linux by Fedoraproject
View all CVEs affecting Extra Packages For Enterprise Linux →
Fedora by Fedoraproject
Fedora by Fedoraproject
Fedora by Fedoraproject
Fedora by Fedoraproject
Fedora by Fedoraproject
Fedora by Fedoraproject
Rdo by Rdo Project
⚠️ Risk & Real-World Impact
Worst Case
Complete service unavailability due to memory exhaustion, requiring service restart and potentially causing extended downtime.
Likely Case
Service degradation or temporary unavailability under targeted attack, requiring intervention to restore normal operation.
If Mitigated
Minimal impact with proper request filtering or updated library version, maintaining normal service availability.
🎯 Exploit Status
Attack requires sending HTTP requests with non-standard methods; no authentication needed.
🛠️ Fix & Mitigation
✅ Official Fix
Patch Version: 1.11.1
Vendor Advisory: https://github.com/prometheus/client_golang/security/advisories/GHSA-cg3q-j54f-5p7p
Restart Required: Yes
Instructions:
1. Update go.mod to require github.com/prometheus/client_golang v1.11.1 or later. 2. Run 'go mod tidy'. 3. Rebuild and redeploy application. 4. Restart affected services.
🔧 Temporary Workarounds
Remove method label
allRemove 'method' label name from counter/gauge metrics used in InstrumentHandler middleware
Modify metric definitions to exclude 'method' label
Add request sanitization middleware
allAdd custom middleware before promhttp handler to sanitize request methods
Implement middleware that validates HTTP methods before passing to promhttp
🧯 If You Can't Patch
- Deploy reverse proxy or WAF configured to only allow standard HTTP methods (GET, POST, PUT, DELETE, etc.)
- Disable affected promhttp handlers or implement rate limiting on non-standard method requests
🔍 How to Verify
Check if Vulnerable:
Check if application uses client_golang <1.11.1 and uses promhttp.InstrumentHandler* middleware with method labels
Check Version:
grep 'github.com/prometheus/client_golang' go.mod
Verify Fix Applied:
Verify client_golang version is >=1.11.1 and test with non-standard HTTP method requests
📡 Detection & Monitoring
Log Indicators:
- Unusual HTTP method names in access logs
- Memory usage spikes
- Increased error rates
Network Indicators:
- High volume of requests with non-standard HTTP methods
- Unusual request patterns to instrumented endpoints
SIEM Query:
http.method NOT IN ("GET", "POST", "PUT", "DELETE", "HEAD", "OPTIONS", "PATCH") AND dest_port IN (application_ports)
🔗 References
- https://github.com/prometheus/client_golang/pull/962
- https://github.com/prometheus/client_golang/pull/987
- https://github.com/prometheus/client_golang/releases/tag/v1.11.1
- https://github.com/prometheus/client_golang/security/advisories/GHSA-cg3q-j54f-5p7p
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/2IK53GWZ475OQ6ENABKMJMTOBZG6LXUR/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/2PFW6Q2LXXWTFRTMTRN4ZGADFRQPKJ3D/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/36GUEPA5TPSC57DZTPYPBL6T7UPQ2FRH/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/3L6GDN5S5QZSCFKWD3GKL2RDZQ6B4UWA/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/4KDETHL5XCT6RZN2BBNOCEXRZ2W3SFU3/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/5OGNAFVXSMTTT2UPH6CS3IH6L3KM42Q7/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/7V7I72LSQ3IET3QJR6QPAVGJZ4CBDLN5/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/AK7CJBCGERCRXYUR2EWDSSDVAQMTAZGX/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/DLUJZV3HBP56ADXU6QH2V7RNYUPMVBXQ/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/FY3N7H6VSDZM37B4SKM2PFFCUWU7QYWN/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/HLAQRRGNSO5MYCPAXGPH2OCSHOGHSQMQ/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/J5WPM42UR6XIBQNQPNQHM32X7S4LJTRX/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/KBMVIQFKQDSSTHVVJWJ4QH6TW3JVB7XZ/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/MH6ALXEQXIFQRQFNJ5Y2MJ5DFPIX76VN/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/RN7JGC2LVHPEGSJYODFUV5FEKPBVG4D7/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/SASRKYHT5ZFSVMJUQUG3UAEQRJYGJKAR/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/ZKORFJTRRDJCWBTJPISKKCVMMMJBIRLG/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/ZY2SLWOQR4ZURQ7UBRZ7JIX6H6F5JHJR/
- https://github.com/prometheus/client_golang/pull/962
- https://github.com/prometheus/client_golang/pull/987
- https://github.com/prometheus/client_golang/releases/tag/v1.11.1
- https://github.com/prometheus/client_golang/security/advisories/GHSA-cg3q-j54f-5p7p
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/2IK53GWZ475OQ6ENABKMJMTOBZG6LXUR/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/2PFW6Q2LXXWTFRTMTRN4ZGADFRQPKJ3D/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/36GUEPA5TPSC57DZTPYPBL6T7UPQ2FRH/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/3L6GDN5S5QZSCFKWD3GKL2RDZQ6B4UWA/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/4KDETHL5XCT6RZN2BBNOCEXRZ2W3SFU3/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/5OGNAFVXSMTTT2UPH6CS3IH6L3KM42Q7/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/7V7I72LSQ3IET3QJR6QPAVGJZ4CBDLN5/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/AK7CJBCGERCRXYUR2EWDSSDVAQMTAZGX/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/DLUJZV3HBP56ADXU6QH2V7RNYUPMVBXQ/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/FY3N7H6VSDZM37B4SKM2PFFCUWU7QYWN/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/HLAQRRGNSO5MYCPAXGPH2OCSHOGHSQMQ/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/J5WPM42UR6XIBQNQPNQHM32X7S4LJTRX/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/KBMVIQFKQDSSTHVVJWJ4QH6TW3JVB7XZ/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/MH6ALXEQXIFQRQFNJ5Y2MJ5DFPIX76VN/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/RN7JGC2LVHPEGSJYODFUV5FEKPBVG4D7/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/SASRKYHT5ZFSVMJUQUG3UAEQRJYGJKAR/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/ZKORFJTRRDJCWBTJPISKKCVMMMJBIRLG/
- https://lists.fedoraproject.org/archives/list/package-announce%40lists.fedoraproject.org/message/ZY2SLWOQR4ZURQ7UBRZ7JIX6H6F5JHJR/