CVE-2024-10190
📋 TL;DR
Horovod versions up to v0.28.1 are vulnerable to unauthenticated remote code execution via malicious pickle objects in PUT requests. Attackers can execute arbitrary code on servers running vulnerable Horovod instances. This affects any system using Horovod for distributed deep learning training.
💻 Affected Systems
- Horovod
📦 What is this software?
Horovod by Horovod
⚠️ Risk & Real-World Impact
Worst Case
Complete compromise of the Horovod server with attacker gaining full control over the system, potentially leading to data theft, ransomware deployment, or lateral movement within the network.
Likely Case
Attackers exploit vulnerable internet-facing Horovod instances to install cryptocurrency miners, establish persistent backdoors, or exfiltrate sensitive training data and models.
If Mitigated
With proper network segmentation and access controls, impact is limited to the isolated Horovod environment, preventing lateral movement to critical systems.
🎯 Exploit Status
Exploit requires sending a malicious pickle object via HTTP PUT request to the vulnerable endpoint. The vulnerability is straightforward to exploit with publicly available proof-of-concept code.
🛠️ Fix & Mitigation
✅ Official Fix
Patch Version: v0.29.0 and later
Vendor Advisory: https://github.com/horovod/horovod/security/advisories/GHSA-8q4r-m3rh-8qj2
Restart Required: Yes
Instructions:
1. Update Horovod to version 0.29.0 or later using pip: pip install --upgrade horovod==0.29.0
2. Restart all Horovod processes and services
3. Verify the update with: pip show horovod
🔧 Temporary Workarounds
Network Access Restriction
linuxRestrict network access to Horovod services to trusted IP addresses only
iptables -A INPUT -p tcp --dport 12345 -s trusted_ip_range -j ACCEPT
iptables -A INPUT -p tcp --dport 12345 -j DROP
Disable ElasticRendezvousHandler
allConfigure Horovod to use alternative coordination mechanisms if possible
export HOROVOD_ELASTIC=0
🧯 If You Can't Patch
- Isolate Horovod instances in a dedicated network segment with strict egress filtering
- Implement application-level authentication and authorization for Horovod endpoints
🔍 How to Verify
Check if Vulnerable:
Check Horovod version with: pip show horovod | grep Version
If version is 0.28.1 or earlier, the system is vulnerable.Check Version:
pip show horovod | grep VersionVerify Fix Applied:
Verify Horovod version is 0.29.0 or later: pip show horovod | grep Version
Test with a benign pickle payload to ensure the vulnerability is patched.📡 Detection & Monitoring
Log Indicators:
- Unusual PUT requests to Horovod endpoints
- Base64-encoded pickle objects in HTTP traffic
- Suspicious process execution from Horovod services
Network Indicators:
- HTTP PUT requests to Horovod ports (typically 12345) with base64 payloads
- Outbound connections from Horovod servers to suspicious IPs
SIEM Query:
source="horovod.log" AND (method="PUT" AND uri="/put" AND size>1000)