Hugging Face, the widely used platform for hosting AI and machine learning models, recently faced cybersecurity concerns after researchers discovered malicious models embedded with malware.
A report by cybersecurity firm ReversingLabs highlighted that attackers exploited vulnerabilities to distribute harmful code, potentially affecting developers and organizations relying on open-source AI solutions.
Exploitation via Pickle File Serialization
According to ReversingLabs, the malicious models leveraged Pickle file serialization, a method that enables Python code execution during model loading.
While this serialization technique is efficient, it is often flagged as insecure because it allows arbitrary code execution, making it susceptible to misuse in open-source environments.
Hugging Face, as an open-source platform, provides broad access to ML models, which attackers exploited to distribute malware-laden files that initially evaded detection by the platform’s security tools.
How the Attack Worked
The exploit involved compressing models using the 7z format, which interfered with Hugging Face’s Picklescan security tool.
Typically, models stored in the PyTorch format use ZIP compression, allowing Picklescan to scan for harmful content. However, the use of the less common 7z format rendered these scans ineffective.
ReversingLabs described the exploitation technique as “nullifAI,” emphasizing its ability to bypass existing security protocols. The cybersecurity firm warned that developers who downloaded the compromised models might have unknowingly introduced malware into their systems.
Technical Observations
- Bypassing Security Tools: The compressed files in PyTorch format prevented Picklescan from detecting malicious payloads.
- Broken Pickle Files: These files contained “broken” serialization data, halting processes shortly after execution, which allowed malicious functions to bypass security checks.
- Malicious Payloads: The embedded code reportedly enabled reverse shell connections, allowing attackers to establish communication with affected systems and execute commands remotely.
Hugging Face’s Response
Upon receiving a report from ReversingLabs on January 20, Hugging Face acted swiftly to address the issue.
The malicious models were removed within 24 hours. Additionally, the platform updated its Picklescan tool to better identify threats, including those present in compressed or broken Pickle files.
Security Recommendations
This incident highlights the challenges of maintaining security on collaborative open-source platforms. ReversingLabs urged developers to be cautious when downloading third-party models and to consider safer practices,
- Avoiding reliance on inherently insecure serialization formats like Pickle.
- Keeping custom loading functions separate from serialized model data.
- Regularly reviewing and documenting loading procedures.
Frequently Asked Questions
What is Pickle file serialization and why is it risky?
Pickle file serialization is a Python method for converting objects into byte streams. Its risk lies in its ability to execute arbitrary code during deserialization, which attackers can exploit by embedding malicious payloads.
How did attackers bypass Hugging Face’s security tools?
Attackers exploited vulnerabilities by compressing ML models with the 7z format instead of the typical ZIP compression. This unconventional format prevented Picklescan from detecting harmful code embedded within the models.
What measures has Hugging Face taken to secure the platform?
Once the malicious models were discovered, Hugging Face swiftly removed them and updated their Picklescan tool to better detect threats, including those hidden in compressed or broken Pickle files.
What precautions should developers take when using third-party ML models?
Developers are advised to verify the integrity and provenance of models, avoid relying solely on Pickle serialization, implement custom loading functions with strict validation, and monitor for any unusual behavior when integrating third-party models.
What is a reverse shell and why is it significant here?
A reverse shell is a technique that allows an attacker to gain remote access to a compromised system by initiating an outbound connection. In this context, it enabled attackers to execute commands on affected systems, emphasizing the severe security risks involved.
