Introduction
Apache Spark is a distributed open-source system for large-scale data processing and fast general-purpose calculation. It also supports a wide range of advanced tools, including Spark SQL for SQL and DataFrames, pandas API on Spark for pandas workloads, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing. Its powerful features combined with free access make it widely used worldwide. As a result, the shell command injection vulnerability has a broad impact.
Vulnerability
The shell command injection vulnerability revolves around manipulating application ACLs, which allows users/groups – excluding the owner who starts the application – access to an application. The Apache Spark UI offers the possibility to enable ACLs via the configuration option spark.acls.enable. If ACLs are enabled, Apache spark will go through the HttpSecurityFilter filter to verify the user’s permissions. In the code of HttpSecurityFilter, the username can be entered arbitrarily and the fields will be concatenated later. Therefore, a malicious user might then be able to inject a piece of shell command by separating the characters of the command to several username fields. Then after the fields are concatenated into a piece of the shell command the executeAndGetOutput function will cause the command to execute.
Affected Version
Spark Core – Apache<=3.0.3
3.1.1 <= Spark Core – Apache <=3.1.2
3.2.0 <= Spark Core – Apache <=3.2.1
Remediation
1. Official fix: The vendor has released the patches, and we recommend that users update Apache Spark to 3.1.3, 3.2.2, 3.3.0, or a higher version. Please visit:
https://spark.apache.org/downloads.html
2. Reference methods on GitHub: We can fix the vulnerability by removing the call to bash in the ShellBasedGroupsMappingProvider function. Please visit:
https://github.com/apache/spark/pull/36315/files
Implementing the Fix
The Hillstone IPS can support the detection and protection of this vulnerability.
Figure 1. Apache Spark Shell Command Injection Vulnerability detected by Hillstone IPS
The hotspot intelligence of this vulnerability is available on Hillstone iSource as well.
Figure 2. Hotspot intelligence on Hillstone iSource
