On-Chip System Call Tracing: A Feasibility Study and Open Prototype
Authors
Abstract
There exist several techniques and tools for pro- gram tracing and introspection. These tools can be used for analyzing potentially malicious or untrusted programs. In this setting it is important to prevent that the target program determines whether it is being traced or not. This is typically achieved by minimizing the code of the introspection routines and any artifact or side-effect that the program can leverage. Indeed, the most recent approaches consist of lightly instrumented operating systems or thin hypervisors running directly on bare metal. Following this research trend, we investigate the feasibility of transparently tracing a Linux/ARM program without modifying the software stack, while keeping the analysis cost and flexibility compatible with state of the art, emulation- or bare-metal-based approaches. As for the typical program tracing task, our goal is to reconstruct the stream of system call invocations along with the respective un-marshalled arguments.We propose to leverage the availability of on-chip debugging interfaces of modern ARM systems, which are accessible via JTAG. More precisely, we developed OpenST, an open-source prototype tracer that allowed us to analyze the performance overhead and to assess the transparency with respect to evasive, real-world malicious programs. OpenST has two tracing modes: In-kernel dynamic tracing and external tracing. The in-kernel dynamic tracing mode uses the JTAG interface to “hot-patch” the system calls at runtime, injecting proper introspection code. This mode is more transparent than emulator based approaches, but assumes that the traced program does not have access to the kernel memory—where the introspection code is loaded. The external tracing mode removes this assumption by using the JTAG interface to manage hardware breakpoints. As we can expect, the in-kernel dynamic tracing mode is more efficient and less transparent than the external tracing mode. Our tests show that OpenST’s greater transparency comes at the price of about 74× performance penalty in the worst case. However, with a cost model, we show that OpenST scales better than the state of the art, bare-metal-based approach, while remaining equally stealthy to evasive malware.