Exploring the VirusTotal Dataset | An Analyst's Guide to Effective Threat Research

Overview

The VirusTotal Dataset

The top-level structure of the VirusTotal dataset

The communicating_files relationship

Querying VirusTotal

• 
  
  : This endpoint retrieves the tools, or custom scripts. /api/v3/files/{hash}/behaviours retrieves sandbox-generated data for files specified by their hash value. URLs discovered in process memory are stored in the memory_pattern_urls field returned by the endpoint. For files that meet other search criteria, users can programmatically extract the URLs and keep the file in consideration if a URL aligns with a specific search requirement.
  
  
  
  

  memory_pattern_urls values 
  
  
  
  
  
  The API may provide more information than what is visible to users in the GUI. For example, there can be discrepancies in sandbox-generated data provided to users through the GUI and the API. For example, the /api/v3/files/{hash}/behaviours endpoint retrieves all data generated by the sandbox is VirusTotal's in-house AI engine implementation, based on Google’s is a collection of third-party AI engines contributed by the community and is continuously enriched with new additions.
  
  
  
  Depending on their training and design, the AI engines specialize in analyzing specific file types. For example, the ByteDefend AI engine is designed to analyze macro code in Microsoft Office files, including Word, Excel, and PowerPoint documents. 
  
  
  
  The Code Insight engine focuses on script files, such as PowerShell, Python, and Ruby scripts. It excludes from analysis any script files that exceed a set file size or similarity threshold (these values are currently undocumented). The VirusTotal platform compares each script's code with scripts previously analyzed by the AI engine and calculates a similarity value. This value is then evaluated against the similarity threshold.
  
  
  
  While we were writing this post, Google ) binary files. This feature is enabled by three interconnected phases: 
  
  
  
  
  
  • 
    
    Unpacking binaries using the malware analysis service Mandiant Backscatter: This step reveals the underlying code of potentially obfuscated (packed) malicious binaries submitted to VirusTotal, which is the intended subject of analysis.
    
    
  
  
  • 
    
    Decompilation of the unpacked binaries using engine, the platform released two new search modifiers: bytedefend_ai_analysis and bytedefend_ai_verdict.
    
    
    
    The AI search modifiers can be combined with other AI search modifiers or with any other modifiers supported by VirusTotal using the logical operators AND, OR, and NOT. For example, the search query crowdsourced_ai_analysis:"inject" AND crowdsourced_ai_analysis:"explorer.exe" can be used to identify files that perform injection involving the explorer.exe process. The results returned from VirusTotal include the PowerShell script Shell object to execute the PowerShell process powershell.exe for conducting further activities. The results returned from VirusTotal include the Visual Basic script into suspected China-nexus actors targeting Southeast Asian gambling companies. The investigation led to the , AdventureQuest.exe is likely part of the same activity cluster. 
    The certificate’s serial number provides a starting point for identifying any other malware loaders submitted to VirusTotal that are signed with the same certificate and share implementation characteristics with AdventureQuest.exe, suggesting a potential link to the same threat actor or campaign.
    VirusTotal uses the , and on Operation ChattyGoblin, we know that the threat actors do not disguise their malware as Ivacy VPN components. This suggests that the files that had been located in the Ivacy VPN installation directory before submission to VirusTotal may be false positives. An analysis of some of these files using a .NET decompiler revealed that they are indeed legitimate Ivacy VPN components.
    Building on the previous search, the query signature:"0E3E037C57A5447295669A3DB1A28B8A" AND tag:"peexe" AND magic:".NET" AND (NOT metadata:".pdb") AND (NOT name:"Program Files (x86)\ivacy") further narrows the results to submitted executables that do not have PDB paths and had not been located in the Ivacy VPN installation directory before submission. The query returns one result, AdventureQuest.exe. 
    This suggests that VirusTotal does not host other malware loaders, which are signed with the same certificate as AdventureQuest.exe and are likely linked to the investigated threat cluster. However, the extensive number of VirusTotal search modifiers allows for the identification of such loaders based on characteristics beyond the used code signing certificate. For example, querying VirusTotal for a code segment specific to AdventureQuest.exe using the content modifier leads to further malware that is likely part of the same activity cluster. We leave this as an exercise for the reader.
    

    Search queries and results
    
    
    

    Clustering With Search Modifiers | Limitations

    Certain aspects of how VirusTotal collects information on submitted artifacts, which users can query using search modifiers, may increase the likelihood of missing relevant findings in some search scenarios. This is particularly relevant given the third-party tools and functionalities that VirusTotal uses for collecting this information, such as sandboxes and detection engines. Each of these tools has specific limitations, which affect the quality and quantity of information VirusTotal collects and stores in its dataset. In this section, we highlight some of these limitations to help users understand how they impact querying VirusTotal with search modifiers.
    As mentioned earlier, VirusTotal techniques exhibited during execution. This is accomplished through a set of rules that map observed behaviors to MITRE ATT&CK techniques. 
    
    
    For each submitted file, the sandboxes generate a report documenting captured activities, which are accessible to VirusTotal users. To facilitate systematic searching of sandbox-generated data, VirusTotal stores this data in an object of type file_behaviour. This object of search modifiers, such as behavior_created_processes (searches for a name of a created process), behavior_files (searches for a name or path of an opened, written, deleted, or dropped file), or attack_technique (searches for a MITRE ATT&CK , VirusTotal automatically executes the vast volume of executable files continuously submitted to the platform's public corpus in sandboxes, without customizing their execution or execution environments. As a result, search queries with modifiers applied to sandbox-generated data, like behavior_files or attack_technique, may return incomplete results.
    
    
    
    For example, the BlackCat ransomware , the CAPE sandbox has not captured any file system activities, such as deleting, creating, or modifying files. When the correct access token is provided, this sample enumerates the files and folders on the filesystem of a compromised system and encrypts files as specified in embedded configuration data.
    
    
    
    

    The CAPE sandbox report for veros3.exe
    
    
    
    
    
    In addition to running executable files in sandboxes to capture their behaviors, VirusTotal is capable of extracting malware configurations from these files. To achieve this, the platform uses the Mandiant Backscatter malware analysis service, which implements configuration extraction modules to automatically extract configurations based on known implementation patterns. Users can search through extracted configuration data using the malware_config search modifier. However, the automated extraction might not work when the analyzed malware uses new or changed methods for storing its configuration data, which are not covered by the existing modules. As a result, search queries involving malware_config may return incomplete results.
    
    
    
    It is also important to note that some information related to submitted artifacts may change over time. For example, the last_analysis_stats attribute of the file object stores the number of third-party detection engines that have labeled the corresponding submitted file as malicious. Users can use the positives search modifier to narrow searches based on whether this number is less than, greater than, or equal to a user-specified value. For example, positives:20+ narrows searches to files that have been labeled as malicious by more than 20 engines.
    
    
    
    

    last_analysis_stats attribute
    
    
    
    
    
    Setting positives to a relatively high number is a way to focus searches on files that are likely to be malicious. However, the returned results may not include malicious files for which an insufficient number of third-party detection engines have developed detections. The development of detections is fully in control of the engines' vendors and may depend on a variety of factors.
    
    
    
    A common factor prompting vendors to develop a detection for a specific malware implementation is the public release of a threat research report listing files that implement the malware. For example, on September 21, 2023, SentinelLabs released a report on the file as malware used by this group. Prior to this date, on March 15, 22, and 29, 2023, the number of engines detecting the file as malware was 5, 6, and 7, respectively. Shortly after the release of the report, this number spiked to 17 and reached 53 by September 29, 2023.
    
    
    
    

    Number of engines detecting the UpdateCheck.dll malware
    
    
    
    
    

    Conclusions

    
    
    
    Effectively using VirusTotal for threat research requires a good understanding of the platform’s wide range of querying capabilities, the scenarios in which these capabilities return informative results beneficial to investigations, and the factors that may impact the completeness or relevance of the data returned.
    
    
    
    While the GUI provides an agile and user-friendly way to query VirusTotal, the API enables large-scale querying, offers expanded querying capabilities, and allows for retrieving more extensive information. Additionally, the AI engines that VirusTotal integrates can significantly speed up malware analysis efforts; however, their outputs should be considered as part of a broader analysis strategy as they may lack sufficient or useful information due to limitations in design or training data. Moreover, the extensive set of search modifiers provides flexible search capabilities, but the relevance and completeness of results can be impacted by false positives and false negatives.
    
    
    
    SentinelLabs and VirusTotal are committed to sharing information and insights that help new users gain a solid understanding of the platform’s capabilities, enabling them to make full use of the available VirusTotal features and conduct thorough investigations.
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    Vollständiger Original-Bericht

    Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf blog.virustotal.com.

    ↗ Original-Artikel auf blog.virustotal.com lesen