1. IT-Security >
  2. Cyber Security Nachrichten >
  3. Malcolm - A Powerful, Easily Deployable Network Traffic Analysis Tool Suite For Full Packet Capture Artifacts (PCAP Files) And Zeek Logs


Malcolm - A Powerful, Easily Deployable Network Traffic Analysis Tool Suite For Full Packet Capture Artifacts (PCAP Files) And Zeek Logs

RSS Kategorie Pfeil IT Security Nachrichten vom | Quelle: feedproxy.google.com Direktlink öffnen

Malcolm is a powerful network traffic analysis tool suite designed with the following goals in mind:
  • Easy to use – Malcolm accepts network traffic data in the form of full packet capture (PCAP) files and Zeek (formerly Bro) logs. These artifacts can be uploaded via a simple browser-based interface or captured live and forwarded to Malcolm using lightweight forwarders. In either case, the data is automatically normalized, enriched, and correlated for analysis.
  • Powerful traffic analysis – Visibility into network communications is provided through two intuitive interfaces: Kibana, a flexible data visualization plugin with dozens of prebuilt dashboards providing an at-a-glance overview of network protocols; and Moloch, a powerful tool for finding and identifying the network sessions comprising suspected security incidents.
  • Streamlined deployment – Malcolm operates as a cluster of Docker containers, isolated sandboxes which each serve a dedicated function of the system. This Docker-based deployment model, combined with a few simple scripts for setup and run-time management, makes Malcolm suitable to be deployed quickly across a variety of platforms and use cases, whether it be for long-term deployment on a Linux server in a security operations center (SOC) or for incident response on a Macbook for an individual engagement.
  • Secure communications – All communications with Malcolm, both from the user interface and from remote log forwarders, are secured with industry standard encryption protocols.
  • Permissive license – Malcolm is comprised of several widely used open source tools, making it an attractive alternative to security solutions requiring paid licenses.
  • Expanding control systems visibility – While Malcolm is great for general-purpose network traffic analysis, its creators see a particular need in the community for tools providing insight into protocols used in industrial control systems (ICS) environments. Ongoing Malcolm development will aim to provide additional parsers for common ICS protocols.
Although all of the open source tools which make up Malcolm are already available and in general use, Malcolm provides a framework of interconnectivity which makes it greater than the sum of its parts. And while there are many other network traffic analysis solutions out there, ranging from complete Linux distributions like Security Onion to licensed products like Splunk Enterprise Security, the creators of Malcolm feel its easy deployment and robust combination of tools fill a void in the network security space that will make network traffic analysis accessible to many in both the public and private sectors as well as individual enthusiasts.
In short, Malcolm provides an easily deployable network analysis tool suite for full packet capture artifacts (PCAP files) and Zeek logs. While Internet access is required to build it, it is not required at runtime.

Quick start

Getting Malcolm
For a TL;DR example of downloading, configuring, and running Malcolm on a Linux platform, see Installation example using Ubuntu 18.04 LTS.

Source code
The files required to build and run Malcolm are available on the Idaho National Lab's GitHub page. Malcolm's source code is released under the terms of a permissive open source software license (see see License.txt for the terms of its release).

Building Malcolm from scratch
The build.sh script can Malcolm's Docker from scratch. See Building from source for more information.

Pull Malcolm's Docker images
Malcolm's Docker images are periodically built and hosted on Docker Hub. If you already have Docker and Docker Compose, these prebuilt images can be pulled by navigating into the Malcolm directory (containing the docker-compose.yml file) and running docker-compose pull like this:
$ docker-compose pull
Pulling elasticsearch ... done
Pulling kibana ... done
Pulling elastalert ... done
Pulling curator ... done
Pulling logstash ... done
Pulling filebeat ... done
Pulling moloch ... done
Pulling file-monitor ... done
Pulling pcap-capture ... done
Pulling upload ... done
Pulling htadmin ... done
Pulling nginx-proxy ... done
You can then observe that the images have been retrieved by running docker images:
$ docker images
malcolmnetsec/moloch 1.4.0 xxxxxxxxxxxx 27 minutes ago 517MB
malcolmnetsec/htadmin 1.4.0 xxxxxxxxxxxx 2 hours ago 180MB
malcolmnetsec/nginx-proxy 1.4.0 xxxxxxxxxxxx 4 hours ago 53MB
malcolmnetsec/file-upload 1.4.0 xxxxxxxxxxxx 24 hours ago 198MB
malcolmnetsec/pcap-capture 1.4.0 xxxxxxxxxxxx 24 hours ago 111MB
malcolmnetsec/file-monitor 1.4.0 xxxxxxxxxxxx 24 hours ago 355MB
malcolmnetsec/logstash-oss 1.4.0 xxxxxxxxxxxx 25 hours ago 1.2 4GB
malcolmnetsec/curator 1.4.0 xxxxxxxxxxxx 25 hours ago 303MB
malcolmnetsec/kibana-oss 1.4.0 xxxxxxxxxxxx 33 hours ago 944MB
malcolmnetsec/filebeat-oss 1.4.0 xxxxxxxxxxxx 11 days ago 459MB
malcolmnetsec/elastalert 1.4.0 xxxxxxxxxxxx 11 days ago 276MB
docker.elastic.co/elasticsearch/elasticsearch-oss 6.8.1 xxxxxxxxxxxx 5 weeks ago 769MB
You must run auth_setup.sh prior to running docker-compose pull. You should also ensure your system configuration and docker-compose.yml settings are tuned by running ./scripts/install.py or ./scripts/install.py --configure (see System configuration and tuning).

Import from pre-packaged tarballs
Once built, the malcolm_appliance_packager.sh script can be used to create pre-packaged Malcolm tarballs for import on another machine. See Pre-Packaged Installation Files for more information.

Starting and stopping Malcolm
Use the scripts in the scripts/ directory to start and stop Malcolm, view debug logs of a currently running instance, wipe the database and restore Malcolm to a fresh state, etc.

User interface
A few minutes after starting Malcolm (probably 5 to 10 minutes for Logstash to be completely up, depending on the system), the following services will be accessible:


Malcolm processes network traffic data in the form of packet capture (PCAP) files or Zeek logs. A packet capture appliance ("sensor") monitors network traffic mirrored to it over a SPAN port on a network switch or router, or using a network TAP device. Zeek logs are generated containing important session metadata from the traffic observed, which are then securely forwarded to a Malcolm instance. Full PCAP files are optionally stored locally on the sensor device for examination later.
Malcolm parses the network session data and enriches it with additional lookups and mappings including GeoIP mapping, hardware manufacturer lookups from organizationally unique identifiers (OUI) in MAC addresses, assigning names to network segments and hosts based on user-defined IP address and MAC mappings, performing TLS fingerprinting, and many others.
The enriched data is stored in an Elasticsearch document store in a format suitable for analysis through two intuitive interfaces: Kibana, a flexible data visualization plugin with dozens of prebuilt dashboards providing an at-a-glance overview of network protocols; and Moloch, a powerful tool for finding and identifying the network sessions comprising suspected security incidents. These tools can be accessed through a web browser from analyst workstations or for display in a security operations center (SOC). Logs can also optionally be forwarded on to another instance of Malcolm.
For smaller networks, use at home by network security enthusiasts, or in the field for incident response engagements, Malcolm can also easily be deployed locally on an ordinary consumer workstation or laptop. Malcolm can process local artifacts such as locally-generated Zeek logs, locally-captured PCAP files, and PCAP files collected offline without the use of a dedicated sensor appliance.

Malcolm leverages the following excellent open source tools, among others.
  • Moloch - for PCAP file processing, browsing, searching, analysis, and carving/exporting; Moloch itself consists of two parts:
    • moloch-capture - a tool for traffic capture, as well as offline PCAP parsing and metadata insertion into Elasticsearch
    • viewer - a browser-based interface for data visualization
  • Elasticsearch - a search and analytics engine for indexing and querying network traffic session metadata
  • Logstash and Filebeat - for ingesting and parsing Zeek Log Files and ingesting them into Elasticsearch in a format that Moloch understands and is able to understand in the same way it natively understands PCAP data
  • Kibana - for creating additional ad-hoc visualizations and dashboards beyond that which is provided by Moloch Viewer
  • Zeek - a network analysis framework and IDS
  • ClamAV - an antivirus engine for scanning files extracted by Zeek
  • CyberChef - a "swiss-army knife" data conversion tool
  • jQuery File Upload - for uploading PCAP files and Zeek logs for processing
  • Docker and Docker Compose - for simple, reproducible deployment of the Malcolm appliance across environments and to coordinate communication between its various components
  • nginx - for HTTPS and reverse proxying Malcolm components
  • ElastAlert - an alerting framework for Elasticsearch. Specifically, the BitSensor fork of ElastAlert, its Docker configuration and its corresponding Kibana plugin are used.

Checking out the Malcolm source code results in the following subdirectories in your malcolm/ working copy:
  • curator - code and configuration for the curator container which define rules for closing and/or deleting old Elasticsearch indices
  • Dockerfiles - a directory containing build instructions for Malcolm's docker images
  • docs - a directory containing instructions and documentation
  • elastalert - code and configuration for the elastalert container which provides an alerting framework for Elasticsearch
  • elasticsearch - an initially empty directory where the Elasticsearch database instance will reside
  • elasticsearch-backup - an initially empty directory for storing Elasticsearch index snapshots
  • filebeat - code and configuration for the filebeat container which ingests Zeek logs and forwards them to the logstash container
  • file-monitor - code and configuration for the file-monitor container which can scan files extracted by Zeek
  • file-upload - code and configuration for the upload container which serves a web browser-based upload form for uploading PCAP files and Zeek logs, and which serves an SFTP share as an alternate method for upload
  • htadmin - configuration for the htadmin user account management container
  • iso-build - code and configuration for building an installer ISO for a minimal Debian-based Linux installation for running Malcolm
  • kibana - code and configuration for the kibana container for creating additional ad-hoc visualizations and dashboards beyond that which is provided by Moloch Viewer
  • logstash - code and configuration for the logstash container which parses Zeek logs and forwards them to the elasticsearch container
  • moloch - code and configuration for the moloch container which handles PCAP processing and which serves the Viewer application
  • moloch-logs - an initially empty directory to which the moloch container will write some debug log files
  • moloch-raw - an initially empty directory to which the moloch container will write captured PCAP files; as Moloch as employed by Malcolm is currently used for processing previously-captured PCAP files, this directory is currently unused
  • nginx - configuration for the nginx reverse proxy container
  • pcap - an initially empty directory for PCAP files to be uploaded, processed, and stored
  • pcap-capture - code and configuration for the pcap-capture container which can capture network traffic
  • scripts - control scripts for starting, stopping, restarting, etc. Malcolm
  • shared - miscellaneous code used by various Malcolm components
  • zeek-logs - an initially empty directory for Zeek logs to be uploaded, processed, and stored
and the following files of special note:
  • auth.env - the script ./scripts/auth_setup.sh prompts the user for the administrator credentials used by the Malcolm appliance, and auth.env is the environment file where those values are stored
  • cidr-map.txt - specify custom IP address to network segment mapping
  • host-map.txt - specify custom IP and/or MAC address to host mapping
  • docker-compose.yml - the configuration file used by docker-compose to build, start, and stop an instance of the Malcolm appliance
  • docker-compose-standalone.yml - similar to docker-compose.yml, only used for the "packaged" installation of Malcolm
  • docker-compose-standalone-zeek-live.yml - identical to docker-compose-standalone.yml, only Filebeat is configured to monitor live Zeek logs (ie., being actively written to)

Building from source
Building the Malcolm docker images from scratch requires internet access to pull source files for its components. Once internet access is available, execute the following command to build all of the Docker images used by the Malcolm appliance:
$ ./scripts/build.sh
Then, go take a walk or something since it will be a while. When you're done, you can run docker images and see you have fresh images for:
  • malcolmnetsec/curator (based on debian:buster-slim)
  • malcolmnetsec/elastalert (based on bitsensor/elastalert)
  • malcolmnetsec/file-monitor (based on debian:buster-slim)
  • malcolmnetsec/file-upload (based on debian:buster-slim)
  • malcolmnetsec/filebeat-oss (based on docker.elastic.co/beats/filebeat-oss)
  • malcolmnetsec/htadmin (based on debian:buster-slim)
  • malcolmnetsec/kibana-oss (based on docker.elastic.co/kibana/kibana-oss)
  • malcolmnetsec/logstash-oss (based on centos:7)
  • malcolmnetsec/moloch (based on debian:stretch-slim)
  • malcolmnetsec/nginx-proxy (based on jwilder/nginx-proxy:alpine)
  • malcolmnetsec/pcap-capture (based on debian:buster-slim)
Additionally, the command will pull from Docker Hub:
  • docker.elastic.co/elasticsearch/elasticsearch-oss

Pre-Packaged installation files

Creating pre-packaged installation files
scripts/malcolm_appliance_packager.sh can be run to package up the configuration files (and, if necessary, the Docker images) which can be copied to a network share or USB drive for distribution to non-networked machines. For example:
$ ./scripts/malcolm_appliance_packager.sh 
You must set a username and password for Malcolm, and self-signed X.509 certificates will be generated
Administrator username: analyst
analyst password:
analyst password (again):

(Re)generate self-signed certificates for HTTPS access [Y/n]?

(Re)generate self-signed certificates for a remote log forwarder [Y/n]?

Store username/password for forwarding Logstash events to a secondary, external Elasticsearch instance [y/N]?
Packaged Malcolm to "/home/user/tmp/malcolm_20190513_101117_f0d052c.tar.gz"

Do you need to package docker images also [y/N]? y
This might take a few minutes...

Packaged Malcolm docker images to "/home/user/tmp/malcolm_20190513_101117_f0d052c_images.tar.gz"

To install Malcolm:
1. Run install.py
2. Follow the prompts

To start, stop, restart, etc. Malcolm:
Use the control scripts in the "scripts/" dir ectory:
- start.sh (start Malcolm)
- stop.sh (stop Malcolm)
- restart.sh (restart Malcolm)
- logs.sh (monitor Malcolm logs)
- wipe.sh (stop Malcolm and clear its database)
- auth_setup.sh (change authentication-related settings)

A minute or so after starting Malcolm, the following services will be accessible:
- Moloch: https://localhost/
- Kibana: https://localhost:5601/
- PCAP Upload (web): https://localhost:8443/
- PCAP Upload (sftp): sftp://[email protected]:8022/files/
- Account management: https://localhost:488/
The above example will result in the following artifacts for distribution as explained in the script's output:
$ ls -lh
total 2.0G
-rwxr-xr-x 1 user user 61k May 13 11:32 install.py
-rw-r--r-- 1 user user 2.0G May 13 11:37 malcolm_20190513_101117_f0d052c_images.tar.gz
-rw-r--r-- 1 user user 683 May 13 11:37 malcolm_20190513_101117_f0d052c.README.txt
-rw-r--r-- 1 user user 183k May 13 11:32 malcolm_20190513_101117_f0d052c.tar.gz

Installing from pre-packaged installation files
If you have obtained pre-packaged installation files to install Malcolm on a non-networked machine via an internal network share or on a USB key, you likely have the following files:
  • malcolm_YYYYMMDD_HHNNSS_xxxxxxx.README.txt - This readme file contains a minimal set up instructions for extracting the contents of the other tarballs and running the Malcolm appliance.
  • malcolm_YYYYMMDD_HHNNSS_xxxxxxx.tar.gz - This tarball contains the configuration files and directory configuration used by an instance of Malcolm. It can be extracted via tar -xf malcolm_YYYYMMDD_HHNNSS_xxxxxxx.tar.gz upon which a directory will be created (named similarly to the tarball) containing the directories and configuration files. Alternately, install.py can accept this filename as an argument and handle its extraction and initial configuration for you.
  • malcolm_YYYYMMDD_HHNNSS_xxxxxxx_images.tar.gz - This tarball contains the Docker images used by Malcolm. It can be imported manually via docker load -i malcolm_YYYYMMDD_HHNNSS_xxxxxxx_images.tar.gz
  • install.py - This install script can load the Docker images and extract Malcolm configuration files from the aforementioned tarballs and do some initial configuration for you.
Run install.py malcolm_XXXXXXXX_XXXXXX_XXXXXXX.tar.gz and follow the prompts. If you do not already have Docker and Docker Compose installed, the install.py script will help you install them.

Preparing your system

Recommended system requirements
Malcolm needs a reasonably up-to-date version of Docker and Docker Compose. In theory this should be possible on Linux, macOS, and recent Windows 10 releases, although so far it's only been tested on Linux and macOS hosts.
To quote the Elasticsearch documentation, "If there is one resource that you will run out of first, it will likely be memory." The same is true for Malcolm: you will want at least 16 gigabytes of RAM to run Malcolm comfortably. For processing large volumes of traffic, I'd recommend at a bare minimum a dedicated server with 16 cores and 16 gigabytes of RAM. Malcolm can run on less, but more is better. You're going to want as much hard drive space as possible, of course, as the amount of PCAP data you're able to analyze and store will be limited by your hard drive.
Moloch's wiki has a couple of documents (here and here and here and a calculator here) which may be helpful, although not everything in those documents will apply to a Docker-based setup like Malcolm.

System configuration and tuning
If you already have Docker and Docker Compose installed, the install.py script can still help you tune system configuration and docker-compose.yml parameters for Malcolm. To run it in "configuration only" mode, bypassing the steps to install Docker and Docker Compose, run it like this:
sudo ./scripts/install.py --configure
Although install.py will attempt to automate many of the following configuration and tuning parameters, they are nonetheless listed in the following sections for reference:

docker-compose.yml parameters
Edit docker-compose.yml and search for the ES_JAVA_OPTS key. Edit the -Xms4g -Xmx4g values, replacing 4g with a number that is half of your total system memory, or just under 32 gigabytes, whichever is less. So, for example, if I had 64 gigabytes of memory I would edit those values to be -Xms31g -Xmx31g. This indicates how much memory can be allocated to the Elasticsearch heaps. For a pleasant experience, I would suggest not using a value under 10 gigabytes. Similar values can be modified for Logstash with LS_JAVA_OPTS, where using 3 or 4 gigabytes is recommended.
Various other environment variables inside of docker-compose.yml can be tweaked to control aspects of how Malcolm behaves, particularly with regards to processing PCAP files and Zeek logs. The environment variables of particular interest are located near the top of that file under Commonly tweaked configuration options, which include:
  • INITIALIZEDB – indicates to Malcolm to create (or recreate) Moloch’s internal settings database on startup; this setting is managed by the wipe.sh and start.sh scripts and does not generally need to be changed manually
  • MANAGE_PCAP_FILES – if set to true, all PCAP files imported into Malcolm will be marked as available for deletion by Moloch if available storage space becomes too low (default false)
  • ZEEK_AUTO_ANALYZE_PCAP_FILES – if set to true, all PCAP files imported into Malcolm will automatically be analyzed by Zeek, and the resulting logs will also be imported (default false)
  • MOLOCH_ANALYZE_PCAP_THREADS – the number of threads available to Moloch for analyzing PCAP files (default 1)
  • ZEEK_AUTO_ANALYZE_PCAP_THREADS – the number of threads available to Malcolm for analyzing Zeek logs (default 1)
  • LOGSTASH_JAVA_EXECUTION_ENGINE – if set to true, Logstash will use the new Logstash Java Execution Engine which may significantly speed up Logstash startup and processing (default false, as it is currently considered experimental)
  • LOGSTASH_OUI_LOOKUP – if set to true, Logstash will map MAC addresses to vendors for all source and destination MAC addresses when analyzing Zeek logs (default true)
  • LOGSTASH_REVERSE_DNS – if set to true, Logstash will perform a reverse DNS lookup for all external source and destination IP address values when analyzing Zeek logs (default false)
  • ES_EXTERNAL_HOSTS – if specified (in the format ''), logs received by Logstash will be forwarded on to another external Elasticsearch instance in addition to the one maintained locally by Malcolm
  • ES_EXTERNAL_SSL – if set to true, Logstash will use HTTPS for the connection to external Elasticsearch instances specified in ES_EXTERNAL_HOSTS
  • ES_EXTERNAL_SSL_CERTIFICATE_VERIFICATION – if set to true, Logstash will require full SSL certificate validation; this may fail if using self-signed certificates (default false)
  • KIBANA_OFFLINE_REGION_MAPS – if set to true, a small internal server will be surfaced to Kibana to provide the ability to view region map visualizations even when an Internet connection is not available (default true)
  • CURATOR_CLOSE_COUNT and CURATOR_CLOSE_UNITS - determine behavior for automatically closing older Elasticsearch indices to conserve memory; see Elasticsearch index curation
  • CURATOR_DELETE_COUNT and CURATOR_DELETE_UNITS - determine behavior for automatically deleting older Elasticsearch indices to reduce disk usage; see Elasticsearch index curation
  • CURATOR_DELETE_GIGS - if the Elasticsearch indices representing the log data exceed this size, in gigabytes, older indices will be deleted to bring the total size back under this threshold; see Elasticsearch index curation
  • CURATOR_SNAPSHOT_DISABLED - if set to False, daily snapshots (backups) will be made of the previous day's Elasticsearch log index; see Elasticsearch index curation
  • AUTO_TAG – if set to true, Malcolm will automatically create Moloch sessions and Zeek logs with tags based on the filename, as described in Tagging (default true)
  • BEATS_SSL – if set to true, Logstash will use require encrypted communications for any external Beats-based forwarders from which it will accept logs; if Malcolm is being used as a standalone tool then this can safely be set to false, but if external log feeds are to be accepted then setting it to true is recommended (default false)
  • ZEEK_EXTRACTOR_MODE – determines the file extraction behavior for file transfers detected by Zeek; see Automatic file extraction and scanning for more details
  • EXTRACTED_FILE_IGNORE_EXISTING – if set to true, files extant in ./zeek-logs/extract_files/ directory will be ignored on startup rather than scanned
  • EXTRACTED_FILE_PRESERVATION – determines behavior for preservation of Zeek-extracted files
  • VTOT_API2_KEY – used to specify a VirusTotal Public API v.20 key, which, if specified, will be used to submit hashes of Zeek-extracted files to VirusTotal
  • EXTRACTED_FILE_ENABLE_CLAMAV – if set to true (and VTOT_API2_KEY is unspecified), Zeek-extracted files will be scanned with ClamAV
  • EXTRACTED_FILE_ENABLE_FRESHCLAM – if set to true, ClamAV will periodically update virus databases
  • PCAP_ENABLE_NETSNIFF – if set to true, Malcolm will capture network traffic on the local network interface(s) indicated in PCAP_IFACE using netsniff-ng
  • PCAP_ENABLE_TCPDUMP – if set to true, Malcolm will capture network traffic on the local network interface(s) indicated in PCAP_IFACE using tcpdump; there is no reason to enable both PCAP_ENABLE_NETSNIFF and PCAP_ENABLE_TCPDUMP
  • PCAP_IFACE – used to specify the network interface(s) for local packet capture if PCAP_ENABLE_NETSNIFF or PCAP_ENABLE_TCPDUMP are enabled; for multiple interfaces, separate the interface names with a comma (eg., 'enp0s25' or 'enp10s0,enp11s0')
  • PCAP_ROTATE_MEGABYTES – used to specify how large a locally-captured PCAP file can become (in megabytes) before it closed for processing and a new PCAP file created
  • PCAP_ROTATE_MINUTES – used to specify an time interval (in minutes) after which a locally-captured PCAP file will be closed for processing and a new PCAP file created
  • PCAP_FILTER – specifies a tcpdump-style filter expression for local packet capture; leave blank to capture all traffic

Linux host system configuration

Installing Docker
Docker installation instructions vary slightly by distribution. Please follow the links below to docker.com to find the instructions specific to your distribution:
After installing Docker, because Malcolm should be run as a non-root user, add your user to the docker group with something like:
$ sudo usermod -aG docker yourusername
Following this, either reboot or log out then log back in.
Docker starts automatically on DEB-based distributions. On RPM-based distributions, you need to start it manually or enable it using the appropriate systemctl or service command(s).
You can test docker by running docker info, or (assuming you have internet access), docker run --rm hello-world.

Installing docker-compose
Please follow this link on docker.com for instructions on installing docker-compose.

Operating system configuration
The host system (ie., the one running Docker) will need to be configured for the best possible Elasticsearch performance. Here are a few suggestions for Linux hosts (these may vary from distribution to distribution):
  • Append the following lines to /etc/sysctl.conf:
# the maximum number of open file handles

# the maximum number of user inotify watches

# the maximum number of memory map areas a process may have

# decrease "swappiness" (swapping out runtime memory vs. dropping pages)

# the maximum number of incoming connections

# the % of system memory fillable with "dirty" pages before flushing

# maximum % of dirty system memory before committing everything
  • Depending on your distribution, create either the file /etc/security/limits.d/limits.conf containing:
# the maximum number of open file handles
* soft nofile 65535
* hard nofile 65535
# do not limit the size of memory that can be locked
* soft memlock unlimited
* hard memlock unlimited
OR the file /etc/systemd/system.conf.d/limits.conf containing:
# the maximum number of open file handles
# do not limit the size of memory that can be locked
  • Change the readahead value for the disk where the Elasticsearch data will be stored. There are a few ways to do this. For example, you could add this line to /etc/rc.local (replacing /dev/sda with your disk block descriptor):
# change disk read-adhead value (# of blocks)
blockdev --setra 512 /dev/sda
  • Change the I/O scheduler to deadline or noop. Again, this can be done in a variety of ways. The simplest is to add elevator=deadline to the arguments in GRUB_CMDLINE_LINUX in /etc/default/grub, then running sudo update-grub2
  • If you are planning on using very large data sets, consider formatting the drive containing elasticsearch volume as XFS.
After making all of these changes, do a reboot for good measure!

macOS host system configuration

Automatic installation using install.py
The install.py script will attempt to guide you through the installation of Docker and Docker Compose if they are not present. If that works for you, you can skip ahead to Configure docker daemon option in this section.

Install Homebrew
The easiest way to install and maintain docker on Mac is using the Homebrew cask. Execute the following in a terminal.
$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ brew install cask
$ brew tap caskroom/versions

Install docker-edge
$ brew cask install docker-edge
This will install the latest version of docker and docker-compose. It can be upgraded later using brew as well:
$ brew cask upgrade --no-quarantine docker-edge
You can now run docker from the Applications folder.

Configure docker daemon option
Some changes should be made for performance (this link gives a good succinct overview).
  • Resource allocation - For a good experience, you likely need at least a quad-core MacBook Pro with 16GB RAM and an SSD. I have run Malcolm on an older 2013 MacBook Pro with 8GB of RAM, but the more the better. Go in your system tray and select DockerPreferencesAdvanced. Set the resources available to docker to at least 4 CPUs and 8GB of RAM (>= 16GB is preferable).
  • Volume mount performance - You can speed up performance of volume mounts by removing unused paths from DockerPreferencesFile Sharing. For example, if you’re only going to be mounting volumes under your home directory, you could share /Users but remove other paths.
After making these changes, right click on the Docker icon in the system tray and select Restart.

Windows host system configuration
There are several ways of installing and running docker with Windows, and they vary depending on the version of Windows you are running, whether or not Hyper-V must be enabled (which is a requirement for VMWare, but is precluded by the recent non-virtual machine release of Docker).
As the author supposes that the target audience of this document are more likely to be running macOS or Linux, detailed instructions for Docker setup under Windows are not included here. Instead, refer to the following links:

Running Malcolm

Configure authentication
Run ./scripts/auth_setup.sh before starting Malcolm for the first time in order to:
  • define the administrator account username and password
  • specify whether or not to (re)generate the self-signed certificates used for HTTPS access
    • key and certificate files are located in the nginx/certs/ directory
  • specify whether or not to (re)generate the self-signed certificates used by a remote log forwarder (see the BEATS_SSL environment variable above)
    • certificate authority, certificate, and key files for Malcolm’s Logstash instance are located in the logstash/certs/ directory
    • certificate authority, certificate, and key files to be copied to and used by the remote log forwarder are located in the filebeat/certs/ directory
  • specify whether or not to store the username/password for forwarding Logstash events to a secondary, external Elasticsearch instance (see the ES_EXTERNAL_HOSTS, ES_EXTERNAL_SSL, and ES_EXTERNAL_SSL_CERTIFICATE_VERIFICATION environment variables above)
    • these parameters are stored securely in the Logstash keystore file logstash/certs/logstash.keystore

Account management
auth_setup.sh is used to define the username and password for the administrator account. Once Malcolm is running, the administrator account can be used to manage other user accounts via a Malcolm User Management page served over HTTPS on port 488 (eg., https://localhost:488 if you are connecting locally).
Malcolm user accounts can be used to access the interfaces of all of its components, including Moloch. Moloch uses its own internal database of user accounts, so when a Malcolm user account logs in to Moloch for the first time Malcolm creates a corresponding Moloch user account automatically. This being the case, it is not recommended to use the Moloch Users settings page or change the password via the Password form under the Moloch Settings page, as those settings would not be consistently used across Malcolm.
Users may change their passwords via the Malcolm User Management page by clicking User Self Service. A forgotten password can also be reset via an emailed link, though this requires SMTP server settings to be specified in htadmin/config.ini in the Malcolm installation directory.

Starting Malcolm
Docker compose is used to coordinate running the Docker containers. To start Malcolm, navigate to the directory containing docker-compose.yml and run:
$ ./scripts/start.sh
This will create the containers' virtual network and instantiate them, then leave them running in the background. The Malcolm containers may take a several minutes to start up completely. To follow the debug output for an already-running Malcolm instance, run:
$ ./scripts/logs.sh
You can also use docker stats to monitor the resource utilization of running containers.

Stopping and restarting Malcolm
You can run ./scripts/stop.sh to stop the docker containers and remove their virtual network. Alternately, ./scripts/restart.sh will restart an instance of Malcolm. Because the data on disk is stored on the host in docker volumes, doing these operations will not result in loss of data.
Malcolm can be configured to be automatically restarted when the Docker system daemon restart (for example, on system reboot). This behavior depends on the value of the restart: setting for each service in the docker-compose.yml file. This value can be set by running ./scripts/install.py --configure and answering "yes" to "Restart Malcolm upon system or Docker daemon restart?."

Clearing Malcolm’s data
Run ./scripts/wipe.sh to stop the Malcolm instance and wipe its Elasticsearch database (including index snapshots).

Capture file and log archive upload
Malcolm serves a web browser-based upload form for uploading PCAP files and Zeek logs over HTTPS on port 8443 (eg., https://localhost:8443 if you are connecting locally).

Additionally, there is a writable files directory on an SFTP server served on port 8022 (eg., sftp://[email protected]:8022/files/ if you are connecting locally).
The types of files supported are:
  • PCAP files (of mime type application/vnd.tcpdump.pcap or application/x-pcapng)
    • PCAPNG files are partially supported: Zeek is able to process PCAPNG files, but not all of Moloch's packet examination features work correctly
  • Zeek logs in archive files (application/gzip, application/x-gzip, application/x-7z-compressed, application/x-bzip2, application/x-cpio, application/x-lzip, application/x-lzma, application/x-rar-compressed, application/x-tar, application/x-xz, or application/zip)
    • where the Zeek logs are found in the internal directory structure in the archive file does not matter
Files uploaded via these methods are monitored and moved automatically to other directories for processing to begin, generally within one minute of completion of the upload.

In addition to be processed for uploading, Malcolm events will be tagged according to the components of the filenames of the PCAP files or Zeek log archives files from which the events were parsed. For example, records created from a PCAP file named ACME_Scada_VLAN10.pcap would be tagged with ACME, Scada, and VLAN10. Tags are extracted from filenames by splitting on the characters "," (comma), "-" (dash), and "_" (underscore). These tags are viewable and searchable (via the tags field) in Moloch and Kibana. This behavior can be changed by modifying the AUTO_TAG environment variable in docker-compose.yml.
Tags may also be specified manually with the browser-based upload form.

Processing uploaded PCAPs with Zeek
The browser-based upload interface also provides the ability to specify tags for events extracted from the files uploaded. Additionally, an Analyze with Zeek checkbox may be used when uploading PCAP files to cause them to be analyzed by Zeek, similarly to the ZEEK_AUTO_ANALYZE_PCAP_FILES environment variable described above, only on a per-upload basis. Zeek can also automatically carve out files from file transfers; see Automatic file extraction and scanning for more details.

Live analysis

Capturing traffic on local network interfaces
Malcolm's pcap-capture container can capture traffic on one or more local network interfaces and periodically rotate these files for processing with Moloch and Zeek. The pcap-capture Docker container is started with additional privileges (IPC_LOCK, NET_ADMIN, NET_RAW, and SYS_ADMIN) in order for it to be able to open network interfaces in promiscuous mode for capture.
The environment variables prefixed with PCAP_ in the docker-compose.yml file determine local packet capture behavior. Local capture can also be configured by running ./scripts/install.py --configure and answering "yes" to "Should Malcolm capture network traffic to PCAP files?."
Note that currently Microsoft Windows and Apple macOS platforms run Docker inside of a virtualized environment. This would require additional configuration of virtual interfaces and port forwarding in Docker, the process for which is outside of the scope of this document.

Webseite öffnen Komplette Webseite öffnen

Kommentiere zu Malcolm - A Powerful, Easily Deployable Network Traffic Analysis Tool Suite For Full Packet Capture Artifacts (PCAP Files) And Zeek Logs

Ähnliche Beiträge