1. Server >
  2. Unix Server >
  3. Fabian Arrotin: Implementing Zabbix custom LLD rules with Ansible


Fabian Arrotin: Implementing Zabbix custom LLD rules with Ansible

Unix Server vom | Direktlink: arrfab.net Nachrichten Bewertung

While I have to admit that I'm using Zabbix since the 1.8.x era, I also have to admit that I'm not an expert, and that one can learn new things every day. I recently had to implement a new template for a custom service, that is multi-instances aware, and so can be started multiple times with various configurations, and so with its own set of settings, like tcp port on which to listen, etc .. , but also the number of instances running as it can be different from one node to the next one.

I was thinking about the best way to implement this through Zabbix, and my initial idea was to just have one template per possible instance type, that would though use macros defined at the host level, to know which port to check, etc .. so in fact backporting into zabbix what configuration management (Ansible in our case) already has to know to deploy such app instance.

But parallel to that, I always liked the fact that Zabbix itself has some internal tools to auto-discover items and so triggers for those : That's called Low-level Discovery (LLD in short).

By default, if you use (or have modified) some zabbix templates, you can see those in actions for the mounted filesystems or even the present network interfaces in your linux OS. That's the "magic" : you added a new mount point or a new interface ? Zabbix discovers it automatically and start monitoring it, and also graph values for those.

So back to our monitoring problem and the need for multiple templates : what if we could use LLD too and so have Zabbix automatically checking our deployed instances (multiple ones) automatically ? The good is that we can : one can create custom LLD rules and so it would work OOTB when only one template would be added for those nodes.

If you read the link above for custom LLD rule, you'll see some examples about a script being called at the agent level, from the zabbix server, at periodic interval, to "discover" those custom discovery checks. The interesting part to notice is that it's a json that is returned to zabbix server , pointing to a new key, that is declared at the template level.

So it (usually) goes like this :

  • create a template
  • create a new discovery rule, give it a name and a key (and also eventually add Filters)
  • deploy a new UserParameter at the agent level reporting to that key the json string it needs to declare to zabbix server
  • Zabbix server receives/parses that json and based on the checks/variables declared in that json, it will create , based on those returned macros, some Item Prototypes, Trigger prototypes and so on ...

Magic! ... except that in my specific case, for some reasons I never allowed the zabbix user to really launch commands, due to limited rights and also the Selinux context in which it's running (for interested people, it's running in the zabbix_agent_t context)

I suddenly didn't want to change that base rule for our deployments, but the good news is that you don't have to use UserParameter for LLD ! . It's true that if you look at the existing Discovery Rules for "Network interface discovery", you'll see the key net.if.discovery, that is used for everything after, but the Type is "Zabbix agent". We can use something else in that list, like we already do for a "normal" check

I'm already (ab)using the Trapper item type for a lot of hardware checks : reason is simple : as zabbix user is limited (and I don't want to grant more rights for it), I have some scripts checking for hardware raid controllers (if any), etc, and reporting back to zabbix through zabbix_sender.

Let's use the same logic for the json string to be returned to Zabbix server for LLD. (as yes, Trapper is in the list for the discovery rule Type.

It's even easier for us, as we'll control that through Ansible : It's what is already used to deploy/configure our RepoSpanner instances so we have all the logic there.

Let's first start by creating the new template for repospanner, and create a discovery rule (detecting each instances and settings) :


You can then apply that template to host[s] and wait ... but first we need to report back from agent to server which instances are deployed/running. So let's see how to implement that through ansible.

To keep it short, in Ansible we have the following (default values, not the correct ones) variables (from roles/repospanner/default.yml):

  - name: default
    admin_cli: False
    rpc_port: 8443
    http_port: 8444
    tls_ca_cert: ca.crt
    tls_cert: nodea.regiona.crt
    tls_key: nodea.regiona.key
    my_cn: localhost.localdomain
    master_node : nodea.regiona.domain.com # to know how to join a cluster for other nodes
    init_node: True # To be declared only on the first node

That simple example has only one instance, but you can easily see how to have multiple ones, etc So here is the logic : let's have ansible, when configuring the node, create the file that will be used zabbix_sender (triggered by ansible itself) to send the json to zabbix server. zabbix_sender can use a file that is separated (man page) like this :

  • hostname (or '-' to use name configured in zabbix_agentd.conf)
  • key
  • value

Those three fields have to be separated by one space only, and important : you can't have extra empty line (but something can you easily see when playing with this the first time)

How does our file (roles/repospanner/templates/zabbix-repospanner-lld.j2) look like ? :

- repospanner.lld.instances { "data": [ {% for instance in repospanner_instances -%} { "{{ '{#INSTANCE}' }}": "{{ instance.name }}", "{{ '{#RPCPORT}' }}": "{{ instance.rpc_port }}", "{{ '{#HTTPPORT}' }}": "{{ instance.http_port }}" } {%- if not loop.last -%},{% endif %} {% endfor %} ] }

If you have already used jinja2 templates for Ansible, it's quite easy to understand. But I have to admit that I had troubles with the {#INSTANCE} one : that one isn't an ansible variable, but rather a fixed name for the macro that we'll send to zabbix (and so reused as macro everywhere). But ansible, when trying to translate the jinja2 template, was complaining about missing "comment' : Indeed {# ... #} is a comment in jinja2. So the best way (thanks to people in #ansible for that trick) is to include it in {{ }} brackets but then escape it so that it would be rendered as {#INSTANCE} (nice to know if you have to do that too ....)

The rest is trival : excerpt from monitoring.yml (included in that repospanner role) :

- name: Distributing zabbix repospanner check file
    src: "{{ item }}.j2"
    dest: "/usr/lib/zabbix/{{ item }}"
    mode: 0755
    - zabbix-repospanner-check
    - zabbix-repospanner-lld
  register: zabbix_templates   
    - templates

- name: Launching LLD to announce to zabbix
  shell: /bin/zabbix_sender -c /etc/zabbix/zabbix_agentd.conf -i /usr/lib/zabbix/zabbix-repospanner-lld
  when: zabbix_templates is changed

And this is how is rendered on one of my test node :

- repospanner.lld.instances { "data": [ { "{#INSTANCE}": "namespace_rpms", "{#RPCPORT}": "8443", "{#HTTPPORT}": "8444" }, { "{#INSTANCE}": "namespace_centos", "{#RPCPORT}": "8445", "{#HTTPPORT}": "8446" }  ] }

As ansible auto-announces/push that back to zabbix, zabbix server can automatically start creating (through LLD, based on the item prototypes) some checks and triggers/graphs and so start monitoring each newly instance. You want to add a third one ? (we have two in our case) : ansible pushes the config, would modify the .j2 template and would notify zabbix server. etc, etc ...

The rest is just "normal" operation for zabbix : you can create items/trigger prototypes and just use those special Macros coming from LLD :


It was worth spending some time in the LLD doc and in #zabbix to discuss LLD, but once you see the added value, and that you can automatically configure it through Ansible, one can see how powerful it can be.



Externe Webseite mit kompletten Inhalt öffnen

Kommentiere zu Fabian Arrotin: Implementing Zabbix custom LLD rules with Ansible

➤ Ähnliche Beiträge

  • 1.

    Zabbix Threat Control - Zabbix Vulnerability Assessment Plugin

    vom 1278.35 Punkte ic_school_black_18dp
    This plugin transforms your Zabbix monitoring system into vulnerability, risk and security managment system for your infrastructure.What the plugin doesIt provides Zabbix with information about vulnerabilities existing in your entire infrastructure and suggests eas
  • 2.

    Fabian Arrotin: Implementing Zabbix custom LLD rules with Ansible

    vom 1224.03 Punkte ic_school_black_18dp
    While I have to admit that I'm using Zabbix since the 1.8.x era, I also have to admit that I'm not an expert, and that one can learn new things every day. I recently had to implement a new template for a custom service, that is multi-instances aware,
  • 3.

    Fabian Arrotin: Linking Foreman with Zabbix through MQTT

    vom 340.43 Punkte ic_school_black_18dp
    It's been a while since I thought about this design, but I finally had time to implement it the proper way, and "just in time" as I needed recently to migrate our Foreman instance to another host (from CentOS 6 to CentOS 7) Within the CentOS Infra, we u
  • 4.

    DevAudit - Open-source, Cross-Platform, Multi-Purpose Security Auditing Tool

    vom 272.07 Punkte ic_school_black_18dp
    DevAudit is an open-source, cross-platform, multi-purpose security auditing tool targeted at developers and teams adopting DevOps and DevSecOps that detects security vulnerabilities at multiple levels of the solution stack. DevAudit provides a wide array
  • 5.

    Dr. ROBOT - Tool To Enumerate The Subdomains Associated With A Company By Aggregating The Results Of Multiple OSINT Tools

    vom 262.64 Punkte ic_school_black_18dp
    Dr. ROBOT is a tool for Domain Reconnaissance and Enumeration. By utilizing containers to reduce the overhead of dealing with dependencies, inconsistency across operating sytems, and different languages, Dr. ROBOT is built to be highly portable and configurable.
  • 6.

    zabbix bis 1.8.20/2.0.12/2.2.4/2.3.1 XML Data XML Request XXE erweiterte Rechte

    vom 225.05 Punkte ic_school_black_18dp
    Es wurde eine Schwachstelle in zabbix bis 1.8.20/2.0.12/2.2.4/2.3.1 gefunden. Sie wurde als kritisch eingestuft. Es betrifft eine unbekannte Funktion der Komponente XML Data Handler. Durch Manipulation durch XML Request kann eine erweiterte Rechte-Sc
  • 7.

    Resource-Counter - This Command Line Tool Counts The Number Of Resources In Different Categories Across Amazon Regions

    vom 175.13 Punkte ic_school_black_18dp
    This command line tool counts the number of resources in different categories across Amazon regions. This is a simple Python app that will count resources across different regions and display them on the command line. It first shows the dictionary of the results for the monitored services on a per-region basis, then it shows totals across all regions in a friendlier format. It tries to use the most-efficie
  • 8.

    PHPStan - PHP Static Analysis Tool (Discover Bugs In Your Code Without Running It!)

    vom 171.46 Punkte ic_school_black_18dp
    PHPStan focuses on finding errors in your code without actually running it. It catches whole classes of bugs even before you write tests for the code. It moves PHP closer to compiled languages in the sense that the correctness of each line of the code
  • 9.


    vom 169.38 Punkte ic_school_black_18dp
  • 10.

    Zabbix bis 2.0.17/2.2.12/3.0.2 Configuration Script userparameter_mysql.conf) mysql.size erweiterte Rechte

    vom 160.75 Punkte ic_school_black_18dp
    Es wurde eine kritische Schwachstelle in Zabbix bis 2.0.17/2.2.12/3.0.2 entdeckt. Es geht dabei um eine unbekannte Funktion der Datei userparameter_mysql.conf) der Komponente Configuration Script. Dank der Manipulation des Arguments mysql.size mit einer
  • 11.

    USN-4072-1: Ansible vulnerabilities

    vom 142 Punkte ic_school_black_18dp
    ansible vulnerabilities A security issue affects these releases of Ubuntu and its derivatives: Ubuntu 19.04 Ubuntu 18.04 LTS Ubuntu 16.04 LTS Summary Several security issues were fixed in Ansible. Software Description ansible - Configuration man
  • 12.

    Ansible bis 1.6.5 User Module erweiterte Rechte

    vom 132.54 Punkte ic_school_black_18dp
    Eine kritische Schwachstelle wurde in Ansible bis 1.6.5 gefunden. Hierbei geht es um eine unbekannte Funktion der Komponente User Module. Mittels Manipulieren mit einer unbekannten Eingabe kann eine erweiterte Rechte-Schwachstelle ausgenutzt werden. Kla