How it works

How it works - basics

NERD receives data about malicious behavior of IP addresses (or potentially other network entities) from various sources and keeps a record for each such malicious entity. The record contains metadata about detected events (alerts) and various other data related to the entity. The goal is to have all security related information about malicious IPs in one place.

Data

Basic data classification

The main data element in NERD is entity record, a record representing an entity (e.g. IP address) by a set of attributes. All the data in NERD can be divided into two classes:

• Primary data
  • Data about malicious behavior of IP addresses that may result in creation of a new IP record (i.e. if the IP is not in the Entity DB yet, it's added)
  • Currently the only primary data are alerts from Warden (messages about detected security events in IDEA format).
    • The main NERD instance uses data from CESNET's main Warden server (available to a closed community only).
• Secondary data
  • All other data attached to entity records.
  • Some are queried from external data sources (e.g. hostname, geolocation, presence on blacklists, whois data, ...), some are computed from other data already present in database (e.g. reputation score computed from metadata about detected events, or tags based on keywords in hostname)

Entities

The main type of entity processed by NERD is an IP address.

There are currently other types of records in Entity DB - for BGP prefixes, Autonomous systems (ASN), IP blocks (or networks, from whois database) and Organizations. These serve mostly as groupings of IP addresses. These records are created only if there is some IP record belonging to the group.

Record creation and updates

When an alert about previously unseen IP address is received, a new record for that IP is created in Entity DB and various other information (secondary data) about the IP are fetched from external sources and/or computed. If needed, a record for corresponding ASN, IP block, etc. is created. When an alert about already known address is received, only metadata about the alerts (e.g. number of alerts per category and day) are updated and attributes depending on it (e.g. reputation score) recomputed.

All data are periodically updated, usually once per day or week. For some data history is stored as well (currently only for blacklists).

IP records are deleted after a defined time of inactivity, i.e. when no alert about the IP is received for that time. If some alert is received, it will extend IP record lifetime for X days. X depends on the source the alert came from. By default it is 14 days if the alert was from Warden and 180 days if the alert was from MISP.

Modules

All data processing, i.e. receiving primary data and computing and updating secondary data, is done by a set of processes forming the backend system of NERD. This system is highly modular, with a core processing component (workers), primary modules for receiving data from primary sources (potentially creating new records) and secondary modules for fetching or computing the supplementary data. Each module is usually responsible for a single data source, setting one attribute or a set of related attributes.

See Architecture page for more details.

TODO

• principle of update requests, handler functions, update cascade
  • logically - entity record has attributes, these may be asynchronously updated (e.g. by receiving new alert, periodic update); values of some attributes may depend on values of other attributes, therefore, an update of one attribute may trigger recomputation of another one - thus a whole cascade of changes may be triggered by a single change (or addition) of an attribute.
  • how is it implemented (queue of update requests, handler functions, worker threads, ...), maybe as a separate page
• probably many other things...