Puppet module for official Beats

.fixtures.yml

@@ -0,0 +1,6 @@
+fixtures:
+  forge_modules:
+    apt: puppetlabs/apt
+    elastic_stack: elastic/elastic_stack
+    stdlib: puppetlabs/stdlib
+zypprepo: darin/zypprepo

README.md

@@ -15,57 +15,132 @@
 
 ## Description
 
-Start with a one- or two-sentence summary of what the module does and/or what problem it solves. This is your 30-second elevator pitch for your module. Consider including OS/Puppet version it works with.       
-
-You can give more descriptive information in a second paragraph. This paragraph should answer the questions: "What does this module *do*?" and "Why would I use it?" If your module has a range of functionality (installation, configuration, management, etc.), this is the time to mention it.
+A Puppet module for managing and configuring official Elastic beats. Works best with RPM/DEB installs of Beats packages but tries to handle custom installations.
 
 ## Setup
 
-### What beats affects **OPTIONAL**
-
-If it's obvious what your module touches, you can skip this section. For example, folks can probably figure out that your mysql_instance module affects their MySQL instances.
+### What beats affects
 
-If there's more that they should know about, though, this is the place to mention:
+* Elastic stack repository files.
+* Each Beats package.
+* Each Beats configuration file.
+* Each Beats service file.
+* Metricbeat module configuration files.
 
-* Files, packages, services, or operations that the module will alter, impact, or execute.
-* Dependencies that your module automatically installs.
-* Warnings or other important notices.
+### Setup Requirements
 
-### Setup Requirements **OPTIONAL**
+* The [stdlib](https://forge.puppetlabs.com/puppetlabs/stdlib) Puppet library.
 
-If your module requires anything extra before setting up (pluginsync enabled, another module, etc.), mention it here. 
-
-If your most recent release breaks compatibility or requires particular steps for upgrading, you might want to include an additional "Upgrading" section here.
+#### Repository management
 
-### Beginning with beats  
+When using the repository management, the following module dependencies are required:
 
-The very basic steps needed for a user to get the module up and running. This can include setup steps, if necessary, or it can be an example of the most basic use of the module.
+* Debian/Ubuntu: [Puppetlabs/apt](http://forge.puppetlabs.com/puppetlabs/apt)
+* OpenSuSE/SLES: [Darin/zypprepo](https://forge.puppetlabs.com/darin/zypprepo)
 
-## Usage
+### Beginning with beats
 
-This section is where you describe how to customize, configure, and do the fancy stuff with your module here. It's especially helpful if you include usage examples and code samples for doing things with your module.
+Include the `beats` class and pass a list of individual Beats to manage with `beats_manage`:
 
-## Reference
+```puppet
+class { 'beats':
+  $beats_manage => ['metricbeat','auditbeat','heartbeat','packetbeat']
+}
+```
 
-Users need a complete list of your module's classes, types, defined types providers, facts, and functions, along with the parameters for each. You can provide this list either via Puppet Strings code comments or as a complete list in the README Reference section.
+## Usage
 
-* If you are using Puppet Strings code comments, this Reference section should include Strings information so that your users know how to access your documentation.
+### Main class
+
+There is very few parameters you should need to customise.  The most useful would be `config_root` which allows you to control where this module expects the individual Beats configuration files to live:
+
+```puppet
+class { 'beats':
+  beats_manage => ['metricbeat','auditbeat','heartbeat','packetbeat'],
+  config_root  => '/opt/beats'
+}
+```
+
+#### Beats custom configuration
+
+This module recommends using Hiera for configuration data.  You can either specify your complete Beats configuration in Hiera or as a Puppet URL under `beats::<beat_name>::settings`.
+
+Configure in Hiera:
+
+```yaml
+beats::auditbeat::settings:
+  auditbeat.modules:
+  - module: auditd
+    audit_rules: |
+      -w /etc/group -p wa -k identity
+      -w /etc/passwd -p wa -k identity
+      -w /etc/gshadow -p wa -k identity
+      -w /etc/shadow -p wa -k identity
+      -w /etc/security/opasswd -p wa -k identity
+      -a always,exit -F arch=b64 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EACCES -F auid>=1000 -F auid!=4294967295 -k access
+      -a always,exit -F arch=b32 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EACCES -F auid>=1000 -F auid!=4294967295 -k access
+      -a always,exit -F arch=b64 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EPERM -F auid>=1000 -F auid!=4294967295 -k access
+      -a always,exit -F arch=b32 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EPERM -F auid>=1000 -F auid!=4294967295 -k access
+  - module: file_integrity
+    paths:
+    - /bin
+    - /usr/bin
+    - /sbin
+    - /usr/sbin
+    - /etc
+```
+
+Or pass a Puppet URL that will be used as the source of configuration:
+
+```yaml
+beats::auditbeat::settings: 'puppet:///somefileshare/auditbeat.yml'
+```
+
+### Beats specific usage
+
+#### Metricbeat modules
+
+This class can handle enabling/disabling Metricbeat modules for you.
+
+To enable/disable a list of modules, ensure you manage Metricbeat with this class:
+
+```puppet
+class { 'beats':
+  beats_manage => ['metricbeat'],
+}
+```
+
+And then configure the modules you want to enable/disable in Hiera:
+
+```yaml
+beats::metricbeat::modules:
+  'present':
+    - docker
+    - system
+  'absent':
+    - kafka
+```
+
+If you need to define custom settings for a particular module, add those in Hiera under `beats::metricbeat::modules::settings`. For example:
+
+```yaml
+beats::metricbeat::modules::settings:
+  - module: docker
+    metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
+    hosts: ["unix:///var/run/docker.sock"]
+    period: 10s
+  - module: redis
+    period: 10s
+```
 
-* If you are not using Puppet Strings, include a list of all of your classes, defined types, and so on, along with their parameters. Each element in this listing should include:
+## Reference
 
-  * The data type, if applicable.
-  * A description of what the element does.
-  * Valid values, if the data type doesn't make it obvious.
-  * Default value, if any.
+TBD
 
 ## Limitations
 
-This is where you list OS compatibility, version compatibility, etc. If there are Known Issues, you might want to include them under their own heading here.
+TBD
 
 ## Development
 
-Since your module is awesome, other users will want to play with it. Let them know what the ground rules for contributing are.
-
-## Release Notes/Contributors/Etc. **Optional**
-
-If you aren't using changelog, put your release notes here (though you should consider using changelog). You can also add any additional sections you feel are necessary or important to include here. Please use the `## ` header. 
+Please see the [CONTRIBUTING.md](CONTRIBUTING.md) file for instructions regarding development environments and testing.

data/common.yaml

@@ -0,0 +1,8 @@
+beats::beats_manage: ['auditbeat','heartbeat','metricbeat','packetbeat']
+beats::package_ensure: present
+beats::package_manage: true
+beats::service_enable: true
+beats::service_ensure: running
+beats::service_manage: true
+beats::service_provider: ~
+beats::config_root: /etc

data/windows-family.yaml

@@ -0,0 +1 @@
+beats::config_root: C:\Program Files

functions/metricbeat/get_module_settings.pp

@@ -0,0 +1,18 @@
+# beats::metricbeat::get_module_settings
+#
+# @api private
+# This function fetches all of a specific Metricbeat module settings from
+# the beats::metricbeat::modules::settings key in Hiera.
+# Avoid modifying private classes.
+#
+# @param module
+#   A Metricbeat module to fetch settings for.
+#
+function beats::metricbeat::get_module_settings(String $module) >> Array[Hash] {
+  $module_settings = lookup("beats::metricbeat::modules::settings", Array[Hash], 'deep', undef)
+  $settings = $module_settings.filter |$v| {
+    $value = $v
+    $value =~ Hash and $value[module] == $module
+  }
+  $settings
+}

hiera.yaml

@@ -0,0 +1,22 @@
+---
+version: 5
+
+defaults:
+  datadir: 'data'
+  data_hash: 'yaml_data'
+
+hierarchy:
+  - name: 'Full Version'
+    path: '%{facts.os.name}-%{facts.os.release.full}.yaml'
+
+  - name: 'Major Version'
+    path: '%{facts.os.name}-%{facts.os.release.major}.yaml'
+
+  - name: 'Distribution Name'
+    path: '%{facts.os.name}.yaml'
+
+  - name: 'Operating System Family'
+    path: '%{facts.os.family}-family.yaml'
+
+  - name: 'common'
+    path: 'common.yaml'

lib/puppet/provider/metricbeat_module/metricbeat.rb

@@ -0,0 +1,48 @@
+require 'yaml'
+
+Puppet::Type.type(:metricbeat_module).provide(:metricbeat) do
+
+  commands :metricbeat => 'metricbeat'
+
+  # Metricbeat module file.
+  #
+  # @return String
+  def module_file
+    File.join(
+    @resource[:module_dir],
+    "#{resource[:name]}.yml"
+    )
+  end
+
+  def exists?
+    if !File.exists?(module_file)
+      debug("Module file #{module_file} does not exist")
+      return false
+    else
+      debug "Module exists"
+      return true
+    end
+  end
+
+  def create
+    retry_count = 3
+    retry_times = 0
+    begin
+      info("Enabling Metricbeat module")
+      metricbeat(['modules','enable',resource[:name]])
+    rescue Puppet::ExecutionFailure => e
+      retry_times += 1
+      debug("Failed to enable module. Retrying... #{retry_times} of #{retry_count}")
+      sleep 2
+      retry if retry_times < retry_count
+      raise "Failed to enable module. Received error: #{e.inspect}"
+    end
+  end
+
+  # Remove this plugin from the host.
+  def destroy
+    info("Disabling Metricbeat module")
+    metricbeat(['modules','disable',@resource[:name]])
+  end
+
+end

lib/puppet/type/metricbeat_module.rb

@@ -0,0 +1,15 @@
+Puppet::Type.newtype(:metricbeat_module) do
+    @doc = 'Manage Metricbeat modules'
+
+    ensurable
+
+    newparam(:name, :namevar => true) do
+        desc 'An arbitrary name used as the identity of the resource.'
+    end
+
+    newparam(:module_dir) do
+        desc 'Path to the Metricbeat modules directory'
+        defaultto '/etc/metricbeat/modules.d'
+    end
+
+end

manifests/config.pp

@@ -0,0 +1,42 @@
+# @api private
+# This class handles the configuration files for beats. Avoid modifying private classes.
+class beats::config {
+  $beats::beats_manage.each |String $beat| {
+    case $facts['os']['family'] {
+      'windows': {
+        $beat_config = "${beats::config_root}\\${beat}\\${beat}.yml"
+      }
+      default: {
+        $beat_config = "${beats::config_root}/${beat}/${beat}.yml"
+      }
+    }
+    $settings = lookup("beats::${beat}::settings", Data, 'deep', undef)
+    if $settings {
+      case type($settings) {
+        String: {
+          file { "${beat}_config":
+            ensure => file,
+            path   => $beat_config,
+            owner  => 0,
+            group  => 0,
+            mode   => '0600',
+            source => $settings,
+          }
+        }
+        default: {
+          file { "${beat}_config":
+            ensure  => file,
+            path    => $beat_config,
+            owner   => 0,
+            group   => 0,
+            mode    => '0600',
+            content => epp('beats/beat.yml.epp', { settings => $settings }),
+          }
+        }
+      }
+    }
+    if $beat == 'metricbeat' and lookup('beats::metricbeat::modules', Data, 'deep', undef) {
+      require beats::metricbeat::config
+    }
+  }
+}

manifests/init.pp

@@ -0,0 +1,51 @@
+# beats
+#
+# @param beats_manage
+#   The names of Beats to manage with this module.
+#
+# @param package_ensure
+#   Whether to install Beats packages, and what version to install. Values: 'present', 'latest', or a specific version number.
+#   Default value: 'present'.
+#
+# @param package_manage
+#   Whether to manage the Beats packages. Default value: true.
+#
+# @param service_enable
+#   Whether to enable the Beats services at boot. Default value: true.
+#
+# @param service_ensure
+#   Whether the Beats services should be running. Default value: 'running'.
+#
+# @param service_manage
+#   Whether to manage the Beats services.  Default value: true.
+#
+# @param service_provider
+#   Which service provider to use for Beats. Default value: 'undef'.
+#
+# @param config_root
+#   The root directory under which individual beats config directories are found. Default value: '/etc'.
+#
+# @param [Boolean] manage_repo
+#   Enable repository management. Configure the official repositories.
+#
+class beats (
+  Array[String] $beats_manage,
+  String $package_ensure,
+  Boolean $package_manage,
+  Boolean $service_enable,
+  Enum['running', 'stopped'] $service_ensure,
+  Boolean $service_manage,
+  Optional[String] $service_provider,
+  String $config_root,
+  Boolean $manage_repo = true
+  ) {
+    contain beats::install
+    contain beats::config
+    contain beats::service
+    if ($manage_repo == true) {
+      include elastic_stack::repo
+    }
+    Class['::beats::install']
+    -> Class['::beats::config']
+    -> Class['::beats::service']
+}

manifests/install.pp

@@ -0,0 +1,9 @@
+# @api private
+# This class handles beats packages. Avoid modifying private classes.
+class beats::install {
+  if $beats::package_manage {
+    package { $beats::beats_manage:
+      ensure => $beats::package_ensure
+    }
+  }
+}