Rollout all your Prometheus exporters with Puppet!

# Rollout all your Prometheus exporters with Puppet!

## (Proudly powered by Vox Pupuli Puppet modules)

???

---

## $ whoami

* Tim 'bastelfreak' Meusel

* Puppet Contributor since 2012

* Merging stuff on [Vox Pupuli](https://voxpupuli.org/)  (Puppet Community) since 2015

* Vox Pupuli Project Management Committee member

???

---

# Let us go on a monitoring journey

---

* Assume you maintain a few thousand servers

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

---

* Assume you maintain a few thousand servers

* Assume you have at least one coworker that develops software

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

---

* Assume you maintain a few thousand servers

* Assume you have at least one coworker that develops software

* You have to maintain that software in production

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

---

* Assume you maintain a few thousand servers

* Assume you have at least one coworker that develops software

* You have to maintain that software in production

* All monitoring you have is really bad

* It has false positives, it isn't efficient, it's complex

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

---

* Assume you maintain a few thousand servers

* Assume you have at least one coworker that develops software

* You have to maintain that software in production

* All monitoring you have is really bad

* It has false positives, it isn't efficient, it's complex

* Or you have no monitoring at all
]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

---

# Chapter 1: Exploration

## Or: The Happy Phase
---

* You attend your most favourite config management conference

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

---

* You attend your most favourite config mangement conference

* Somebody recommends you [node_exporter](https://github.com/prometheus/node_exporter/#node-exporter)

* "Just download the binary, run it as root and it works" they said...

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

start prometheus-node-exporter
* curl http://localhost:9100/metrics --silent

explain HELP/TYPE

* curl http://localhost:9100/metrics --silent | grep -v '^#'
* curl http://localhost:9100/metrics --silent | grep -v '^#' -c

---

* Reads many metrics from your box, expose them via HTTP

* No HTTPS, no authentication

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

* Prometheus collectors https://github.com/prometheus/node_exporter/#enabled-by-default
---

* Reads many metrics from your box, expose them via HTTP

* No HTTPS, no authentication

* Has many collectors, that can be enabled/disabled

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

* Prometheus collectors https://github.com/prometheus/node_exporter/#enabled-by-default
---

* Reads many metrics from your box, expose them via HTTP

* No HTTPS, no authentication

* Has many collectors, that can be enabled/disabled

* Can read values from 3rd parties from a txt file

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

* Prometheus collectors https://github.com/prometheus/node_exporter/#enabled-by-default
---

* Reads many metrics from your box, expose them via HTTP

* No HTTPS, no authentication

* Has many collectors, that can be enabled/disabled

* Can read values from 3rd parties from a txt file

* Single binary, written in go

* Nice for dependency management, but please consider building packages

* Do not download random binaries from the internet...

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

* Prometheus collectors https://github.com/prometheus/node_exporter/#enabled-by-default

---

```sh
# HELP Total number of scrapes by HTTP status code.
# TYPE counter
promhttp_metric_handler_requests_total{code="200"} 5
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

```

* Each metric has a name and a value

* There should be a HELP

* There should be a type

* A metric can have labels (`code="XXX"` in this case)
]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

---

* Tell your developers to expose data as prometheus metrics
]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

* Most frameworks have builtin features to expose metrics
* Many of them already in a prometheus style

---

* Tell your developers to expose data as prometheus metrics

* Enrich your exporter with 3rd party data

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

* metadata, for example from cloud-init
* from your inventory system
* data collected by systemd timers/cronjobs

Let them write into a txt file that node_exporter can read

---

* Tell your developers to expose data as prometheus metrics

* Enrich your exporter with 3rd party data

* Many applications already provide prometheus-style metrics

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

* kubernetes for example

---

* Tell your developers to expose data as prometheus metrics

* Enrich your exporter with 3rd party data

* Many applications already provide prometheus-style metrics

* There are [hundreds of exporters](https://prometheus.io/docs/instrumenting/exporters/#exporters-and-integrations) available
]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

* most of them are single go binaries

---

# Chapter 2: Integration

## Or: Million ways to shoot yourself in the foot

---

With the [puppet/prometheus](https://forge.puppet.com/puppet/prometheus#puppet-prometheus) module:
```puppet
include prometheus::node_exporter
```
]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

---

* Prometheus database scrapes all exporters

* Everybody can make backups of your metrics \o/

* (Because they are available to the whole internet)
]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

---

* Prometheus database needs to know the targets

* Register each exporter in Consul as a service

* Feed services into Prometheus

* Use [KyleAnderson/consul](https://forge.puppet.com/KyleAnderson/consul#puppet-consul) Puppet module to deploy consul

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

---

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

---

Take a look at [./examples/node_exporter_consul.pp](https://github.com/bastelfreak/prometheusdemo/blob/production/examples/node_exporter_consul.pp)
]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

---

* Deploy nginx in front of node_exporter

* Require a valid client certificate from prometheus database

* [Vox Pupuli](https://voxpupuli.org) provides [puppet/nginx](https://forge.puppet.com/puppet/nginx#nginx-module-for-puppet) \o/
]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

---

Take a look at [./examples/node_exporter_profile.pp](https://github.com/bastelfreak/prometheusdemo/blob/production/examples/node_exporter_profile.pp)

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

---

* No authentication by default

* Can also use TLS client certificates / a secret key

* Puppet certificates can be reused

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

* Consul has a gossip protocol. it's used to manage membership and broadcast messages to the cluster.
* gossip is encrypted with the key
* agent to server communication will be handled with TLS. Client certificates are validated.

---

.left-column[
## Starting Point
## Exploration
## Integration
### Discovery
### Authentication
### Firewalling
]

* Only Prometheus database should be able to access exporters

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

* Obvious.

---

.left-column[
## Starting Point
## Exploration
## Integration
### Discovery
### Authentication
### Firewalling
]

* Only Prometheus database should be able to access exporters

* Firewalling should be automated

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

* manual work costs time
* manual work doesn't make any fun
* manual work leads to errors

---

.left-column[
## Starting Point
## Exploration
## Integration
### Discovery
### Authentication
### Firewalling
]

* Only Prometheus database should be able to access exporters

* Firewalling should be automated

* IP addresses might change often
]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

* Especially in a cloud environment, you don't have permanent ip addresses
* boxes are often reprovisioned
* or you add more boxes

---

.left-column[
## Starting Point
## Exploration
## Integration
### Discovery
### Authentication
### Firewalling
]

.right-column[
```puppet
class profiles::prometheus {
  @@ferm::rule{"prometheus2$trusted['certname']":
    ensure => 'present',
    chain  => 'INPUT',
    saddr  => facts['networking']['ip6'],
    dport  => 9100,
    proto  => 'tcp',
    policy => 'ACCEPT',
    tag    => 'prometheus2node_exporter',
  }
}

class profiles::node_exporter {
  Ferm::Rule <<| tag == 'prometheus2node_exporter' |>>
}
```

* Exported resources to the rescue
* [Vox Pupuli](https://voxpupuli.org) provides [puppet/ferm](https://forge.puppet.com/puppet/ferm#puppet-ferm)

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

---

.left-column[
## Starting Point
## Exploration
## Integration
### Discovery
### Authentication
### Firewalling
]

Consul:

* Consul creates a meshed network

* iptables doesn't scale with thousands of entries

* filtering pakets and changing rules will be delayed

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

* Every consul node needs to talk to every other consul instance

* Consul is currently plaintext

---

.left-column[
## Starting Point
## Exploration
## Integration
### Discovery
### Authentication
### Firewalling
]

ipsets and separate chains to the rescue!

* handle consul traffic in a separate iptables chain

* ip sets are a hashmap of CIDR ranges

* In memory, scale very good

* iptables can check against this hashmap

* You cannot mix IPv4/IPv6 in one hashmap

* Updating the ip set is independet from iptables rules

* [Vox Pupuli](https://voxpupuli.org) provides [puppet/ipset](https://forge.puppet.com/puppet/ferm#puppet-ferm)
]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

---

.left-column[
## Starting Point
## Exploration
## Integration
### Discovery
### Authentication
### Firewalling
]

Take a look at [./examples/ipsets.pp](https://github.com/bastelfreak/prometheusdemo/blob/production/examples/ipsets.pp)
]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

---

# Chapter 3: Conclusion

## Or: \o/ ?

---

* Rolling out an exporter with Puppet is easy

* 1 line of code! (for a dev environment)

* Required config management code for a production setup is still reasonable
]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

---

* Rolling out an exporter with Puppet is easy

* [puppet/prometheus](https://forge.puppet.com/puppet/prometheus#puppet-prometheus) supports 20 famous exporters

* You can easily add support for another one

* Ping the [Vox Pupuli](https://voxpupuli.org/) people and ask for help
]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

---

* Rolling out an exporter with Puppet is easy

* [puppet/prometheus](https://forge.puppet.com/puppet/prometheus#puppet-prometheus) supports 20 famous exporters

* Consul as service discovery

\+ Probably the solution that scales best

\+ Since you now have it, you can use it for even more stuff

\- More configuration effort than an actual exporter

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???

---

* Rolling out an exporter with Puppet is easy

* [puppet/prometheus](https://forge.puppet.com/puppet/prometheus#puppet-prometheus) supports 20 famous exporters

* Consul as service discovery

* [github.com/bastelfreak/prometheusdemo](https://github.com/bastelfreak/prometheusdemo) repo with slides + all examples + Vagrantfile to show your coworkers

* Contact: [tim@bastelfreak.de](mailto:tim@bastelfreak.de) or bastelfreak on freenode

* [Collection of related talks](https://github.com/bastelfreak/talks#collection-of-talks-proposals-and-related-stuff)

### Thanks for your attention!

]

.footnote[[@bastelsblog](https://twitter.com/bastelsblog) for [@voxpupuliorg](https://twitter.com/voxpupuliorg)]

???