Puppet hacks you didn't know you were looking for

# Puppet Hacks

???

---
class: center, middle, inversedoubleheading

# Puppet Hacks

## you didn't know you were looking for

???

---
## $ whoami
<img src="bastelfreak2.jpg" alt="drawing" style="float: right;width: 400px;"/>

* Tim 'bastelfreak' Meusel

* Puppet Contributor since 2012

* Merging stuff on [Vox Pupuli](https://voxpupuli.org/) (Puppet Community) since 2015

* Vox Pupuli Project Management Committee member

* Senior IT Automation Consultant at [betadots](https://betadots.de/)

<a href='https://betadots.de' alt='company website betadots'><img src="logo-1181x1181.png" alt="drawing" style="position: fixed; bottom: 12px; left: 20px; width: 100px;"/></a>
???

* who has seen this picture before because I reviewed/merged your pull request?

---
class: center, middle, inverse

# Resource Abstraction Layer

???

---
.left-column[
## RAL
]

]

<a href='https://betadots.de' alt='company website betadots'><img src="logo-1181x1181.png" alt="drawing" style="position: fixed; bottom: 12px; left: 20px; width: 100px;"/></a>
.footnote[[bastelfreak](https://github.com/bastelfreak) for [Vox Pupuli](https://voxpupuli.org)]

???

Who knows the RAL?

---
.left-column[
## RAL
]

* Abstracts package manager away into Puppet resources

]

---

* Abstracts package manager away into Puppet resources

* Implemented via types and providers

```puppet
package { 'htop':
  ensure => 'installed',
}
```
]

---

* Abstracts package manager away into Puppet resources

* Implemented via types and providers

```puppet
package { 'htop':
  ensure => 'installed',
}
```

* List available types on CLI

* `puppet resource --types`
]

???

* Switch to terminal and show? Works as root/normal user

---

* `puppet resource package htop`

```puppet
package { 'htop':
  ensure   => '3.4.1-1',
  provider => 'pacman',
}
```
]

---

* `puppet resource package htop`

```puppet
package { 'htop':
  ensure   => '3.4.1-1',
  provider => 'pacman',
}
```

* Can output all instances of a resource type

* `puppet resource package`

```puppet
package { 'yamllint':
  ensure   => '1.33.0-1',
  provider => 'pacman',
}
package { 'yard':
  ensure   => ['0.9.26'],
  provider => 'gem',
}
```
]

???

* Puppet needs to know the current state of a resource before code is applied
* required to detect if changes are needed
* prefetch needs to be implemented in the provider
* check yard ensure, array

---

* `puppet resource package htop ensure=absent`

```puppet
Notice: /Package[htop]/ensure: removed
package { 'htop':
  ensure   => 'absent',
  provider => 'pacman',
}
```
]

???

* You can also pass the other parameters
---

* `puppet resource package htop ensure=absent`

```puppet
Notice: /Package[htop]/ensure: removed
package { 'htop':
  ensure   => 'absent',
  provider => 'pacman',
}
```

* Can modify a resource instance with debug output!

* `puppet resource package htop ensure=absent --debug`
]

???

---

* `puppet resource package htop ensure=absent`

```puppet
Notice: /Package[htop]/ensure: removed
package { 'htop':
  ensure   => 'absent',
  provider => 'pacman',
}

```
* Can modify a resource instance with debug output!

* `puppet resource package htop ensure=absent --debug`

* Can simulate a change!

* `puppet resource package htop ensure=absent --debug --noop`
]

???

* Puppet CLI args like debug and noop can be passed to every `puppet resource package`
---

.left-column[
## RAL
### Find Resources
### Modify Resources
### Purge Resources
]
.right-column[
* [puppet.com/docs/puppet/8/type#resources](https://www.puppet.com/docs/puppet/8/type#resources)

![img](resources.png)
]

???

* Who knows the resources resource type?

* Who used it?

* Who likes it?

* I took the screenshots some time ago in case perforce fucks up their docs. well they did now
---

.left-column[
## RAL
### Find Resources
### Modify Resources
### Purge Resources
]
.right-column[
```puppet
resources { 'User':
  purge => true,
}

resources { 'Package':
  purge => true,
}

```
]

???

* purge all users not managed by puppet
* show resources.pp
* Resource title has the resource capitalized

---

.left-column[
## RAL
### Find Resources
### Modify Resources
### Purge Resources
]
.right-column[
```puppet
resources { 'User':
  purge => true,
}

resources { 'Package':
  purge => true,
}

resources { 'Service':
  purge => true,
  noop  => false,
}
```
]

???

* noop is nice because it will inform you about unmanaged/unexpected resources
* not so useful for packages or users
* but what about database users
* backends in your F5?
---

.left-column[
## RAL
### Find Resources
### Modify Resources
### Purge Resources
]
.right-column[
* [forge.puppet.com/crayfishx/purge](https://forge.puppet.com/modules/crayfishx/purge/readme)

![img](purge.png)

]

---

.left-column[
## RAL
### Find Resources
### Modify Resources
### Purge Resources
]
.right-column[
![img](purge2.png)

]

---

![img](tidy.png)

```puppet
tidy { '/tmp':
  age     => '1w',
  recurse => 1,
  matches => [ '[0-9]pub*.tmp', '*.temp', 'tmpfile?' ],
}
```
]

???

* uses the RAL to collect all files
* applies the different matchers
* purges all matching files

* sucks, because it adds one file resource for each file to the catalog
---

* Purging files and directories

* [enterprisemodules.com/blog/2022/02/cleanup-temporary-files-in-puppet](https://www.enterprisemodules.com/blog/2022/02/cleanup-temporary-files-in-puppet/)

![img](cleanup.png)

]

???

* works on dirs and and files
* purges the files at the end of the catalog application, without any dependencies to file resources

* best solution is a systemd timer or systemd tmpfs configuration
---

* https://petersouter.xyz/the-puppet-resource-abstraction-layer-ral-explained-part-1/

* https://www.puppet.com/docs/puppet/7/man/resource
]

---
class: center, middle, inverse

# Puppet Faces

???

---

* https://puppetcommunity.slack.com/archives/C0W1X7ZAL/p1705520404613479?thread_ts=1705511900.536469&cid=C0W1X7ZAL

![img](2024-01-17_22-17.png)

]

???

* An awesome way to extend the Puppet CLI
* Allows you to access puppet as a lib

---

.right-column[
* Faces were "deprecated" but won't be removed: https://www.puppet.com/docs/puppet/5.5/deprecated_api.html

* Kelsey Hightower explaining faces in 2011: https://www.youtube.com/watch?v=WUlYEJ-fpfU

* Faces are used heavily within Puppet: https://github.com/puppetlabs/puppet/tree/main/lib/puppet/face
]

???

---

.right-column[
* Faces were "deprecated" but won't be removed: https://www.puppet.com/docs/puppet/5.5/deprecated_api.html

* Kelsey Hightower explaining faces in 2011: https://www.youtube.com/watch?v=WUlYEJ-fpfU

* Faces are used heavily within Puppet: https://github.com/puppetlabs/puppet/tree/main/lib/puppet/face

* "Puppet Applications" are the successor for Faces

* [github.com/puppetlabs/puppet/blob/main/lib/puppet/application.rb](https://github.com/puppetlabs/puppet/blob/49d74c94c848d4d4f4d022b58ff8135b2f46e7b0/lib/puppet/application.rb#L10)
]

???

---

* `puppet config print certname`

* `puppet config print certname --section`

* `puppet config set 'option' 'value' --section 'section'`

* Valid sections: main, user, agent, server/master
]

???

* config face is builtin
* can read and write from puppet.conf
* can disable defaults from the code that aren't in puppet.conf
* identifies path to puppet.conf on its own

---

.right-column[
* [github.com/voxpupuli/puppet-catalog_diff](https://github.com/voxpupuli/puppet-catalog_diff/tree/master?tab=readme-ov-file#overview)

* The tool can automatically compile the catalogs for both your new and older servers/environments. It can ask the master to use PuppetDB to compile the catalog for the last known environment with the last known facts. It can then validate against PuppetDB that the node is still active. This filtered list should contain only machines that have not been decommissioned in PuppetDB (important as compiling their catalogs would also reactive them and their exports otherwise).
]

???

* `catalog` face exists in core, `catalog diff` is custom, maintained by Vox Pupuli

---

.right-column[
* `puppet catalog diff puppet.example.com/production puppet.example.com/staging --filter_old_env`

* `puppet catalog diff /foo/old/node1.example.com.json /foo/new/node1.example.com.json`

* [voxpupuli.org/puppet-catalog-diff-viewer](https://voxpupuli.org/puppet-catalog-diff-viewer/)
]

???

* diff catalogs from PuppetDB or compile new ones
* diff between environments
* diff between different Puppet infrastructures
* Visualize reports
* deserves its own talk, Raphael did some in the past, linked in the README.md

---

---

* provide a list of exported resources in PuppetDB

* `watch 'puppet node exports'`

* [forge.puppet.com/zack/exports](https://forge.puppet.com/modules/zack/exports/readme#example-usage)

]

---

* provide a list of exported resources in PuppetDB

* `watch 'puppet node exports'`

* [forge.puppet.com/zack/exports](https://forge.puppet.com/modules/zack/exports/readme#example-usage)

* Lists all exported resources

* Can be filtered for resource types (for example File)
]

---

* provide a list of exported resources in PuppetDB

* `watch 'puppet node exports'`

* [forge.puppet.com/zack/exports](https://forge.puppet.com/modules/zack/exports/readme#example-usage)

* Lists all exported resources

* Can be filtered for resource types (for example File)

* Doesn't work properly with Puppet 8 :sadface:

![img](exports.png)
]

???

It uses a deprecated HTTP client
---

# Puppet CLI

???

---

* `puppet agent -t`

]

---

count:false

* `puppet agent -t`

* `puppet agent --test`
]

???

those are equal

---

count:false

* `puppet agent -t`

* `puppet agent --test`

* `puppet agent --verbose --no-daemonize --no-usecacheonfailure --detailed-exitcodes --no-splay --show_diff`
]

---

* `puppet agent -t`

* `puppet agent --test`

* `puppet agent --verbose --no-daemonize --no-usecacheonfailure --detailed-exitcodes --no-splay --show_diff`

* Who knows more CLI invocations?

]

???

---

* `puppet agent -t`

* `puppet agent --test`

* `puppet agent --verbose --no-daemonize --no-usecacheonfailure --detailed-exitcodes --no-splay --show_diff`

* Who knows more CLI invocations?

* Some Options are only available on the CLI: `puppet agent --help`

* Every config option can be set and overwritten on CLI: https://www.puppet.com/docs/puppet/8/configuration.html

* `puppet agent -t --no-noop`

]

???

* puppet.conf can be overwritten on CLI
* See --no-splay above to disable options

* Add noop=true to puppet.conf
* when you want to apply changes, run with --no-noop

---

* available in puppet faces as well

![img](trace.png)

]

???

* Remember our error from earlier? It had no line/file information
* now it has, exports.rb line 63

---

* `puppet agent -t --debug`

* Prints all HTTP calls to puppetserver

* Lists all resources and if they are already in the desired state
]

???

---

count:false

* `puppet agent -t --debug`

* Prints all HTTP calls to puppetserver

* Lists all resources and if they are already in the desired state

* prints all debug messages

* Facter: `Facter.debug('my debug text')`

* Puppet Types/Provider: `Puppet.debug('my other debug text')`
]

???

* Now we can extend our code with many debug messages
* but what if we need to dig deep
---

---

* Dumps every HTTP Payload and Headers from/to Puppetserver
]

---

* Dumps every HTTP Payload and Headers from/to Puppetserver

* As base64
]

---

* prints all resources and their evaluation time

![img](eval.png)
]

???

* Who knows that option?
* naming things is hard. evaltrace written together. http_debug uses an underscore
---

* All options can be combined

]

---

* All options can be combined

* You will need a very big screen!
]

???

* This is a lot of output

---

---

* `puppet facts show $fact` # show a single fact named fact
]

---

* `puppet facts show $fact` # show a single fact named fact

* `puppet facts show --timing` # timing per fact resolution
]

---

---
class: center, middle, inverse

# Design Pattern and Hacks

???

---

* IEEE-802.1AB standard

* exchange information between direct layer 2 peers

* Common protocol for Routers, Switches *and servers*

]

---

* IEEE-802.1AB standard

* exchange information between direct layer 2 peers

* Common protocol for Routers, Switches *and servers*

* Linux: [github.com/lldpd/lldpd](https://github.com/lldpd/lldpd)

* Puppet Module: [forge.puppet.com/puppet/lldpd](https://forge.puppet.com/modules/puppet/lldpd/readme)
]

---

???

* VMs/Hypervisor run LLDP
* VM knows on which HV it is running
* Hypervisor knows to wich switches it's connected
* Switches know which hypervisors and routers are connected
* All those information can be stored to PuppetDB
* the puppet module has a fact that exports the data
* you can query PuppetDB to find out which devices are connected to each other and plot them live!
* I would love to throw this into a graph database at some point
---

.right-column[
* [github.com/voxpupuli/puppet-puppet_ca_utils](https://github.com/voxpupuli/puppet-puppet_ca_utils)

* Can crosssign Puppet CAs
]

???

* Helpful if you've a new and old Infra and want to migrate agents
* You can configure if Agents should be able to talk to both CAs or Agent from A only to A and B but B not do A

---

.right-column[
* [github.com/voxpupuli/puppet-puppet_ca_utils](https://github.com/voxpupuli/puppet-puppet_ca_utils)

* Can crosssign Puppet CAs

* [github.com/voxpupuli/puppet-puppet_certificate](https://github.com/voxpupuli/puppet-puppet_certificate)

* Allow Agents to renew their own certificate

* Update CSR/SAN attributes

```puppet
file { '/etc/puppetlabs/puppet/csr_attributes.yaml':
  ensure  => file,
  owner   => 'root',
  group   => 'root',
  mode    => '0440',
  content => epp('example/csr_attributes.yaml.epp'),
}

~> puppet_certificate { $certname:
  ensure      => present,
  waitforcert => 60,
  onrefresh   => regenerate,
}
```
]

???

---

.right-column[
* [github.com/voxpupuli/puppet-puppet_ca_utils](https://github.com/voxpupuli/puppet-puppet_ca_utils)

* Can crosssign Puppet CAs

* [github.com/voxpupuli/puppet-puppet_certificate](https://github.com/voxpupuli/puppet-puppet_certificate)

* Allow Agents to renew their own certificate

* Update CSR/SAN attributes

```puppet
file { '/etc/puppetlabs/puppet/csr_attributes.yaml':
  ensure  => file,
  owner   => 'root',
  group   => 'root',
  mode    => '0440',
  content => epp('example/csr_attributes.yaml.epp'),
}

~> puppet_certificate { $certname:
  ensure      => present,
  waitforcert => 60,
  onrefresh   => regenerate,
}
```

* Puppet 8.2 supports auto cert renewal: [puppet.com/docs/puppet/8/release_notes](https://www.puppet.com/docs/puppet/8/release_notes_puppet#release_notes_puppet_x-8-2-0)
]

???

---

* Use personal certificates to authenticate to local services

* `puppet ssl bootstrap --certname "$(hostname -f)-${USER}"`

![img](ca.png)
]

???

* Nginx for example can validate client TLS certificates

* admins can use that for internal sites
---

.right-column[
* [github.com/m0dular/ca_extend](https://github.com/m0dular/ca_extend?tab=readme-ov-file#ca_extend)

* Extend a Open Source CA
]

???

---

.right-column[
* [github.com/m0dular/ca_extend](https://github.com/m0dular/ca_extend?tab=readme-ov-file#ca_extend)

* Extend a Open Source CA

* [github.com/puppetlabs/ca_extend](https://github.com/puppetlabs/ca_extend?tab=readme-ov-file#ca_extend)

* Extend a PE CA
]

???

---

.right-column[
* [github.com/m0dular/ca_extend](https://github.com/m0dular/ca_extend?tab=readme-ov-file#ca_extend)

* Extend a Open Source CA

* [github.com/puppetlabs/ca_extend](https://github.com/puppetlabs/ca_extend?tab=readme-ov-file#ca_extend)

* Extend a PE CA

* [My Shell script to extend a CA](https://gist.github.com/bastelfreak/034fca2e7719351fe42c28e7844dace5)
]

???
---

* Use PuppetDB to dynamically build your clusters

```puppet
# Get FQDNs for all cluster peers
$query = "inventory[certname] {
  trusted.extensions.pp_role = 'consul' and trusted.extensions.pp_region = 'eu-01'
}"
$nodes = puppetdb_query($query).map |$value| { $value["certname"] }
```

]

---

* Use PuppetDB to dynamically build your clusters

```puppet
@@sshkey{$trusted['certname']:
 host_aliases => [$facts['ipaddress6'], $facts['ipaddress']],
 type => "ssh-${type}",
 key => $facts['ssh'][$type]['key'],
 # defaults to /etc/ssh/ so all users can use it
 #target => '/root/.ssh/known_hosts',
 tag => $trusted['extensions']['pp_role'],
}
## import host key
Sshkey <<| tag == $trusted['extensions']['pp_role'] and title != $trusted['certname'] |>>
```
]

???

* Often systems want a list of all backend systems, consul for example

* Works of course for IPs and ssh_authorized_keys and others as well
---
.left-column[
## Hacks & Patterns
### LLDP
### Puppet CA
### Clusterfoo
### Updating PuppetDB
]

* https://www.puppet.com/docs/puppet/7/configuration#resubmit-facts

* `puppet config set resubmit_facts true`

* `puppet agent -t --resubmit_facts`

* Evaluate all facts *after* catalog application and send them to PuppetDB
]

???

* by default, facts are only gathered and stored before the catalot is applied

---
.left-column[
## Hacks & Patterns
### LLDP
### Puppet CA
### Clusterfoo
### Updating PuppetDB
### puppetmoduleinfo
]

* [www.puppetmodule.info/modules](https://www.puppetmodule.info/modules)

* Similar to http://rubydoc.info/, puppetmodule.info renders puppet modules and puppet-strings annotations

* static site with module documentation

]

???

---

.left-column[
## Hacks & Patterns
### LLDP
### Puppet CA
### Clusterfoo
### Updating PuppetDB
### puppetmoduleinfo
### healthchecks
]

* [forge.puppet.com/puppet/healthcheck](https://forge.puppet.com/modules/puppet/healthcheck)

* Types that block catalog application until a TCP or HTTP(S) connection is successful

```puppet
tcp_conn_validator { 'foo-machine ssh service' :
  host => '192.168.0.42',
  port   => 22,
}
```

```puppet
http_conn_validator { 'foo-machine home' :
  host    => '127.0.0.1',
  port    => 8081,
  use_ssl => true,
}
```
]

???

---