Titelkarte für Ansible Vault-Blogbeitrag

This is a follow up to our previous post on how to speed up mass templating in Ansible.

It is common practice to leverage the Ansible Vault’s encrypt_string feature to encrpyt single values in Ansible variable files. While particularly useful for keeping all the values of one configuration together, rather than splitting off the secrets and managing multiple files, the downside is that editing such files cannot be done with the standard ‘ansible-vault edit’ command, because Ansible Vault expects the whole file to be encrypted – and has no way of knowing which specific entries in a variable file to encrypt after an edit.

In this blog post we will look at a way to update a partially encrypted variable file utilizing the Ansible Python modules.

How Ansible Vault is used in this Scenario

</>

We assume that we have a very simple software using CSV to setup its users. The passwords are stored in cleartext in CSV, along with Username and Department:

Roland,demopwd,Helpdesk
Sandro,demopwd,Engineering
Torsten,demopwd,Engineering

To automate the creation of this file, we have developed a playbook which deploys a template and fills the values based on entries in a group_vars file. The passwords in this file are encrypted with Ansible Vault: group_vars/all/users.yml

users:
Roland:
  department: Helpdesk
  password: !vault |
    $ANSIBLE_VAULT;1.1;AES256
    38373664613433633764613131366330343133623138623636366238303538633464383735316139
    3334616234383636326533633366353233396538346138660a613634333935313630313933396163
    37653937303733386232623863643439333666376538383632383134633830383534376536626635
    3131613131396662610a396165636331363835646466643135323638353765323865336432333630
    6133
Sandro:
  department: Engineering
  password: !vault |
    $ANSIBLE_VAULT;1.1;AES256
    61316132333737666162333661393866653537373036313731646334303132306666653131363038
    6137656431663163623539646361333332666236643332340a393235393866303664663535323962
    38636234326335333736363738326366353635393233356636363737373565326333346239323433
    6330656638326632620a633465393938373464306132386130656536333335306432623332393632
    3635
Torsten:
  department: Engineering
  password: !vault |
    $ANSIBLE_VAULT;1.1;AES256
    32313237633563313631623663373261636439313732323061386361623432346462356635383232
    3635303265363361323939646536623831316233383639620a376666656264336237636335373066
    36353230353732616461626161393438363833663465346462326266353965386161316230333439
    3834646166323738360a323961303034356563343663333533366662363530623536336235336261
    3337

Playbook:

- hosts: localhost
tasks:
  - template:
      src: templates/password_file.j2
      dest: /tmp/password_file

Template:

{% for user, details in users.items() %}
{{ user }},{{ details.password }},{{ details.department }}
{% endfor %}

Let us assume we receive an update for our application, the CSV gets a new field to specify in which office the user is located.

Naturally, with just three users it would be fastest to just add the new fields manually, but since there might also be 200 users, we want to automate this step. We need to read in the group_vars file, modify it to our requirements and write it back to the folder.

Updating the partially Ansible Vault encrypted Yaml

</>

Digression: YaML tags

When we look at our Vault encrypted string, we notice it begins with ‘!vault’. The exclamation mark denotes a tag. A tag in YaML instructs the interpreter how to handle the data. With the standard tag ‘!!str’ we declare the data to be of type string, and it gets loaded as such.

As an example, we load a Yaml with the content ‘amount: 14’ in python, with and without the ‘!!str’ tag

>>> text = 'amount: 14'
>>> yaml.safe_load(text)
{'amount': 14}
>>> text = 'amount: !!str 14'
>>> yaml.safe_load(text)
{'amount': '14'}

As we can see from the single quotes, without the tag, 14 was interpreted as a number; with the tag as a string.

Tags allow the storage i.e. declaration of various types, even whole classes within YaML. And of course Ansible Vault encrypted content as well.

Handling the !vault tag generated by Ansible Vault

We now know that the !vault tag denotes Ansible Vault encrpyted content. To enable our yaml loader to recognize this tag, we need to add a constructor, to write it back we need a representer for the dumper:

yaml.SafeLoader.add_constructor(u'!vault', vault_string_constructor)
yaml.Dumper.add_representer(VaultString, vault_string_representer)

For the constructor the tag is the first argument, a function that constructs the object is the second.

The representer takes the type and representer function as an argument.

To allow the dumper to discern a vault string from a regular string, we create a new class VaultString. The constructor function then returns an instance of this class:

class VaultString:
  def __init__(self, string):
    self.string = string
  def __repr__(self):
    return self.string

def vault_string_constructor(loader, node):
    return VaultString(loader.construct_scalar(node))

Note the style argument in the representer function to keep the pipe-multiline notation:

def vault_string_representer(dumper, data):
return dumper.represent_scalar(u'!vault', u'%s' % data, style='|')

Bonus-Tip: Using the Ansible Vault VaultLib

</>

To facilitate encrypting and decrypting we need to import a few things from our Ansible installation:

from ansible.constants import DEFAULT_VAULT_ID_MATCH
from ansible.parsing.vault import VaultLib, VaultSecret

After we read our vault password from the password file, VaultLib can be initialized in a single line:

with open(os.environ.get('ANSIBLE_VAULT_PASSWORD_FILE')) as password_file:
    vault_pass = password_file.read().strip().encode('utf8')

vault = VaultLib([(DEFAULT_VAULT_ID_MATCH, VaultSecret(vault_pass))])

Encrypted values can now be decrypted with vault.decrypt(), and encrypted with vault.encrypt(). If we need to update the passwords or view them in our script, we could therefore extend our VaultString class as follows:

class VaultString:
def __init__(self, string):
  self.string = string
def __repr__(self):
  return self.string
def decrypt(self):
  return vault.decrypt(self.string)
def update(self, new_value):
  self.string = vault.encrypt(new_value).decode('utf8')
  return 'value updated'

Putting it together

</>

We have a way to load the vault strings; if necessary decrypt, modify, encrypt and a way to write them back into yaml format. We now only need to put it all together and add our logic adding the new value.

We keep it simple, the engineering department is on the second floor Office, helpdesk on the third:

"""Read group_vars/all/users.yml and add a new attribute"""
import os
import yaml
from ansible.constants import DEFAULT_VAULT_ID_MATCH
from ansible.parsing.vault import VaultLib, VaultSecret

# Init Vault
with open(os.environ.get('ANSIBLE_VAULT_PASSWORD_FILE')) as password_file:
    vault_pass = password_file.read().strip().encode('utf8')
vault = VaultLib([(DEFAULT_VAULT_ID_MATCH, VaultSecret(vault_pass))])

class VaultString:
  def __init__(self, string):
    self.string = string
  def __repr__(self):
    return self.string
  def decrypt(self):
    return vault.decrypt(self.string)
  def update(self, new_value):
    self.string = vault.encrypt(new_value).decode('utf8')
    return 'value updated'

def vault_string_constructor(loader, node):
    return VaultString(loader.construct_scalar(node))

def vault_string_representer(dumper, data):
    return dumper.represent_scalar(u'!vault', str(data), style='|')

yaml.SafeLoader.add_constructor(u'!vault', vault_string_constructor)
yaml.Dumper.add_representer(VaultString, vault_string_representer)

# Read group_vars
with open('group_vars/all/users.yml') as users_file:
     file_content = yaml.safe_load(users_file)

for user in file_content['users'].values():
  # Show the password
  # print(user['password'].decrypt())
  # Update the password
  # user['password'].update('newpwd')
  if user['department'] == 'Engineering':
    user['office'] = '2nd Floor'
  if user['department'] == 'Helpdesk':
    user['office'] = '3rd Floor'

with open('group_vars/all/users.yml','w') as users_file:
  users_file.write(yaml.dump(file_content))

Result:

users:
Roland:
  department: Helpdesk
  office: 3rd Floor
  password: !vault |
    $ANSIBLE_VAULT;1.1;AES256
    38373664613433633764613131366330343133623138623636366238303538633464383735316139
    3334616234383636326533633366353233396538346138660a613634333935313630313933396163
    37653937303733386232623863643439333666376538383632383134633830383534376536626635
    3131613131396662610a396165636331363835646466643135323638353765323865336432333630
    6133
Sandro:
  department: Engineering
  office: 2nd Floor
  password: !vault |
    $ANSIBLE_VAULT;1.1;AES256
    61316132333737666162333661393866653537373036313731646334303132306666653131363038
    6137656431663163623539646361333332666236643332340a393235393866303664663535323962
    38636234326335333736363738326366353635393233356636363737373565326333346239323433
    6330656638326632620a633465393938373464306132386130656536333335306432623332393632
    3635
Torsten:
  department: Engineering
  office: 2nd Floor
  password: !vault |
    $ANSIBLE_VAULT;1.1;AES256
    32313237633563313631623663373261636439313732323061386361623432346462356635383232
    3635303265363361323939646536623831316233383639620a376666656264336237636335373066
    36353230353732616461626161393438363833663465346462326266353965386161316230333439
    3834646166323738360a323961303034356563343663333533366662363530623536336235336261
    3337

Conclusion

</>

The trick of working with encrypted strings in variable files is to make the yaml loader aware on how to handle it. By using Ansibles Vault implementation in our own Scripts, we open up countless possibilities mutating these files.

Fragen?

Wir helfen gern, unverbindlich, kompetent und fast automatisch.

+41 43 543 50 40 info@newbit.ch