Managing Secrets with Ansible Vault – The Missing Guide (Part 1 of 2)

(This post is part 1/2 in a series. For part 2 see: Managing Secrets with Ansible Vault – The Missing Guide (Part 2 of 2))

Background and Introduction to Ansible Vault

Once you’ve started using Ansible to codify the configuration of your infrastructure, you will undoubtedly run into a situation where you need to manage some of your infrastructure’s “secrets”. Examples of such secrets include SSH private keys, SSL certificates, or passwords. How do you codify and automate the distribution of these secrets? By checking these secrets into a source control system or posting for review in a code review tool in plain-text, you’d be instantly making them visible to a large number of people within your organization.

Luckily Ansible has created a tool to address this: Ansible Vault. The documentation for Ansible Vault describes its easy to use interface for encrypting, decrypting, and re-keying your secrets for storing in source control. Unfortunately the documentation provides little information on best practices for how to use Ansible Vault to deploy those secrets via a playbook, how to prevent the contents of those secrets from being echoed in plain-text to STDOUT when run with “–verbose” mode (ouch!), and how to test your playbooks when they contain such encrypted secrets, and how to integrate this into Jenkins.

Having recently spent time writing an Ansible role for deploying an OpenVPN server and having had to figure out the answer to a lot of these issues, I’m now happy to present “The Missing Guide to Managing Secrets with Ansible Vault”.

Storing and Deploying Secret Files

The first mental-hurdle to overcome around deploying secret files (ex. SSH private keys) with Ansible Vault is that that one must use a totally different mechanism for deploying files than with the traditional Ansible copy mechanism. Ex: Typically one would check in non-secret files into the “files/” directory of their Ansible role, and drop those files into place on the remote host with Ansible’s “copy” module using the “src” and “dest” parameters. Easy as pie.

Things work quite differently for encrypted secret files however, as mentioned in this StackOverflow post. Instead of checking in an encrypted version of the file to the “files/” subdirectory, one must place the contents of the file into an Ansible variable and deploy that file using the “contents” arg of the copy module. Here’s a working example:

# Unencrypted version of “vars/vpn-secrets.yml”
---
vpn_secret_files:
  /etc/openvpn/easy-rsa/keys/ec2-openvpn.key:
    owner: root
    group: root
    mode: "u=r,go="
    content: |
      -----BEGIN PRIVATE KEY-----
      MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQD5koXgI24E360f
      nhxCfOPVORzFW1CN7u/zOQdvKoIStogF0UQifDCnY/POEjoBmzBrg/UyAmsqLIli
      xMtRIuvEhwaGEUQPoZNCaRW+1XtJ3kDvr9MVTlJTcNGOlGe/E+HyAKBq5vinxzzM
      9ba8M9Nc1PQ93B1OTUY1QGHVYRvSFYDJ5Fnz23xKeNsnY3hmRkV7CDZXSdy9nbmy
      1X9uz7z5bG7PKUVD3JZjI75CHAEDJKtscBv9ez/z16YTxwahIL3CXfqBq8peyAZ0
      n4Mzj4Lt8Cwaw2Kw3w3gMhbhf4fy284+hYqHe9uqYJC6dJJSKDIXqoLSD+e8aN+v
      BAEQcAWXAgMBAAECggEAbmHJ6HqDHJC5h3Rs11NZiWL7QKbEmCIH6rFcgmRwp0oo
      GzqVQhNfiYmBubECCtfSsJrqhbXgJAUStqaHrlkdogx+bCmSyr8R3JuRzJerMd6l
      Jd3EJHZBnzoU1VT6Fd77Xge868tASySp1ZUPv2nEoBhn9jw2kf1HgiH5o2CR53ZP
      pnL72Ng7MHpKuyoAZ9DtUU7yGG4RTCN2JuPGD6IwKoXBs1b7tqsMncz86u6Iibwk
      Np4j3vPmSLfQxvBP85T0xzSURlnP+bFCaJDPfXYIgDLROkrFAgJ2ADCm4gwfk93i
      Z/wnk8tFjnxUy2V5UbtWqqkVHmvdHHCc/6bZfcNOsQKBgQD/v94YX3vhgZRiz1kZ
      c0v2lxFZqNgMPC7EADmO34nFq7KtmVXYQfpoiooGDfQXTqfVGQsyTcpg5HLZvlyb
      qm9oaXpZY4yP/SLF6Pc00/iDTleSxGROyqhsaBotXpqSSC3rv92D9Zas/Xdz3lHD
      NSY9EVsiFId7O4OkvLuZVDvZQwKBgQD50Rs873/yUdyCwKx9/GF4yWVRg7//FTyQ
      Cj1KCBK5tDqOc+hiIS1GF0HRkcvIot71owTe+PG9OouXlUuxWrtc+fzgGSPaYjMp
      Ub69EcSNtUsK8MUS+VADbR5VDzS27OM1g+pJO7BbHpPWuEI1cjYmW/+3cCzFYnIV
      5z6OctbjHQKBgEVQWP8+EbMijXbiP4G4T+Q7OUaVjkhynzIb5X2ldA+Q41JNdoiw
      CRAATDwr1/XhKXeF3BT8JFdyUvZUs4C1BpDD1ZcYdeYocx40b5tvv7DGsNFkTNNV
      9aO76yxUsYvn6Bo22/CBxR6Ja7CJlptTclOmuo5YBggOLzWcuTNrMvVFAoGASIoV
      lK4ewuhOVZFJBRRB4Wbpiq/tEk7CVTkD7vlFJrNUxYSWl9f2Y4HhVM83Ez1n7H+3
      rF8xIrdbTVrGresguLDGYvQp2wHkxTy9W/1Ky7M25ShgsU+/kh8fTaeqsOs8Vo/F
      ehpg7TSFzTWX1Bkj7COOr19dQLuDUSTin05tY2kCgYB35ZHVDMR6TlW0Kp/l7gAx
      FQx5hojllzHr3RRv8a4rBbhsdAJGBr5QHZbzVeuw1z6NlDc/4brer3y52FnnHbD3
      fkUrvh+g1xHeXF4Yekr5Mu2D7PoQoFRRai2hjPnIHRLmHI45EPri3USoHuNPl+qB
      l23chS70zQ9VDmqEs9gjLA==
      -----END PRIVATE KEY-----
  /etc/openvpn/easy-rsa/keys/ca.key:
    owner: root
    group: root
    mode: "u=r,go="
    content: |
      -----BEGIN PRIVATE KEY-----
      MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDPm22e2QTeTnLN
      PT//6kyB8tM/2kE6+LsFD3TFA4XvS3gwNZLybjXpPtncF4qLxjq3c4uSBp2tuAa2
      VvWUCAyQX4EcOuCFhh1AIUHX9O4F2JhLtNH366D6LmfGE7Lck85R6bzErYJ5OzBN
      /3WSGtWmLbQWhXTvNwG5re17Ds7DLQ6/XRXCg91lAbtGqYCvw9F6X8N3VNdcovqN
      Ud+tJ4XjmGfPD8ZgSk/iVKeLzz5fuNxON+ygdUJ9IQJGu7kvJOhWD1F3p3lzuS4E
      7zyR8r9QK6lGdk2/ifmY5f+tmI92fvVl2HD2DroEVp42hCYEpNogm8BKXHFHBA9N
      0mugGMzVAgMBAAECggEAAK5a2rWNjYkmWUQFLLrBC4AXb1Mw+ZeNTYPydx7+1n0h
      5M6YL9Fqvdwl7NHq83BwCuAHKjB5XfOHmhuI7LZmDCc0DjqnN+jruaUiSSoVidFf
      Foh+U9jjC08RqhWwdYbKm3wv0VlcXzdxfiADa7pIzyXBPH2tl4dPqyNF7yxqQzum
      F42D4IExbYYkGR7bP6RePrUiaO3iU/EwDL5Dey4+93K+EaxbdxIhMLclvnQ8I0tl
      tFGn4AbbOqPqzPxWZhWk2gT//jMTtJh6FxQLQkvDoEnta5UYQ2E38r33jK+Wasga
      lGZEyNOTMq1MMdPrCzXloJSnerCXC4vTFt62AOdIQQKBgQD3SvUeaXV7Xf67vL0t
      EdBG9YL0Zz2MxxoVAth44svMzQ4gR6/pkakEhMzR51I/Skl/wzCJFHY1Z4nq9DoA
      RY5APjO63uHdZEKKYZ1MTXmO6F+IkUY5MCBvyCtLsnkAcToyuyDuhV4NBfjydw6E
      L5S1H9NI7klvaPxq5I7KkzSeJQKBgQDW6sDBvi6ctV3w8GCTUjP5Ker1FuKYL7Yn
      HI6RIGnWB2hS8NbEe8ODgzsVOVnC6x+WCNBiu/GmF8wlue7PCH7rLEa8diiM+J9/
      QYXtezfLIhPqhPJZDj5IX7bIotkvUzv+ywvUfCtJ3aCAu8DMi09x1GRgU6go/4ZK
      SCmVmj588QKBgQCNhr2gCRTuZM37nbnayF4drjajL06/eddIfRdsn8epTxWtjbl0
      gCNt7Z7W5n9gr2A/GXN2kFpSmA4LhHiJXUVbKP4sDZDQRqf6UIFYgOJ30i+SlinN
      Yui9cJ6utNahVSvMiuH/AB7iby+ZfF+3cQ+3VR5zl8Q5WalUd7fs4bB0bQKBgBI1
      x+lipO5wS6pro7M35uF41Mi5jK+ac1OzDr1rQqx46jUE5R224uUUzH/K4Tkr1PxQ
      eN+0zw/kuk6EB6ERNjfVA5VaaaswMcuFkMSDiUGz/H4Fj8dN9qcJPSKY8dAZvF6l
      c7YoYz6aAcyGnBp4v12EwpCK5he7NvS6UpOzgxHxAoGBAOjiBQtwikKLzLYwg1gF
      QYh1TLvEJIRFYEFQveVUKxmSskN4W6VQrTrcqobYHM9tOSbSe+Ib/y/khpaEz0PE
      E5gxeUbxhTj0PVvOKJmyCKWDPL8o61MGVhX1nAJarfbdP1XM9fl4S3pZH14bIhOU
      FG0e4jNsDq6vdwytV9R/GyAv
      -----END PRIVATE KEY-----
...<snip>...

#######

# tasks/main.yml

#
# Use "no_log: true" to keep from echoing secrets to stdout.
# See: http://docs.ansible.com/faq.html#how-do-i-keep-secret-data-in-my-playbook
#
---
- name: VPN Server | Load VPN secret keys
  include_vars: "vpn-secrets.yml"
  no_log: true

- name: VPN Server | Copy secret files
  copy:
    dest="{{ item.key }}"
    content="{{ item.value.content }}"
    owner="{{ item.value.owner }}"
    group="{{ item.value.group }}"
    mode="{{ item.value.mode }}"
  with_dict: vpn_secret_files
  no_log: true
  notify:
    - restart openvpn

Here we see that “vars/vpn-secrets.yml” contains a multi-level hash where the first level is the destination file name (ex. “/etc/openvpn/easy-rsa/keys/ec2-openvpn.key”) and the secondary level for each filename contains respective attributes for the secret file (ex. owner, group, mode, and file contents). Those attributes are then passed straight through as args to the “copy” command which is iterating over the keys of that hash via the “with_dict: vpn_secret_files” argument.

Also note the use of “no_log: true” for both the “include_vars” and “copy” commands, above. This is necessary otherwise Ansible will echo the contents of your secret files to STDOUT when executing those commands.

So what does this look like when run with “ansible-playbook –ask-vault-pass …”?

       TASK: [dlpx.vpn-server | VPN Server | Load VPN secret keys] *******************
       ok: [localhost] => {"censored": "results hidden due to no_log parameter"}

       TASK: [dlpx.vpn-server | VPN Server | Copy secret files] **********************
       changed: [localhost] => {"censored": "results hidden due to no_log parameter", "changed": true}
       changed: [localhost] => {"censored": "results hidden due to no_log parameter", "changed": true}
       changed: [localhost] => {"censored": "results hidden due to no_log parameter", "changed": true}
       changed: [localhost] => {"censored": "results hidden due to no_log parameter", "changed": true}
       changed: [localhost] => {"censored": "results hidden due to no_log parameter", "changed": true}
       changed: [localhost] => {"censored": "results hidden due to no_log parameter", "changed": true}

       NOTIFIED: [dlpx.vpn-server | restart openvpn] *********************************
        REMOTE_MODULE service name=openvpn state=restarted
       changed: [localhost] => {"changed": true, "name": "openvpn", "state": "started"}

Hooray! Encrypted files are copied to the remote host securely. We now have a logical framework to re-use throughout our Ansible code base.

An Aside on Unix File Modes (Here be Dragons!)

I typically use the octal representation for Unix file modes instead of the string-based symbolic representation, but I had difficulty using octal representations with this deployment method. Although one could represent the file mode as an integer within the Ansible variable file, the “mode” arg to “copy” needs to have that value quoted as a string because of the double curly braces that Jinja needs to interpolate the variable. (Curly braces have a special meaning in YAML and thus need to be quoted).

One could theoretically re-cast that string back to an int() using Jinja’s “|int” filter, but I couldn’t seem to get this to work, so I eventually broke down and used symbolic file modes. Oh well, we can always write Test Kitchen tests to verify the correct permissions on these files later…

The Next Steps

Thus far we’ve covered how to use Ansible Vault to store your secrets safely in source control, and how to organize your Ansible variables/tasks to securely deploy those secrets. In Part 2 of this guide we’ll go over how to use Vault when testing with Test Kitchen, and also different ways that this could be integrated into a Jenkins job.

On to part 2 of “Managing Secrets with Ansible Vault – The Missing Guide”

ansible_logo_black_square

Advertisements

5 thoughts on “Managing Secrets with Ansible Vault – The Missing Guide (Part 1 of 2)

  1. Pingback: Managing Secrets with Ansible Vault – The Missing Guide (Part 2 of 2) | Dan Tehranian's Blog

  2. Pingback: Encrypting Login Credentials in Ansible Vault | Dan Tehranian's Blog

  3. Note that you don’t need to put the file contents into a variable anymore, you can use `lookup` to automatically decrypt the file, like this:

    “`
    – name: Copy ssh private key
    copy:
    dest: /var/lib/jenkins/.ssh/id_rsa
    content: “{{ lookup(‘file’, ‘files/ssh_privkey’) }}”
    owner: jenkins
    group: jenkins
    mode: 0600
    “`

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s