In my previous blog Concourse CI: A new way to approach CI-CD, I introduced Concourse CI. It is a tool to implement CI-CD pipelines to deliver software. If you have not read that post, please read that first. It will only take 15-20 minutes and make it easy for you to understand what I am trying to explain in this blog.
As explained in the introductory blog, all pipelines are in the form of code. Pipeline code contains resources, jobs and tasks. This same pipeline code will also need credentials to connect to SCM (Source Control Management) tools like Git or PaaS such as Cloud Foundry. This pipeline code will also be stored in a source code repository either with the application code or in a separate repository. This will give access to these secure credentials to anyone who has access to the repository.
One simple approach to hide these credentials is to use a separate params.yml file. We can define template variables in our pipeline code. Values for these template variables is stored in a separate file like params.yml. Actual values for these variables will be populated in the pipeline code while creating the pipeline using --var or --load-vars-from flags.
Let me explain this by an example. Below is a snippet of pipeline code containing Git resource definition.
resources:
- name: source
type: git
source:
branch: git-hub-branch
username: ((github_username))
password: ((github_password))
In this example, github_username and github_password are two template variables. We can store the actual values for these variables into a separate file for example params.yml as shown below.
github_username: actual_username
github_password: actual_password
We can then define the pipeline using below command
$ fly --target local set-pipeline --pipeline sample --config pipeline.yml --load-vars-from params.yml
or in short form
$ fly -t local set-pipeline -p sample -c pipeline.yml -l params.yml
Using this approach, we can store the pipeline code in a source repository and share it with others without exposing the credentials used in the pipeline.
This approach is good up to an extent. Still, anyone who has access to Concourse server can use fly CLI to obtain the complete pipeline code. This will also reveal the template variables values.
$ fly -t local get-pipeline -p sample
A better approach will be to use a Credentials management tool like Hashicorp's Vault or Credhub. In this blog, I will explain basics of Vault and how it can be used with Concourse to securely store credentials.
Introduction to Vault:
Vault is a tool to securely access secrets. A secret can be a password, SSH key, or a certificate. Vault provides a unified interface to any secret with tight access control and a detailed audit log. Vault is completely open source and free.
Installing Vault is very simple. Just download the Vault package from their official website page. Vault is packaged as zip file. Vault runs a single binary named vault. Just make sure that this binary is available on the PATH. On Linux, you can put it under /usr/local/bin which is already part of the PATH for simplification.
Once done, you can run vault command on the command line and it will show all the available sub commands. Or vault --version to check the version.
$ vault --version
Vault runs in a client/server mode. Vault server directly interacts with the data storage and backends. Commands issued using CLI interact with server over a TLS connection.
Just to test, Vault server can be started in dev mode using below command. In this case, Vault runs completely in-memory and starts unsealed with a single unseal key. This should be only used to test and play around before you actually start using Vault for other purposes.
$ vault server -dev
Now lets configure Vault to run in server mode. Vault is configured using HCL files. Configuration is very simple and contains two major components, storage and listener.
Storage is used to mention the physical backed used for storage. There are many options available to select appropriate storage backend (e.g. Consul, Etcd, S3, MySQL etc.). I will select File storage for simplification.
Listener mentions how Vault listens for API requests.
Below is a sample configuration file. Lets name it config.hcl and store it at /etc/vault. We will use it later to define Vault as a service also.
storage "file" {
path = "/mnt/vault/data"
}
listener "tcp" {
address = "127.0.0.1:8200"
tls_disable = 1
}
You can start the Vault server now from command line using vault server command.
$ cd /etc/vault
$ sudo vault server -config=config.hcl
Once Vault server has started, we need to initialize the Vault. This happens only once for each backend. To do so run the following command. Before running the initialization command, please set environment variable VAULT_ADDR as shown below. You can permanently save its value in .bashrc or other profile files.
$ export VAULT_ADDR=http://127.0.0.1:8200
$ vault operator init
Please note that older version of Vault had the command "vault init" and not "vault operator init". I have tested all these commands against version 0.10.
This will generate encryption key, create unseal keys, and the initial root token. It generates total 5 unseal keys and 1 root token. Vault needs minimum three keys to unseal. Ideally all these keys should not be saved together and distributed. So one single person cannot unseal the Vault. They should also be stored using some form of encryption.
Before we can start using the Vault, we need to unseal. This is where the unseal keys generated during initialization process will be used. Run vault unseal command three times with any of three different keys out of five.
$ vault operator unseal <key1>
With each iteration of unseal command, you will be able to see the progress. When Vault Sealed value changes from true to false, it will be ready for use.
Finally we need to authenticate with initial root token. It was also generated with initialization process. This process needs to be done only once.
$ vault login <root_token>
Vault can be sealed back using the vault seal command.
$ vault operator seal
Finally lets configure it as a service so we don't have to start using vault server command every time. Below instructions are for Ubuntu 16.04 which uses systemd. First create a file vault.conf in /etc/vault and define environment variable VAULT_ADDR=http://127.0.0.1:8200 in that file.
Once done, navigate to /etc/systemd/system directory. Create an empty file named vault-server.service there. You will need sudo privilege to do so. Put the following contents inside that file.
[Unit]
Description=Vault Server
[Service]
User=root
Restart=on-failure
EnvironmentFile=/etc/vault/vault.conf
ExecStart=/usr/local/bin/vault server -config=/etc/vault/config.hcl
[Install]
WantedBy=multi-user.target
Once done, run following commands to start it as service, check status and enable it at the boot time. Vault should be ready for use once the service starts successfully.
$ sudo systemctl start vault-server
$ sudo systemctl status vault-server
$ sudo systemctl enable vault-server
Concourse and Vault Integration:
As our Vault is ready for use, lets save some secrets. I will take the scenario mentioned at the beginning of the blog where we need to store github_username and github_password. We can write secrets into Vault using vault write command.
$ vault write secret/github_username value=actual_username
Vault uses different mount points to store different kinds of data. To store, key-value pairs, it uses mount point secret. We provide the key-name after the mount point and its value with value option. To read the value, use the vault read command.
$ vault read secret/github_username
To delete a secret, use the vault delete command.
$ vault delete secret/github_username
To make these credentials accessible to Concourse, we need to write them with a particular convention. When Concourse is integrated with Vault, it first looks for keys defined for a pipeline and then for the team. By default Concourse creates a team named main.
So if we wanted to make above mentioned username/password accessible only to sample pipeline defined under team main, we need to write them with following convention.
$ vault write secret/main/sample/github_username value=actual_username
If we wanted to make the credentials accessible to all pipelines under team main. We can use the below command.
$ vault write secret/main/github_username value=actual_username
Before we can integrate Vault with Concourse, we need to generate a token which Concourse can use to access Vault. Use vault token-create command as shown below to generate the token.
$ vault token-create -ttl=24h -period=24h -renewable=true
We start Concourse web server using concourse web command. We can provide Vault related information with flags --vault-url, --vault-path-prefix, --vault-client-token, and --vault-insecure-skip-verify flags as shown below. All the flag values can be store in an environment file as environment variables which can be used by Concourse service.
$ concourse web \
--basic-auth-username myuser \
--basic-auth-password mypass \
--session-signing-key session-signing_key \
--tsa-host-key tsa_host_key \
--tsa-authorized-keys authorized_worker_keys \
--external-url http://my-ci.example.com \
--vault-url http://my-vault.example.com \
--vault-path-prefix /secret \
--vault-client-token vault-token \
--vault-insecure-skip-verify true
This is it on Concourse and Vault. You can write to me at kpsingh.chouhan@gmail.com. My Twitter handle is @ChouhanKP84. In the next edition, I will cover how we can use Grafana to create dashboards for Concourse metrics.
Comments
Post a Comment