So I just lost about 16 hours to this, and I haven't even been able to evaluate whether it'll work for me. On the one hand I suppose I could've not secured anything, but personally I feel you want to know what the production configuration looks like before you evaluate (and in my case, I like to default my docker containers to "would not be wrong to roll this into production").

So: how does TLS work for CockroachDB? Well the problem is CockroachDB has atrocious logging for its TLS certificate errors in v2.0.1.

The Problem

The problem was basically that CockroachDB expects a very specific format for it's x509 certificate data - outlined here https://github.com/cockroachdb/cockroach/issues/24621

I have a small utility I use for test certificates called makecerts which exists basically to have a much simpler static binary that does something like cfssl but with looser defaults. But the problem would apply to both scenarios.

In short: organization needs to be set to cockroach for node certificates, and the commonName needs to be set to node. I was generating certificates with a commonName of my docker-compose test network - 172.20.0.1 and the like, which is perfectly valid, validates correctly with the CA, and can be used to initialize the cluster - but none of the nodes will connect to each other.

And as noted in the Github issue produces no logs actually describing the problem.

The Solution

So there you have it - with makecerts the line I needed for the test docker-compose file was:

makecerts --O=cockroach --CN=generated \
    172_20_0_1=node,172.20.0.1,localhost,127.0.0.1 \
    172_20_0_2=node,172.20.0.2,localhost,127.0.0.1 \
    172_20_0_3=node,172.20.0.3,localhost,127.0.0.1 \
    172_20_0_4=node,172.20.0.4,localhost,127.0.0.1 \
    172_20_0_5=node,172.20.0.5,localhost,127.0.0.1 \
    root

Note on how this works: this command above is saying "generate 172_20_0_1.crt and 172_20_0_1.pem for the certificate and key respectively, assign a commonName of node and then generate SANs for the commonName and all common-separated values."

Since makecerts is simple minded it also just signs the cert for all use-cases - it's very much a testing tool.

The final docker-compose I used to get this started was:

version: '2'

networks:
  roachnet:
    driver: bridge
    ipam:
      driver: default
      config:
      - subnet: 172.20.0.0/24
        gateway: 172.20.0.254

services:
  roach1:
    image: cockroachdb/cockroach:v2.0.1
    command: start --host=172.20.0.1 --logtostderr=INFO --certs-dir=/certs --join=172.20.0.1,172.20.0.2,172.20.0.3,172.20.0.4,172.20.0.5
    volumes:
    - ./roach1:/cockroach/cockroach-data
    - ./172_20_0_1.crt:/certs/node.crt
    - ./172_20_0_1.pem:/certs/node.key
    - ./root.crt:/certs/client.root.crt
    - ./root.pem:/certs/client.root.key
    - ./generated.crt:/certs/ca.crt
    - ./generated.crt:/usr/local/share/ca-certificates/ca.crt
    networks:
      roachnet:
        ipv4_address: 172.20.0.1

  roach2:
    image: cockroachdb/cockroach:v2.0.1
    command: start --host=172.20.0.2 --logtostderr=INFO --certs-dir=/certs --join=172.20.0.1,172.20.0.2,172.20.0.3,172.20.0.4,172.20.0.5
    volumes:
    - ./roach2:/cockroach/cockroach-data
    - ./172_20_0_2.crt:/certs/node.crt
    - ./172_20_0_2.pem:/certs/node.key
    - ./root.crt:/certs/client.root.crt
    - ./root.pem:/certs/client.root.key
    - ./generated.crt:/certs/ca.crt
    - ./generated.crt:/usr/local/share/ca-certificates/ca.crt
    networks:
      roachnet:
        ipv4_address: 172.20.0.2

  roach3:
    image: cockroachdb/cockroach:v2.0.1
    command: start --host=172.20.0.3 --logtostderr=INFO --certs-dir=/certs --join=172.20.0.1,172.20.0.2,172.20.0.3,172.20.0.4,172.20.0.5
    volumes:
    - ./roach3:/cockroach/cockroach-data
    - ./172_20_0_3.crt:/certs/node.crt
    - ./172_20_0_3.pem:/certs/node.key
    - ./root.crt:/certs/client.root.crt
    - ./root.pem:/certs/client.root.key
    - ./generated.crt:/certs/ca.crt
    - ./generated.crt:/usr/local/share/ca-certificates/ca.crt
    networks:
      roachnet:
        ipv4_address: 172.20.0.3

  roach4:
    image: cockroachdb/cockroach:v2.0.1
    command: start --host=172.20.0.4 --logtostderr=INFO --certs-dir=/certs --join=172.20.0.1,172.20.0.2,172.20.0.3,172.20.0.4,172.20.0.5
    volumes:
    - ./roach4:/cockroach/cockroach-data
    - ./172_20_0_4.crt:/certs/node.crt
    - ./172_20_0_4.pem:/certs/node.key
    - ./root.crt:/certs/client.root.crt
    - ./root.pem:/certs/client.root.key
    - ./generated.crt:/certs/ca.crt
    - ./generated.crt:/usr/local/share/ca-certificates/ca.crt
    networks:
      roachnet:
        ipv4_address: 172.20.0.4

  roach5:
    image: cockroachdb/cockroach:v2.0.1
    command: start --host=172.20.0.5 --logtostderr=INFO --certs-dir=/certs --join=172.20.0.1,172.20.0.2,172.20.0.3,172.20.0.4,172.20.0.5
    volumes:
    - ./roach5:/cockroach/cockroach-data
    - ./172_20_0_5.crt:/certs/node.crt
    - ./172_20_0_5.pem:/certs/node.key
    - ./root.crt:/certs/client.root.crt
    - ./root.pem:/certs/client.root.key
    - ./generated.crt:/certs/ca.crt
    - ./generated.crt:/usr/local/share/ca-certificates/ca.crt
    networks:
      roachnet:
        ipv4_address: 172.20.0.5

and you need to run a once-off init phase to start the cluster:

#!/bin/bash
docker-compose exec roach1 ./cockroach init --certs-dir=/certs/ --host=172.20.0.1

A final note - why does makecerts exist?

I really want to like cfssl, but it still just seems like too much typing for when you're setting up test scenarios. It's a production tool for Cloudflare, whereas the goal with makecerts was to make it as easy as possible to generate TLS certs for test cases on the desktop and thus force myself to always turn TLS on when developing - since obviously I'm always going to be using it in production, so I should test with it.