A (very) brief intro to TLS

TLS (Transport Layer Security) is a protocol designed to enable client-server
communication over the Internet in a way that prevents eavesdropping, tampering
and message forgery. It’s described in RFC 8446.

TLS relies on state-of-the art cryptography; this is also why it’s recommended
using the newest version of TLS available, which is 1.3 (as of early 2021).
Revisions of the TLS protocol clean up potentially unsafe corner cases, remove
weak encryption algorithms and generally try to make the protocol more secure.

When a client connects to a server with plain HTTP, it starts sending plaintext
data wrapped in TCP packets right after completing the standard TCP handshake
(SYN -> SYN-ACK -> ACK). Using TLS, the situation is somewhat more complicated
[1]:

TLS diagram

After completing the TCP handshake, the server and client perform a TLS
handshake to agree on a shared secret that’s unique only to them (and to this
specific session). This shared secret is then used to securely encrypt all
data being exchanged between them. While there’s a lot going on here, it’s
something the TLS layer implements for us. We just have to set up the TLS server
(or client) properly; the actual diff between an HTTP and an HTTPS server in Go
is minimal.

TLS certificates

Before we jump to the code showing how to set up an HTTPS server in Go using
TLS, let’s talk about certificates. In the diagram above, you’ll notice that
the server sends a certificate to the client as part of its very first
ServerHello message. Formally these are known as X.509 certificates,
described by RFC 5280.

Public key cryptography plays a major part in TLS. A certificate is a standard
way to wrap the server’s public key, along with its identity and a signature by
a trusted authority (typically a Certificate Authority). Suppose you want to
talk to https://bigbank.com; how do you know it’s really Big Bank there asking
for your password? What if someone is sitting on your cable connection,
intercepting all traffic and pretending to be Big Bank (classical MITM –
man-in-the-middle attack).

The certificate process is designed to prevent this scenario. When your client’s
underlying TLS implementation accesses https://bigbank.com, it expects Big
Bank’s certificate with a public key, signed by a trusted certificate authority
(CA). Certificate signatures can form a tree (bank’s key signed by A, which is
signed by B, which is signed by C, etc.), but at the end of the chain it must
have some certificate authority trusted by your client. Modern browsers have a
list of pre-trusted CAs (along with their own certificates) built-in. Since your
cable snooper cannot forge a trusted certificate’s signature, they cannot
impersonate Big Bank.

Generating self-signed certificates in Go

For local testing, it’s often very useful to be able to work with self-signed
certificates. A self-signed certificate is a certificate for some entity E with
a public key P, but the key is signed not by a known certificate authority, but
rather by P itself. While self-signed certificates have some additional
legitimate uses, we’ll focus on their use for testing here.

Go’s standard library has excellent support for everything related to crypto,
TLS and certificates. Let’s see how to generate a self-signed certificate in Go!

This code uses the crypto/ecdsa, crypto/elliptic and crypto/rand
packages to generate a new key pair [2], using the P-256 elliptic curve,
which is one of the allowed curves in TLS 1.3.

Next, we’ll create a certificate template:

Each certificate needs a unique serial number; typically, certificate
authorities will have these stored in some database but for our local needs a
random 128-bit number will do. This is what the first few lines of the snippet
are doing.

Next comes the x509.Certificate template. For more information on what the
fields mean, see the crypto/x509 package docs, as well as RFC 5280. We’ll just note that the certificate
is valid for 3 hours, and is only valid for the localhost domain.

Next:

The certificate is created from the template, and is signed with the private key
we’ve generated earlier. Note that &template is passed in both for the
template and parent parameters of CreateCertificate. The latter is
what makes this certificate self-signed.

This is it, we have the private key for our server and its certificate (which
contains the public key, among other information). All that’s left now is to
serialize them into files. First, the certificate:

And then, the private key:

We serialize the certificate and the key into PEM files, which looks like this
(for the certificate):

If you’ve ever set up SSH keys, the format should look familiar. We can use
the openssl command-line tool to show its contents:

HTTPS server in Go

Now that we have the certificate and private key in hand, we are ready to run an
HTTPS server! Once again, the standard library makes it very easy, though it’s
important to mention that security is a very tricky issue. Before exposing your
server to the public Internet, consider consulting with a security engineer
about best practices and what configuration options to be aware of [3].

Here’s a basic HTTPS server in Go:

It serves a single handler on the root path. The interesting part is the
TLS configuration, as well as the ListenAndServeTLS call, which takes the
paths to a certificate file and a private key file (in PEM format, just as
we generated them earlier). The TLS configuration has many possible fields;
here, I opted for a relatively strict protocol of forcing TLS 1.3 at minimum.
TLS 1.3 comes with strong security out of the box, so this is a good option if
you can ensure all your clients understand this version (and in 2021, they
should!)

The diff from a plain HTTP server is fewer than 10 lines of code! The bulk of
the server’s code (handlers for specific routes) is completely oblivious to the
underlying protocol and won’t change.

With this server running locally (and serving on port 4000 by default), Chrome
will initially balk when accessing it:

That’s because a web browser will not, by default, accept a self-signed
certificate. As mentioned above, browsers come with a hard-coded list of CAs
they trust, and our self-signed certificate is obviously not one of them. We
can still proceed to the server by clicking Advanced and then allowing Chrome
to go on, accepting the risk explicitly. It will then show us the website,
albeit grudgingly (with a red “Not secure” sign in the address bar).

If we try to curl to the server, we’ll also get an error [4]:

By reading the docs, we can find that curl can be made to trust our server
by providing it with the server’s certificate into the –cacert flag. If
we try that:

Success!

We can also talk to our server using a custom HTTPS client written in Go. Here’s
the code:

The only part making this different from a standard HTTP client is the TLS
setup. The important bit is setting up the RootCAs field of the
tls.Config struct. This is telling Go which certificates the client can
trust.

Other options for generating certificates

You may not know that Go comes with a tool to generate self-signed
TLS certificates, right in the standard installation. If you have Go installed
at /usr/local/go, you can run this tool with:

In general, it accomplishes the same goal as the first code snippet in this
post, but whereas my snippet makes some opinionated decisions about what to
generate, generate_cert is configurable with flags and supports several
different options.

As we’ve seen, while self-signed certificates can work for testing, they’re not
ideal for all scenarios. For example, it’s difficult to make browsers trust
them, and even then the user experience doesn’t fully replicate the “real world”
one.

Another option for generating local certificates for testing is the
mkcert tool. It creates a local
certificate authority (CA), and adds it to your system’s trusted list of CAs.
It then generates certificates signed by this authority for you, so as far as
the browser is concerned, they’re fully trusted.

If we run our simple HTTPS server with a certificate/key generated by
mkcert, Chrome will happily access it without warnings; we can also see
the details in the Security tab of developer tools:

curl will also successfully contact the server without requiring a
cacert flag, because it checks the system’s trusted CAs already.

If you’re looking for real certificates, Let’s Encrypt is of course a natural
option, using the certbot client or something
similar. In Go, libraries like certmagic can automate the interaction with
Let’s Encrypt for servers.

Client authentication (mTLS)

So far the examples we’ve seen have the server providing its (CA-signed)
certificate to the client to prove that the server is legitimately who it claims
to be (e.g. your bank’s website, before you agree to provide your password).

This idea is easy to extend to mutual authentication, where the client also
has a signed certificate to prove its identity. In the world of TLS, this is
called mTLS (for mutual TLS), and could be useful in many settings where
internal services have to communicate with each other securely. Public-key
crypto is generally considered more secure than passwords.

Here’s a simple HTTPS server with client authentication. The lines that changed
from the earlier HTTPS server are highlighted:

The changes are pretty much what you’d expect; in addition to setting up its
own certificate, key and TLS config, the server loads a client certificate and
sets TLSConfig to trust it. Naturally, this could also be the certificate
of a locally trusted CA which signs client certificates.

And this is an HTTPS client which authenticates itself when connecting to a
server; again, the lines that changed from the earlier (non-mTLS) client are
highlighted:

Before we try it, we’ll need to change our certificate generating script to
generate certificates suitable for clients as well. The change is in this line:

Which changes to:

Now let’s do a trial run. Start by generating separate certificates/keys for
client and server:

Running the mTLS server, it should pick up the right files based on flag
defaults:

In a separate window, if we run the older (non-mTLS) client, we get an error:

And the server log will show that “client didn’t provide a certificate”.
However, if we run the new mTLS client, it works:

While this demonstrates the mechanics of running mTLS servers and clients, in
reality there’d be a lot more to do, especially to manage certificates,
certificate renewal and revocation, and trusted CAs. This is called Public Key
Infrastructure (PKI), and it’s a large topic outside the scope of this modest
post.

[1]
This image is taken from hpbn.co,
licensed under CC BY-NC-ND 4.0.

[2]
The value returned by GenerateKey has type ecdsa.PrivateKey,
which also embeds an ecdsa.PublicKey, so it’s actually a key pair.

[3]
Another option to consider is setting up a reverse proxy between your
service and the outside world. Reverse proxies like Nginx or Caddy (which itself is written in Go)
are battle-tested and have clear instructions for secure setup. An
additional benefit of reverse proxies is that they can provide simple
load-balancing for heavily loaded services.

[4]
curl’s verbose log here is an interesting description of the process of
a TLS handshake!