IKEv2 VPN on OpenWrt – Part II

In this second blog post we will switch from a username/password authentication to a certificate based authentication using EAP-TLS. We will go into detail on the x.509 certificates employed for both the VPN initiator and VPN responder.

Changes to previous setup

In the first part of this blog post we drew a big picture, made some assumptions, defined preconditions and agreed on a network setup. All these points actually won’t change for the second part, as the only thing we’re changing is authentication. In other words we only change the way how a client (the VPN initiator) authenticates itself to the server (the VPN responder) and vice versa.

Extensible Authentication Protocol

When talking about EAP-TLS we’re using a framework called the Extensible Authentication Protocol, which is commonly used in network communications and also is a big part of the 802.1x standard.

What happens in a nutshell is that upon establishment of a VPN connection, TLS handshakes are being used to mutually authenticate client and server. While this method consumes a lot of extra messages during key exchange (~6 to 10 extra IKE messages), it has a lot of advantages. One of those advantages is the authentication against an AAA backend (a RADIUS server). Another advantage is that all modern operating systems for mobile and desktop devices support authentication via this method.

Certificates

The first thing we need to clarify is how our certificates can be verified by both the VPN initiator and the VPN responder. For that to happen, certificates must be derived from a certificate authority that everybody trusts. While it is of course possible to obtain a certificate from a certificate authority that is represented in Mozilla’s, Apple’s or Microsoft’s root store, it is not necessary. Besides the associated cost for such certificates, we don’t actually need anybody to trust our certificates. Only actual parties involved in our VPN connection need to trust our certificates.

Long stories short – we will use our own root certificate authority to issue our certificates. Ideally we actually don’t even use the root certificate authority, but an intermediate certificate authority to issue our certificates.

    +-------------+
    |             |
    |   ROOT CA   |
    |             |
    +------+------+
           |
           v
+----------+----------+
|                     |
|   INTERMEDIATE CA   |
|                     |
+---------+-+---------+
          | |
          | |
          | |    +------------------+
          | |    |                  |
          | +--->+ HOST CERTIFICATE |
          |      |                  |
          |      +------------------+
          |
          |
          |      +--------------------+
          |      |                    |
          +----->+ CLIENT CERTIFICATE |
                 |                    |
                 +--------------------+

Host Certificate Deployment

As stated above, the VPN responder needs not only the host certificate, but only the certificate issuing chain for verification. Let’s have a quick look at those parts of the directory structure that is important on our VPN responder (our OpenWrt router):

/etc/ipsec.d
├── cacerts
│   ├── tinkivity-rootca.pem
│   └── tinkivity-stepca.pem
├── certs
│   └── servercert.cert.pem
└── private
    └── servercert.key.pem

O.K. – but what goes where?

cacerts

In this directory we put the root certificate as well as all other intermediate certificates. We need to know that strongSwan only reads the first certificate in every certificate file (i.e. .pem file), so placing one full-chain certificate file will do us no good. Instead we will put one certificate for every certification authority in our chain in that directory. In our case this directory will contain two certificates. Upon start the ipsec daemon will automatically load all certificates it can find in this directory. That means that no configuration entries are needed to make ipsec aware of certificates in that folder.

certs

As part of the configuration in /etc/ipsec.conf we can define certificates to be used (i.e. via left|rightcert statements). This folder contains such certificates and in our case it will contain our host certificate.

private

For every certificate that we use as part of our /etc/ipsec.conf configuration, a private key is needed. This folder will contain one private key for every corresponding certificate mentioned in the left|rightcert statements in our configuration. In our case this directory will contain the private key of our host certificate.

For strongSwan to properly load the private key (including application of the passphrase that protects it), the following entry is needed in the /etc/ipsec.secrets configuration:

# /etc/ipsec.secrets - strongSwan IPsec secrets file

: RSA servercert.key.pem "INSERT PASSPHRASE HERE"

Host Certificate Generation

In general we follow the common recipe for generating any x.509 certificate. We create a private key, a certificate signing request and we sign the certificate.

Private Key

We choose RSA as algorithm and create a 4096 bit long private key that we protect by a password. Rather than using the legacy genrsa command in openssl, we go with the newer genpkey command, as it allows for better fine-tuning. We do not put the passphrase on the command line, but define a file (passphrase.txt) that we include into our command.

andreas@intermediateca ➜  ~ openssl genpkey -algorithm RSA -aes-256-cbc -pkeyopt rsa_keygen_bits:4096 -pass file:passphrase.txt -out servercert.key.pem

Certificate Signing Request

Because we want to limit the length and complexity of our command line, we use an openssl configuration file to determine what we want in our certificate.

[req]
prompt			= no
distinguished_name	= req_dn
req_extensions		= req_ext

[req_dn]
CN			= vpn.example.com

[req_ext]
subjectAltName		= @alt_names

[alt_names]
DNS.1			= vpn
DNS.2			= vpn.example.com
DNS.3			= internalhostname
DNS.4			= internalhostname.lan.local

The canonical name (CN) must be set to match one of the subject alternative names (as shown in lines 13-16 above). Moreover, there must be match between one of the subject alternative names and the leftid used in the configuration at /etc/ipsec.conf for the certificate to be useful for strongSwan. Eventually we issue the certificate signing request with the following command:

andreas@intermediateca ➜  ~ openssl req -new -config config.cnf -key servercert.key.pem -passin file:passphrase.txt -out request.csr

Certificate Signing

For convenience we use our smallstep intermediate certificate authority for signing our certificate. However, we could just as easily sign our certificate directly with openssl.

andreas@intermediateca ➜  ~ step ca sign --ca-url https://acme.tinkivity.home:8443 --root /etc/ssl/tinkivity.pem --provisioner vpnserver-prov --not-after 2021-12-31T23:59:59Z request.csr servercert.cert.pem

More important than how we sign the certificate is the question what our certificate looks like? We use openssl to inspect the content of the certificate:

andreas@intermediateca ➜  ~ openssl x509 -noout -text -in servercert.cert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            46:79:09:ef:cf:6a:8a:55:93:17:6c:fe:44:38:0a:89
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = DE, ST = Saxony, O = Tinkivity, OU = Tinkivity Intermediate Certificate Authority, CN = Smallstep Intermediate CA, emailAddress = xxx@xxx.com
        Validity
            Not Before: Jan  9 14:16:05 2021 GMT
            Not After : Dec 31 23:59:59 2021 GMT
        Subject: C = DE, ST = Saxony, L = Dresden, street = "Musterstrasse 1, 01234 Dresden, Germany", O = Tinkivity, OU = vpn servers via manual vpnserver-prov, CN = vpn.example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (4096 bit)
                Modulus:
                    00:cb:9a:1a:dd:28:a7:84:8e:15:e7:83:c0:64:1c:
                    ...
                    << REDACTED >>
                    ...
                    b9:36:25
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            X509v3 Subject Key Identifier: 
                90:F9:FF:02:CD:CA:5E:2F:61:43:EA:8B:CB:B0:FE:DD:7A:16:77:D4
            X509v3 Authority Key Identifier: 
                keyid:87:32:28:49:63:29:06:79:96:13:DE:47:14:9F:EF:C0:DD:EC:4D:C3

            X509v3 Subject Alternative Name: 
                DNS:vpn, DNS:vpn.example.com, DNS:internalhostname, DNS:internalhostname.lan.local
            1.3.6.1.4.1.37476.9000.64.1: 
                0@.....vpnserver-prov.+jtrIkQa8-b_nzz2OU4DlffIMHmDZmhVV1R3rBBC46qo
    Signature Algorithm: sha256WithRSAEncryption
         4a:3a:26:fa:dc:3b:08:be:df:03:a1:cf:d7:47:e5:98:10:ac:
         ...
         << REDACTED >>>
         ...
         30:25:5a:cb:94:dd:4a:24

Obviously we use a dedicated provisioner for VPN server certificates (vpnserver-prov), but other than automatic application of custom subject settings for our certificate, there is no special magic happening in the provisioner. Still, you can read my earlier blog post if you like to understand more how step ca templates can be used.

Either way, the important part of the host certificate from an ipsec point of view are the subject alternative names (line 32 above). Matching between leftid in the configuration, the Remote ID declared on the client side and one of the subject alternative names is crucial.

VPN Responder Configuration

In the first part of this blog post we defined EAP-MSCHAPv2 as authentication method.

/etc/ipsec.conf

Compared to the configuration we’ve used for username/password authentication, only a very few lines need to change. For better housekeeping (and debugging) we have renamed our connection (line 16) to rwEAPTLS. In line 19 we changed our rightid into vpn-via-eaptls and in line 20 we switched to eap-tls for an authentication method.

config setup

conn %default
        keyexchange=ikev2
        ike=aes256-aes128-sha256-sha1-modp3072-modp2048
        esp=aes128-aes256-sha256-modp3072-modp2048,aes128-aes256-sha256
        left=%any4
        leftauth=pubkey
        leftcert=servercert.cert.pem
        leftid=vpn.example.com
        right=%any4
        rightsourceip=10.10.97.2-10.10.97.6
        eap_identity=%identity
        auto=add

conn rwEAPTLS
	leftsendcert=always
	leftsubnet=10.10.97.0/29,10.10.90.0/24
	rightid=vpn-via-eaptls
	rightauth=eap-tls

/etc/ipsec.secrets

In our previous configuration we had one line per every user that wanted to connect via username/password authentication with our VPN (i.e. myuser : EAP “mypassword”). As we don’t use usernames and passwords any more we can delete those lines. Alternatively we can leave them in the configuration, as strongSwan will disregard them on its own if they are not needed.

: RSA servercert.key.pem "INSERT PASSPHRASE HERE"

However, as mentioned above in this blog post, we need to put in the passphrase (if any) that is needed to decode the private key. If no passphrase is set for the private key, the configuration line ends after the name of the private key file (no empty quotes needed).

Client Certificate

As part of EAP-TLS clients need a certificate too. From an x.509 point of view there is no dramatic difference between a client certificate and host certificate. The same recipe as for host certificates does apply.

Private Key

Same as for hosts, we create ourselves a 4096 bit long RSA key and protect it with a passphrase. We follow the exact procedure as for the host private key above.

Certificate Signing Request

Same as for hosts, we create a configuration. What’s different this time though, is the content.

We use the canonical name to define the name of the user (line 7).

For the subject alternative names we like to get the user’s email address into our certificate. Besides DNS names, the standard for the subject alternative names extension allows to put email as value. We do that in line 13.

In line 14 we use DNS to specify what this client will be allowed to do in match of a rightid in our /etc/ipsec.conf configuration (see above). This is a very important part of the authentication. While the certificate chain helps strongSwan to verify the authenticity of the client certificate, the subject alternative name vpn-via-eaptls in the certificate will be matched to potential connections in the VPN responder side. As it happens we have defined one such connection and this will be the reason that the client will be allowed to connect.

[req]
prompt             = no
distinguished_name = req_dn
req_extensions     = req_ext

[req_dn]
CN                 = John Doe

[req_ext]
subjectAltName     = @alt_names

[alt_names]
email.1            = john.doe@example.com
DNS.1              = vpn-via-eaptls

As for the generation of the CSR (certificate signing request), we follow the exact procedure as for the host certificate signing request.

Certificate Signing

Again, for convenience we use our smallstep ca to issue the certificate. This time we use a different provisioner that applies some different claims. However, the provisioner is not that important really, as besides administrative differences there is nothing going on that would require it. I mainly use provisioners for better housekeeping (i.e. certificate expiration claims).

andreas@intermediateca ➜  ~ step ca sign --ca-url https://acme.tinkivity.home:8443 --root /etc/ssl/tinkivity.pem --provisioner vpnclient-prov --not-after 2021-12-31T23:59:59Z request.csr usercert.pem

Client Certificate Deployment

When it comes to deployment to our client we actually don’t want to send a bunch of files (keys and certs). Ideally we want to only send around one archive that contains it all. We use the PKCS#12 format for that. Besides the private key and the user certificate, we also put our root certificate into the PKCS#12 archive.

andreas@intermediateca ➜  ~ openssl pkcs12 -export -inkey userkey.pem -in usercert.pem -name SomeFriendlyName -certfile /etc/ssl/tinkivity.pem -passin file:passphrase.txt -out john.doe.vpncert.p12
Enter Export Password:
Verifying - Enter Export Password:

The export password that is asked for, is not the same as the passphrase of the private key. All it does is to protect the p12 archive during transport. With a strong export password it will be perfectly safe to distribute the p12 archive via insecure media (such as email, dropbox, etc.).

But careful! When creating a PKCS#12 archive, the private key in that archive will be stripped of its password. When retrieving a private key from a PKCS#12 archive, the private key will not be password protected.

andreas@intermediateca ➜  ~ openssl pkcs12 -nodes -nocerts -in john.doe.vpncert.p12
Enter Import Password:
Bag Attributes
    localKeyID: B8 B9 AA 7E B5 A4 BC BD 91 B2 6A 8D 11 99 59 31 EC 34 30 CF 
    friendlyName: SomeFriendlyName
Key Attributes: <No Attributes>
-----BEGIN PRIVATE KEY-----
MIIJQQIBADANBgkqhkiG9w0BAQEFAASCCSswggknAgEAAoICAQChH5dr3zCmsJtf
...
<< REDACTED >>
...
8rz6Fz++G18dufZ66uzObluJ/+mw
-----END PRIVATE KEY-----

Importing the user certificate

This part is specific to the operating system of the client, but typically a double-click on the file does the trick. Anyway, as our client receives and imports the .p12 archive, the root certificate, that allows that client to trust our host, is automatically being installed on the client as well.

Client Configuration

Compared to the first part of this blog post, there are actually only very slight differences in setup. The first step of configuring a new connection (go to System Preferences –> Network -> “plus sign” button), is still the same.

What happens after we click the Create button is a little different. We need to use our new Local ID for EAP-TLS, as we’re not using username/password authentication any longer.

Server Address: xxx.xxx.xxx.xxx
     Remote ID: vpn.example.com
      Local ID: vpn-via-eaptls

Also, when we click on the Authentication Settings button, we need to use Certificate for the Authentication Settings and we need to use the Select… button to use our imported user certificate.

Authentication Settings: Certificate
            Certificate: Select... John Doe

Making a connection

That part has not changed. However, what you need to keep in mind now are the expiration dates of the certificates. By the time a user certificate expires, that user will not be able to use VPN any longer. When the host certificate expires, no user will be able to connect via VPN any longer.

Reviving a bricked TP Link WR1043ND Router

I have this old TP Link router that I thought I could tech some tricks. Unfortunately I flashed it with an image that contained a broken configuration and I successfully locked myself out completely from the all network access to it.

TP Link TL-WR1043ND
TP Link TL-WR1043ND

Newer models support a failsafe method in which very early on in the boot process the router might look for a specific image via TFTP. This model (at least this revision) didn’t let me grasp that straw though.

Serial Console to the rescue

OpenWrt’s wiki provides extremely good documentation on a seemlingly endless list of devices. Of course there is also a long and detailed entry for the TL-WR1043ND model. Near the bottom of the wiki entry it mentions UART ports that lie dormant on the PCB. All that is needed is to “crack” open the router and to connect 3 wires to the right places.

I am extremely unexperienced when it comes to soldering, but adding three wires in place is a task that I can (barely) get done.

UART ports with wires soldered into them

On the other end of my wires I connected a USB to serial adapter, to be more precise I own a FT232RL that can be bought for less than 10 bucks on AMAZON (or Ali Express if you have enough time to wait for it to be shipped in from China). The most important thing when connecting the FTDI to the board is the voltage. The board runs on 3.3V and it is of utmost importance to adjust the FTDI to that exact voltage, as otherwise the circuit will fry. Other than that the RX port on the board needs to be hooked up with the TX port of the FTDI and the TX port of the board goes into the RX of the FTDI. When ready my setup looked like this:

Router with UART ports connected to FTDI

Connect to the console

I am sure there are plenty of programs out there that allow connecting the serial console to a terminal. With OSX there is no need to install anything, as you just have to open a terminal and use the screen command. The first thing though is to find out the name of the device.

andreas@AnDeSu16 ➜  ~ ls -lah /dev/tty.*
crw-rw-rw-  1 root  wheel   18,   0 Dec 10 14:48 /dev/tty.Bluetooth-Incoming-Port
crw-rw-rw-  1 root  wheel   18,   2 Dec 10 14:48 /dev/tty.JabraEVOLVE65-SPPDev
crw-rw-rw-  1 root  wheel   18,   4 Jan  8 14:38 /dev/tty.usbserial-AB0KGVU8

As shown above the device we want to connect to it is listed as last entry. Now all we need to do is to use the screen command with the device name and the correct baud rate. In our case the router uses 115200 baud and so we adhere…

andreas@AnDeSu16 ➜  ~ screen /dev/tty.usbserial-AB0KGVU8 115200

Last but not least comes the moment of truth and we switch on our router…

U-Boot 1.1.4 (Nov 17 2009 - 11:56:26)

AP83 (ar9100) U-boot 0.0.11
DRAM:  
sri
32 MB
id read 0x100000ff
flash size 8MB, sector count = 128
Flash:  8 MB
Using default environment

In:    serial
Out:   serial
Err:   serial
Net:   ag7100_enet_initialize...
No valid address in Flash. Using fixed address
: cfg1 0xf cfg2 0x7114
eth0: 00:03:7f:09:0b:ad
eth0 up
eth0
Autobooting in 1 seconds## Booting image at bf020000 ...
   Uncompressing Kernel Image ... OK

Starting kernel ...

Perfect. Everything after that is a piece of cake, as we get into a root console and can do whatever we like with it. Well, obviously we use it to flash a working image that allows us remote access again 😉

Employing SSH host certificates

Most of us have been in a situation at least once where we need to follow the TOFU (trust on first use) pattern. A popular example would be a new host that we try to login for the first time via SSH. We might see something like this:

andreas@AnDeSu16 ➜  ~ ssh andreas@192.168.1.1
The authenticity of host '192.168.1.1 (192.168.1.1)' can't be established.
ECDSA key fingerprint is SHA256:fTGKcndGlhvoHoNI8HGu6ErQpKer495rMRJrrQok+ok.
Are you sure you want to continue connecting (yes/no/[fingerprint])? 

Just to be clear: this type of warning is exactly the same situation as if our web browser is asking us if we want to continue because the browser cannot verify the authenticity of the website’s certificate (see below).

Solution Approach

The ‘fix’ is actually much easier than one might think. Similar to TLS certificates we can use SSH certificates. The idea is simple: we create ourselves a certificate authority for SSH and sign public host keys with it. As a result we will create host certificates that we can send back to the host. After that hosts will present such certificates to clients. All that clients need to do in order to verify the authenticity of such certificate is to trust our global (internal) SSH Certificate Authority.

While TLS certificates make use of a standardized format (x.509), SSH certificates follow a proprietary format that is widely used. However, the principles behind the certification process are the same. Let’s summarize the process is plain steps briefly…

  • Certificate Authority has a private key and a public key
  • Client trusts the public key of Certificate Authority
  • Host has a key-pair consisting of one private and one public key
  • Host keeps its private key secret but submits its public key to Certificate Authority
  • Certificate Authority uses its private key to sign the host’s public key
  • Certificate Authority’s signature has yielded a host certificate
  • Host imports the host certificate
  • Client is presented with host’s certificate upon login
  • Because Client trusts the Certificate Authority’s public key, it implicitly trusts Host’s certificate

Certificate Authority Layout

Let’s have a look at the directory structure we have in place for our SSH Certificate Authority. Our master keypair is stored in the private and public folder. The certs folder is meant for the certificates we issue.

/
└── ca
	└── SSH
		└── host
			├── certs
			├── private
			│   ├── tinkivity_host_ecdsa_key
			│   ├── tinkivity_host_ed25519_key
			│   └── tinkivity_host_rsa_key
			└── public
				├── tinkivity_host_ecdsa_key.pub
				├── tinkivity_host_ed25519_key.pub
				└── tinkivity_host_rsa_key.pub

Import Host’s public key(s)

Our practical example is creating a certificate for a new host (newhost.tinkivity.home), which we want to be valid for 5 years. The first thing we have to do is to create a directory under the certs folder and to change into it.

andreas@rootca ➜  SSH mkdir /ca/SSH/host/certs/newhost.tinkivity.home
andreas@rootca ➜  SSH cd /ca/SSH/host/router/newhost.tinkivity.home

In order to issue a certificate we need the public keys from the new host. Technically one public key would be enough, but there are 3 common (and considered safe) algorithms out there (ECDSA, RSA and ED25519), which each have their own key-pair and can have a corresponding certificate.

NEVER EVER should private keys leave the machine or host at which they have been generated! Please do not export private keys from your host ever!

Let’s have a quick look into the directory structure after import of the public host keys.

andreas@rootca ➜  newhost.tinkivity.home ls -lah
total 23
drwxr-xr-x  2 root  wheel     5B Jan  4 18:54 .
drwxr-xr-x  7 root  wheel     7B Jan  4 18:42 ..
-r--r-----  1 root  wheel   188B Jan  4 18:54 ssh_host_ecdsa_key.pub
-r--r-----  1 root  wheel   108B Jan  4 18:54 ssh_host_ed25519_key.pub
-r--r-----  1 root  wheel   408B Jan  4 18:54 ssh_host_rsa_key.pub

Issue Host certificate(s)

For each of the 3 keys we can issue a certificate. There are some parameters we will supply to the ssh-keygen command. Let’s go through those parameters one by one:

parametermeaning
-ssays we want to sign a certificate and the next parameter to follow must be the private key of the certificate authority
-hindicates that we are about to sign a host certificate (has no parameter value to follow)
-Isays we want to give the certificate an ID and the next parameter to follow must be the string representing the certificate ID
-nsays we want to set principal name(s) and the next parameter to follow must be a comma separated list of principal names (no white spaces please)
-Vsays we want to set the validity interval and the next parameter must be a validity interval (please read the ssh-keygen man page for format instructions)
our router’s public host key

Having understood the meaning of those parameters, we can get to work and issue our first certificate. We go alphabetically and start with the ECDSA certificate.

andreas@rootca ➜  newhost.tinkivity.home ssh-keygen -s ../../private/tinkivity_host_ecdsa_key -h -I newhost_v01 -n newhost,newhost.tinkivity.home -V 'always:20260131' ssh_host_ecdsa_key.pub
Enter passphrase: 
Signed host key ssh_host_ecdsa_key-cert.pub: id "newhost_v01" serial 0 for newhost,newhost.tinkivity.home valid before 2026-01-31T00:00:00

We repeat the same procedure for ED25519 and RSA.

andreas@rootca ➜  newhost.tinkivity.home ssh-keygen -s ../../private/tinkivity_host_ed25519_key -h -I newhost_v01 -n newhost,newhost.tinkivity.home -V 'always:20260131' ssh_host_ed25519_key.pub
andreas@rootca ➜  newhost.tinkivity.home ssh-keygen -s ../../private/tinkivity_host_rsa_key -h -I newhost_v01 -n newhost,newhost.tinkivity.home -V 'always:20260131' ssh_host_rsa_key.pub

When done we should find 3 certificates in our folder. The names of our certificates are automatically being generated. The pattern is that xxx_key.pub is being expanded to xxx_key-cert.pub.

andreas@rootca ➜  newhost.tinkivity.home ls -lah *-cert.pub
-r--r-----  1 root  wheel   873B Jan  4 18:21 ssh_host_ecdsa_key-cert.pub
-r--r-----  1 root  wheel   521B Jan  4 18:22 ssh_host_ed25519_key-cert.pub
-r--r-----  1 root  wheel   2.0K Jan  4 18:22 ssh_host_rsa_key-cert.pub

Verifying host certificates

We can use the ssh-keygen command to check the content of the certificate.

andreas@rootca ➜  newhost.tinkivity.home ssh-keygen -Lf ssh_host_ecdsa_key-cert.pub 
ssh_host_ecdsa_key-cert.pub:
        Type: ecdsa-sha2-nistp521-cert-v01@openssh.com host certificate
        Public key: ECDSA-CERT SHA256:b17eQwm1UGqUIISx1rulZt7yKypRa8zBuuBBsf7EtwU
        Signing CA: ECDSA SHA256:RElqZXAHlXvULMiwDK1OaYgQtTyxY9iLlhbctQgKRic
        Key ID: "newhost_v01"
        Serial: 0
        Valid: before 2026-01-31T00:00:00
        Principals: 
                newhost
                newhost.tinkivity.home
        Critical Options: (none)
        Extensions: (none)

Installing host certificates

At our host we need to edit the /etc/ssh/sshd_config and insert the following lines.

HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub
HostCertificate /etc/ssh/ssh_host_ecdsa_key-cert.pub
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub

Now the SSH Daemon will present the listed SSH certificates to clients. Don’t forget to restart the SSH Daemon after the configuration change.

Client Trust

As mentioned before, clients now don’t have to trust every certificate, but they only have to trust our Certificate Authority (one time). For that to happen we add the following lines into our ~/.ssh/known_hosts file.

andreas@AnDeSu16 ➜  ~ head -n 3 ~/.ssh/known_hosts 
@cert-authority *.tinkivity.home ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBABmiuAXjy7orTZPxVrzSRe73/Cbd32skx7ESOr/pDx7Sf56uimPrjpj3/iwkx7qdjSOLVNgwyfYftlJl+GOSz/teQFleLuvNOq134YJEYX7dFh5osZTGtzndRQbFOGZ/R4zGgY1I499PdQxzN0r3pWBgR1Ch9fj6PFmu8QaeqjOWXe9Yw== tinkivity ssh ca host key
@cert-authority *.tinkivity.home ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBTLDMY7h06Hcw2O7dyh9jCN+V+g17ZXSE14aSDR25nR tinkivity ssh ca host key
@cert-authority *.tinkivity.home ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCwc0SM2eEDSPE5tV5A/H8ImtTfuZupnN9EObetGvSyTyo8K/qHLm8qpdy0AJbssm5O+Sxcy0TWjV3fEHeVABnn0FS7KGVDu6RcgJSfjszmLe1L+nhYF1jtLm1tco1EMir2iyUwLNQGsBn89auSyYF/K8109ILo06a4DErKcI3hSp/itB55dws2p2XLWtvWPhFH5tp8gCSCc90DRlRiWyyrlMYxQWnJfJbNVxc5g8D9R5JT7Kj+Yj1KyTolrF75N9x53TeOrhbJu4cBkr/Inpp+uI4bz0+ZNJE5KTrtW1NvxEicFw3R2aqjtpBaIY6ZDFW3SM7zHT49MrI4Q8Vd9y4i+q5KruPMSjCUhIPB05yyWJk9vaOyNdyCgluIWOUM81Bey9b13gvSfNMAx+8O29jD5dCBaAHn6lHu+/i67tHxvR5kAgA/XXZfjNNXrAb4PcWlasOXRLNsgJCEb/DDxzNA4dVuhhdV4+KC2qZheGn6YROpBqCHCrL8ITE2hPK+j30DkFTH4jw69tThQjZZZDo9jqoI0kVpDroFskUI8fPZBZY+k7/lTUEPxH+JaDi80fNFWuROYiAwF44NCp1I7GdtqyVdU+WNUaz6NPAaKvFZqGdcwmCJqpi7yCeS4w7vERGTzQ1V4ZJZzgDUPOrthrVP4XBoJtvsBZ3l5KdAO96Ciw== tinkivity ssh ca host key

With the amendment to your ~/.ssh/known_hosts file above you should be done. If you now connect to a host matching the domain filter (*.tinkivity.home in the case above) and still get told the authenticity cannot be verified, you should be concerned for a reason 😉

Exception: create private keys for future host

In exceptional cases we can store the private key of a host in our Certificate Authority. One of those rare exceptions is when we want to build an appliance that we want to setup via an image in a one-stop-shop approach (i.e. using Crochet-FreeBSD or YOCTO). In such cases it makes sense to generate the private key(s) upfront, because the first time such device will actually boot might be somewhere in the field (where we don’t have access). What we want are 3 keys – one for each commonly accepted and considered safe cryptographic algorithms.

Do not set a password if you want the host to have the possibility to work unattended!

andreas@rootca ➜  image123.tinkivity.home ssh-keygen -t ecdsa -b 521 -C "image123 host key" -f ssh_host_ecdsa_key
andreas@rootca ➜  image123.tinkivity.home ssh-keygen -t ed25519 -C "image123 host key" -f ssh_host_ed25519_key
andreas@rootca ➜  image123.tinkivity.home ssh-keygen -t rsa -b 4096 -C "image123 host key" -f ssh_host_rsa_key

SSL cert from local ACME server for OpenWrt

There is an acme package available for OpenWrt which allows to obtain SSL certificates from Let’s Encrypt. Wouldn’t it be nice if we could have OpenWrt obtain SSL certificates from our own private ACME server (within our local network)? This blog post explains the easy steps in order to achieve just that.

What’s wrong with the default ACME package?

Actually, there is nothing wrong with the default package. However, it doesn’t allow for any ACME server URL other than Let’s Encrypt’s URL. These are our requirements:

  1. make use of the –server parameter so that we can supply the URL of our internal ACME server
  2. make use of the –ca-bundle parameter so that we can verify the SSL certificate of our internal ACME server
  3. make use of the –days parameter so that we can set automatic renewal of certificates to something (way) less than the default 60 days

What needs to change?

There is one config file that we have to customize and there is a script that we need to extend. Also, we need to install our internal root certificate so that acme.sh can verify the SSL certificate of our internal ACME server.

/etc/config/acme

As soon as you install the acme package, a default configuration file will be created. Within the acme section we need to supply our credentials (our email address). There is one cert section that we can assign a name to it, although the name doesn’t really matter as in most cases we will be ok with just having the one cert section anyway. The last 3 lines in that configuration file are our own and will not be parsed by the default acme package (we will handle that in a minute). All 3 options correspond to the command line parameters of the official acme.sh script.

config acme
        option state_dir        '/etc/acme'
        option account_email    'john.doe@example.com'
        option debug            '0'

config cert 'localserver'
        option keylength        '2048'
        option update_uhttpd    '1'
        list domains            'openwrt.local'
        list domains            'openwrt'
        option use_staging      '0'
        option enabled          '1'
        option server           'https://acme.local:8443/acme/weekly/directory'
        option cabundle         '/etc/ssl/certs/tinkivity.pem'
        option renewal_days     '3'

/usr/lib/acme/run-acme

While OpenWrt’s acme package calls an unmodified acme.sh script, it does so via a wrapper script. The name of that wrapper script is run-acme and we need to insert 9 lines of code to handle our own configuration parameters.

Find the issue_cert() subroutine and go to the first line that modifies the acme_args parameter. Now insert the following lines (see marked in red below) into the script.

    [ "$enabled" -eq "1" ] || return

    [ "$DEBUG" -eq "1" ] && acme_args="$acme_args --debug"

    local server
    local cabundle
    local renewal_days
    config_get server "$section" server
    config_get cabundle "$section" cabundle
    config_get renewal_days "$section" renewal_days
    [ -n "$cabundle" ] && acme_args="$acme_args --ca-bundle $cabundle"
    [ -n "$server" ] && acme_args="$acme_args --server $server"
    [ -n "$renewal_days" ] && acme_args="$acme_args --days $renewal_days"

    set -- $domains
    main_domain=$1

Our first 3 custom lines (lines 5, 6 and 7 above) declare a local variable within the shell script. The next 3 lines (lines 8, 9 and 10 above) use OpenWrt’s functional API to get our custom parameters from the /etc/config/acme configuration file. The last 3 lines (lines 11, 12 and 13 above) append our custom configuration values to the command line parameters that will be handed over to acme.sh.

Test run

The only thing left to do is to test our modifications…

root@OpenWrt:~# /usr/lib/acme/run-acme
acme: Running pre checks for openwrt.local.
4+0 records in
4+0 records out
Generating a RSA private key
..........................+++++
......................+++++
writing new private key to '/etc/acme/openwrt.local/openwrt.local.key.new'
acme: Running ACME for openwrt.local
acme: Using standalone mode
Standalone mode.
Create account key ok.
Registering account
Registered
ACCOUNT_THUMBPRINT='2dYMwVSlI3Ewk69JJIpdhDa63uYdmoRiAWrBP64O8zg'
Creating domain key
The domain key is here: /etc/acme/openwrt.local/openwrt.local.key
Single domain='openwrt.local'
Getting domain auth token for each domain
Getting webroot for domain='openwrt.local'
Verifying: openwrt.local
Standalone mode server
Success
Verify finished, start to sign.
Lets finalize the order, Le_OrderFinalize: https://acme.local:8443/acme/weekly/order/mdApyrB1mSTDfrG0oJP97hCfo9MapD4g/finalize
Download cert, Le_LinkCert: https://acme.local:8443/acme/weekly/certificate/zT0aO2NdTXhIJZXE8DsSGdw1d6dw0Pbs
Cert success.
-----BEGIN CERTIFICATE-----
MIIFrDCCA5SgAwIBAgIQbFGdrtEr8eQq2McxMc2+ojANBgkqhkiG9w0BAQsFADCB
...
<< REDACTED >>
...
DPnJoXMPK0WR2dkRHtpZDQ==
-----END CERTIFICATE-----
Your cert is in  /etc/acme/openwrt.local/openwrt.local.cer 
Your cert key is in  /etc/acme/openwrt.local/openwrt.local.key 
The intermediate CA cert is in  /etc/acme/openwrt.local/ca.cer 
And the full chain certs is there:  /etc/acme/openwrt.local/fullchain.cer 
acme: Running post checks (cleanup).

IKEv2 VPN on OpenWrt – Part I

This blog post is the first in a series of three blog posts. Obviously we talk about VPN and somehow we involve a device that runs OpenWrt. To be more clear on what exactly we’re trying to achieve by using which means, let’s elaborate our challenge and solution proposal thereof.

The big picture

We have “somebody” that likes to connect to a network from a remote location. That person will use a device such as a computer, laptop, tablet or mobile phone. Often throughout this blog post we will refer to this person as Road Warrior. The network that the road warrior wants to access can be a home or office network, but in either event is reachable via the public internet and it lives behind a router with a firewall. In this blog post we call that router a Gateway. In terms of virtual private network communication in such scenario we will assign the road warrior the role of an Initiator while the gateway serves as a Responder.

There might be many solutions to the challenge posted via the general setup as described above. We will solve it with a particular architecture and plan to employ the following components and methods:

  • OpenWrt (as network router)
  • strongSwan (as VPN responder)
  • OSX / iOS (as VPN initiator)
  • IKEv2 (for key exchange)
  • EAP-TLS (for authentication)

This 3 Part tutorial will show how the complete setup and configuration on a step by step basis. In Part I (this part) we will show how to setup the VPN server and how to establish a simple connection from a client using a username/password authentication. In Part II we will show how to create certificates that are needed for EAP-TLS and in Part III we talk about site-to-site VPN.

Assumptions

We will assume a moderate knowledge on OpenWrt and general networking. That said, we will not go into details on how to install OpenWrt onto a router and we will also not explain how to install a package onto an OpenWrt router or even how to custom build OpenWrt from scratch. If that should be a concern, please follow the instructions from the OpenWrt user guide or for more advanced topics please look up the developer guide.

Furthermore we assume the following network setup:

[user device] <---> [public internet] <---> [router] <---> [target network]

In our case we will use 2 different user devices. One user device will be a laptop running OSX and the other device will be a phone running IOS.

Devices running Windows, Android, Linux, etc., equally work without problems, but are not used as examples in our blog post.

Although we will ultimately use EAP-TLS for authentication, we will use EAP-MSCHAPv2 in the first part of our blog post. The main reason is that we try with a much simpler username/password authentication in a first step for easier debugging. Once that works, we can move on to the more complex client certificates.

Preconditions

In my case I decided to custom build OpenWrt for my specific router, because a custom build can yield a much smaller footprint. In my build configuration I selected the following packages to be built into the system.

CONFIG_PACKAGE_strongswan=y
CONFIG_PACKAGE_strongswan-charon=y
CONFIG_PACKAGE_strongswan-default=y
CONFIG_PACKAGE_strongswan-ipsec=y
CONFIG_PACKAGE_strongswan-libtls=y
CONFIG_PACKAGE_strongswan-mod-aes=y
CONFIG_PACKAGE_strongswan-mod-attr=y
CONFIG_PACKAGE_strongswan-mod-connmark=y
CONFIG_PACKAGE_strongswan-mod-constraints=y
CONFIG_PACKAGE_strongswan-mod-des=y
CONFIG_PACKAGE_strongswan-mod-dnskey=y
CONFIG_PACKAGE_strongswan-mod-eap-identity=y
CONFIG_PACKAGE_strongswan-mod-eap-mschapv2=y
CONFIG_PACKAGE_strongswan-mod-eap-tls=y
CONFIG_PACKAGE_strongswan-mod-fips-prf=y
CONFIG_PACKAGE_strongswan-mod-gmp=y
CONFIG_PACKAGE_strongswan-mod-hmac=y
CONFIG_PACKAGE_strongswan-mod-kernel-netlink=y
CONFIG_PACKAGE_strongswan-mod-md4=y
CONFIG_PACKAGE_strongswan-mod-md5=y
CONFIG_PACKAGE_strongswan-mod-nonce=y
CONFIG_PACKAGE_strongswan-mod-openssl=y
CONFIG_PACKAGE_strongswan-mod-pem=y
CONFIG_PACKAGE_strongswan-mod-pgp=y
CONFIG_PACKAGE_strongswan-mod-pkcs1=y
CONFIG_PACKAGE_strongswan-mod-pubkey=y
CONFIG_PACKAGE_strongswan-mod-random=y
CONFIG_PACKAGE_strongswan-mod-rc2=y
CONFIG_PACKAGE_strongswan-mod-resolve=y
CONFIG_PACKAGE_strongswan-mod-revocation=y
CONFIG_PACKAGE_strongswan-mod-sha1=y
CONFIG_PACKAGE_strongswan-mod-sha2=y
CONFIG_PACKAGE_strongswan-mod-socket-default=y
CONFIG_PACKAGE_strongswan-mod-sshkey=y
CONFIG_PACKAGE_strongswan-mod-stroke=y
CONFIG_PACKAGE_strongswan-mod-updown=y
CONFIG_PACKAGE_strongswan-mod-x509=y
CONFIG_PACKAGE_strongswan-mod-xauth-generic=y
CONFIG_PACKAGE_strongswan-mod-xcbc=y
CONFIG_PACKAGE_strongswan-pki=y

Note that there are other packages that these packages might depend on. As explanations in custom building OpenWrt is not the scope of this blog post, please make sure all dependencies are satisfied (i.e. use make menuconfig to automatically resolve dependencies).

You don’t have to custom build, but can use an off-the-shelf OpenWrt image and simply install the strongSwan packages listed above.

The target network is one of potentially many networks that are routed by the router. In our case we have a network called lab, that is separated from other networks. We want road warriors to be able to get to the lab network.

Network setup

Understanding our configuration later on requires to understand our network setup on the router in the first place. In our setup we have 3 networks, all of which have an active DHCP server that hands out ip addresses to clients.

        Main Network: [10.10.10.0/24]
Road Warrior Network: [10.10.97.0/28]
         Lab Network: [10.10.90.0/24]

The main network is the default network that would also exist if there would be only one network setup. Assuming we have a home network, this is where your computers and laptops go. Devices in this network are generally trusted and allowed to see each other. Also, the main network most likely will have a route to the internet as well as all other networks configured. In our case the main network is connected to all lan ports of the router and which ever device that plugs into the router via cable will get an ip address in the main network.

The road warrior network only serves one purpose, that is to assign an ip address to incoming road warriors. Road warriors might be allowed to see each other – or not – depending on your preference. Also, the firewall as well as the ipsec settings will determine into which other networks road warriors are allowed to get into them. In our case we will allow road warriors to access the lab network, but deny them the main network. Also we will allow road warriors to see each other.

The lab network is an isolated network that may host a couple of servers and some clients. Either way, the lab network doesn’t route into the main network but might have internet access. In our case the lab network cannot go anywhere and it is isolated from the rest of the world.

strongSwan configuration

More or less there are 3 different configuration files that we need to adopt. Let’s start with the easiest modification.

/etc/strongswan.d/charon/attr.conf

In this file we need to add 2 lines for DNS. We make use of two different IKEv2 configuration payload attribute types in order to tell incoming road warrior which DNS server to use for our target network(s). The ip address 10.10.97.1, that points to the DNS server of the road warrior network, is declared in line 5 by using attribute type 3 (INTERNAL_IP4_DNS). The domain name of our local network, for which our internal DNS server should be used, is declared in line 6 by using attribute type 25 (INTERNAL_DNS_DOMAIN).

In summary, the road warrior network gateway assumes the DNS server function in our setup and we tell road warriors that connect via VPN to submit all DNS queries that match our domain (locallab.home) to it.

attr {
    3  = 10.10.97.1
    25 = locallab.home
    load = yes
}

/etc/ipsec.secrets

In the first line we declare the type and name of the private key for the server certificate that will be offered to VPN clients as part of the key exchange. The key is in a file named servercert.key.pem and should be treated as an RSA key. The key has no password. The key is located in the default folder for private keys that strongSwan looks into – in our case /etc/ipsec.d/private, thus the full path to the key is /etc/ipsec.d/private/servercert.key.pem. The second line defines a user by the name of myuser with the password mypassword.

: RSA servercert.key.pem
myuser : EAP "mypassword"

/etc/ipsec.conf

Line 1 declares a section for general configuration parameters. We don’t have any of those, so we leave the block empty in our config.

Line 3 declares the default configuration parameter block that all specific connections (i.e. also our connection in line 16) will inherit from. Let’s look at every line in that block in more detail:

Line 4 says that we want to use an ikev2 key exchange. In line 5 we define the cipher suites to be used for our IKE and ISAKMP security associations and in line 6 we do the same for ESP.

Before we continue to the next line, we need to understand the concept of left and right parameters for strongSwan. In easy words, we do have two communication partners and thus 2 sides of our communication. More or less you can think of the left side of the local side and the right side of the remote side. The config below is for our VPN responder and we do consider it the local config in this context. That’s why ‘left’ parameters refer to the settings on our OpenWrt router.

Line 7 defines the public network interface for our router. We don’t have a static ip address, so we leave it as %any4, which signifies an IPv4 address to be filled in during negotiation.

Line 8 defines that our router will be authenticated via public key cryptography – or in layman’s terms: our router will present a server certificate to the client.

Line 9 defines the file name of the server certificate.

Line 10 specifies how the server (our router) should be identified for authentication. We will talk about certificates in Part II, so for now we leave it like that.

Line 11 specifies the public ip address for our road warriors. At this point we don’t know from where they’re coming, so we do not impose filters such as a specific geographic ip address regions or domain names. We exclude IPv6 addresses and limit our road warriors to use IPv4 addresses only.

Line 12 specifies the ip address to be used in a tunnel for the road warrior. We define a pool of addresses in the small subnet 10.10.97.0/29. We exclude the first host address (10.10.97.1) as it would collide with the gateway. We also exclude the broadcast address (10.10.97.7). In summary a road warrior will be assigned one ip address between 10.10.97.2 and 10.10.97.6.

Line 13 defines the identity a client uses to reply to an EAP Identity request. In our case we set it to %identity and by that indicate to the EAP identity method that during EAP authentication the client should be asked to provide an EAP identity.

Line 14 tells the IPSec daemon to load this configuration automatically during startup via the add parameter. In other scenarios such as site-to-site VPN tunnels we could use start as a parameter and that way tell the IPSec daemon to immediately start establish the VPN tunnel upon daemon initialization.

Line 16 starts a new configuration block that contains all the settings for our specific connection which we arbitrary name rwMSCHAPV2.

In line 17 we tell the IPSec daemon that incoming road warriors should always (as in automatically) be sent our server certificate. Although part of IKEv2 would be for the client to ask for the certificate explicitly, some clients don’t ask and just expect the certificate, which is why we set it up like that.

In line 18 we define to which subnets the incoming road warrior should gain access to. If we would define 0.0.0.0/0 here, our road warriors would send all traffic through the VPN tunnel and thus our router. What we want is a so called split-tunnel, hence we provide a comma separated list of sub networks which a road warrior should route through the VPN tunnel. Note that the road warrior subnet itself includes all 6 possible hosts. We excluded the gateway (10.10.97.1) from the road warrior address pool but we include it in the routing table for the road warrior. Otherwise road warriors would not be able to route traffic in that subnet.

In line 19 we define the identity that a client must use for authentication. For a username/password authentication the identity can be almost any string, as long as it matches responder and initiator configuration. In other authentication mechanisms (i.e. eap-tls the identity must be confirmed via certificate).

Finally, line 20 defines that this connection shall employ MSCHAPv2 as authentication method. Again, other methods such as pubkey or eap-tls could be used instead, but are not in scope of Part I (this part) of the blog post.

config setup

conn %default
        keyexchange=ikev2
        ike=aes256-aes128-sha256-sha1-modp3072-modp2048
        esp=aes128-aes256-sha256-modp3072-modp2048,aes128-aes256-sha256
        left=%any4
        leftauth=pubkey
        leftcert=servercert.cert.pem
        leftid=vpn.example.com
        right=%any4
        rightsourceip=10.10.97.2-10.10.97.6
        eap_identity=%identity
        auto=add

conn rwMSCHAPV2
        leftsendcert=always
        leftsubnet=10.10.97.0/29,10.10.90.0/24
        rightid=vpn-via-mschapv2
        rightauth=eap-mschapv2

Network configuration

Obviously we need to configure 2 extra networks – one for our road warriors and another one as our target.

/etc/config/network

We add 2 new interfaces, which we both declare as a virtual lan.

The vpn interface will point to VLAN 97 and be defined as a 29 bit network, which provides for a total of 6 hosts. That number will be more than enough as a) our home network will not have more than 2 or 3 parallel VPN tunnels established concurrently and b) the hardware we’re using will not have enough computing power for the cryptographic operations needed to sustain 10s of VPN tunnels at the same time.

The lab interface will point to VLAN 90 and be defined as a 24 bit network, which provides for a total of 254 hosts. I doubt we will have that many hosts connected at home, but we don’t need to limit the amount of supported hosts for performance reasons either.

config interface 'vpn'
        option stp '1'
        option type 'bridge'
        option ifname 'eth0.97'
        option proto 'static'
        option ipaddr '10.10.97.1'
        option netmask '255.255.255.248'
        option delegate '0'

config interface 'lab'
        option stp '1'
        option type 'bridge'
        option ifname 'eth0.90'
        option proto 'static'
        option ipaddr '10.10.90.1'
        option netmask '255.255.255.0'
        option delegate '0'

/etc/config/dhcp

We add 2 dhcp configuration blocks. The first block is for our vpn network and we define that no DHCP server should be started in that network, as IPSec takes care of assigning IP addresses to road warriors. The second block defines a DHCP server for our lab network.

config dhcp 'vpn'
        option interface 'vpn'
        option ignore '1'

config dhcp 'lab'
        option start '100'
        option leasetime '12h'
        option limit '150'
        option interface 'lab'

Firewall configuration

Having IPSec setup is not enough, as road warriors will bounce at the firewall before even getting to the IPSec daemon. Again, there are 3 configuration files that we need to adopt to get the firewall in order.

/etc/config/firewall

We have to add a number of blocks into the existing firewall configuration. The brief summary is as follows:

We create 2 new zones – one for our vpn network and another one for the lab network (lines 1-13). We allow traffic from our vpn zone to be forwarded to the lab zone (lines 14-16). We add 4 rules that open up our wan zone for incoming VPN connections.

config zone
        option name             vpn
        list network            'vpn'
        option input            ACCEPT
        option output           ACCEPT
        option forward          ACCEPT

config zone               
        option name             lab
        list network            'lab'    
        option input            ACCEPT    
        option output           ACCEPT
        option forward          REJECT

config forwarding
        option src              vpn
        option dest             lab

config rule                                            
        option name             Allow-IPSec-ESP        
        option src              wan                    
        option proto            esp                    
        option target           ACCEPT                 
                                                       
config rule                                            
        option name             Allow-ISAKMP           
        option src              wan                    
        option dest_port        500                    
        option proto            udp                    
        option target           ACCEPT                 
                                                       
config rule                                            
        option name             Allow-IPSec-NAT        
        option src              wan                    
        option dest_port        4500                   
        option proto            udp                    
        option target           ACCEPT                 
                                                       
config rule                                            
        option name             Allow-IPSec-AH         
        option src              wan                    
        option proto            ah                     
        option target           ACCEPT                 

/etc/firewall.user

We have to accept input, forward and output traffic originated from and directed to clients matching an IPsec policy. If we want to be able to reach or ping road warriors, we need the rule in line 5, as it exempts traffic that matches an IPsec policy from being NAT-ed before tunneling. Optionally, we can setup NAT (or SNAT) as shown in line 6 in order for our road warriors to be able to access the internet.

iptables -I INPUT  -m policy --dir in --pol ipsec --proto esp -j ACCEPT
iptables -I FORWARD  -m policy --dir in --pol ipsec --proto esp -j ACCEPT
iptables -I FORWARD  -m policy --dir out --pol ipsec --proto esp -j ACCEPT
iptables -I OUTPUT   -m policy --dir out --pol ipsec --proto esp -j ACCEPT
iptables -t nat -I POSTROUTING -m policy --pol ipsec --dir out -j ACCEPT
iptables -t nat -I POSTROUTING -s 10.10.97.0/29 -o wan -j MASQUERADE

Putting it all together

First of all, let’s have a quick look again at all the files that we’ve had to create or modify in order to get our setup done. Please note that there might be other directories that would fall into the tree structure shown below, but for easier reading all those are not listed. Below only shows those directories and files we had to modify. Also, in this part of the blog post we will not talk about the certificates, but we will magically assume those to be in place and correctly setup. We will talk about certificates in Part II.

└── etc
    ├── config
    │   ├── dhcp
    │   ├── firewall
    │   └── network
    ├── firewall.user
    ├── ipsec.conf
    ├── ipsec.d
    │   ├── cacerts
    │   │   ├── rootca.pem
    │   │   └── intermediateca.pem
    │   ├── certs
    │   │   └── servercert.cert.pem
    │   └── private
    │       └── servercert.key.pem
    ├── ipsec.secrets
    └── strongswan.d
        └── charon
            └── attr.conf

IPSec Status

The first thing we do is to login to our router on a local interface via ssh (or alternatively a serial console). The IPSec daemon should be started automatically upon boot, otherwise a quick /etc/init.d/ipsec start command will do the trick. Time to use the ipsec statusall command to see what’s going on.

root@OpenWrt:~# ipsec statusall
Status of IKE charon daemon (strongSwan 5.8.2, Linux 4.14.209, mips):
  uptime: 84 seconds, since Jan 01 19:10:14 2021
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 0
  loaded plugins: charon aes des rc2 sha2 sha1 md4 md5 random nonce x509 revocation constraints pubkey pkcs1 pgp dnskey sshkey pem openssl fips-prf gmp xcbc hmac attr kernel-netlink resolve socket-default connmark stroke updown eap-identity eap-mschapv2 eap-tls xauth-generic
Virtual IP pools (size/online/offline):
  10.10.97.2-10.10.97.6: 5/0/0
Listening IP addresses:
  10.10.10.1
  10.10.90.1
  10.10.97.1
  xxx.xxx.xxx.xxx
Connections:
  rwMSCHAPV2:  %any4...%any4  IKEv2
  rwMSCHAPV2:   local:  [vpn.example.com] uses public key authentication
  rwMSCHAPV2:    cert:  "C=DE, ST=Saxony, L=Dresden, STREET=Musterstrasse 1, 01234 Dresden, Germany, O=Tinkivity, OU=vpn gateways via manual vpngate-prov, CN=vpn.example.com"
  rwMSCHAPV2:   remote: [vpn-via-mschapv2] uses EAP_MSCHAPV2 authentication with EAP identity '%any'
  rwMSCHAPV2:   child:  10.10.97.0/29 10.10.90.0/24 192.168.1.0/24 === dynamic TUNNEL
Security Associations (0 up, 0 connecting):
  none

Based on a good understanding of the configuration discussed in this blog post so far, most of the lines shown in the status should start to make real sense now. Line 12 is special, as it will not show triple-x pairs in reality, but show the public ip address of your wan port on your router.

Client configuration

Let’s use OSX’s build in network manager to configure a new VPN. Go to System Preferences –> Network and click the + (plus sign) button to configure a new VPN connection. Select VPN from the drop down list of interfaces and create a service as follows:

   Interface: VPN
    VPN Type: IKEv2
Service Name: myFirstVPN

Click the Create button and you will see a new connection that awaits some more configuration. Please key in the following values:

Server Address: xxx.xxx.xxx.xxx
     Remote ID: vpn.example.com
      Local ID: vpn-via-mschapv2

For the server address you can either type in the public ip address (if it doesn’t change in the next 5 minutes) or a fully qualified domain name (if you have dynamic DNS setup). The Remote ID and the Local ID should ring a bell for you, as it will match leftid and rightid from the /etc/ipsec.conf we have setup earlier. Finally, you will need to click on the Authentication Settings button to open a popup menu where you type in the following values:

Authentication Settings: Username
               Username: myuser
               Password: mypassword

Username and password will obviously match our configuration in /etc/ipsec.secrets.

Making a Connection

As soon as you click that Connect button, you should get a connection. If you still have your local console on the router you can tail the router log via the following command for good debugging:

root@OpenWrt:~# logread && logread -f

Once you have successfully established a connection, you should run the ipsec statusall command again and check out what is different now.

root@OpenWrt:~# ipsec statusall
Status of IKE charon daemon (strongSwan 5.8.2, Linux 4.14.209, mips):
  uptime: 99 seconds, since Jan 01 19:10:29 2021
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 0
  loaded plugins: charon aes des rc2 sha2 sha1 md4 md5 random nonce x509 revocation constraints pubkey pkcs1 pgp dnskey sshkey pem openssl fips-prf gmp xcbc hmac attr kernel-netlink resolve socket-default connmark stroke updown eap-identity eap-mschapv2 eap-tls xauth-generic
Virtual IP pools (size/online/offline):
  10.10.97.2-10.10.97.6: 5/0/0
Listening IP addresses:
  10.10.10.1
  10.10.90.1
  10.10.97.1
  xxx.xxx.xxx.xxx
Connections:
  rwMSCHAPV2:  %any4...%any4  IKEv2
  rwMSCHAPV2:   local:  [vpn.example.com] uses public key authentication
  rwMSCHAPV2:    cert:  "C=DE, ST=Saxony, L=Dresden, STREET=Musterstrasse 1, 01234 Dresden, Germany, O=Tinkivity, OU=vpn gateways via manual vpngate-prov, CN=vpn.example.com"
  rwMSCHAPV2:   remote: [vpn-via-mschapv2] uses EAP_MSCHAPV2 authentication with EAP identity '%any'
  rwMSCHAPV2:   child:  10.10.97.0/29 10.10.90.0/24 192.168.1.0/24 === dynamic TUNNEL
Security Associations (1 up, 0 connecting):
  rwMSCHAPV2[1]: ESTABLISHED 2 seconds ago, xxx.xxx.xxx.xxx[vpn.example.com]...192.168.0.2[vpn-via-mschapv2]
  rwMSCHAPV2[1]: Remote EAP identity: ding
  rwMSCHAPV2[1]: IKEv2 SPIs: 91b67839639696d2_i 04224a0ddb30a1e2_r*, public key reauthentication in 2 hours
  rwMSCHAPV2[1]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
  rwMSCHAPV2{1}:  INSTALLED, TUNNEL, reqid 1, ESP SPIs: cfff2908_i 0dfa6773_o
  rwMSCHAPV2{1}:  AES_CBC_256/HMAC_SHA2_256_128, 1097 bytes_i (8 pkts, 3s ago), 289 bytes_o (4 pkts, 3s ago), rekeying in 48 minutes
  rwMSCHAPV2{1}:   10.10.90.0/24 10.10.97.0/29 === 10.10.97.2/32

Congratulations – that’s it. In the next part of this blog post we will look at the certificates in more detail.

OpenWrt Mesh WLAN

What we want to do is to create an 802.11s mesh wireless lan. We will use OpenWrt (version 19.07) and B.A.T.M.A.N. (Better Approach To Mobile Adhoc Networking) to accomplish our goal. We will play it easy first and start with 2 nodes:

  • TP-Link Archer C7 AC1750 (~60€)
  • Netgear EX3700 / EX3800 (~30€)

The technical terms under which we address the 2 above nodes require some quick clarification. A device that provides a gateway into another network, like an uplink to the internet (i.e. via cable connection), can be called an exit point in a mesh setup. A device that enables clients such as mobile phones and laptops to connect to the mesh network can be called entry point in a mesh setup.

mesh setup

Having said that, the Netgear device will be mesh node with no connection to any other networks. It will only have the ability to reach other networks by going through the mesh network it is a part of. It will pose as a wireless access point and provide users with the ability to connect to “the” network. Hence, the Netgear device is a plain entry point in our mesh setup.

The TP-Link device has two roles! It will be a mesh node that, while being connected to other mesh nodes, will pose as a wireless access point. The TP-Link device will be an entry point in our mesh setup. At the same time the TP-Link device will have a connection to other networks than the mesh network. It will pose as a gateway out of the mesh network into other networks. Therefore, the TP-Link device will be an exit point as well.

Assumptions / Scope

This post assumes that you can manage to install OpenWrt onto the TP-Link and the Netgear device on your own. Also, this post doesn’t cover initial network setup for the TP-Link device in terms of WAN interface, DNS Server, DHCP Server or Firewall. It is assumed you know your way around the basic workings of how computer networks are coming together.

Network Setup

We will use the network 192.168.28.0/24 and setup the following configuration:

DeviceConfig KeyConfig Value
TP-Linkipaddr192.168.28.1
gateway
SSID
Netgearipaddr192.168.28.3
gateway192.168.28.1
SSID24TEST

TP-Link Archer C7 AC1750

While this device could of course offer wifi, in our initial setup we decide not to do that at first. The reason is that it might help understand that wifi communication for mesh and wifi communication for access points (client devices) are totally separate from on another. Actually, different radios can be used all together.

Let’s start by figuring out how many radios our device has and which frequencies are supported:

root@AC1750:~# iw phy | grep 'MHz \['
			* 2412 MHz [1] (24.0 dBm)
			* 2417 MHz [2] (24.0 dBm)
			* 2422 MHz [3] (24.0 dBm)
			* 2427 MHz [4] (24.0 dBm)
			* 2432 MHz [5] (24.0 dBm)
			* 2437 MHz [6] (24.0 dBm)
			* 2442 MHz [7] (24.0 dBm)
			* 2447 MHz [8] (24.0 dBm)
			* 2452 MHz [9] (24.0 dBm)
			* 2457 MHz [10] (24.0 dBm)
			* 2462 MHz [11] (24.0 dBm)
			* 2467 MHz [12] (disabled)
			* 2472 MHz [13] (disabled)
			* 2484 MHz [14] (disabled)
			* 5180 MHz [36] (23.0 dBm)
			* 5200 MHz [40] (23.0 dBm)
			* 5220 MHz [44] (23.0 dBm)
			* 5240 MHz [48] (23.0 dBm)
			* 5260 MHz [52] (23.0 dBm) (radar detection)
			* 5280 MHz [56] (23.0 dBm) (radar detection)
			* 5300 MHz [60] (23.0 dBm) (radar detection)
			* 5320 MHz [64] (23.0 dBm) (radar detection)
			* 5500 MHz [100] (23.0 dBm) (radar detection)
			* 5520 MHz [104] (23.0 dBm) (radar detection)
			* 5540 MHz [108] (23.0 dBm) (radar detection)
			* 5560 MHz [112] (23.0 dBm) (radar detection)
			* 5580 MHz [116] (23.0 dBm) (radar detection)
			* 5600 MHz [120] (23.0 dBm) (radar detection)
			* 5620 MHz [124] (23.0 dBm) (radar detection)
			* 5640 MHz [128] (23.0 dBm) (radar detection)
			* 5660 MHz [132] (23.0 dBm) (radar detection)
			* 5680 MHz [136] (23.0 dBm) (radar detection)
			* 5700 MHz [140] (23.0 dBm) (radar detection)
			* 5720 MHz [144] (23.0 dBm) (radar detection)
			* 5745 MHz [149] (30.0 dBm)
			* 5765 MHz [153] (30.0 dBm)
			* 5785 MHz [157] (30.0 dBm)
			* 5805 MHz [161] (30.0 dBm)
			* 5825 MHz [165] (30.0 dBm)
			* 5845 MHz [169] (disabled)
			* 5865 MHz [173] (disabled)

Well, we kind of knew upfront that the device has both a 2.4 GHz and a 5 GHz radio, but now we can explicitly see all the frequencies and channels that are supported by the device. What we need to check next is if (and how) the device supports 802.11s:

root@AC1750:~# iw phy | grep -i mesh
		 * mesh point
		 * #{ managed } <= 2048, #{ AP, mesh point } <= 8, #{ P2P-client, P2P-GO } <= 1, #{ IBSS } <= 1,
		 * mesh point
		 * #{ AP, mesh point } <= 8, #{ managed } <= 1,

Looks good. Next step is to install the B.A.T.M.A.N. kernel module as well as the full version of batctl.

root@AC1750:~# opkg update
root@AC1750:~# opkg install kmod-batman-adv
root@AC1750:~# opkg install batctl-full

We also need to install wpad-mesh, but it conflicts with wpad-basic – so we need to send that one packing upfront.

root@AC1750:~# opkg remove wpad-basic
root@AC1750:~# opkg install wpad-mesh-openssl

Wireless Config

We are going to create a mesh network with the name MyMesh and encrypt it with psk2 and ccmp. We will put our mesh network on our 5 GHz radio and create an internal network device called nwi_mesh0. We need to edit the file /etc/config/wireless and insert the following:

config wifi-iface 'wifinetmesh0'
        option device 'radio5'
        option ifname 'mesh0'
        option network 'nwi_mesh0'
        option mode 'mesh'
        option mesh_fwding '0'
        option mesh_id 'MyMesh'
        option encryption 'psk2+ccmp'
        option key 'mysecretpassword'

We have to make sure that our device name (line 2) matches that of our 5 GHz radio. In a fresh installation of OpenWrt radios are often numbered through starting from zero (radio0, radio1, etc.). I like to rename my radios to radio24 (2.4 GHz) and radio5 (5 GHz) for better readability. Anyway, the other thing that is important is the name of our internal network device (line 4) which we will create later via /etc/config/network. All other options in the configuration stanza above must be the same on every other mesh node in our mesh network! There are 3 other settings that also must be the same across all mesh nodes in the mesh network: wireless protocol, radio frequency (channel) and channel width. Below are settings for channel 48 (5240 MHz), wireless 802.11a and an 80MHz throughput.

config wifi-device 'radio5'
        option type 'mac80211'
        option channel '48'
        option hwmode '11a'
        option htmode 'VHT80'
        option path 'pci0000:00/0000:00:00.0'
 

Network Config

We need to create two interfaces and bridge them with our local area network. The first interface we need to specify is bat0, which we need to adhere to the batman advanced protocol (batadv). The second interface we need is nwi_mesh0, as this is the exact name we gave in the wireless config earlier on. That one will adhere to the batman advanced hard interface protocol (batadv_hardif). Also, because B.A.T.M.A.N. is a layer 2 protocol, we should increase the MTU above the typical value of 1500 so that we can avoid packet fragmentation. We add the following stanza in the file /etc/config/network.

config interface 'nwi_mesh0'
        option proto 'batadv_hardif'
        option mtu '2304'
        option master 'bat0'

config interface 'bat0'
        option proto 'batadv'

Last but not least we need to bridge the bat0 interface with our local area network by adding it to the interface definition of lan. We change the following line in /etc/config/network.

config interface 'lan'
        option stp '1'
        option type 'bridge'
        option ifname 'eth0.1 bat0'
        option proto 'static'
        option ipaddr '192.168.28.1'
        option netmask '255.255.255.0'
        option delegate '0'

That’s it for the TP-Link device. We should reboot to make sure all changes will be applied.

Netgear EX3700 / EX3800

Mostly we will apply the exact same setup, but with two differences:

  1. the device will not be connected to the internet, so we will need to copy the packages onto the device via scp
  2. the device will also provide a wireless access point for client devices to connect to it

Let’s start with checking the preconditions. Without showing lengthy console dumps to prove it, it is safe to say that this device also supports 802.11s as well as both a 2.4 GHz and a 5 GHz radio. On the 5 GHz radio it supports one less channel than the TP-Link device (channel 144 is missing), but other than than things look pretty compatible.

Copy & Install packages

For the packages needed, there are some dependencies that we need to supply as well. We need to copy the following packages onto the Netgear device (i.e. via scp):

batctl-full_2019.2-3_mipsel_24kc.ipk
kmod-batman-adv_4.14.171+2019.2-5_mipsel_24kc.ipk
kmod-cfg80211_4.14.171+4.19.98-1-1_mipsel_24kc.ipk
kmod-crypto-crc32c_4.14.171-1_mipsel_24kc.ipk
kmod-crypto-hash_4.14.171-1_mipsel_24kc.ipk
kmod-lib-crc16_4.14.171-1_mipsel_24kc.ipk
kmod-lib-crc32c_4.14.171-1_mipsel_24kc.ipk
libopenssl1.1_1.1.1d-2_mipsel_24kc.ipk
librt_1.1.24-2_mipsel_24kc.ipk
wpad-mesh-openssl_2019-08-08-ca8c2bd2-2_mipsel_24kc.ipk

Next we follow the exact same installation procedure as for the TP-Link device. One thing we should watch out for is flash memory… the Netgear device doesn’t offer much ‘free disk space’, so it might be wise to copy, install and delete one package as a time.

Wireless config

We actually use the exact same wireless config stanza as for the TP-Link device. As far as the radio is concerned, we have to make sure that wireless protocol, radio frequency (channel) and channel width match with the settings for the TP-Link device.

In addition we want our Netgear device to be a wireless access point. We define a new wireless access point named , so we need to define 24TEST for our 2.4 GHz radio. Note that what happens on the 2.4 GHz radio and the access point 24TEST have nothing to do with our mesh network. Both configurations are totally separate from one another. The interesting part of 24TEST though is line 21, which defines the network interface to the wireless access point.

config wifi-device 'radio5'
        option type 'mac80211'
        option channel '48'
        option hwmode '11a'
        option path 'pci0000:00/0000:00:00.0/0000:01:00.0'
        option htmode 'VHT80'

config wifi-device 'radio24'
        option type 'mac80211'
        option channel '11'
        option hwmode '11g'
        option path 'platform/10180000.wmac'
        option htmode 'HT40'

config wifi-iface 'wifinet24test'
        option device 'radio24'
        option mode 'ap'
        option key 'anothersecretpassword'
        option encryption 'psk2'
        option ssid '24TEST'
        option network 'lan'

config wifi-iface 'wifinetmesh0'
        option device 'radio5'
        option ifname 'mesh0'
        option network 'nwi_mesh0'
        option mode 'mesh'
        option mesh_fwding '0'
        option mesh_id 'MyMesh'
        option encryption 'psk2+ccmp'
        option key 'mysecretpassword'

Network Config

Again, we do the exact same things as for the TP-Link device in /etc/config/network.

config interface 'nwi_mesh0'
        option proto 'batadv_hardif'
        option mtu '2304'
        option master 'bat0'

config interface 'bat0'
        option proto 'batadv'

Where things look a little bit different, is the definition of the local area network (lan) in /etc/config/network. Of course we have a different IP address (line 6), but we also must declare the TP-Link device as gateway (line 8) and dns server (line 10).

config interface 'lan'
        option stp '1'
        option type 'bridge'
        option ifname 'eth0 bat0'
        option proto 'static'
        option ipaddr '192.168.28.3'
        option netmask '255.255.255.0'
        option gateway '192.168.28.1'
        option delegate '0'
        list dns '192.168.28.1'

Putting things up for a test (layer 2)

B.A.T.M.A.N. is a layer 2 protocol, which means there is a bunch of stuff that we can do without an IP address. First, we can log in to either device and the first thing we should make sure is that the mesh devices mesh0 and bat0 are up and running:

root@EX3700:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br-lan state UNKNOWN qlen 1000
    link/ether 3c:37:86:60:f7:1f brd ff:ff:ff:ff:ff:ff
5: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 3c:37:86:60:f7:1f brd ff:ff:ff:ff:ff:ff
6: mesh0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2304 qdisc noqueue master bat0 state UP qlen 1000
    link/ether 3c:37:86:60:f7:1e brd ff:ff:ff:ff:ff:ff
7: bat0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UNKNOWN qlen 1000
    link/ether 96:e1:f8:8e:fc:d7 brd ff:ff:ff:ff:ff:ff
8: wlan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-lan state UP qlen 1000
    link/ether 3c:37:86:60:f7:1f brd ff:ff:ff:ff:ff:ff
9: wlan0-1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether 3e:37:86:60:f7:1e brd ff:ff:ff:ff:ff:ff

Next we should make sure that the nodes can find each other. If the following command returns nothing, there is probably a typo in the config, otherwise you should see this:

root@EX3700:~# iw dev mesh0 station dump
Station b0:be:76:e9:90:ab (on mesh0)
	inactive time:	0 ms
	rx bytes:	12437433
	rx packets:	141145
	tx bytes:	58226
	tx packets:	363
	tx retries:	26
	tx failed:	1
	rx drop misc:	4
	signal:  	-69 [-69, -71] dBm
	signal avg:	-66 [-66, -68] dBm
	Toffset:	6797824060 us
	tx bitrate:	351.0 MBit/s VHT-MCS 4 80MHz VHT-NSS 2
	rx bitrate:	263.3 MBit/s VHT-MCS 6 80MHz VHT-NSS 1
	rx duration:	26267 us
	last ack signal:0 dBm
	expected throughput:	60.333Mbps
	mesh llid:	0
	mesh plid:	0
	mesh plink:	ESTAB
	mesh local PS mode:	ACTIVE
	mesh peer PS mode:	ACTIVE
	mesh non-peer PS mode:	ACTIVE
	authorized:	yes
	authenticated:	yes
	associated:	yes
	preamble:	long
	WMM/WME:	yes
	MFP:		yes
	TDLS peer:	no
	DTIM period:	2
	beacon interval:100
	connected time:	6559 seconds

Now we should see our mesh neighbor with batctl.

root@EX3700:~# batctl o
[B.A.T.M.A.N. adv openwrt-2019.2-5, MainIF/MAC: mesh0/3c:37:86:60:f7:1e (bat0/96:e1:f8:8e:fc:d7 BATMAN_IV)]
   Originator        last-seen (#/255) Nexthop           [outgoingIF]
 * b0:be:76:e9:90:ab    0.420s   (255) b0:be:76:e9:90:ab [     mesh0]

If we want to know how much throughput we can expect, we can run a throughput test with batctl as well. We will use the MAC address from our neighbor.

root@EX3700:~# batctl tp b0:be:76:e9:90:ab
Test duration 10430ms.
Sent 58511592 Bytes.
Throughput: 5.35 MB/s (44.88 Mbps)

Putting things up for a test (Layer 3)

Finally, we can send a regular ping to see if layer 3 is fine as well.

root@EX3700:~# ping -c 3 192.168.28.1
PING 192.168.28.1 (192.168.28.1): 56 data bytes
64 bytes from 192.168.28.1: seq=0 ttl=64 time=1.540 ms
64 bytes from 192.168.28.1: seq=1 ttl=64 time=2.160 ms
64 bytes from 192.168.28.1: seq=2 ttl=64 time=1.240 ms

--- 192.168.28.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 1.240/1.646/2.160 ms

Summary

Beyond that point you should connect to the 24TEST wifi with your mobile phone and check that things are working fine there, too.

Other than that mesh with 802.11s only starts to make sense with 3 devices and more, so while our little test setup is a nice POC, fun doesn’t begin until many more devices (2- or 3-digit numbers of devices). That said, the list of neighbors with the following command can become much (much) longer:

root@EX3700:~# batctl n
[B.A.T.M.A.N. adv openwrt-2019.2-5, MainIF/MAC: mesh0/3c:37:86:60:f7:1e (bat0/96:e1:f8:8e:fc:d7 BATMAN_IV)]
IF             Neighbor              last-seen
        mesh0	  b0:be:76:e9:90:ab    0.570s

OpenLDAP Server

Installing an OpenLDAP Server for central user management

This post explains how to setup a LDAP DNS server inside a jail and surround it with a phpLDAPadmin web UI inside that same jail. Whenever a new client is served an ip address from the DHCP server, the DHCP server will update the DNS server with the new ip address dynamically.

OpenLDAP

The first thing we need to do is to install the needed openldap-server package. Before we do that we set the time zone though:

tzsetup Europe/Berlin
pkg install openldap-server

After installation of the openldap-server package we already get two messages – one from the openldap-client package and one from the openldap-server package. Let’s look at the openldap-client message:

Message from openldap-client-2.4.45:

************************************************************

The OpenLDAP client package has been successfully installed.

Edit
  /usr/local/etc/openldap/ldap.conf
to change the system-wide client defaults.

Try `man ldap.conf' and visit the OpenLDAP FAQ-O-Matic at
  http://www.OpenLDAP.org/faq/index.cgi?file=3
for more information.

************************************************************

We will actually follow that right away and edit our ldap.conf file as follows:

BASE    dc=home,dc=local 
URI     ldap://ldap1.home.local ldap://ldap1.home.local:666

SIZELIMIT       0
TIMELIMIT       15
DEREF           never

The other message we get is from openldap-server:

Message from openldap-server-2.4.45_4:

************************************************************

The OpenLDAP server package has been successfully installed.

In order to run the LDAP server, you need to edit
  /usr/local/etc/openldap/slapd.conf
to suit your needs and add the following lines to /etc/rc.conf:
  slapd_enable="YES"
  slapd_flags='-h "ldapi://%2fvar%2frun%2fopenldap%2fldapi/ ldap://0.0.0.0/"'
  slapd_sockets="/var/run/openldap/ldapi"

Then start the server with
  /usr/local/etc/rc.d/slapd start
or reboot.

Try `man slapd' and the online manual at
  http://www.OpenLDAP.org/doc/
for more information.

slapd runs under a non-privileged user id (by default `ldap'),
see /usr/local/etc/rc.d/slapd for more information.

************************************************************

We will copy the following entries into our /etc/rc.conf:

slapd_enable="YES"
slapd_flags='-h "ldapi://%2fvar%2frun%2fopenldap%2fldapi/ ldap://0.0.0.0/"'
slapd_sockets="/var/run/openldap/ldapi"

Also, we have to seriously edit our /usr/local/etc/openldap/slapd.conf file and make sure that the following lines are in the config file:

# Include 4 important schema files at least
include         /usr/local/etc/openldap/schema/core.schema
include         /usr/local/etc/openldap/schema/cosine.schema
include         /usr/local/etc/openldap/schema/inetorgperson.schema
include         /usr/local/etc/openldap/schema/nis.schema

# Define the storage location for pid and argument file
pidfile         /var/run/openldap/slapd.pid
argsfile        /var/run/openldap/slapd.args

# set the log level so we stat log connections/operations/results
loglevel        256

# Load dynamic backend modules:
modulepath      /usr/local/libexec/openldap
moduleload      back_mdb

#######################################################################
# MDB database definitions
#######################################################################
database        mdb
maxsize         1073741824

# Define our domain suffix and root dn
suffix          "dc=home,dc=local"
rootdn          "cn=Manager,dc=home,dc=local"

# Use of strong authentication for password
rootpw        {SSHA}SYPSMKmTgbugazvkHadr47fra83ISyON
password-hash {SSHA}

# The database directory MUST exist prior to running slapd AND 
# should only be accessible by the slapd and slap tools.
# Mode 700 recommended.
directory       /var/db/openldap-data
# Indices to maintain
index   objectClass     eq

Now you might wonder where the root password comes from? Very easy – you need to create the hash using the slappasswd utility on the command line.

slappasswd -h "{SSHA}"

You will be asked to enter a password and in return you will receive a strong hash for your password (in the example above the password test123 has produced the hash SYPSMKmTgbugazvkHadr47fra83ISyON).

Now we can already start our OpenLDAP server using the following command:

service slapd start

If we look into /var/log/messages we can see the correct start of the server. However, ideally we want a separate log file just for OpenLDAP. We can create that logfile by adding the following two lines into the /etc/syslog.conf file:

!slapd
*.*                                             /var/log/slapd.log

We need to create the empty log file and restart the syslogd daemon next:

touch /var/log/slapd.log
service syslogd restart
service slapd restart

With that done we should see a separate log file with the following content:

Mar 18 11:07:28 ldap slapd[54700]: @(#) $OpenLDAP: slapd 2.4.45 (Mar 15 2018 21:57:40) $        root@111amd64-default-job-17:/wrkdirs/usr/ports/net/openldap24-server/work/openldap-2.4.45/servers/slapd
Mar 18 11:07:28 ldap slapd[54701]: slapd starting

Finally we can create ourselves a small LDIF file and import that into our database.

dn: dc=home,dc=local
objectclass: dcObject
objectclass: organization
o: home
dc: home

dn: cn=Manager,dc=home,dc=local 
objectclass: organizationalRole
cn: Manager

When importing the file you will be asked for a password. The password you have to enter is the same you have put as a hash into the /usr/local/etc/openldap/slapd.conf file (in our example we used test123).

cd /usr/local/etc/openldap
ldapadd -D "cn=Manager,dc=home,dc=local" -W -f import.ldif

Congratulations! Our OpenLDAP server is setup and ready to be used. Special note here: in case you need a SAMBA schema you will note the schema files are not provided by the OpenLDAP installation. Don’t worry, you can find the schema here: https://raw.githubusercontent.com/samba-team/samba/master/examples/LDAP/samba.schema

Nginx

We actually would like to have a web frontend for our OpenLDAP server. One easy way is to install phpLDAPadmin which requires Nginx and PHP. Let’s start by installing Nginx:

pkg install nginx

Right after installation we can enable the service in /etc/rc.conf

sysrc nginx_enable=YES
nginx_enable:  -> YES

After that we can start the service and test it:

service nginx start
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Starting nginx.

If you have a SSL certificate you need to put the following into your server section of your nginx.conf file:

    server {
        listen          443 ssl;
        server_name     ldap1 ldap1.home.local;

        ssl_certificate         ldap1.home.local.bundle.pem;
        ssl_certificate_key     ldap1.home.local.key.pem;

Above will take care of serving content via https. In addition you could redirect all http traffic to https by putting in an additional server section:

    server {
        listen          80 default;
        server_name     ldap1 ldap1.home.local;
        access_log      off;
        error_log       off;
        ## redirect http to https ##
        return          301 https://$server_name$request_uri;
    }

PHP

For phpLDAPadmin we are going to install version 5.6 of PHP. Let’s install the package via package manager:

pkg install php56

First of all we need a php.ini file. We can just copy the production example that comes along with the package:

cp /usr/local/etc/php.ini-production /usr/local/etc/php.ini

In the php.ini file we should apply the following fix:

;cgi.fix_pathinfo=1
cgi.fix_pathinfo=0

Next we need to adapt /usr/local/etc/php-fpm.conf to setup fpm for our needs. Find the line where it says at which port to listen and replace that line with an instruction to use a socket file instead.

;listen = 127.0.0.1:9000
listen = /var/run/php-fpm.sock

In the same file find the three lines about listener ownership and uncomment them. Please note that you have to change the listen.mode from 0660 to 0666 as otherwise phpLDAPadmin will complain about missing _SESSION variables (don’t ask me why… didn’t further investigate this).

listen.owner = www
listen.group = www
listen.mode = 0666

Next we have to take care of a couple of things in the Nginx configuration. Edit your nginx.conf file to replace the location section as follows:

#location / {
#    root   /usr/local/www/nginx;
#    index  index.html index.htm;
#}

location / {
    try_files $uri $uri/ =404;
}

root /usr/local/www/nginx;
index index.php index.html index.htm;

location ~ \.php$ {
    try_files $uri =404;
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    fastcgi_pass unix:/var/run/php-fpm.sock;
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME $request_filename;
    include fastcgi_params;
}

Now let’s enable the php fpm service.

sysrc php_fpm_enable=YES

php_fpm_enable:  -> YES

And of course, let’s start the php fpm service.

service php-fpm start

Performing sanity check on php-fpm configuration:
[25-Mar-2018 13:09:41] NOTICE: configuration file /usr/local/etc/php-fpm.conf test is successful

Starting php_fpm.

Last but not least we need to restart Nginx.

service nginx restart

Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Stopping nginx.
Waiting for PIDS: 30037.
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Starting nginx.

phpLDAPadmin

Let’s install the package via package manager:

pkg install phpldapadmin

The package manager will tell you that it installed phpLDAPadmin into folder /usr/local/www/phpldapadmin. As a first activity, we need to edit the config.php configuration file as we need to setup our LDAP connection details. For that please change the following lines as follows:

// $servers->setValue('server','host','127.0.0.1');
$servers->setValue('server','host','ldap://ldap1.home.local');

// $servers->setValue('server','port',389);
$servers->setValue('server','port',389);

// $servers->setValue('login','auth_type','session');
$servers->setValue('login','auth_type','session');

Eventually we have to edit our nginx.conf again as we need to point the web server root to phpLDAPadmin.

#root /usr/local/www/nginx;
root /usr/local/www/phpldapadmin/htdocs;

For the change to take effect please restart Nginx:

service nginx restart

When creating new posix accounts we should make sure that user ids do not collide with local system users. The way we do it is to make sure new users are being created with ids of 10.000 and above. We will edit the posixAccount.xml template that phpLDAPadmin provides for posix account creation. Specifically we will override the value for the uidNumber attribute:

        UID Number
        terminal.png
        6
        1
        1
<!--    <value>=php.GetNextNumber(/;uidNumber)</value> -->
        =php.GetNextNumber(/;uidNumber;;;;10000)

Neat trick 😉 …now we do the same thing for the gidNumber attribute in the posixGroup.xml template:

        GID Number
        2
        1
        1
        1
<!--    <value>=php.GetNextNumber(/;gidNumber)</value> -->
        =php.GetNextNumber(/;gidNumber;;;;10000)

Taking things for a spin

Last but not least we should check that we can do an ldap search from a remote host somewhere. We use the following command:

ldapsearch -x -b 'dc=home,dc=local' -h ldap.home.local -D 'cn=myuser,ou=users,dc=home,dc=local' -W

If things go to plan you are being prompted for the password of your user (“myuser” in the example above). The query should return something ending with these lines:

# search result
search: 2
result: 0 Success

The importing thing in the result above being the result code 0 Success.

Bonus: Change DN layout for new users

Right now phpLDAPadmin creates new users with a distinguished name that is driven by cn, however we might want users to become created driven by uid. Changing that requires us to go back to the posixAccount.xml template. There we need to do one more so phpLDAPadmin will create new users with a DN driven by uid:

<!-- <rdn>cn</rdn> -->
uid

Why do we do that? Consider this:

dn: cn=sample user,ou=users,dc=home,dc=local
objectClass: top
objectClass: inetOrgPerson
cn: sample user
uid: sampleuser

but on the other hand consider this:

dn: uid=sampleuser,ou=users,dc=home,dc=local
objectClass: top
objectClass: inetOrgPerson
cn: sample user
uid: sampleuser

Even though the attributes are identical, the DN is the primary key and the entries given above, are two complete separate entries with two different DNs. Now if we want to login via uid later on, rather than cn, we need that fix.

Certificate Authority

Creating a local certificate authority

This post explains how to setup a local CA (certificate authority) from which to issue server certificates for a local network.

Our Setup

At home we also want to have SSL certificates for our servers, so we can leverage encryption of http traffic and turn it into https. Also, some infrastructure servers (i.e. GIT) will rely on https, so it will be a good idea to be prepared. Our setup will consist of three layers:

Root CA <-> Intermediate CA <-> Servers

The general reason we don’t issue server certificates directly from the root CA is that it is 1) rather uncommon to do so and 2) we might end up with the need for more than one CA to issue server certificates in our network (i.e. using lets encrypt)

Root CA

The first thing we need to do is to setup a root CA which more or less is nothing more than a structure of directories on any host. The host actually doesn’t need to be a server and literally can be your laptop. No services need to be started and in reality the root CA often is an air gapped computer.

mkdir /root/ca
cd /root/ca
mkdir certs crl newcerts private
chmod 700 private
touch index.txt
echo 1000 > serial

Now we need to create a config file for openssl

touch caconfig.cnf

The file content is shown below. Please note that the root directory in line 7 of the file needs to match the directory on your drive. If you decide to create the directory somewhere else, please adapt line 7. The same goes for the subdirectories between line 8 and line 21.

[ ca ]
# `man ca`
default_ca = CA_default

[ CA_default ]
# Directory and file locations.
dir               = /root/ca
certs             = $dir/certs
crl_dir           = $dir/crl
new_certs_dir     = $dir/newcerts
database          = $dir/index.txt
serial            = $dir/serial
RANDFILE          = $dir/private/.rand

# The root key and root certificate.
private_key       = $dir/private/ca.key.pem
certificate       = $dir/certs/ca.cert.pem

# For certificate revocation lists.
crlnumber         = $dir/crlnumber
crl               = $dir/crl/ca.crl.pem
crl_extensions    = crl_ext
default_crl_days  = 30

# SHA-1 is deprecated, so use SHA-2 instead.
default_md        = sha256

name_opt          = ca_default
cert_opt          = ca_default
default_days      = 375
preserve          = no
policy            = policy_strict

[ policy_strict ]
# The root CA should only sign intermediate certificates that match.
# See the POLICY FORMAT section of `man ca`.
countryName             = match
stateOrProvinceName     = match
organizationName        = match
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ policy_loose ]
# Allow the intermediate CA to sign a more diverse range of certificates.
# See the POLICY FORMAT section of the `ca` man page.
countryName             = optional
stateOrProvinceName     = optional
localityName            = optional
organizationName        = optional
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ req ]
# Options for the `req` tool (`man req`).
default_bits        = 2048
distinguished_name  = req_distinguished_name
string_mask         = utf8only

# SHA-1 is deprecated, so use SHA-2 instead.
default_md          = sha256

# Extension to add when the -x509 option is used.
x509_extensions     = v3_ca

[ req_distinguished_name ]
# See .
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name
localityName                    = Locality Name
0.organizationName              = Organization Name
organizationalUnitName          = Organizational Unit Name
commonName                      = Common Name
emailAddress                    = Email Address

# Optionally, specify some defaults.
countryName_default             = DE
stateOrProvinceName_default     = Saxony
localityName_default            = Dresden
0.organizationName_default      = Example
organizationalUnitName_default  = Example Certificate Authority
emailAddress_default            = john.doe@example.com

[ v3_ca ]
# Extensions for a typical CA (`man x509v3_config`).
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ v3_intermediate_ca ]
# Extensions for a typical intermediate CA (`man x509v3_config`).
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ v3_intermediate_ca ]
# Extensions for a typical intermediate CA (`man x509v3_config`).
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ usr_cert ]
# Extensions for client certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = client, email
nsComment = "OpenSSL Generated Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, emailProtection

[ server_cert ]
# Extensions for server certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

[ crl_ext ]
# Extension for CRLs (`man x509v3_config`).
authorityKeyIdentifier=keyid:always

[ ocsp ]
# Extension for OCSP signing certificates (`man ocsp`).
basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, digitalSignature
extendedKeyUsage = critical, OCSPSigning

Now it is time to create the root key. Because we are creating a private key for a root CA we should use 4096 bit encryption.

cd /root/ca
openssl genrsa -aes256 -out private/ca.key.pem 4096

You will be prompted to give a password:

Enter pass phrase for ca.key.pem: mysecretpassword
Verifying - Enter pass phrase for ca.key.pem: mysecretpassword

After the certificate is created we should make sure nobody else can read it on the filesystem.

chmod 400 private/ca.key.pem

Now we create a signing request for a certificate that is valid for 20 years. We also implicitly self sign the request:

cd /root/ca
openssl req -config caconfig.cnf -key private/ca.key.pem -new -x509 -days 7300 -sha256 -extensions v3_ca -out certs/ca.cert.pem


When we are being prompted for the password, we shall give the private key password from above.

Enter pass phrase for ca.key.pem: mysecretpassword
You are about to be asked to enter information that will be incorporated
into your certificate request.
-----
Country Name (2 letter code) [XX]:DE
State or Province Name []:Saxony
Locality Name []: Dresden
Organization Name []:Example
Organizational Unit Name []:Example Certificate Authority
Common Name []:Example Root CA
Email Address []:john.doe@example.com

The public certificate can be read by everybody, as long as nobody can write it. If desired we can also use openssl to marvel at our fresh certificate and verify it.

chmod 444 certs/ca.cert.pem
openssl x509 -noout -text -in certs/ca.cert.pem

Intermediate CA

Same as for the root CA, we now setup an intermediate CA. The intermediate CA could be at the same host, but it really doesn’t have to be though.

mkdir /root/ca/ca-intermediate
cd /root/ca/ca-intermediate
mkdir certs crl csr newcerts private
chmod 700 private
touch index.txt
echo 1000 > serial
echo 1000 > crlnumber

Again, we need to create a config file for openssl

touch ca-intermediate.cnf

The file content is shown below. Not much has changed compared to the root CA. The policy we apply is now loose (see line 32) and we copy possible extensions (see line 35). Again, the root directory in line 7 of the file needs to match the directory on your drive. If you decide to create the directory somewhere else, please adapt line 7. The same goes for the subdirectories between line 8 and line 21.

[ ca ]
# `man ca`
default_ca = CA_default

[ CA_default ]
# Directory and file locations.
dir               = /root/ca/ca-intermediate
certs             = $dir/certs
crl_dir           = $dir/crl
new_certs_dir     = $dir/newcerts
database          = $dir/index.txt
serial            = $dir/serial
RANDFILE          = $dir/private/.rand

# The root key and root certificate.
private_key     = $dir/private/intermediate.key.pem
certificate     = $dir/certs/intermediate.cert.pem

# For certificate revocation lists.
crlnumber         = $dir/crlnumber
crl               = $dir/crl/intermediate.crl.pem
crl_extensions    = crl_ext
default_crl_days  = 30

# SHA-1 is deprecated, so use SHA-2 instead.
default_md        = sha256

name_opt          = ca_default
cert_opt          = ca_default
default_days      = 375
preserve          = no
policy            = policy_loose

# <<< I M P O R T A N T >>>
copy_extensions   = copy

[ policy_strict ]
# The root CA should only sign intermediate certificates that match.
# See the POLICY FORMAT section of `man ca`.
countryName             = match
stateOrProvinceName     = match
organizationName        = match
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ policy_loose ]
# Allow the intermediate CA to sign a more diverse range of certificates.
# See the POLICY FORMAT section of the `ca` man page.
countryName             = optional
stateOrProvinceName     = optional
localityName            = optional
organizationName        = optional
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ req ]
# Options for the `req` tool (`man req`).
default_bits        = 2048
distinguished_name  = req_distinguished_name
string_mask         = utf8only

# SHA-1 is deprecated, so use SHA-2 instead.
default_md          = sha256

# Extension to add when the -x509 option is used.
x509_extensions     = v3_ca

[ req_distinguished_name ]
# See .
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name
localityName                    = Locality Name
0.organizationName              = Organization Name
organizationalUnitName          = Organizational Unit Name
commonName                      = Common Name
emailAddress                    = Email Address

# Optionally, specify some defaults.
countryName_default             = DE
stateOrProvinceName_default     = Saxony
localityName_default            = Dresden
0.organizationName_default      = Example
organizationalUnitName_default  = Example Certificate Authority
emailAddress_default            = john.doe@example.com

[ v3_ca ]
# Extensions for a typical CA (`man x509v3_config`).
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ v3_intermediate_ca ]
# Extensions for a typical intermediate CA (`man x509v3_config`).
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ usr_cert ]
# Extensions for client certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = client, email
nsComment = "OpenSSL Generated Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, emailProtection

[ server_cert ]
# Extensions for server certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

[ crl_ext ]
# Extension for CRLs (`man x509v3_config`).
authorityKeyIdentifier=keyid:always

[ ocsp ]
# Extension for OCSP signing certificates (`man ocsp`).
basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, digitalSignature
extendedKeyUsage = critical, OCSPSigning

Next we can create the intermediate private key (please note that we are still in the ca-intermediate folder):

openssl genrsa -aes256 -out private/intermediate.key.pem 4096

We are being prompted to give a password:

Enter pass phrase for intermediate.key.pem: anothersecretpassword
Verifying - Enter pass phrase for intermediate.key.pem: anothersecretpassword

We should make sure that nobody can read the private key:

chmod 400 private/intermediate.key.pem

As a next step we now create a signing request:

openssl req -config ca-intermediate.cnf -new -sha256 -key private/intermediate.key.pem -out csr/intermediate.csr.pem

As we do so, we are being prompted for the password that protects the private key of our intermediate certificate authority.

Enter pass phrase for intermediate.key.pem: anothersecretpassword
You are about to be asked to enter information that will be incorporated
into your certificate request.
-----
Country Name (2 letter code) [XX]:DE
State or Province Name []:Saxony
Locality Name []:Dresden
Organization Name []:Example
Organizational Unit Name []:Example Certificate Authority
Common Name []:Example Intermediate CA
Email Address []:john.doeh@example.com

We have successfully created a certificate signing request. Now let’s go back to the Root CA and sign the request. Let’s make sure we use the Root CA’s openssl config for doing that:

cd /root/ca
openssl ca -config caconfig.cnf -extensions v3_intermediate_ca -days 3650 -notext -md sha256 -in ca-intermediate/csr/intermediate.csr.pem -out ca-intermediate/certs/intermediate.cert.pem

The password we are being prompted for is the password of the Root CA – makes sense, because the Root CA is asked to sign something.

Enter pass phrase for ca.key.pem: mysecretpassword
Sign the certificate? [y/n]: y

It should be allowed that everybody can read the public certificate. At the same time nobody should be allowed to write it.

chmod 444 ca-intermediate/certs/intermediate.cert.pem

We could verify the intermediate certificate with the following command:

openssl x509 -noout -text -in ca-intermediate/certs/intermediate.cert.pem

However, so much more important is to verify the chain of the intermediate certificate against the root certificate:

openssl verify -CAfile certs/ca.cert.pem ca-intermediate/certs/intermediate.cert.pem

intermediate.cert.pem: OK

On that note… let’s create a certificate chain that we can later on hand out to web servers (i.e. apache has an option for that):

cat ca-intermediate/certs/intermediate.cert.pem certs/ca.cert.pem > ca-intermediate/certs/ca-chain.cert.pem
chmod 444 ca-intermediate/certs/ca-chain.cert.pem

Server Certificate

Finally, we can issue a server certificate from our intermediate CA. At first, create the server key (make sure it is 2048 bit). As we are creating a certificate for a service (i.e. Nginx or Apache) we should skip the -aes256 option as we otherwise would create a private key with a password and would need to enter the password every time we start the service.

cd /root/ca/ca-intermediate
openssl genrsa -out private/myserver.example.com.key.pem 2048
chmod 400 private/myserver.example.com.key.pem

It makes a lot of sense to create a config file for openssl!

cd /root/ca/ca-intermediate
touch myserver.cnf

And here is why: look at lines 72, 96 and 99 until 101. Our server will have more than just one simple domain name service entry. It might be available under the full name myserver.example.com but also under the short name myserver. We need alternative DNS entries for the server certificate.

[ ca ]
# `man ca`
default_ca = CA_default

[ CA_default ]
# Directory and file locations.
dir               = /root/ca/ca-intermediate
certs             = $dir/certs
crl_dir           = $dir/crl
new_certs_dir     = $dir/newcerts
database          = $dir/index.txt
serial            = $dir/serial
RANDFILE          = $dir/private/.rand

# The root key and root certificate.
private_key     = $dir/private/intermediate.key.pem
certificate     = $dir/certs/intermediate.cert.pem

# For certificate revocation lists.
crlnumber         = $dir/crlnumber
crl               = $dir/crl/intermediate.crl.pem
crl_extensions    = crl_ext
default_crl_days  = 30

# SHA-1 is deprecated, so use SHA-2 instead.
default_md        = sha256

name_opt          = ca_default
cert_opt          = ca_default
default_days      = 375
preserve          = no
policy            = policy_loose

# <<< I M P O R T A N T >>>
# Extension copying option: use with caution.
copy_extensions   = copy

[ policy_strict ]
# The root CA should only sign intermediate certificates that match.
# See the POLICY FORMAT section of `man ca`.
countryName             = match
stateOrProvinceName     = match
organizationName        = match
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ policy_loose ]
# Allow the intermediate CA to sign a more diverse range of certificates.
# See the POLICY FORMAT section of the `ca` man page.
countryName             = optional
stateOrProvinceName     = optional
localityName            = optional
organizationName        = optional
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ req ]
# Options for the `req` tool (`man req`).
default_bits        = 2048
distinguished_name  = req_distinguished_name
string_mask         = utf8only

# SHA-1 is deprecated, so use SHA-2 instead.
default_md          = sha256

# Extension to add when the -x509 option is used.
x509_extensions     = v3_ca

# <<< I M P O R T A N T >>>
req_extensions      = v3_req

[ req_distinguished_name ]
# See .
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name
localityName                    = Locality Name
0.organizationName              = Organization Name
organizationalUnitName          = Organizational Unit Name
commonName                      = Common Name
emailAddress                    = Email Address

# Optionally, specify some defaults.
countryName_default             = DE
stateOrProvinceName_default     = Saxony
localityName_default            = Dresden
0.organizationName_default      = Example
organizationalUnitName_default  = Example Web Services
emailAddress_default            = john.doe@example.com

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
# <<< I M P O R T A N T >>>
subjectAltName = @alt_names

# <<< I M P O R T A N T >>>
[ alt_names ]
DNS.0 = myserver.example.com
DNS.1 = myserver

[ v3_ca ]
# Extensions for a typical CA (`man x509v3_config`).
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ v3_intermediate_ca ]
# Extensions for a typical intermediate CA (`man x509v3_config`).
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ usr_cert ]
# Extensions for client certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = client, email
nsComment = "OpenSSL Generated Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, emailProtection

[ server_cert ]
# Extensions for server certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

[ crl_ext ]
# Extension for CRLs (`man x509v3_config`).
authorityKeyIdentifier=keyid:always

[ ocsp ]
# Extension for OCSP signing certificates (`man ocsp`).
basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, digitalSignature
extendedKeyUsage = critical, OCSPSigning

Now we create the signing request (make sure you’re in the intermediate ca folder and use the special server config as it contains the SAN):

openssl req -config myserver.cnf -key private/myserver.example.com.key.pem -new -sha256 -out csr/myserver.example.com.csr.pem

There should be no password prompt, as our private key is unprotected.

You are about to be asked to enter information that will be incorporated
into your certificate request.
-----
Country Name (2 letter code) [XX]:DE
State or Province Name []:Saxony
Locality Name []:Dresden
Organization Name []:Example
Organizational Unit Name []:Example Web Services
Common Name []:myserver.example.com
Email Address []:john.doe@example.com

Now we sign the request with the intermediate CA and write protect it:

openssl ca -config ca-intermediate.cnf -extensions server_cert -days 375 -notext -md sha256 -in csr/myserver.example.com.csr.pem -out certs/myserver.example.com.cert.pem
chmod 444 certs/myserver.example.com.cert.pem

You can verify the certificate:

openssl x509 -noout -text -in certs/myserver.example.com.cert.pem

However, very important would be to create a bundle that chains the intermediate certificate with the server certificate:

cat certs/myserver.example.com.cert.pem certs/intermediate.cert.pem > certs/myserver.example.com.bundle.pem

Last but not least lets verify the chain of the server certificate against the root certificate:

cd /root/ca
openssl verify -CAfile certs/ca.cert.pem ca-intermediate/certs/myserver.example.com.bundle.pem

myserver.example.com.cert.pem: OK

…all done!

Create an Infrastructure Service Appliance with BSD Jails

This post will explain how to setup a total of 6 jails distributed over 2 hosts, so that DHCP, DNS and LDAP can be provided into the network.

What we’d like to setup are the following servers:

Of course in an ideal world all 6 servers would be sitting on physically dispersed hardware for good load balancing. However, in our case we will use only 2 physical servers and setup 3 jails in each of server. For now we will use “regular” servers, but later want to run our setup on a single board computer such as a raspberry pi. In this post we will talk about the setup process of either of the “ISA” hosts.

ISA Host Network

First of all we need to get our network setup straight. In our example the infrastructure service appliance will be called isa1 and to the outside world have an ip address of 192.168.23.2/24 (which you can change for your configuration). We edit /etc/resolv.conf to look like this:

search          home.local
nameserver      192.168.23.1
nameserver      8.8.8.8

In the /etc/hosts file we need to put the following:

::1                     localhost isa1.home.local
127.0.0.1               localhost isa1.home.local

Next we will actually create multiple aliases for our network interface to be used by the jails later on. Every jail needs one loopback adapter and one public ip address. On top of that our router sits at 192.168.23.1/24 and we have to put that into /etc/rc.conf as well. Also, we will enable services for sshd and openntpd (openntpd being told to synchronize upon start). We will automatically start ezjail and tell the kernel to produce and persist a core dump in case of software exceptions.

ifconfig_em0="inet 192.168.23.2/24"
ifconfig_em0_alias0="inet 192.168.23.4/32"
ifconfig_em0_alias1="inet 192.168.23.6/32"
ifconfig_em0_alias2="inet 192.168.23.8/32"
defaultrouter="192.168.23.1"
cloned_interfaces="${cloned_interfaces} lo1"
cloned_interfaces="${cloned_interfaces} lo2"
cloned_interfaces="${cloned_interfaces} lo3"
sshd_enable="YES"
#ntpd_enable="YES"
#ntpd_sync_on_start="YES"
openntpd_enable="YES"
openntpd_flags="-s -v"
ezjail_enable="YES"
dumpdev="AUTO"

The network configuration we just put into /etc/rc.conf will not apply unless we reboot. As we don’t need (neither want) to do that, we can make the network adjustments ad-hoc by issuing the following commands in the shell:

service netif cloneup lo1
service netif cloneup lo2
service netif cloneup lo3
ifconfig em0 inet 192.168.23.2/24
ifconfig em0 alias0 inet 192.168.23.4/32
ifconfig em0 alias1 inet 192.168.23.6/32
ifconfig em0 alias2 inet 192.168.23.8/32

Finally we should make sure our log files and everything else relates to our time zone. Let’s execute the following to set our timezone:

tzsetup Europe/Berlin

DHCP jail

Obviously the first command that we need is for creation of the actual jail. We use ezjail and call our jail dhcpjail. We assign network interface em0 with ip address 192.168.23.4 to the outside world and use loopback adapter lo1 with ip 127.0.4.1 internally.

ezjail-admin create dhcpjail 'em0|192.168.23.4,lo1|127.0.4.1'

There is something really special about the jail or DHCP – it needs the bpf (Berkley Packet Filter) device the UDP broadcast messaging. Jails are not just a chroot-ed environment but also are restricted heavily in the resources they can use, especially the device file system (devfs) is not fully exposed. That being said, in this case we need to create a ruleset allowing bpf for our jail, so please make sure your /etc/devfs.rules file contains these lines:

[devfsrules_jail_with_bpf=6]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path 'bpf*' unhide

Above ruleset now shall be applied to our DHCP jail. We have to put the following lines into /usr/local/etc/ezjail/dhcpjail:

export jail_dhcpjail_devfs_ruleset="6"
export jail_dhcpjail_parameters="allow.raw_sockets allow.sysvipc"

Next we will copy our /etc/resolv.conf file into the new jail:

cp /etc/resolv.conf /usr/jails/dhcpjail/etc/

And finally we need to finetune the hosts file in the jail. Please make sure to edit the hosts file in the jail (/usr/jails/dhcpjail/etc/hosts) and not the hosts file of your host environment.

::1                     localhost dhcp1.home.local
127.0.4.1               localhost dhcp1.home.local

Last but not least – and if desired, we can start the loopback interface associated with the jail. We don’t have to do that now, but need to do it before we start the jail (note: if you reboot between now and the time you start the jail, you can skip this step, as the interface will be started as part of parsing /etc/rc.conf during boot).

service netif start lo1

DNS jail

We assign network interface em0 with ip address 192.168.23.6 to the outside world and use loopback adapter lo2 with ip 127.0.6.1 internally.

ezjail-admin create dnsjail 'em0|192.168.23.6,lo2|127.0.6.1'

Next we will copy our /etc/resolv.conf file into the new jail:

cp /etc/resolv.conf /usr/jails/dnsjail/etc/

We finetune the hosts file in the jail.

::1                     localhost dns1.home.local
127.0.6.1               localhost dns1.home.local

Again – optinally, we can start the loopback interface associated with the jail.

service netif start lo2

LDAP jail

We assign network interface em0 with ip address 192.168.23.8 to the outside world and use loopback adapter lo3 with ip 127.0.8.1 internally.

ezjail-admin create ldapjail 'em0|192.168.23.8,lo3|127.0.8.1'

Next we will copy our /etc/resolv.conf file into the new jail:

cp /etc/resolv.conf /usr/jails/ldapjail/etc/

We finetune the hosts file inside the jail.

::1                     localhost ldap1.home.local
127.0.8.1               localhost ldap1.home.local

Again – optinally, we can start the loopback interface associated with the jail.

service netif start lo3

check

Last but not least you want to start all three jails and check that they are running. Please run the following four commands:

ezjail-admin start dhcpjail
ezjail-admin start dnsjail
ezjail-admin start ldapjail
ezjail-admin list

Especially the last command will give you the output we’re looking for. It will list the jails that are installed and their status.

STA JID  IP              Hostname                       Root Directory
--- ---- --------------- ------------------------------ ------------------------
DR  3    192.168.23.8    ldapjail                       /usr/jails/ldapjail
    3    lo3|127.0.8.1
DR  2    192.168.23.6    dnsjail                        /usr/jails/dnsjail
    2    lo2|127.0.6.1
DR  1    192.168.23.4    dhcpjail                       /usr/jails/dhcpjail
    1    lo1|127.0.4.1

DHCP2, DNS2 and LDAP2 jail on ISA2

As you might have guessed, the setup is exactly the same. The only difference are the ip addresses. You can see the ip addresses in the figure up top this article.

Setup KVM – make CentOS a Virtualization Host

This article explains how to setup your CentOS Linux host so you use it as a Virtualization Host with KVM.

If you have a home server that is powerful enough to host a couple of virtual machines, you don’t have to go all ESXi to make your home server a VM Server. Instead you can use your existing CentOS installation with libvirt and KVM to host virtual machines.

Making sure you can run virtualization you need to verify your CPU support. Use the following command line to find out if your CPU supports virtualization. We can use egrep to go through /proc/cpuinfo to look for the appearance of either vmx (Intel) or svm (AMD).

> egrep '(vmx|svm)' /proc/cpuinfo
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 invpcid_single intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm arat pln pts
...

In the example above you can see one core being reported as eligible. We’re good to go and can install the group package for virtualization:

> sudo yum group install "Virtualization Host"


Making sure a normal user can administrate virtualization a policy needs to be created with polkit. Please make sure to replace andreas with your username:

> sudo groupadd virt
> sudo usermod -aG virt andreas
> sudo mkdir -p /etc/polkit-1/localauthority/50-local.d/

Now create the file /etc/polkit-1/localauthority/50-local.d/50-org.example.libvirt-access.pkla with the following contents:

[libvirt Admin Access]
Identity=unix-group:virt
Action=org.libvirt.unix.manage
ResultAny=yes
ResultInactive=yes
ResultActive=yes

That’s it already. On a different host you could install the virt-manager GUI to administrate your VMs from remote. Just use the following command on a third-party CentOS workstation:

> sudo yum install virt-manager

After that start the virtualization manager from the Desktop.