SSH proxy and command logs

Note

Cloudflare recommends configuring SSH command logs through our new Access for Infrastructure workflow. Access for Infrastructure supports differing SSH aliases out-of-the-box, custom SSH ports, and Logpush integrations.

Cloudflare Zero Trust supports SSH proxying and command logging using Secure Web Gateway and the WARP client.

You can create network policies to manage and monitor SSH access to your applications. When a device connects to your origin server over SSH, a session log will be generated showing which user connected, the session duration, and optionally a full replay of all commands run during the session.

Prerequisites

• Install the WARP client on end-user devices.
• Install the Cloudflare root certificate on end-user devices.

1. Ensure Unix usernames match user SSO identities

Cloudflare Gateway will take the identity from a token and, using short-lived certificates, authorize the user on the target infrastructure.

The simplest setup is one where a user’s Unix username matches their email address prefix. Issued short-lived certificates will be valid for the user’s email address prefix. For example, if a user in your Okta or GSuite organization is registered as jdoe@example.com, they would log in to the SSH server as jdoe.

For testing purposes, you can run the following command to generate a Unix user on the machine:

sudo adduser jdoe

Advanced setup: Differing usernames

SSH certificates include one or more principals in their signature which indicate the Unix usernames the certificate is allowed to log in as. Cloudflare Access will always set the principal to the user’s email address prefix. For example, when jdoe@example.com tries to connect, Access issues a short-lived certificate authorized for the principal jdoe.

By default, SSH servers authenticate the Unix username against the principals listed in the user’s certificate. You can configure your SSH server to accept principals that do not match the Unix username.

Note

If you would like to use short-lived certificates with the browser-based terminal, the user’s email address prefix needs to matches their Unix username.

Username matches a different email

To allow jdoe@example.com to log in as the user johndoe, add the following to the server’s /etc/ssh/sshd_config:

Match user johndoe
  AuthorizedPrincipalsCommand /bin/echo 'jdoe'
  AuthorizedPrincipalsCommandUser nobody

This tells the SSH server that, when someone tries to authenticate as the user johndoe, check their certificate for the principal jdoe. This would allow the user jdoe@example.com to sign into the server with a command such as:

ssh johndoe@server

Username matches multiple emails

To allow multiple email addresses to log in as vmuser, add the following to the server’s /etc/ssh/sshd_config:

Match user vmuser
  AuthorizedPrincipalsFile /etc/ssh/vmusers-list.txt

This tells the SSH server to load a list of principles from a file. Then, in /etc/ssh/vmusers-list.txt, list the email prefixes that can log in as vmuser, one per line:

jdoe
bwayne
robin

Username matches all users

To allow any Access user to log in as vmuser, add the following command to the server’s /etc/ssh/sshd_config:

Match user vmuser
  AuthorizedPrincipalsCommand /bin/bash -c "echo '%t %k' | ssh-keygen -L -f - | grep -A1 Principals"
  AuthorizedPrincipalsCommandUser nobody

This command takes the certificate presented by the user and authorizes whatever principal is listed on it.

Allow all users

To allow any Access user to log in with any username, add the following to the server’s /etc/ssh/sshd_config:

AuthorizedPrincipalsCommand /bin/bash -c "echo '%t %k' | ssh-keygen -L -f - | grep -A1 Principals"
AuthorizedPrincipalsCommandUser nobody

Since this will put the security of your server entirely dependent on your Access configuration, make sure your Access policies are correctly configured.

2. Generate a Gateway SSH proxy CA

Instead of traditional SSH keys, Gateway uses short-lived certificates to authenticate traffic between Cloudflare and your origin.

Note

Other short-lived CAs, such as those used to secure SSH servers behind Cloudflare Access, are incompatible with the Gateway SSH proxy. For SSH logging to work, you must create a new CA using the gateway_ca API endpoint.

To generate a Gateway SSH proxy CA and get its public key:

1. Create an API token with the following permissions:

   Type	Item	Permission
   Account	Access: SSH Auditing	Edit

2. If you have not yet generated a Cloudflare SSH CA, make a POST request to the Cloudflare API:

   curl --request POST \
   "https://api.cloudflare.com/client/v4/accounts/{account_id}/access/gateway_ca" \
   --header "Authorization: Bearer <API_TOKEN>"

3. If you have already created a Cloudflare SSH CA or receive the error message access.api.error.gateway_ca_already_exists, make a GET request instead:

   curl https://api.cloudflare.com/client/v4/accounts/{account_id}/access/gateway_ca \
   --header "Authorization: Bearer <API_TOKEN>"

4. Copy the public_key value returned in the response.

3. Save the public key

1. Use the following command to change directories to the SSH configuration directory on the remote target machine:

   cd /etc/ssh

2. Once there, you can use the following command to both generate the file and open a text editor to input/paste the public key.

   vim ca.pub

3. In the ca.pub file, paste the public key without any modifications.

   The ca.pub file can hold multiple keys, listed one per line. Empty lines and comments starting with # are also allowed.

4. Save the ca.pub file. In some systems, you may need to use the following command to force the file to save depending on your permissions:

   :w !sudo tee %
   :q!

4. Modify your SSHD config

The following procedure makes two changes to the sshd_config file on the remote target machine. The first change requires that you uncomment a field already set in most default configurations; the second change adds a new field.

1. While staying within the /etc/ssh directory on the remote machine, open the sshd_config file.

   vim /etc/ssh/sshd_config

2. Go to the row named PubkeyAuthentication. In most default configurations, the row will appear commented out as follows:

   # PubkeyAuthentication yes

3. Remove the # symbol to uncomment the line; keep the setting yes enabled.

4. Next, add a new line below PubkeyAuthentication as follows:

   TrustedUserCAKeys /etc/ssh/ca.pub

   Save the file and quit the editor. You might need to use the following command again to save and exit.

   :w !sudo tee %
   :q!

5. Check your SSH port number

Cloudflare’s SSH proxy only works with servers running on the default port 22. Open the sshd_config file and verify that no other Port values are specified.

cat /etc/ssh/sshd_config

6. Restart your SSH server

Once you have modified your SSHD configuration, restart the SSH service on the remote machine.

Debian/Ubuntu

For older Debian/Ubuntu versions:

sudo service ssh restart

For newer Debian/Ubuntu versions:

sudo systemctl restart ssh

CentOS/RHEL

For CentOS/RHEL 6 and older:

sudo service sshd restart

For CentOS/RHEL 7 and newer:

sudo systemctl restart sshd

7. Create an Audit SSH policy

1. In Zero Trust ↗, go to Gateway > Firewall policies.

2. In the Network tab, create a new network policy.

3. Name the policy and specify the Destination IP for your origin server.

   You can enter either a public or private IP. To use a private IP, refer to Connect private networks.

4. Add any other conditions to your policy. If a user does not meet the criteria, they will be blocked by default.

5. In the Action dropdown, select Audit SSH.

6. (Optional) Enable SSH Command Logging. If you have not already uploaded an SSH encryption public key, follow the steps in Configure SSH Command Logging.

7. Save the policy.

8. Connect as a user

Users can use any SSH client to connect to the target resource, as long as they are logged into the WARP client on their device. Cloudflare Zero Trust will authenticate, proxy, and optionally encrypt and record all SSH traffic through Gateway.

Users must specify their desired username to connect with as part of the SSH command:

ssh <username>@<hostname>

Note

If the target resource is already in a user’s .ssh/known_hosts file, the user must first remove existing SSH keys before attempting to connect:

ssh-keygen -R <targetIP or hostname>

(Optional) Configure SSH Command Logging

To log SSH commands, you will need to generate an HPKE key pair and upload the public key to Cloudflare.

1. Download ↗ the Cloudflare ssh-log-cli utility.

2. Using the ssh-log-cli utility, generate a public and private key pair.

   ./ssh-log-cli generate-key-pair -o sshkey
   ls

   README.md    ssh-log-cli    sshkey    sshkey.pub

   This command outputs two files, an sshkey.pub public key and a matching sshkey private key.

3. In Zero Trust ↗, go to Settings > Network.

4. In SSH encryption public key, paste the contents of sshkey.pub and select Save. Note that this a different public key from the ca.pub file you used to configure the SSH server.

All proxied SSH commands are immediately encrypted using this public key. The matching private key is required to view logs.

View SSH Logs

1. In Zero Trust ↗, go to Logs > Gateway > SSH.

2. If you enabled the SSH Command Logging feature, you can Download a session’s command log.

3. To decrypt the log, follow the instructions in the SSH Logging CLI repository ↗. In the following example, sshkey is the private key that matches the public key uploaded to Cloudflare.

   ./ssh-log-cli decrypt -i sshlog -k sshkey

   This command outputs a sshlog-decrypted.zip file with the decrypted logs.

Limitations

SSH Command Logging does not support SFTP since it cannot be inspected and logged.