Skip to content

Instantly share code, notes, and snippets.

@lizthegrey
Last active February 24, 2024 14:11
Show Gist options
  • Save lizthegrey/9c21673f33186a9cc775464afbdce820 to your computer and use it in GitHub Desktop.
Save lizthegrey/9c21673f33186a9cc775464afbdce820 to your computer and use it in GitHub Desktop.
Hardening SSH with 2fa
default['sshd']['sshd_config']['AuthenticationMethods'] = 'publickey,keyboard-interactive:pam'
default['sshd']['sshd_config']['ChallengeResponseAuthentication'] = 'yes'
default['sshd']['sshd_config']['PasswordAuthentication'] = 'no'

Hi! I'm Liz, a Developer Advocate at honeycomb.io, and I spent my first weeks at the company doing security hardening of our infrastructure. I'd like to share what I'd learned with you, so that you can benefit from my reading of dozens of scattered pages of documentation and my ruling out of numerous dead ends.

Why you should take security and usability seriously

Developers and administrators have historically used SSH keys to provide authentication between hosts. By adding passphrase encryption, the private keys become resistant to theft when at rest. But what about when in use? Unfortunately, the usability challenges of re-entering the passphrase on every connection means that engineers began caching keys unencrypted in memory of their workstations, and worse yet, forwarding the agent to allow remote hosts to use the cached keys without further confirmation. The recent breach at Matrix underscores how dangerous it is to allow authenticated sessions to propagate across hosts and environments without a human in the loop.

Thus, we need solutions that prevent key theft from the systems we connect to, while maintaining ease of use. Two-factor authentication stops malicious automated propagation in its tracks by having a second factor protect use of our keys. There are two primary ways of preventing an attacker from misusing our credentials: either using a separate device that generates, using a shared secret, numerical codes that we can transfer over out of band and enter alongside our key, or having the separate device perform all the cryptography for us only when physically authorized by us.

Google, where I previously worked, employs short-lived SSH certificates issued by a central piece of infrastructure, stored on secure hardware tokens. But this is a serious change to developer workflow, and requires extensive infrastructure to set up. What will work for a majority of developers who are used to simply loading their SSH key into the agent at the start of their login session and SSHing everywhere?

Design considerations & threat models

I'm assuming that you have a publicly exposed bastion host for each environment that intermediates accesses to the rest of each environment's VPC, and use SSH keys to authenticate from laptops to the bastion and from the bastion to each VM/container in the VPC. If you don't yet have a bastion host and a VPC, start there!

It was important to me to make Honeycomb safe from compromise, even if malicious worm-like code were executed on a developer's laptop while SSH keys were unlocked, or if a developer accidentally forwarded an SSH agent to a hostile remote system. I also thought it important to build on existing work to disk encrypt all endpoints by ensuring the loss of physical control over a phone or hardware token could not itself grant production access. However, I consider it out of scope to prevent active local intervention and session hijacking (since someone who controls your active console or keyboard has you pretty well pwned).

I'm also assuming you have a mix of operating systems, hardware, and preferences about carrying dongles vs. wanting to use phones for second factor, etc.

How to get started!

First, start by enabling numerical time-based one time password (TOTP) for SSH authentication. Is it perfect? No, since a malicious host could impersonate the real bastion (if strict host checking isn't on), intercept your OTP, and then use it to authenticate to the real bastion. But it's better than being wormed or compromised because you forgot to take basic measures against even a passive adversary.

Server-side setup

You'll want a root shell open just in case, and the following snippets added to your Chef cookbooks (from this gist):

  • metadata.rb
  • attributes/default.rb (from attributes.rb)
  • files/sshd
  • recipes/default.rb (copy from recipe.rb)
  • templates/default/users.oath.erb

Okay, now we can set this running on our hosts… and go through the client setup for ourselves at least.

Client-side setup

Now, each user authenticating needs a shared key to be present, encrypted, in SSM (or equivalent for your choice of cloud provider). Have each user install an OTP app such as Google Authenticator, Authy, Duo, or Lastpass, then do the following on their laptop:

Install dependencies:

brew install oath-toolkit OR apt install oathtool openssl

Generate a random base16 string to use as your key:

➜ openssl rand -hex 10
22ea2966afefd82660e1
##### ^^^ that's an example output used here - don't use it!

Convert it and put it into a phone-based authenticator app: Run oathtool -v [key] to convert it to the format (“Base32 secret”) that mobile authenticators use.

➜ oathtool -v 22ea2966afefd82660e1
Hex secret: 22ea2966afefd82660e1
Base32 secret: ELVCSZVP57MCMYHB
... more stuff down here we don't need
  • For 1Password, add a one time password and enter the “Base32 secret” output from oathtool -v [key]
  • For Duo, select “other” and use the Base32 secret.
  • for Authy click “Enter key manually” and use the Base32 secret

Verify that generated codes are correct: Run oathtool --totp [key] and check that it returns the same value as your authenticator application.

➜ oathtool --totp 22ea2966afefd82660e1
693439

Store our key into the cloud secrets manager: Run aws ssm put-parameter --name /2fa/totp/$USER --value [key] --type SecureString --key-id alias/parameter_store_key to put your key into SSM Parameter Store. $USER should be the same as the username you use when you log in to a bastion. If you are updating the key instead of pushing it for the first time, add the --overwrite flag to the end of the command.

➜ aws ssm put-parameter --name /2fa/totp/lizf --value 22ea2966afefd82660e1 --type SecureString --key-id alias/parameter_store_key 
{
    "Version": 1
}

Log in for the first time: Now, when we ssh to the bastion host, we can ensure that the SSH agent can only be trampolined to other hosts within the VPC, but any attempt to programatically use from the outside the forwarded agent (or loaded in-memory keys) to access a bastion will fail because no TOTP from the separate mobile device was provided.

Let's check that we're asking for TOTPs:

➜ ssh -A bastion
Enter passphrase for key '[snip]': 
One-time password (OATH) for '[user]': 
Welcome to Ubuntu 18.04.1 LTS...

Now there's a value proposition for hardware auth…

People might get sick and tired of entering a numerical OTP every time they have to log into the bastion! It's almost like the old days of passphrase-encrypted SSH keys that motivated us to use agents! So let's leverage this inherent laziness to get people more, rather than less, secure!

Server-side setup

Change the beginning of files/sshd in your Chef module to begin as follows:

auth	required	pam_permit.so
auth	optional	pam_cap.so

# If it's a hardware or secure enclave SSH key, no need for a numerical OTP.
auth    sufficient      pam_ssh_agent_auth.so file=/etc/2fa_token_keys

# Check a TOTP code as a second resort, using a time slip of +/- 150 seconds.
auth	sufficient	pam_oath.so usersfile=/etc/users.oath digits=6 window=5

# People without OTPs will need to add an OTP secret to AWS SSM and wait an hour.
auth	requisite	pam_deny.so
...

And if your openssh is older than 7.8 and/or your libpam-ssh-agent-auth is older than 0.10.4, add the following additional lines to recipes/default.rb (a note to the nervous: my source modifications to openssh-server and libpam-ssh-agent-auth are available from Launchpad):

apt_repository 'openssl-pam-bindings' do
  uri 'ppa:honeycomb.io/ssh-2fa'
end

packages = %w{ openssh-server libpam-ssh-agent-auth }
packages.each do |p|
  r = package p do
    action :upgrade
  end
end

service 'sshd' do
  subscribes :reload, 'package[openssh-server]'
end

Now you'll need to use Chef to populate /etc/2fa_token_keys with keys that you know are generated and stored securely (e.g. using one of the below methods). I don't know how you maintain your lists of ssh key mappings to users, nor how you add ssh keys to your ~/.ssh/authorized_keys files, so I can't provide general advice.

Mac client setup

People with Touchbar Macs should use TouchID to authenticate logins, as they'll have their laptop and their fingers with them anyways. sekey lets us support this.

Install the binary:

brew cask install sekey

Add to ~/.ssh/config on your local machine:

IdentityAgent ~/.sekey/ssh-agent.ssh

Generate a key and export it:

sekey --generate-keypair "bastion key"
sekey --export-key $(sekey --list-keys|grep "bastion key"|grep --only-matching -E '[a-f0-9]{40}')

And then store the resulting key to /etc/2fa_token_keys and ~/.ssh/authorized_keys in Chef.

Krypt.co setup for iOS and Android

Instead of generating OTPs and sending them over manually with our fingers, our mobile devices can securely store our SSH keys and only remotely authorize usage (and send the signed challenge to the remote server) if a human presses a button on the phone.

This is the theory behind krypt.co, and is even more secure than a TOTP app so long as you supply appropriate parameters to force hardware coprocessor storage (NIST P-256 for iOS, and 3072-bit RSA for Android, on new enough devices). Make sure people use screen locks!

Follow the instructions here: https://krypt.co/docs/start/upload-your-ssh-publickey.html and then supply the generated key to both ~/.ssh/authorized_keys and /etc/2fa_token_keys in your Chef automation, and you won't be prompted for a TOTP.

YubiKey hardware token & Linux/ChromeOS client setup

Initial per-YubiKey setup

Follow these instructions from a Linux host to set up a basic working hardened YubiKey SSH key:

Install Dependencies

sudo apt-add-repository ppa:yubico/stable && sudo apt-get update
sudo apt-get install gpg yubikey-manager-qt pinentry-curses scdaemon pcscd
echo "reader-port Yubico YubiKey" > .gnupg/scdaemon.conf

Hardening to prevent a rogue host from authenticating without your permission

ykman openpgp touch sig on
ykman openpgp touch aut on
ykman openpgp touch enc on

Hardening in case your security key is stolen

gpg --change-pin

Default user pin is 123456 and admin pin is 12345678, change both of them to something more secure; they can both be the same PIN.

Generate a random 24-byte hex-encoded reset key and save it somewhere, GPG encrypted with your normal daily use keys (ykman-gui can generate a 24-byte string for you in “PIV → Configure PINs → Change Management Key”)

Generating the keys:

gpg --card-edit
admin
generate
  • Input 4096 for all three modes (you'll need to enter the admin and user pins)
  • Don't back up the stubs when prompted.
  • Enter your full name and email address; make sure you leave a comment (e.g. desk computer) so you know which stub key is which in your GPG keyring.

Wait a minute, then enter the user PIN one more time, then wait about 5-10 minutes for the generation process to complete. It will print the UID of the master key before returning you to the card-edit prompt.

quit

gpg --export-ssh-key UID_of_master_key

This will print out the ssh pubkey string you'll need to add to the remote ~/.ssh/authorized_keys and /etc/2fa_token_keys in Chef.

Usage for authentication

Linux

Once the per-key setup is done, the configured Yubikey can be used in a Linux machine configured like so: gpg --with-keygrip -K

Save the keygrip of the master key you just generated to .gnupg/sshcontrol

Ensure that you have gpg-agent configured correctly: Set curses pinentry. Why? So you don't randomly get X passphrase/passcode prompts all over the place (esp remotely):

Edit ~/.gnupg/gpg-agent.conf to contain:

pinentry-program /usr/bin/pinentry-curses
enable-ssh-support

You'll update your ~/.bashrc to contain the following lines:

export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket)
export GPG_TTY=$(tty)
gpg-connect-agent updatestartuptty /bye >/dev/null

Run ssh-add -l to confirm you see your key in the list (it'll show 4096 SHA256:... cardno:... (RSA) in the listing).

When you ssh from a terminal into a bastion (remember to ssh -A for agent forwarding!), it'll prompt in the terminal that you most recently opened for your PIN on initial usage of the key. You'll complete that step, then tap your key to confirm. You're in!

ChromeOS

Install these two Chrome apps:

Then open the Secure Shell App (this won't work yet from the Crostini Terminal app because Crostini doesn't have USB pass-through yet, although it's coming in Chrome 75!)

Within the secure shell app's configuration screen for the bastion host:

  • relay server option: --ssh-agent=gsc
  • ssh option: -A

You'll then enter the user PIN when prompted, and tap the security key to confirm when logging into the bastion.

Further reading

The folks at krypt.co have written some fantastic blogs on securing SSH that go beyond the basic hardening I recommend here.

Hope this helps! Send me a Twitter DM (@lizthegrey) or email (lizf@honeycomb.io) if you have improvements to suggest!

name "bastion"
description "special hardening for bastions"
version "0.0.1"
depends "aws"
depends "sshd"
package "libpam-oath" do
action :upgrade
end
aws_ssm_parameter_store 'getOTPsecrets' do
path '/2fa/totp/' # or your own choice of SSM path.
recursive true
with_decryption true
return_key 'totp_secrets'
action :get_parameters_by_path
# No need for aws_access_key and aws_secret_access_key due to implicit EC2 grant.
sensitive true
end
# Populate the oath file.
template '/etc/users.oath' do
source 'users.oath.erb'
owner 'root'
group 'root'
mode '0600'
variables(
:users => lazy { node.run_state['totp_secrets'] }
)
sensitive true
end
cookbook_file '/etc/pam.d/sshd' do
source 'sshd'
owner 'root'
group 'root'
mode '0644'
action :create
end
# Force ssh to consult PAM as well as using SSH keys for primary auth..
include_recipe 'sshd'
auth required pam_permit.so
auth optional pam_cap.so
# Check a TOTP code, using a time slip of +/- 150 seconds.
auth sufficient pam_oath.so usersfile=/etc/users.oath digits=6 window=5
# People without OTPs will need to add an OTP secret to AWS SSM and wait an hour.
auth requisite pam_deny.so
account required pam_nologin.so
@include common-account
session [success=ok ignore=ignore module_unknown=ignore default=bad] pam_selinux.so close
session required pam_loginuid.so
session optional pam_keyinit.so force revoke
@include common-session
session optional pam_motd.so motd=/run/motd.dynamic
session optional pam_motd.so noupdate
session optional pam_mail.so standard noenv
session required pam_limits.so
session required pam_env.so
session required pam_env.so user_readenv=1 envfile=/etc/default/locale
session [success=ok ignore=ignore module_unknown=ignore default=bad] pam_selinux.so open
@include common-password
# To update this file, generate a SSM parameter in /2fa/totp.
# See this URL for examples:
# https://console.aws.amazon.com/systems-manager/parameters/?region=us-east-1#list_parameter_filters=Path:Recursive:%2F2fa%2F
<% @users.each do |key, value| %>
HOTP/T30 <%= key %> - <%= value %>
<% end %>
@trknov
Copy link

trknov commented Apr 20, 2019

I see you like Hardening

@FernandoMiguel
Copy link

I prefer to use (and have my devs use) ed25519 keys instead of RSA.
I do it both on kryptonco and on laptops

@anthonyclarka2
Copy link

This is fantastic, thank you for writing it!

I'd be interested to see how this could be adapted to work with FreeIPA. (Identity management via ldap, Kerberos, sssd and a certificate authority)

@onestsam
Copy link

Thank you for sharing your knowledge and expertise for free. People like you make the world a better place.

@cluelessperson
Copy link

I would just use gpg hardware keys with SSH.

Then you don't have to mess with 2 factor.

@Winkster
Copy link

Winkster commented Apr 20, 2019

This is awesome, Liz, thanks so much. Very thorough and thoughtful post. All angles considered. You rock.

@MrStonedOne
Copy link

I'm a little annoyed this entire thing assumes we have Chef.

Requirements should be stated at the beginning, or at the least, you should not launch directly into "add blah into Chef" but instead "First you'll need Chef and to add blah into it".

Its presumptuous and rude to assume everybody reading has the requirements and you should instead explitically state them.

@christian-ehrisman
Copy link

christian-ehrisman commented Apr 20, 2019

Thank you for this write-up, it's a lot of great information. I don't think it is either presumptuous or rude to document the actual common and normal software you used, though some may be offended by your use of code. I must admit I'm left confused why anyone would have a problem with someone posting on github the thing they did with common software.

@robertkraig
Copy link

robertkraig commented Apr 20, 2019

I'd love to see a video posted on youtube or attached in gif in fast-forward mode to see the whole process work / look.

@sbgoodm
Copy link

sbgoodm commented Apr 21, 2019

Great write-up, thanks for sharing!

@relaxdiego
Copy link

relaxdiego commented Apr 21, 2019

Area man says it’s presumptuous and rude to share what you know for free on the Interwebz. In other news, research shows that open source is bad for the world because of the multitude of languages and frameworks in use. “It’s just too gosh darn rude and offensive!” according to one bystander.

@kinduff
Copy link

kinduff commented Apr 21, 2019

This is awesome, thanks a lot for sharing!

@julianxhokaxhiu
Copy link

Just wanted to thank you, this is VERY valuable nowadays.

@classmember
Copy link

Thank you, Liz!

@pedrocarrico
Copy link

It works great! Thanks!

@espoelstra
Copy link

This is awesome, and if you disable the Smart Card Connector app you can pass Yubikey devices into Crostini quite nicely these days. There appears to be an "ownership" conflict where it demands exclusive access that prevents sharing into Crostini using vmc usb-attach termina x:yyy if you don't disable the connector.

@lizthegrey
Copy link
Author

This is awesome, and if you disable the Smart Card Connector app you can pass Yubikey devices into Crostini quite nicely these days. There appears to be an "ownership" conflict where it demands exclusive access that prevents sharing into Crostini using vmc usb-attach termina x:yyy if you don't disable the connector.

@espoelstra OMG thanks! super glad to hear yubikeys can be attached via USB sharing, I thought it was broken, but apparently was due to the app!

@lizthegrey
Copy link
Author

@espoelstra I've now confirmed this works, so will update the guide accordingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment