SSH Config and Key Troubleshooting: Manage Multiple Hosts, Agents, and Jump Servers

Overview

SSH configuration becomes difficult when one workstation connects to several environments: personal servers, cloud instances, Git hosts, test machines, and private systems reachable only through a jump host. Reusing a short host name can hide important differences in the real destination, username, port, key, or network path.

The first step is to separate three questions:

• Where is SSH connecting? Check the hostname, resolved address, port, and any ProxyJump path.
• Which identity is it offering? Check IdentityFile, the SSH agent, and whether SSH is trying additional keys.
• Which configuration is effective? Check the result after command-line options, user configuration, included files, and system defaults have been considered.

Do not begin by deleting keys or disabling host-key checks. First capture the effective settings and verbose output. That preserves useful evidence and avoids weakening the connection while diagnosing it.

What to track

1. Host aliases and destinations

Use an alias that describes the role or environment, not an ambiguous name such as server. The alias is only a local label; HostName identifies the actual destination.

Host prod-api
    HostName api.example.net
    User deploy
    Port 22
    IdentityFile ~/.ssh/id_ed25519_prod
    IdentitiesOnly yes

Host staging-api
    HostName staging.example.net
    User deploy
    IdentityFile ~/.ssh/id_ed25519_staging
    IdentitiesOnly yes

Tracking aliases in one place makes it easier to notice when a deployment command points at staging instead of production. Keep the alias, destination, account, and purpose documented together. If a server moves, update the alias configuration rather than scattering a new hostname through shell history, scripts, and documentation.

2. Identity files and agent state

IdentityFile tells the client which private key to use, but an SSH agent may also offer keys loaded into its socket. This is a common reason SSH appears to use the wrong key. Inspect the agent with:

ssh-add -l

To test a host with only the configured identity, use IdentitiesOnly yes in the relevant host block or run a one-off command:

ssh -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_prod prod-api

Use separate keys when environments have different trust boundaries, and give files descriptive names. A key file should normally be readable only by its owner:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/config ~/.ssh/id_ed25519_prod

These permissions are a useful baseline, but the exact error should guide the diagnosis. A permissions warning may prevent SSH from loading a key; it does not prove that the key belongs to the intended account.

3. User, port, and jump-host settings

A valid key can still fail if SSH logs in as the wrong user or reaches the wrong port. Record those values explicitly for hosts that differ from your defaults.

For private systems, ProxyJump is usually clearer than manually constructing a multi-command tunnel:

Host private-db
    HostName 10.20.0.15
    User dbadmin
    Port 22
    IdentityFile ~/.ssh/id_ed25519_private
    IdentitiesOnly yes
    ProxyJump bastion

Host bastion
    HostName bastion.example.net
    User jumpuser
    IdentityFile ~/.ssh/id_ed25519_bastion
    IdentitiesOnly yes

The destination connection and the jump connection can use different accounts and keys. Troubleshoot them separately: verify ssh bastion first, then verify the private host through the alias.

4. Configuration precedence and host matching

SSH can read a system configuration, your user configuration, files referenced by Include, and options supplied on the command line. Host blocks can also overlap. In practice, broad blocks such as Host * should appear before more specific exceptions when you want later host-specific settings to take effect; OpenSSH uses the first obtained value for many options, so ordering matters.

Rather than guessing which file won, ask SSH to print the effective configuration:

ssh -G prod-api | less

Look for hostname, user, port, identityfile, identitiesonly, and proxyjump. This command is one of the most useful developer productivity tools for SSH because it turns layered configuration into inspectable output.

Cadence and checkpoints

SSH configuration does not need daily maintenance, but it benefits from a small review whenever your development environment changes. A monthly or quarterly checkpoint is appropriate for machines used across several projects. Keep a short inventory containing the alias, destination, login account, key path, jump host, and last verification date.

At each checkpoint:

1. Run ssh -G alias for important aliases and confirm the effective hostname, user, port, identity, and proxy settings.
2. Run ssh-add -l and remove stale or unexpected agent identities from the active session.
3. Check that referenced private-key files still exist and have appropriate permissions.
4. Test a representative direct host and a representative jump-host connection.
5. Review aliases that no longer correspond to active systems, projects, or accounts.
6. Confirm that scripts and Git remotes use intended aliases rather than obsolete hostnames.

Do not treat a successful connection as proof that the configuration is clean. An agent may have supplied a key by accident, a DNS record may have changed, or a broad Host * block may be masking a missing host-specific value.

How to interpret changes

When SSH fails, add verbose output without exposing private keys or copying sensitive logs into public issue trackers:

ssh -vvv prod-api

Read the output in stages:

• Configuration: Identify which files are read and which host alias matches. If the expected block is absent, check the alias spelling and any Include path.
• Network: Confirm the final hostname and port. A timeout or connection refusal usually points to routing, firewall, service availability, or an incorrect destination rather than key selection.
• Authentication: Watch for lines indicating which identity files are offered. If the intended key never appears, inspect IdentityFile, file permissions, and the alias being used.
• Agent behavior: If unexpected keys are offered, inspect the agent and test with IdentitiesOnly=yes. This narrows the experiment without permanently changing the configuration.
• Server response: “Permission denied” after a key is offered can mean the public key is not authorized for that account, the account is wrong, or the server applies an authentication restriction.
• Host identity: A host-key warning is a separate issue from user authentication. Do not blindly remove entries from known_hosts; verify that the destination really changed before updating the record.

Compare a failing alias with a known-working one using ssh -G. Differences in User, Port, IdentityFile, or ProxyJump often reveal the problem faster than reading the entire verbose log.

For a one-time diagnostic override, prefer an explicit command such as ssh -o IdentitiesOnly=yes -o PreferredAuthentications=publickey alias. Avoid making permanent changes until the test identifies a real configuration requirement.

When to revisit

Revisit this guide after rotating a credential, adding a host, changing a login account, moving to a new workstation, switching SSH agents, or introducing a bastion. Also review it after a client or server configuration change that affects accepted key types, authentication methods, or host-key handling. Defaults and supported algorithms can change between OpenSSH releases, so confirm current behavior with the client’s documentation and local help output rather than copying an old workaround.

Use the following practical reset whenever an SSH connection starts behaving unexpectedly:

1. Write down the exact alias and command that fails.
2. Run ssh -G alias and verify destination, user, port, identity, and jump host.
3. Run ssh-add -l and test again with IdentitiesOnly=yes.
4. Check key-file permissions and confirm the matching public key is authorized for the intended account.
5. Use ssh -vvv to distinguish network, configuration, authentication, and host-identity errors.
6. Test the bastion independently if ProxyJump is involved.
7. Record the confirmed fix in the host block and remove temporary debug options.

A small, explicit configuration and a quarterly review are usually more valuable than a growing collection of copied command-line flags. Treat SSH aliases, keys, agents, and jump servers as part of your developer workflow: document them, inspect their effective state, and update them when the systems around them change.