Public Key Authentication getting errors

Linux on Servers
1

Hi, Everyone.

I am trying to learn Linux and watched both Jay and Andrew Mallet tutorial on how to configure Public Authentication but I am hitting a brick wall with 2 problems.

1. Problem is, although I was successfully copied over the Public Key to target server I am still getting prompt for a password instead of passphrase 
 2. If set the "PasswordAuthentication" to no I get hit in "sshd_config" I get an error of Permission denied (publickey,gssapi-keyex,gssapi-with-mic)

Here is the configuration from sshd_config file:

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

#Ciphers and keying
#RekeyLimit default none

#Logging
#SyslogFacility AUTH
SyslogFacility AUTHPRIV
#LogLevel INFO

#Authentication:

#LoginGraceTime 2m
PermitRootLogin no
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

PubkeyAuthentication yes

#The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2
#but this is overridden so installations will only check .ssh/authorized_keys
AuthorizedKeysFile ~/.ssh/authorized_keys

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

#For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
#Change to yes if you don’t trust ~/.ssh/known_hosts for
#HostbasedAuthentication
#IgnoreUserKnownHosts no
#Don’t read the user’s ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

#To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no
PasswordAuthentication yes

#Change to no to disable s/key passwords
#ChallengeResponseAuthentication yes
ChallengeResponseAuthentication no

#Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no
#KerberosUseKuserok yes

#GSSAPI options
GSSAPIAuthentication yes
GSSAPICleanupCredentials no
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no
#GSSAPIEnablek5users no

#Set this to ‘yes’ to enable PAM authentication, account processing,
#and session processing. If this is enabled, PAM authentication will
#be allowed through the ChallengeResponseAuthentication and
#PasswordAuthentication. Depending on your PAM configuration,
#PAM authentication via ChallengeResponseAuthentication may bypass
#the setting of “PermitRootLogin without-password”.
#If you just want the PAM account and session checks to run without
#PAM authentication, then enable this but set PasswordAuthentication
#and ChallengeResponseAuthentication to ‘no’.
#WARNING: ‘UsePAM no’ is not supported in Red Hat Enterprise Linux and may cause several
#problems.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
#PrintMotd yes
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#UsePrivilegeSeparation sandbox
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#ShowPatchLevel no
#UseDNS yes
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

#no default banner path
#Banner none

#Accept locale-related environment variables
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS

#override default of no subsystems
Subsystem sftp /usr/libexec/openssh/sftp-server

#Example of overriding settings on a per-user basis
#Match User anoncvs
#X11Forwarding no
#AllowTcpForwarding no
#PermitTTY no
#ForceCommand cvs server

Hope you can point out what I have overlooked, if you need more info please let me know and this is a vm for studying so I can provide everything in this VM.

2

This is the line that jumps out to me, I’ve only ever seen it expressed

AuthorizedKeysFile  .ssh/authorized_keys

Also cat the ~/.ssh/authorized_keys file and ensure that the line for your “from” host matches exactly your id_rsa.pub that you generated on the “from” machine…

2 Likes
3

Thank you for the response, In regards to “AuthorizedKeysFile ~/.ssh/authorized_keys” I have only change while troubleshooting my issue. I was guessing maybe it is not seeing the right location of the authorized_keys

Performing cat ~/.ssh/authorized_keys on my target server and cat ~/.ssh/authorized_keys are the same

[tux@server1 .ssh]$ cat id_rsa2.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYxjBMogG96DCq3VdR0QvaAfYz4mh4RwdNKaxI24v4r0hBi6Yp+mpdWGkK9kcr968NOtCWQV9/zpZqwvv8SnPy/J9otQjRAeFR3PumAgfSD2VMxIrXDY5dameV1Ez+fhcSvCW4h5K1w1HiEu72zn9TC7Btt3DNAsOdeeVpYsmTBEEDPjmMn8XCUNzBlVj0FALiVCTwaXT06blEeQvKW9k6CULyDCeySW7V62KZW3NrUDOqzXAjzcjOMZd8EOWCb8KAG2cOfYcmnB+NWLwJn7WBGXeCKuRjYjwl25TkiA90T1EtmVFk4EzVfbljprQq1LIelknN439qXusImCbtCeMD tux@server1

[tux@server2 ~]$ cat ~/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYxjBMogG96DCq3VdR0QvaAfYz4mh4RwdNKaxI24v4r0hBi6Yp+mpdWGkK9kcr968NOtCWQV9/zpZqwvv8SnPy/J9otQjRAeFR3PumAgfSD2VMxIrXDY5dameV1Ez+fhcSvCW4h5K1w1HiEu72zn9TC7Btt3DNAsOdeeVpYsmTBEEDPjmMn8XCUNzBlVj0FALiVCTwaXT06blEeQvKW9k6CULyDCeySW7V62KZW3NrUDOqzXAjzcjOMZd8EOWCb8KAG2cOfYcmnB+NWLwJn7WBGXeCKuRjYjwl25TkiA90T1EtmVFk4EzVfbljprQq1LIelknN439qXusImCbtCeMD tux@server1

4

Ok, I’ve replicated that error message on a couple of my machines, it’s a client-side issue. It happens when the client does not expose the private key properly.

In any case, the client need not have an authorized_keys file, it only needs to have the private key to log into another host (which then must have the public key in its authorized_keys file). Here’s how my client looks when working properly (‘optional’ means you need not have that file in your client-side .ssh for proper function):

efahl@client$ pwd
/home/efahl

efahl@client$ ll -d .ssh
drwx------ 2 efahl efahl 4096 Dec  2 12:04 .ssh/

efahl@client$ ll .ssh
total 32
-rw-rw-r--  1 efahl efahl 2454 Nov 17 17:39 authorized_keys <-- optional
-rw-r--r--  1 efahl efahl  322 Dec  2 12:04 config          <-- optional
-rw-------  1 efahl efahl 2610 Nov 13 13:53 id_rsa          <-- MANDATORY private key
-rw-r--r--  1 efahl efahl  573 Nov 13 13:53 id_rsa.pub      <-- optional

It may be permissions? (Wild guess, I only tested by removing the private key.) Check and make sure that your .ssh directory and private key looks like above (‘chmod 700 .ssh’ and ‘chmod 600 .ssh/id_rsa’), as I recall having client side issues with that before (on Windows maybe???).

Also that ‘2’ in your key file name looks weird. Did ssh-keygen do that? I’ve only seen the ‘id_rsa’ files without the ‘2’…

5

Yes, your ~/.ssh directory should have group and other permissions removed, as ssh will not work otherwise. This policy is in place because home directories don’t necessarily exclude group and other access.

I wouldn’t consider your public key(s) to be “optional” in any way as they are half of your key pair; it(they) really shouldn’t be removed. Parts of ssh, such as ssh-copy-id, won’t work without it being there.

1 Like
6

In regards to “id_rsa2.pub” this was generated by me. Because when following Andrew Mullet’s tutorial in pluralsight I thought I need to use sudo to generate the keys. Realizing I can use my regular account without using sudo, I regenerated the keys and named it “id_rsa2.pub”

As I check the files and folders, I have the permission already set to 600 for “.ssh/id_rsa” and 700 for .ssh

[tux@server1 ~]$ ll -d .ssh
drwx------. 2 tux tux 131 Dec 7 05:40 .ssh
[tux@server1 ~]$ ll .ssh
total 28
-rw-r–r–. 1 tux root 135 Dec 3 08:01 config
-rw-------. 1 tux tux 1766 Dec 3 07:34 id_rsa
-rw-------. 1 tux tux 1766 Dec 6 05:18 id_rsa2
-rw-r–r–. 1 tux tux 393 Dec 6 05:18 id_rsa2.pub
-rw-r–r–. 1 tux tux 393 Dec 3 07:34 id_rsa.pub
-rw-r–r–. 1 tux tux 347 Dec 7 05:40 known_hosts
-rw-r–r–. 1 tux tux 347 Dec 7 05:40 known_hosts_backup

Also I don’t have “authorized_keys” on my client computer, because I have not tried connecting back from server2 to server1. Is this the possible cause of issue?

In my “config” file here is the content, is this the similar as yours:

[tux@server1 ~]$ cat ~/.ssh/config
Host server1
Hostname 192.168.10.243
User tux
Port 22

Host server2
Hostname 192.168.10.242
User tux
Port 22

Just checking maybe this is the issue.

7

Aha, there’s your issue. Server2 is using the contents of ‘id_rsa2.pub’ via its authorized_keys contents and server1 is using ‘id_rsa.pub’ on the client side. On server1, try doing ‘ssh-copy-id server2’ (it should ask password), then after it completes ‘ssh server2’ should log you in without a password prompt.

When that works, only then can you change server2’s config to enable ‘PasswordAuthentication yes’, restart the sshd on server2 and you’ve secured it…

Since this is just initial config, you probably should just start over:

  1. empty server2 authorized_keys,
  2. delete server1 .ssh directory altogether (well, copy your ‘config’ somewhere),
  3. on server1 ‘ssh-keygen’ to make a new keypair, then ‘ssh-copy-id server2’
  4. ‘ssh server2’ and see what happens

(To be complete, you could use ‘ssh -i ~/.ssh/id_rsa2 tux@server2’ on the client, but that just complicates things, which is probably inappropriate at this point on the learning curve…)

Edit: fix a couple of typos.

8

or if you are using both keys, try adding the line

IdentityFile ~/.ssh/id_rsa2

to the server2 block on your config file. It should look like

Host server2
Hostname 192.168.10.242
User tux
Port 22
IdentityFile ~/.ssh/id_rsa2

Then your ssh connection will automatically use the proper key file. Nice catch efahl!

2 Likes
9

As per advice I have deleted .ssh folder on both server1 and server2. and executed “ssh-keygen -t rsa” on server1 and after generating the public I then created the “config” file and then executed “chown $USER” and “chmod 644” to the “config” file. I copied it to server2 using ssh-copy-id -i id_rsa.pub server2. After doing this it still uses the password instead of the key. I did the same process in server2 to check if I will have the same problem and it worked perfectly. I have a snapshot of server1 from the beginning (no configuration aside from updates and network settings) I followed the same procedure and it worked perfectly. It must be some setting in either sshd_config or somewhere that may be causing this on server1.

Anyway thank you for for trying to help me. Probably I’ll just redo the image for server1.

10

Here are some more things to check:

  1. Make sure PubKeyAuthentication is “yes” in your /etc/ssh/sshd_config file. The default is “yes”, so if it’s commented out, then it’s turned on.

  2. Run “ssh -G serverN” and look at the output. Gah! Too much to pick through by hand, so on each machine do “ssh -G serverN > serverN_output”, then get the two “*_output” files on one machine. Run “diff server1_output server2_output” and you’ll see what’s different between them.

Hopefully the diff output will point to the problem. In any case, report back, we’re curious about what’s going on here! (It should “just work”, but…)

11

This is what I got after comparing the files generated when connecting to each server

[tux@server1 ~]$ diff server2_output server1_output
2c2

hostname 192.168.10.242

hostname 192.168.10.243

12

Well, that didn’t tell us much, did it?

If you’re willing to spend more time on it, you can run sshd in “extended test mode” and have it dump its configuration with ‘sudo sshd -T’ (sshd just reads its configs and keys, dumps everything, then quits). If you redirect the output into files and diff them, maybe that will be more informative (although it looks to me like most of what is important is already in the ‘ssh -G’ output).

13

Thanks for the help man, I am all good in spending more time on this, that is my purpose I want to learn. Unfortunately running “sudo sshd -T server2 > server2_output2” did not return any information:

[tux@server1 ~]$ sudo sshd -T server2 > server2_output2
[sudo] password for tux:
Extra argument server2.

[tux@server1 ~]$ ls -ltr
total 1669924
-------------Output Cut--------------------
-rw-rw-r–. 1 tux tux 3536 Dec 11 22:18 server2_output
-rw-rw-r–. 1 tux tux 3536 Dec 11 22:22 server1_output

14

I was hoping you’d say that regarding the learning part. :grinning: I’ve always hated letting something go until I’ve figured out the root cause, and now that I’m retired and have time to burn, I can afford to do so…

Back to the issue: we’re running the demon here " sshd ", not the client “ssh”, so it doesn’t take a host name, just that “-T” argument:

[tux@server1 ~]$ sudo sshd -T > server1_config

Try running it without redirecting the output ‘sudo sshd -T’ and you’ll see it spew all sorts of stuff when it works properly.

15

Here is what I have got:

[tux@server1 ~]$ diff server1_config server2_config
14c14
< permitrootlogin yes

permitrootlogin no
78c78
< authorizedkeysfile .ssh/authorized_keys

authorizedkeysfile /root/.ssh/authorized_keys

Weird thing is on server2, if I do to “vi /etc/ssh/sshd_config” and check the “autorizedkeysfile” it says “~/.ssh/autorized_keys” can’t remember if I change this before:

# The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2
# but this is overridden so installations will only check .ssh/authorized_keys
AuthorizedKeysFile      ~/.ssh/authorized_keys
16

This shouldn’t affect your ssh connection unless you are trying to use an authorized_keys2 file. If you are anything like me then, it’s probable that you removed the comment on this line to ensure that authorized keys were being used when you were troubleshooting the ssh connection issues.

Is there any chance you have misplaced any settings in the /etc/ssh/ssh_config file by mistake? It’s looking more and more like it’s a client side connection issue to me.

If that’s not it. Try running the sshd in debug mode using a different port by running /usr/sbin/sshd -d -p 2222 on server2. Watch the output when you try to connect from server1 using ssh -p 2222. You should be able to see the public key “handshake” and what went wrong.

2 Likes
17

That /root is probably just because of the sudo, if it were possible to run it as user tux it would probably be /home/tux/.ssh/authorized_keys, so I don’t think that’s an issue.

@hulxmash Yeah, I’ve been thinking it’s a client side issue for a while, too, good tips there.

18

As per instruction I have ran server2 using “sshd -d -p”

[tux@server2 ~]$ sudo /usr/sbin/sshd -d -p 34627
debug1: sshd version OpenSSH_7.4, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: private host key #0: ssh-rsa SHA256:ssb4fn3UAtflCtBLI3N5H5hSXI+Z7KQ6CPguMlAzoNA
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:oV4f6EICfpOJ4YYf7KIWxq8Ga3yPj/6VSBX0yFXG3Uk
debug1: private host key #2: ssh-ed25519 SHA256:1OR73+mSppNAjAoEVLC6vhFl40/e7Py23TdT/MFHXpI
debug1: rexec_argv[0]=‘/usr/sbin/sshd’
debug1: rexec_argv[1]=‘-d’
debug1: rexec_argv[2]=‘-p’
debug1: rexec_argv[3]=‘34627’
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 34627 on 0.0.0.0.
Server listening on 0.0.0.0 port 34627.
debug1: Bind to port 34627 on ::.
Server listening on :: port 34627.

When I try to connect via ssh coming from server1, I got this output:

debug1: userauth-request for user tux service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
debug1: userauth_pubkey: test whether pkalg/pkblob are acceptable for RSA SHA256:Ji1d4762rLaxsP8JZ070JQYjVtBAkcQ8r5Kr/mbcy9g [preauth]
debug1: temporarily_use_uid: 1000/1000 (e=0/0)
debug1: trying public key file /root/.ssh/authorized_keys
debug1: Could not open authorized keys ‘/root/.ssh/authorized_keys’: Permission denied
debug1: restore_uid: 0/0
Failed publickey for tux from 192.168.10.243 port 57974 ssh2: RSA SHA256:Ji1d4762rLaxsP8JZ070JQYjVtBAkcQ8r5Kr/mbcy9g

What I did is delete the .ssh directory under root’s home directory and then went to /etc/ssh/sshd_config and edit the entry under “authorizedkeysfile” back to “.ssh/authorized_kerys” This addressed my issue.

Thank you so much for the help.

1 Like
19

I’m glad you got it working. This was a good one to learn. Thanks for sharing.

20

Thank you for sharing that debug command without it I won’t be able to address my issue and thank to @efahl for providing information that help me understand on how I can troubleshoot ssh connection issue.