MacOS Sierra: Could not add card "/usr/local/lib/opensc-pkcs11.so": agent refused operation (Redux)

[minfrin]

Expected behaviour

What should happen?

Little-Net:~ minfrin$ ssh-add -s /usr/local/lib/opensc-pkcs11.so
Enter passphrase for PKCS#11:

Card added successfully.

Actual behaviour

What happens instead?

Little-Net:~ minfrin$ ssh-add -s /usr/local/lib/opensc-pkcs11.so
Enter passphrase for PKCS#11:
Could not add card "/usr/local/lib/opensc-pkcs11.so": agent refused operation
Little-Net:~ minfrin$ ls -al /usr/local/lib/opensc-pkcs11.so
-rwxr-xr-x 1 root wheel 1679920 May 31 14:35 /usr/local/lib/opensc-pkcs11.so

Steps to reproduce

1. Install OpenSC-0.16.0-179-g777e2a on MacOS Sierra v10.12.5 (16F73)
2. Apply workaround as per https://github.com/OpenSC/OpenSC/pull/1041/files - this works once.
3. Unplug the smartcard, put laptop to sleep, apply more MacOS Sierra updates.
4. Use ssh-add -s /usr/local/lib/opensc-pkcs11.so
5. Could not add card "/usr/local/lib/opensc-pkcs11.so": agent refused operation

There appears to be further regression of MacOS Sierra and https://github.com/OpenSC/OpenSC/pull/1041/files doesn't fix this.

[martinpaljak]

IIRC you need to use an agent from brew or selfcompiled

[frankmorgner]

Unfortunately, I cannot currently test ssh usage. I assumed after getting 👍 in #1041, the patch got tested...

If the problem lies within the PKCS#11 library, you may raise the debug level to see what's going on.

[gae123]

@martinpaljak Can you please elaborate how to "use an agent from brew" or point to some writeup. I understand that macos has a builtin ssh-agent that nicely interacts with the macos keychain and starts when the os boots.

@frankmorgner how can we "raise the debug level" to see what is going on?

[frankmorgner]

@gae123 set debug = 3; in /Library/OpenSC/etc/opensc.conf

[gae123]

I do not see anything suspicious. The message is exactly the same independently on what I pass too ssh-add:

$ ssh-add -s foo
Enter passphrase for PKCS#11:
Could not add card "foo": agent refused operation

ssh-add does not seem to have a -v/-V/--version option:

$ type ssh-add
ssh-add is hashed (/usr/bin/ssh-add)
$ ls -l /usr/bin/ssh-add
-rwxr-xr-x 1 root wheel 1829936 Apr 28 18:59 /usr/bin/ssh-add

[frankmorgner]

It could also be that your ssh-agent is not working (independent from OpenSC)...

@bwithem, @mouse07410 do you have similar problems?

Do you know a simple setup of this use case for local debugging?

[gae123]

No issue with password protected file based keys though:

$ ssh-add ~/.ssh/id_rsa
Enter passphrase for /Users/gae123/.ssh/id_rsa:
Identity added: /Users/gae123/.ssh/id_rsa (/Users/gae123/.ssh/id_rsa)

[Jakuje]

Please, run the ssh-agent in debug mode to see what is the problem in your case:

ssh-agent -d -a ~/.agent.socket

and in different terminal try to add the card:

SSH_AUTH_SOCK="~/.agent.socket" ssh-add -s /usr/local/lib/opensc-pkcs11.so

[gae123]

Magically after I copied again with:

sudo cp -p /Library/OpenSC/lib/opensc-pkcs11.so /usr/local/lib/opensc-pkcs11.so

everything worked. Not sure what has fixed the issue. I have not rebooted the machine since then, the only possibly material change I can think of is that I installed the krypt.co kr tools.

@Jakuje Thank you very much, I did confirm that your local debugging directions work and I will double check them if I run into this again.

[minfrin]

Ran into this again while testing v1.7.0-rc1.

It seems that if the following card is left plugged into the Mac when the machine is rebooted we get the "agent refused operation" from ssh-add:

Using reader with a card: ACS ACR101 ICC Reader
PKCS#15 Card [XXX]:
Version : 0
Serial number : XXX
Manufacturer ID: Aventra Ltd.
Last update : 20170424165641Z
Flags : PRN generation, EID compliant

As soon as the card is physically unplugged from the USB, and then reinserted, retrying the ssh-add works:

[reboot machine with card plugged in]
Little-Net:~ minfrin$ ssh-add -s /usr/local/lib/opensc-pkcs11.so
Enter passphrase for PKCS#11:
Could not add card "/usr/local/lib/opensc-pkcs11.so": agent refused operation
Little-Net:~ minfrin$ pkcs15-tool --verify-pin
No smart card readers found.
[unplug smartcard then plug it back in]
Little-Net:~ minfrin$ ssh-add -s /usr/local/lib/opensc-pkcs11.so
Enter passphrase for PKCS#11:
Card added: /usr/local/lib/opensc-pkcs11.so

This is using the native ssh-add that comes with MacOS Sierra:

Little-Net:~ minfrin$ which ssh-add
/usr/bin/ssh-add
Little-Net:~ minfrin$ ls -al /usr/bin/ssh-add
-rwxr-xr-x 1 root wheel 1829936 Apr 29 03:59 /usr/bin/ssh-add

[frankmorgner]

It seems that macOS has some problems propagating the smart card reader or its state. If so, this should also apply to the OpenSC version from brew @martinpaljak. If I understood correctly, the original problem is fixed. @minfrin if you think you are able to debug the problem about the card reader that's unavailable for OpenSC, although plugged into the mac, then please open a new issue with a log. You may simply copy your comment above to the new issue's description.

[minfrin]

if you think you are able to debug the problem about the card reader
that's unavailable for OpenSC, although plugged into the mac, then please
open a new issue with a log

I don't follow at all?

[minfrin]

Right now, after putting the machine to sleep and waking it up, I need to reboot the machine before the smartcard will give me anything other than "agent refused operation".

[frankmorgner]

Sorry, please ignore my comment from above. I got mixed up with the other issue about the file permissions

The problem with sleep/weakup still needs to be debugged:

1. in OpenSC, see MacOS Sierra: Could not add card "/usr/local/lib/opensc-pkcs11.so": agent refused operation (Redux) #1060 (comment)
2. in ssh-agent, see MacOS Sierra: Could not add card "/usr/local/lib/opensc-pkcs11.so": agent refused operation (Redux) #1060 (comment)

[foozmeat]

I was having this error repeatedly and I discovered that the Yubikey was locked due to too many bad PINs. As soon as I reset the PIN it worked perfectly.

[frankmorgner]

@foozmeat are you referring to the problem when going to sleep and weaking up?

[foozmeat]

@frankmorgner no sorry, my reply was to the original bug report.

I installed OpenSC from homebrew as well as a direct download and could not get the key added to the agent. I confirmed that the library was copied into place and not a symlink. It wasn't until after running pkcs15-tool --verify-pin that I realized what the true problem was.

It's too bad that the real error isn't displayed when adding to the agent.

[frankmorgner]

@foozmeat with #1126 you'll have notifications about a failed (or successful) PIN verification.

[frankmorgner]

@minfrin if you want your problem during wakeup to be resolved, please send a debug log of OpenSC which should contain the specific errors. I'm suspecting that the agent uses an old session after resume, though OpenSC sends errors like CKR_DEVICE_REMOVED, which indicate that a new session needs to be opened. Maybe restarting the agent fixes your problem. But to be sure we would have to look at the debug log.

[gae123]

@frankmorgner I had the problem again this morning. This time, restarting the ssh-agent appeared to be a workaround.

Here is the error I was getting:
$ ssh-add -s /usr/local/lib/opensc-pkcs11.so Enter passphrase for PKCS#11: Could not add card "/usr/local/lib/opensc-pkcs11.so": agent refused operation

I also conclude that the cause might be related to suspend/resume of MacOS. Here is what I saw:

1. I am at Location 1, everything works fine
2. Remove my key, close my laptop lid and go to Location 2
3. I resume my MacOS
4. Several hours later I put the key in and attempt to ssh. I cannot:
   ssh bastion sign_and_send_pubkey: signing failed: agent refused operation no such identity: /Users/<<username>>/.ssh/id_edXXXXXX: No such file or directory Permission denied (publickey).
5. When I do ssh-add -L I see the key along with two other keys as expected but there is no way to remove the key, I get an error. Same when I try to add a key.

[frankmorgner]

Please open a new issue for resume/suspend and use the debugging options mentioned above for both, ssh and OpenSC, thanks.

[minfrin]

In an effort to work out the patterns behind what's going on, I've turned on debug logging and the first thing I've noticed is that if the Mac is restarted with the token still plugged in, the token will be ignored until the token is physically unplugged and plugged back in again. During this state, Firefox claims no token is plugged in, and ssh-add -s says "agent refused operation".

[minfrin]

New behavior this morning - woke my laptop from sleep, plugged the smartcard in, and the smartcard was ignored. No messages indicating the smartcard was plugged or unplugged in /tmp/opensc-debug.log.

[dengert]

Google for: USB not working after sleep mode This could be the reader is powered off.

…

On 10/2/2017 4:01 AM, minfrin wrote: New behavior this morning - woke my laptop from sleep, plugged the smartcard in, and the smartcard was ignored. No messages indicating the smartcard was plugged or unplugged in /tmp/opensc-debug.log. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1060 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA00MZAP5vT5RS_Stt70cZgs7UtrNOTkks5soKZ1gaJpZM4Ny5wZ>.

-- Douglas E. Engert <[email protected]>

[minfrin]

"USB not working after sleep mode" is too general a search, is there a specific known mac issue? I am not finding one.

The USB reader is powered up with LEDs on, so USB is definitely working after sleep.

[jstoja]

Using OpenSC 0.19 installed with brew, I also got Failed to unlock agent: agent refused operation when doing the ssh-add, despite changing the PIN, and the OpenSC notification correctly telling me that the SC was added.

The reason I found is that I had a ECCP256 key that openssh doesn't seem to like. I added a second RSA2048 key and now it works well.
I don't know why but maybe someone here will be able to explain :)

[martinpaljak]

Agent support for ECC keys comes with OpenSSH 8.0

[Jakuje]

I would have to see the logs from the agent. Also trying to login directly without agent would be helpful. Noting what card you have and if yubikey, in which slot the ecdsa key is shuodl also help.

[jstoja]

it's a yubikey and the key is in slot 0:

$ ssh -v gateway
OpenSSH_8.0p1, OpenSSL 1.0.2r  26 Feb 2019
debug1: Reading configuration data /Users/bordel/.ssh/config
debug1: /Users/bordel/.ssh/config line 1: Applying options for *
debug1: /Users/bordel/.ssh/config line 12: Applying options for gateway
debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
debug1: Connecting to 192.168.2.2 [192.168.2.2] port 22.
debug1: Connection established.
debug1: provider /Library/OpenSC/lib/opensc-pkcs11.so: manufacturerID <OpenSC Project> cryptokiVersion 2.20 libraryDescription <OpenSC smartcard framework> libraryVersion 0.19
debug1: provider /Library/OpenSC/lib/opensc-pkcs11.so slot 0: label <ECC-P256> manufacturerID <piv_II> model <PKCS#15 emulate> serial <> flags 0x40d
Enter PIN for 'ECC-P256':
skipping unsupported key type
failed to fetch key
unknown certificate key type
failed to fetch key
debug1: pkcs11_provider_finalize: 0x7fbbfb7000c0 refcount 1 valid 1
debug1: pkcs11_provider_unref: 0x7fbbfb7000c0 refcount 1
debug1: pkcs11_add_provider: provider /Library/OpenSC/lib/opensc-pkcs11.so returned no keys

I don't know if that would help

[dengert]

There may be issues with the keyUsage flags in the certificate. (With the PIV applet the PIV driver reads the certificate to obtain the public key and the namedCurve. And for a non-government issued card, the certificate keyUsage is used to set a number of flags.
If you can run pkcs11-tool -O --login and post the output that would help.

[jstoja]

Here is the output:

$ pkcs11-tool -O --login
Using slot 0 with a present token (0x0)
Logging in to "Julien".
Please enter User PIN:
Private Key Object; EC
  label:      PIV AUTH key
  ID:         01
  Usage:      sign, non-repudiation
Public Key Object; EC  EC_POINT 256 bits
  EC_POINT:   044104c8c6921d67d216b76593b4bcd2a859d98099348d5c0546095e397904343c81ddff6c5222b0f33166779d1d28d93f55641c2205e83e1990006d4c11419b4c86e3
  EC_PARAMS:  06082a8648ce3d030107
  label:      PIV AUTH pubkey
  ID:         01
  Usage:      verify
Certificate Object; type = X.509 cert
  label:      Certificate for PIV Authentication
  subject:    DN: CN=Julien
  ID:         01
Data object 163666992
  label:          'Cardholder Fingerprints'
  application:    'Cardholder Fingerprints'
  app_id:         2.16.840.1.101.3.7.2.96.16
  flags:           private
Data object 163667088
  label:          'Printed Information'
  application:    'Printed Information'
  app_id:         2.16.840.1.101.3.7.2.48.1
  flags:           private
Data object 163667184
  label:          'Cardholder Facial Image'
  application:    'Cardholder Facial Image'
  app_id:         2.16.840.1.101.3.7.2.96.48
  flags:           private
Data object 163666560
  label:          'Card Capability Container'
  application:    'Card Capability Container'
  app_id:         2.16.840.1.101.3.7.1.219.0
  flags:          <empty>
Data object 163666704
  label:          'Card Holder Unique Identifier'
  application:    'Card Holder Unique Identifier'
  app_id:         2.16.840.1.101.3.7.2.48.0
  flags:          <empty>
Data object 163666800
  label:          'Unsigned Card Holder Unique Identifier'
  application:    'Unsigned Card Holder Unique Identifier'
  app_id:         2.16.840.1.101.3.7.2.48.2
  flags:          <empty>
Data object 163666896
  label:          'X.509 Certificate for PIV Authentication'
  application:    'X.509 Certificate for PIV Authentication'
  app_id:         2.16.840.1.101.3.7.2.1.1
  flags:          <empty>
Data object 163667280
  label:          'X.509 Certificate for Digital Signature'
  application:    'X.509 Certificate for Digital Signature'
  app_id:         2.16.840.1.101.3.7.2.1.0
  flags:          <empty>
Data object 163667376
  label:          'X.509 Certificate for Key Management'
  application:    'X.509 Certificate for Key Management'
  app_id:         2.16.840.1.101.3.7.2.1.2
  flags:          <empty>
Data object 163667472
  label:          'X.509 Certificate for Card Authentication'
  application:    'X.509 Certificate for Card Authentication'
  app_id:         2.16.840.1.101.3.7.2.5.0
  flags:          <empty>
Data object 163667568
  label:          'Security Object'
  application:    'Security Object'
  app_id:         2.16.840.1.101.3.7.2.144.0
  flags:          <empty>
Data object 163667664
  label:          'Discovery Object'
  application:    'Discovery Object'
  app_id:         2.16.840.1.101.3.7.2.96.80
  flags:          <empty>

[dengert]

Looks like OpenSSH 8.0 will only support ECDSA with OpenSSL 1.1 You are running with OpenSSL-1.0.2r
configure.ac has:

2975         if test x$enable_nistp256 = x1 || test x$enable_nistp384 = x1 || \
2976             test x$enable_nistp521 = x1; then
2977                 AC_DEFINE(OPENSSL_HAS_ECC, [1], [OpenSSL has ECC])
2978                 AC_CHECK_FUNCS([EC_KEY_METHOD_new])
2979         fi

OpenSSL-1.0.2 does not have EC_KEY_METHOD_new
OpenSSL-1.1 has it in openssl/include/ec.h

[LudovicRousseau]

I think the "agent refused operation" issue is the same as #1007

[michael-scott-howard]

Do this as root (sudo bash)
cd /usr/local/lib
ls -al *opensc *
for each symbolic link that as opensc in the path, remove them one at a time and copy the file pointed to by the each link to /usr/local/lib
(it's an Apple OS X security thing...)

them do:

ssh-add -s /usr/local/lib/opensc-pkcs11.so

and ssh-add -l should show your keys

ssh and carry on....