I'm facing a frustrating issue with SSH connections using my Yubikey for PIV authentication. When I connect to a Windows server, the smartcard works fine, but when I connect from that Windows server to a Linux server using the same smartcard, I experience delays of 5 to 40 seconds for the PIN request. This delay happens regardless of whether I use ssh-agent or third-party tools like open-sc or wincrypt. My colleagues and I have analyzed network traces with Wireshark and monitored processes, but we haven't identified any specific errors, just a lot of smartcard read access logs. While using ssh-add helps speed things up by caching the PIN, we want to avoid that for security reasons. Additionally, without caching the PIN, I can't forward my keys when jumping to another machine via SSH. Has anyone experienced a similar problem or found a solution?
3 Answers
Are you using the Windows middleware that comes with the Yubikey for PIV? You've mentioned trying different tools, but the default middleware might be worth testing too.
Have you tried running the command `ssh -vvv` to check what’s slowing down the logon process? It could be that you have multiple authentication methods enabled, and the server is taking time to negotiate which one to use. I recommend disabling other methods besides the smartcard to see if it helps, especially GSSAPI.
Could the delay be related to a timeout when accessing the Certificate Revocation List (CRL) for your certificate? Those intermittent delays make me think there’s a timeout issue going on.

Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures