So I've managed to get the thing working fully! Watch here:
https://www.youtube.com/watch?v=fl5KW1p3LQ8Regarding the authentication speed, it is now much faster than it was(It's now a little more than a second), but it's still a tad slower than ideal... Caching the certificate helps a ton. Using ECDSA is also noticeably faster than RSA. But I think the problem at the moment is in the software I'm using (OpenSC's
pkcs15-tool) to get the signed nonce from the token is doing a lot of extraneous transactions.
I'll eventually be writing my own software to do this, but I've got so many personal projects going on it may be a bit before I can get around to it. :/
In any case, I hope to write up a blog post about this setup soon. Will post a link when I do!