Update: After improving my testing methodology, I was able to run through several more tests with more targeted challenges.
I'm not 100% certain that I've hit all of the cases yet, but I can say with fairly high confidence that the algorithm below will cover most - if not all - cases.
Based on my latest tests, the algorithm appears to be:
1. If the challenge is greater than 64 bytes long, truncate it to 64 bytes. (This is actually done in YkLib; I have not checked whether the hardware performs the same truncation if passed a longer challenge.)
2. Prepare the challenge:
- If the challenge is empty (zero bytes long), replace it with a single 0x00.
- If the challenge ends with one or more 0x00s, remove them.
- If the challenge is exactly 64 bytes long, remove the last byte and all adjacent bytes of the same value.
3. Calculate the SHA-1 HMAC of the prepared challenge.
I'd still be interested to hear from the Yubico engineers on this. Are there any corner cases I've missed? Is this behavior by design? Can I expect it to be consistent across all types of YubiKeys?