Yubico Forum

[SOLVED] - GethmacSha1 API returning ycRETCODE_FAILED
Page 1 of 1

Author:  evile [ Thu Aug 11, 2016 5:30 pm ]
Post subject:  [SOLVED] - GethmacSha1 API returning ycRETCODE_FAILED


I'm working with the Windows COM library and trying to use the GethmacSha1 API function. I'm receiving the ycRETCODE_FAILED error code.

I've copy and pasted the DoChallengeResponse from https://github.com/Yubico/yubico-window ... ubAuth.cpp into my my project and when I call it, it consistently returns false. The false return code is due to the call to GethmacSha1 failing with ycRETCODE_FAILED. The Windows COM library does not appear to be on github, so thats why I posting this message here.

I'm using Windows 10 64bit
I've tried both 32 and 64bit version of the COM library.

My code which is below checks to see if a device is present, gets the serial number, and then attempts to call the GethmacSha1 function.

Found YubiKey Serial#: 004bXXXX
Do challenge failed

Can anyone tell me what I'm doing wrong.


#include <Windows.h>
#include <iostream>
#include <sstream>
#include <string>
#include <iomanip>
#include "ykdef.h"

#import <YubiClientAPI.dll> no_namespace, named_guids

using namespace std;

IYubiClient* api;

bool DoChallengeResponse(__in BYTE* challenge, __out BYTE* response, __in DWORD len)
   bool res = true;
   variant_t va;
   ostringstream os;
   os << hex << setfill('0');
   for (DWORD i = 0; i < len; i++) {
      os << setw(2) << int(challenge[i]);
   _bstr_t bstr(os.str().c_str());

   va.bstrVal = bstr;
   va.vt = VT_BSTR;
   ycRETCODE ret = api->GethmacSha1(2, ycCALL_BLOCKING);
   if (ret == ycRETCODE_OK) {
      BYTE HUGEP *pb;
      long lbound, hbound;
      SafeArrayGetLBound(api->dataBuffer.parray, 1, &lbound);
      SafeArrayGetUBound(api->dataBuffer.parray, 1, &hbound);
      SafeArrayAccessData(api->dataBuffer.parray, (void **)&pb);
      for (; lbound <= hbound; lbound++) {
         *response++ = *pb++;
      res = true;
   else {
      res = false;

   return res;

int main()
   HRESULT h = CoCreateInstance(CLSID_YubiClient, 0, CLSCTX_ALL, IID_IYubiClient, reinterpret_cast<void **>(&api));
   if (FAILED(h)) {

   ycRETCODE ret = api->GetisInserted();
   if (ret == ycRETCODE_NO_DEVICE)
      cout << "No yubikey device found" << endl;
      return -1;

   api->dataEncoding = ycENCODING::ycENCODING_UINT32;
   ret = api->GetreadSerial(ycCALL_MODE::ycCALL_BLOCKING);
   cout << "Found YubiKey Serial#: " << setw(8) << setfill('0') << hex << api->dataBuffer.intVal << endl;

   DWORD len = 64;
   BYTE challenge[128];
   BYTE response[128];
   memset(challenge, 0, sizeof(challenge));
   memset(response, 0, sizeof(response));
   BCryptGenRandom(NULL, challenge, len, BCRYPT_USE_SYSTEM_PREFERRED_RNG);
   if (DoChallengeResponse(challenge, response, 63))
      cout << "Do challange worked" << endl;
      cout << "Do challenge failed" << endl;
   return 0;

Author:  evile [ Thu Aug 11, 2016 8:07 pm ]
Post subject:  Re: [QUESTION] - GethmacSha1 API returning ycRETCODE_FAILED

Solved. The YubiKey I was using was not personalized for hmac-sha1 using the yubikey-personalization-gui.

Page 1 of 1 All times are UTC + 1 hour
Powered by phpBB® Forum Software © phpBB Group