The admin panel is separate to user login.
It it designed to allow access without SAML (how else can you login if your SAML has gone bad?).
My suggestion would to be use a long complex password for Admin Panel access, and limit the number of administrators.
I have found myself if you login, then click the "Manage this domain" link, you don't need to re-authenticate.
Remember with Google Apps, there are still some services that use your password, Yubikey is not an excuse to use poor passwords underneath. In the ideal world you would use your Yubikey with SAML as a two factor authentication, this way you get the benefits of OTP plus something you know.
Cheers,
Arron
http://www.totali.co.nz/