Removing the YubiKey 4 after you use it and re-inserting it when you need it should require you to re-enter the PIN. An alternative solution would be to add the touch functionality to the keys (
https://developers.yubico.com/PGP/Card_edit.html) - refer to the section "YubiKey 4 touch." If you're using it for signing, you can set forcesig to always require the PIN.
There are cache-timeout and card-timeout options (for gpg-agent and scdaemon, respectively), but they are pretty hit-and-miss on functionality.