To answer some of your questions:
The master device secrets used to generate the site-specific keys are generated on-chip during the programming part of manufacturing (this is also when the attestation certificates are loaded), using the devices RNG. The secret never leaves the device. Let me clearly state that at no point do we ever see these master secrets, and we cannot extract any private keys from any key handles. Just as darco says here and in the old topic (
viewtopic.php?p=6542#p6542), the reason this key isn't settable by the user is that the whole attestation part requires this. The attestation certificate attests to the fact that the key signed by the certificate come from a particular type of device, correctly adhering to the U2F protocol with safeguards in place to prevent device cloning. There is simply no way to do this if the private keys can be re-created outside the device.
In the end it boils down to trust. We are quite open with how our U2F keys are generated, so you can verify that the algorithms used are sound for yourself. You cannot verify that we are truthful, but if you assume that we are lying, then you must also assume that we could potentially take steps to intentionally compromise the security of our devices. Under this assumption, we could do any number of things, such as leak the master key through the ECDSA signatures, or even have a backdoor into the device by sending some secret command. Both of these could be done even with a user supplied device secret.
We could potentially have a command to re-generate the secret on-chip, and still be able to trust attestation. We might do this in the future as a means of "wiping" a device. The reason we have not yet done so is that it potentially introduces new attack vectors (like unwillingly wiping someones device, destroying the credentials). This does NOT however do anything to prevent us from leaking information in the above described way, so you still need the same level of trust.
We could even take more steps to make the whole process of key generation verifiable by the user. I've spent some time thinking about the problem, and have devised a scheme where the user can verify that user-provided entropy has gone into the creation of a random device secret, and that that secret is used for the creation of each key pair and MAC, without giving the user direct access to the device secret or the private keys. However, this approach falls short, as the R value used in the ECDSA signatures cannot be verified to not contain a backdoor without direct access to the private key. Thus, this approach only serves to add a bunch of complexity to the key generation, with no real benefit (and was thus shelved). The bottom line is, you have to place some trust in the device vendor, there will always be room for secret backdoors.
As for OTP vs U2F: U2F imposes guards against several attack vectors that OTP does not. OTP by its nature is simpler and does not require the same type of client support as U2F does. We aren't saying that one is superior to the other, we're offering both and hope to help you choose which one suits your needs the best.