S
Sean Huang
Hi,
May I ask a question about an NTE_BAD_DATA error returned by CryptDecrypt()?
I am working on an application of SSL client/server connection using
Microsoft Certificates and Crypto API. I'm running Windows XP.
Currently the SSL connection between my client and server works, but when I
use IE 6.0 (SSL 3.0/TLS 1.0) to connect to the server, the server failed with
an NTE_BAD_DATA error at CryptDecrypt().
This is the step where the server tries to decrypt the pre-master secret
encrypted by IE. In my own client code, I use CryptEncrypt() to encrypt the
pre-master secret with the server's public key. Not sure if IE uses the same
functions..
Below I've included all the functions I called to decrypt the encrypted
message in the server side:
(1) CryptAcquireContext(&hProv, mcskp->szKeyContainerName,
mcskp->szProvName, PROV_RSA_FULL, 0)
(2) CryptGetUserKey (hProv, AT_KEYEXCHANGE, &hKey)
(3) CryptDecrypt(hKey, 0, TRUE, 0, (BYTE *)tempbuffer,
&tempbufferlength) <--- FALSE was returned here, and NTE_BAD_DATA
(80090005) was reported by GetLastError().
* I've tried four CSPs, but I got the same error in all of them:
"Microsoft Enhanced Cryptographic Provider v1.0", "Microsoft Strong
Cryptographic Provider", "Microsoft RSA Schannel Cryptographic
Provider", and "Microsoft Base Cryptographic Provider v1.0".
* The pre-master secret is supposed to be of size 48 bytes, so setting
the Final flag as TRUE should be correct.
Because NTE_BAD_DATA indicates that the data to be decrypted is invalid, I
logged the value in 'tempbuffer' as below. ('tempbufferlength'==128)
5F EB 8C 0B D7 C2 7C C0 94 8F F9 0F 85 33 30 86
73 B6 DA BA 11 E3 93 AE 3C 02 28 56 E5 06 B4 49
83 2F 44 1B C5 CF C0 21 95 BC 53 0B 03 1C 49 C9
49 0F A6 69 B8 E1 22 9F 04 A2 08 68 43 63 9D 69
53 57 79 C9 FE 5C 02 72 BB 4F 1B 7D EF EB 16 53
E3 62 1A C0 83 D4 71 C5 89 52 E9 1D FE 65 DB C4
CE 33 75 F1 D0 54 E0 87 59 14 43 E5 5E 1C C6 49
62 A7 D4 EC 67 D4 73 7C BB 5F F5 FB 08 EA AC 62
I intended to check why it's invalid using the server's private key, but I
don't know how to print the value of a HCRYPTKEY variable or how to learn it
from the certificate.
I created the user certificate and its trust point, a self-signed
certificate, by the following commands:
* makecert -pe -n "CN=Trusted" -r -b 08/01/2007 -e 12/31/2010 -sp
"Microsoft Strong Cryptographic Provider" -sky exchange -ss ca
* makecert -pe -is ca -in "Trusted" -n "CN=User" -b 08/01/2007 -e
12/31/2010 -sp "Microsoft Strong Cryptographic Provider" -sky exchange
-eku 1.3.6.1.5.5.7.3.1,1.3.6.1.5.5.7.3.2 -ss my
May someone kindly advise what could be the problem in my case? Your help
would be much appreciated.
Thanks,
Sean
May I ask a question about an NTE_BAD_DATA error returned by CryptDecrypt()?
I am working on an application of SSL client/server connection using
Microsoft Certificates and Crypto API. I'm running Windows XP.
Currently the SSL connection between my client and server works, but when I
use IE 6.0 (SSL 3.0/TLS 1.0) to connect to the server, the server failed with
an NTE_BAD_DATA error at CryptDecrypt().
This is the step where the server tries to decrypt the pre-master secret
encrypted by IE. In my own client code, I use CryptEncrypt() to encrypt the
pre-master secret with the server's public key. Not sure if IE uses the same
functions..
Below I've included all the functions I called to decrypt the encrypted
message in the server side:
(1) CryptAcquireContext(&hProv, mcskp->szKeyContainerName,
mcskp->szProvName, PROV_RSA_FULL, 0)
(2) CryptGetUserKey (hProv, AT_KEYEXCHANGE, &hKey)
(3) CryptDecrypt(hKey, 0, TRUE, 0, (BYTE *)tempbuffer,
&tempbufferlength) <--- FALSE was returned here, and NTE_BAD_DATA
(80090005) was reported by GetLastError().
* I've tried four CSPs, but I got the same error in all of them:
"Microsoft Enhanced Cryptographic Provider v1.0", "Microsoft Strong
Cryptographic Provider", "Microsoft RSA Schannel Cryptographic
Provider", and "Microsoft Base Cryptographic Provider v1.0".
* The pre-master secret is supposed to be of size 48 bytes, so setting
the Final flag as TRUE should be correct.
Because NTE_BAD_DATA indicates that the data to be decrypted is invalid, I
logged the value in 'tempbuffer' as below. ('tempbufferlength'==128)
5F EB 8C 0B D7 C2 7C C0 94 8F F9 0F 85 33 30 86
73 B6 DA BA 11 E3 93 AE 3C 02 28 56 E5 06 B4 49
83 2F 44 1B C5 CF C0 21 95 BC 53 0B 03 1C 49 C9
49 0F A6 69 B8 E1 22 9F 04 A2 08 68 43 63 9D 69
53 57 79 C9 FE 5C 02 72 BB 4F 1B 7D EF EB 16 53
E3 62 1A C0 83 D4 71 C5 89 52 E9 1D FE 65 DB C4
CE 33 75 F1 D0 54 E0 87 59 14 43 E5 5E 1C C6 49
62 A7 D4 EC 67 D4 73 7C BB 5F F5 FB 08 EA AC 62
I intended to check why it's invalid using the server's private key, but I
don't know how to print the value of a HCRYPTKEY variable or how to learn it
from the certificate.
I created the user certificate and its trust point, a self-signed
certificate, by the following commands:
* makecert -pe -n "CN=Trusted" -r -b 08/01/2007 -e 12/31/2010 -sp
"Microsoft Strong Cryptographic Provider" -sky exchange -ss ca
* makecert -pe -is ca -in "Trusted" -n "CN=User" -b 08/01/2007 -e
12/31/2010 -sp "Microsoft Strong Cryptographic Provider" -sky exchange
-eku 1.3.6.1.5.5.7.3.1,1.3.6.1.5.5.7.3.2 -ss my
May someone kindly advise what could be the problem in my case? Your help
would be much appreciated.
Thanks,
Sean