Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] CIPHER_LIST on Windows is wrong since in v0.6.x #54

Closed
ed0nkey opened this issue Mar 7, 2024 · 14 comments
Closed

[BUG] CIPHER_LIST on Windows is wrong since in v0.6.x #54

ed0nkey opened this issue Mar 7, 2024 · 14 comments
Assignees
Labels
bug Something isn't working

Comments

@ed0nkey
Copy link

ed0nkey commented Mar 7, 2024

While using impersonate="safari17_2_ios", I get different JA3 digest results between my Windows and macOS machines (8be0b641abb257fae7b13bcfd2657032 on Mac and a76d766e1e01aa4cfaee1331b1bada3b on Windows).
Unfortunately this issue is triggering cloudflare on the Windows machine while on Mac it works just fine every time.
Just for testing I've tried using different python versions with an older openssl version but that is not the problem as the digest doesn't change.

@ed0nkey ed0nkey added the bug Something isn't working label Mar 7, 2024
@perklet
Copy link
Collaborator

perklet commented Mar 7, 2024

It's 773906b0efdefa24a7f2b8eb6985bf37 on my machine, do you mind sharing your code?

@ed0nkey
Copy link
Author

ed0nkey commented Mar 7, 2024

Yes here is on Windows:
windows
{"ja3":"772,4867-4865-4866-49196-49195-52393-49200-49199-52392-49162-49161-49172-49171-157-156-53-47-49160-49170-10,0-23-65281-10-11-16-5-13-18-51-45-43-27,29-23-24-25,0","ja3n":"772,4867-4865-4866-49196-49195-52393-49200-49199-52392-49162-49161-49172-49171-157-156-53-47-49160-49170-10,0-5-10-11-13-16-18-23-27-43-45-51-65281,29-23-24-25,0","ja3_digest":"a76d766e1e01aa4cfaee1331b1bada3b","ja3n_digest":"d3b4c8af5627378f76715637fd69aab6","scrapfly_fp":"version:772|ch_ciphers:GREASE-4867-4865-4866-49196-49195-52393-49200-49199-52392-49162-49161-49172-49171-157-156-53-47-49160-49170-10|ch_extensions:GREASE-0-5-10-11-13-16-18-23-27-43-45-51-65281-GREASE|groups:GREASE-29-23-24-25|points:0|compression:0|supported_versions:GREASE-772-771-770-769|supported_protocols:h2-http11|key_shares:GREASE-29|psk:1|signature_algs:1027-2052-1025-1283-515-2053-2053-1281-2054-1537-513|early_data:0|","scrapfly_fp_digest":"e8f1eaa27db671eea1f1c1c716755780","tls":{"version":"0x0303 - TLS 1.2","ciphers":["0x1A1A","TLS_CHACHA20_POLY1305_SHA256","TLS_AES_128_GCM_SHA256","TLS_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA","TLS_RSA_WITH_AES_256_GCM_SHA384","TLS_RSA_WITH_AES_128_GCM_SHA256","TLS_RSA_WITH_AES_256_CBC_SHA","TLS_RSA_WITH_AES_128_CBC_SHA","TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA","TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA","TLS_RSA_WITH_3DES_EDE_CBC_SHA"],"curves":["TLS_GREASE (0xFAFA)","X25519 (29)","secp256r1 (23)","secp384r1 (24)","secp521r1 (25)"],"extensions":["GREASE (0xAAAA)","server_name (0) (IANA)","extended_master_secret (23) (IANA)","extensionRenegotiationInfo (boringssl) (65281) (IANA)","supported_groups (10) (IANA)","ec_point_formats (11) (IANA)","application_layer_protocol_negotiation (16) (IANA)","status_request (5) (IANA)","signature_algorithms (13) (IANA)","signed_certificate_timestamp (18) (IANA)","key_share (51) (IANA)","psk_key_exchange_modes (45) (IANA)","supported_versions (43) (IANA)","compress_certificate (27) (IANA)","GREASE (0xCACA)","padding (21) (IANA)"],"points":["0x00"],"protocols":["h2","http/1.1"],"versions":["27242","772","771","770","769"],"handshake_duration":"193.217331ms","is_session_resumption":false,"session_ticket_supported":false,"support_secure_renegotiation":true,"supported_tls_versions":[27242,772,771,770,769],"supported_protocols":["h2","http11"],"signature_algorithms":[1027,2052,1025,1283,515,2053,2053,1281,2054,1537,513],"psk_key_exchange_mode":"AQ==","cert_compression_algorithms":"AA==","early_data":false,"using_psk":false,"selected_protocol":"h2","selected_curve_group":29,"selected_cipher_suite":4867,"key_shares":[64250,29]}}

And on Mac:
terminal
{"ja3":"772,4865-4866-4867-49196-49195-52393-49200-49199-52392-49162-49161-49172-49171-157-156-53-47-49160-49170-10,0-23-65281-10-11-16-5-13-18-51-45-43-27,29-23-24-25,0","ja3n":"772,4865-4866-4867-49196-49195-52393-49200-49199-52392-49162-49161-49172-49171-157-156-53-47-49160-49170-10,0-5-10-11-13-16-18-23-27-43-45-51-65281,29-23-24-25,0","ja3_digest":"8be0b641abb257fae7b13bcfd2657032","ja3n_digest":"ee9a64814f953b3433beeb4725f60b5f","scrapfly_fp":"version:772|ch_ciphers:GREASE-4865-4866-4867-49196-49195-52393-49200-49199-52392-49162-49161-49172-49171-157-156-53-47-49160-49170-10|ch_extensions:GREASE-0-5-10-11-13-16-18-23-27-43-45-51-65281-GREASE|groups:GREASE-29-23-24-25|points:0|compression:0|supported_versions:GREASE-772-771-770-769|supported_protocols:h2-http11|key_shares:GREASE-29|psk:1|signature_algs:1027-2052-1025-1283-515-2053-2053-1281-2054-1537-513|early_data:0|","scrapfly_fp_digest":"f638ee5bf20fa34a65437016daa32cf7","tls":{"version":"0x0303 - TLS 1.2","ciphers":["0xBABA","TLS_AES_128_GCM_SHA256","TLS_AES_256_GCM_SHA384","TLS_CHACHA20_POLY1305_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA","TLS_RSA_WITH_AES_256_GCM_SHA384","TLS_RSA_WITH_AES_128_GCM_SHA256","TLS_RSA_WITH_AES_256_CBC_SHA","TLS_RSA_WITH_AES_128_CBC_SHA","TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA","TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA","TLS_RSA_WITH_3DES_EDE_CBC_SHA"],"curves":["TLS_GREASE (0x1A1A)","X25519 (29)","secp256r1 (23)","secp384r1 (24)","secp521r1 (25)"],"extensions":["GREASE (0x1A1A)","server_name (0) (IANA)","extended_master_secret (23) (IANA)","extensionRenegotiationInfo (boringssl) (65281) (IANA)","supported_groups (10) (IANA)","ec_point_formats (11) (IANA)","application_layer_protocol_negotiation (16) (IANA)","status_request (5) (IANA)","signature_algorithms (13) (IANA)","signed_certificate_timestamp (18) (IANA)","key_share (51) (IANA)","psk_key_exchange_modes (45) (IANA)","supported_versions (43) (IANA)","compress_certificate (27) (IANA)","GREASE (0x2A2A)","padding (21) (IANA)"],"points":["0x00"],"protocols":["h2","http/1.1"],"versions":["27242","772","771","770","769"],"handshake_duration":"198.54514ms","is_session_resumption":false,"session_ticket_supported":false,"support_secure_renegotiation":true,"supported_tls_versions":[27242,772,771,770,769],"supported_protocols":["h2","http11"],"signature_algorithms":[1027,2052,1025,1283,515,2053,2053,1281,2054,1537,513],"psk_key_exchange_mode":"AQ==","cert_compression_algorithms":"AA==","early_data":false,"using_psk":false,"selected_protocol":"h2","selected_curve_group":29,"selected_cipher_suite":4865,"key_shares":[6682,29]}}

Hope this helps 👍

@perklet
Copy link
Collaborator

perklet commented Mar 7, 2024

I can verify that you are right. Not only the safari fingerprints on Windows is not correct, but also all other fingerprints are incorrect. The different parts is:

4867-4865-4866

which should be

4865-4866-4867

The fingerprints on macOS and Linux are correct. It seems that the BoringSSL behavior on Windows is not the same with other platforms. I'm not sure it's our mis-configuration or a bug/feature in BoringSSL, still investigating. Thanks for reporting it!

@ed0nkey
Copy link
Author

ed0nkey commented Mar 7, 2024

No problem! Thank you for the rapid replies and for creating all of this. Let me know when you figured it out so I can test on my machine too :)

@perklet perklet changed the title [BUG] Different behavior while impersonating [BUG] CIPHER_LIST on Windows is wrong since in v0.6.x Mar 8, 2024
@perklet
Copy link
Collaborator

perklet commented Mar 8, 2024

I'm moving this issue to upstream, since the bug is not within Python code.

@perklet perklet transferred this issue from lexiforest/curl_cffi Mar 8, 2024
@perklet
Copy link
Collaborator

perklet commented Mar 8, 2024

I may have found the root cause, it should be a side effect of #12, when we enabled OPENSSL_NO_ASM on Windows

If OPENSSL_NO_ASM is defined, then int hwaes_capable() will return 0, which means SSL_CTX_set_cipher_list will put chachaciphers(4867) in the first place, hence the wrong order.

// Order AES ciphers vs ChaCha ciphers based on whether we have AES hardware.
//                                                                           
// TODO(crbug.com/boringssl/29): We should also set up equipreference groups 
// as a server.                                                              
size_t num = 0;                                         
if (has_aes_hw) {                                       
  for (uint16_t id : kAESCiphers) {                     
    co_list[num++].cipher = SSL_get_cipher_by_value(id);
    assert(co_list[num - 1].cipher != nullptr);         
  }                                                     
}                                                       
for (uint16_t id : kChaChaCiphers) {                    
  co_list[num++].cipher = SSL_get_cipher_by_value(id);  
  assert(co_list[num - 1].cipher != nullptr);           
}                                                       
if (!has_aes_hw) {                                      
  for (uint16_t id : kAESCiphers) {                     
    co_list[num++].cipher = SSL_get_cipher_by_value(id);
    assert(co_list[num - 1].cipher != nullptr);         
  }                                                     
}                                                                                                     

This will be automatically sovled once #20 is merged.

@gamer191
Copy link

gamer191 commented Apr 6, 2024

Based on the issue title, does 0.5.10 still have correct fingerprints?

@perklet
Copy link
Collaborator

perklet commented Apr 6, 2024

Based on the issue title, does 0.5.10 still have correct fingerprints?

Yes.

@T-256
Copy link

T-256 commented Apr 16, 2024

#20 got merged.
@ed0nkey can you verify this issue solved?

@perklet
Copy link
Collaborator

perklet commented Apr 16, 2024

@T-256 There is no build yet.

@ed0nkey
Copy link
Author

ed0nkey commented Apr 16, 2024

@yifeikong ping me when I should try it out then

@perklet
Copy link
Collaborator

perklet commented Apr 17, 2024

Please try the new version 0.7.0b2 here.
Not yet, windows build is broken. see #62
Fixed in 0.7.0b4

@perklet
Copy link
Collaborator

perklet commented Apr 17, 2024

@gamer191 The CVE-2023-38545 was also fixed by upgrading to 8.5.0 of curl.

@perklet perklet closed this as completed Apr 18, 2024
@ed0nkey
Copy link
Author

ed0nkey commented Apr 18, 2024

Can confirm it works for me, thank you 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants