Metamask issue - Solved. Question about hanging processes.

[Sore7792]

Good evening.
I'm not very proficient in coding, but I'm gradually figuring it out. ChatGPT has been a great help in this.

I wrote a small script that:

1. Opens my separately created profile through the Chrome driver. Chrome version: 122.0.6261.70. I downloaded a Chrome driver of almost the same version: 122.0.6261.69. Is this critical?
2. Opens the Metamask extension page (where you need to enter a password) and enters the password.
   That's the beginning of the script, but it's enough to describe the problem. The script worked almost flawlessly.

I needed to add your wonderful library to be able to open one secure site - and it worked! I hope I've correctly modified my code and everything is executed correctly.
Specifically, I added/changed 4 lines:
import undetected_chromedriver as uc
options = uc.ChromeOptions()
options.add_argument(f'--user-data-dir={profile_path}')
driver = uc.Chrome(service=service, options=options)

The problem is as follows. The script works, but only on any other sites (it opens the page/interacts with elements). If I try to open the Metamask login page, the script opens it, but as soon as the script tries to enter the password, the profile immediately closes and the script terminates.

I'm not very competent in these matters, and most of the code was helped by ChatGPT, but I assume that undetected_chromedriver changes some variables for masking, and Metamask aggressively reacts to this and doesn't allow clicking elements, leading to a malfunction.

Please suggest some solutions.
Maybe there is a list of all the extensions for undetected_chromedriver, and I can find the one that interferes with Metamask by trial and error.

Thank you, guys! Have a nice weekend!

UPD. I also checked. I have:
Python 3.12.2
Selenium 4.17.2
pip show undetected-chromedriver 3.5.5

[Sore7792]

I found and solved the problem. It turned out that when working through the undetected Chromedriver library, the Metamask extension blocks its operation, and the profile closes. The protective function is called LavaMoat.

I struggled for a long time to disable it, and it turned out to be impossible to maintain good functionality of the extension. So, I went to the official Metamask GitHub and forked the repository. In the index.js file, I canceled the execution of LavaMoat. After that, I created an extension, and the specific protection module was disabled in it.

And I encountered the next problem - such an extension cannot be officially installed in regular Google Chrome, as it is a fork, not an official release.

Therefore, I now use a portable version of Google Chrome for testing. This extension is installed in it, and the profile works correctly.

Now the undetected Chromedriver works well with the Metamask extension.

[Sore7792]

Now I have encountered another problem. I'll describe it:

I have created Google Chrome profiles in separate folders. They are created based on the portable Google Chrome.

To launch each of these profiles, I use the path to the portable chrome.exe and to the specific profile I created. This profile is launched using the Chromedriver, which I also downloaded from the official Google Chrome website. The same place where I downloaded the portable Google Chrome. Each profile folder contains its own Chromedriver. This was done before I started using the undetected Chromedriver. I did this to ensure that the multi-threaded launch of profiles worked correctly.

And now I'm trying to figure out how to deal with the multithreading issue after implementing the undetected Chromedriver.

As far as I understand, using the undetected Chromedriver, it downloads the Chromedriver itself into the folder: C:\Users\user\AppData\Roaming\undetected_chromedriver. Do I understand correctly that when I now use multithreading, all my profiles are launched not from my Chromedriver but from the downloaded one? And it turns out that all processes are loading a single Chromedriver. Is this a problem? Or not?

Probably my code is implemented incorrectly. But at the end of the work, I have unclosed Google Chrome processes left in the task manager, although the profiles themselves have long been closed. And if I set up processing for 100 profiles, my PC just freezes at one moment when there are many such fully unclosed profiles. But this does not happen with every profile, but with some. I don't know how this happens. Maybe someone can advise me?

I can show a piece of code that is responsible for launching my profiles:

import undetected_chromedriver as uc

def process_profile_with_decryption(profile_path, decrypted_password, proxy, use_proxy):
    global window_index
    chromedriver_path = os.path.join(profile_path, "chromedriver-win64", "chromedriver.exe")
    service = Service(executable_path=chromedriver_path)
    options = uc.ChromeOptions()
    options.add_argument(f'--user-data-dir={profile_path}')    
    options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36")     
    #options.add_argument('--log-level=3')
    #options.add_argument("--disable-logging")

    #Портабле версия хрома
    portable_chrome_path = r"D:\ChromBust\PortableChrome\chrome.exe"
    options.binary_location = portable_chrome_path

    # Настройка прокси
    if use_proxy:
        proxy_ip, proxy_port = proxy.split(':')
        #print(f"Профиль №{profile_path} запустился с прокси {proxy_ip}:{proxy_port}")
        options.add_argument(f'--proxy-server={proxy_ip}:{proxy_port}')
    else:
        print(f"Профиль №{profile_path} запустился без прокси")

    driver = uc.Chrome(service=service, options=options)
    #driver.set_window_position(-2560, 0)
    #driver.maximize_window()
 
    # ЭТО НАШЕ ТЕЛО, ВСТАВЛЯЕМ КОД ПО ОБРАЗУ            название_файла.название_процесса_в_файле(driver)
    try:

        import_json.import_json_go(driver, decrypted_password)
        time.sleep(2)

    finally:
        driver.quit()   
    # КОНЕЦ НАШЕГО ТЕЛА.

[Kir-SA]

Русский

Добрый вечер.

В Вашем фрагменте кода на каждый экземпляр Хрома запускается свой экземпляр драйвера из папки в профиле. То есть UC не используется. Чтобы использовался именно UC достаточно опустить аргумент service при запуске Хрома.

Касательно "многопоточности" в плане работы нескольких профилей...
Не работал с питоном и UC пару лет, многое изменилось в плане совместимости и работоспособности, но как делал я:

1. Запускается ОДИН экземпляр драйвера в режиме Service.

from selenium.webdriver.chrome.service import Service

service_args = ["--ignore-ssl-errors=true", "--ssl-protocol=any"]
service = Service(executable_path='./chromedriver', service_args=service_args, port=chromedriver_port)
service.start()

2. Обращаемся к нему через webdriver.Remote, передавая url:port сервиса и свой набор опций экземпляра хрома, но дополнительно порт отладчика (произвольный порт, к которому будет привязан определенный экземпляр).

executor_url = 'http://127.0.0.1:15000'
opts.add_argument('--remote-debugging-port=' + debugger_port)
driver = webdriver.Remote(command_executor=executor_url, desired_capabilities=opts.to_capabilities())

opts - обычный ваш набор опций, включая путь к профилю. но обязательно с указанием порта отладки.

3. В зависимости от задач можем выполнять работу и закрывать хром, либо если нужно переиспользовать открытые экземпляры, получать/запоминать session_id и debugger_port каждого открытого экземпляра и передавать их при последующих обращениях:

from selenium import webdriver
from undetected_chromedriver import ChromeOptions

driver = webdriver.Remote(command_executor=executor_url, options=driver_options)
driver.session_id = instance_data["session_id"]
driver.quit()

*** Дело было в Debian 11, а не в Windows, и проблем с незакрытыми процессами не наблюдалось. Как это поведет себя в вашем случае - одному богу известно. Но есть подозрение, что работа через единственный сервис всё же более правильный путь, чем по драйверу на каждый профиль.

Обращаю внимание, что по какой-то причине (кажется, как раз из-за нерабочих расширений в UC 3.1.3), Service и Remote я тогда использовал из стандартного selenium, но класс опций - из UC, как и автоматически скачанный им драйвер. При чем, это почему-то не приводило к лучшему распознаванию антиботами.

Напомню, что давно с этим не работал, что-то могли изменить или сломать.

Кстати, расширения добавленные через options.add_extension(path_to_crx) не работают и в актуальном 3.5.5 UC. Возможно, и проблема с Metamask в том числе где-то в этой плоскости. Хотя расширения хром уже поддерживает даже в headless=new.

English

Good evening.

In your code, each Chrome instance launches its own driver instance from the folder in the profile. That is, UC is not used. To use UC specifically, it is enough to omit the service argument when launching Chrome.

Regarding "multithreading" in terms of working with multiple profiles...
I haven't worked with Python and UC for a couple of years, a lot has changed in terms of compatibility and performance, but here's how I did it:

1. ONE driver instance is launched in Service mode.

from selenium.webdriver.chrome.service import Service

service_args = ["--ignore-ssl-errors=true", "--ssl-protocol=any"]
service = Service(executable_path='./chromedriver', service_args=service_args, port=chromedriver_port)
service.start()

2. We connect to it through webdriver.Remote, passing the url:port of the service and our set of options for the Chrome instance, but additionally the debugger port (an arbitrary port to which a specific instance will be tied).

executor_url = 'http://127.0.0.1:15000'
opts.add_argument('--remote-debugging-port=' + debugger_port)
driver = webdriver.Remote(command_executor=executor_url, desired_capabilities=opts.to_capabilities())

opts - your usual set of options, including the path to the profile. but necessarily with the indication of the debugging port.

3. Depending on the tasks, we can perform work and close Chrome, or if you need to reuse open instances, get/remember the session_id and debugger_port of each open instance and pass them on subsequent requests:

from selenium import webdriver
from undetected_chromedriver import ChromeOptions

driver = webdriver.Remote(command_executor=executor_url, options=driver_options)
driver.session_id = instance_data["session_id"]
driver.quit()

*** It was in Debian 11, not Windows, and there were no problems with unclosed processes. How this will behave in your case is known only to God. But there is a suspicion that working through a single service is still a more correct path than a driver for each profile.

Please note that for some reason (it seems, precisely because of non-working extensions in UC 3.1.3), I then used Service and Remote from standard selenium, but the options class - from UC, as well as the driver automatically downloaded by it. Moreover, this somehow did not lead to better recognition by antibots.

I remind you that I have not worked with this for a long time, something could have been changed or broken.

By the way, extensions added through options.add_extension(path_to_crx) do not work even in the current 3.5.5 UC. Perhaps the problem with Metamask is also somewhere in this plane. Although Chrome already supports extensions even in headless=new.

[Sore7792]

Can we connect on telegram?
@Sore7792