browse_website fails with non-headless browser #1641
Comments
|
The error you are encountering is related to Chrome not starting correctly when using Selenium WebDriver. You can try adding a few more arguments to the options object to resolve this issue. Update the scrape_text_with_selenium(url) function in your script as follows: File: autogpt/web.py: |
I added it, but it didn't seem to fix the problem unfortunately. Maybe it has something to do with the pathing? I don't know what the file system usually looks like in a docker container? ` options.binary_location = "/usr/local/bin/chromedriver" |
|
I've got this same issue. Tried the suggested arguments to add to options as well as |
|
Make sure chrome is actually installed in your container, the current Dockerfile doesn't install it for example. |
|
Running in a Win10 vm, not a docker container. I'm getting some traction with Still getting some issues: ` DevTools listening on ws://127.0.0.1:52210/devtools/browser/92ead2e2-e12e-4d14-93fd-1ea445cf6fd5 |
|
@aeiberra comments worked along with : |
|
This worked for me! Thanks so much! |
|
looks like web.py got removed from Stable in a recent merge. This broken again and the error persists even with the commands from @darrynv
|
|
I have this error in latest pull |
|
Modifying |
|
For me, the Chat-GPT fixed the problem in web_selenium.py file: I am using headless server for this. |
|
Editing the Auto-GPT/autogpt/commands/web_selenium.py file as follows and suggested by @darkcount2011: Plus running the commands: Solved for me, running on Kali Linux Subsystem for windows. PS: last commands produced the following errors: But browse_website COMMAND works now. Thanks for the continuous support! |
|
I tried this but the suggestion did not work, running on macOS. |
this all worked for me. I am also on macOS |
|
Editing the Auto-GPT/autogpt/commands/web_selenium.py file as follows and suggested by @darkcount2011: Change this part of code: else:
if platform == "linux" or platform == "linux2":
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--remote-debugging-port=9222")
options.add_argument("--no-sandbox") # indent this line
options.add_argument("--headless") # Add this line
driver = webdriver.Chrome(
executable_path=ChromeDriverManager().install(), options=options
)
driver.get(url) |
I'll try again and see if it works, thanks |
|
Grafting together various responses and threads, my web_selenium.py now looks like this and it finally runs without any errors. ` options = options_availableCFG.selenium_web_browser ` |
|
I think this is a specific problem for M1 or other ARM processor users. I assume you also have a problem with macOS. |
Pwuts
changed the title
I add upper code to autogpt/commands/web_selenium.py, but still have problem.
def scrape_text_with_selenium(url: str) -> tuple[WebDriver, str]:
"""Scrape text from a website using selenium
Args:
url (str): The url of the website to scrape
Returns:
Tuple[WebDriver, str]: The webdriver and the text scraped from the website
"""
logging.getLogger("selenium").setLevel(logging.CRITICAL)
options_available = {
"chrome": ChromeOptions,
"safari": SafariOptions,
"firefox": FirefoxOptions,
}
options = options_available[CFG.selenium_web_browser]()
options.binary_location = "/usr/bin/chromedriver"
options.add_argument(
"user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.49 Safari/537.36"
)
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--remote-debugging-port=9222")
options.add_argument("--headless")
if CFG.selenium_web_browser == "firefox":
driver = webdriver.Firefox(
executable_path=GeckoDriverManager().install(), options=options
)
elif CFG.selenium_web_browser == "safari":
# Requires a bit more setup on the users end
# See https://developer.apple.com/documentation/webkit/testing_with_webdriver_in_safari
driver = webdriver.Safari(options=options)
else:
if platform == "linux" or platform == "linux2":
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--remote-debugging-port=9222")
options.add_argument('--headless')
options.add_argument("--no-sandbox")
if CFG.selenium_headless:
options.add_argument("--headless")
options.add_argument("--disable-gpu")
driver = webdriver.Chrome(
executable_path="/usr/bin/chromedriver", options=options
)
#driver = webdriver.Chrome(
# executable_path=ChromeDriverManager().install(), options=options
#)
driver.get(url)
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.TAG_NAME, "body"))
)
# Get the HTML content directly from the browser's DOM
page_source = driver.execute_script("return document.body.outerHTML;")
soup = BeautifulSoup(page_source, "html.parser")
for script in soup(["script", "style"]):
script.extract()
text = soup.get_text()
lines = (line.strip() for line in text.splitlines())
chunks = (phrase.strip() for line in lines for phrase in line.split(" "))
text = "\n".join(chunk for chunk in chunks if chunk)
return driver, text |
|
@busyfree please try with the latest version, and if it's still broken, provide more info about how you are running Auto-GPT, e.g. what OS/environment. |
|
I'm still having this problem on an M1 mac on master with docker-compose. |
|
This exact issue by any chance? |
|
This is the exact error I'm getting: Which seems to be a little different from yours.. more: |
I have update to lastest commit(
|
|
Thanks for posting; please move further discussion about M1-specific issues to #2600 |
|
I tried to add the modifications to posted by @darkcount2011 @dods30 in a Mac 2018 (not M1) using docker run and Im still receiving the error. I tried to execute Auto-GPT with the goal to fix it is own browse_website function, but didn't work 😂 |
|
@Robsonsjre no use applying those mods as a fix is already in |
|
#2600 (comment) This solution worked for me |
using Fukol's 'if' from Significant-Gravitas#1641 (comment)
Thank you this did the trick for me as well |
Duplicates
Steps to reproduce 🕹
3, Have docker desktop up and running.
Current behavior 😯
Once when it tries to browse a website with the command : browse_website and Arguments : {'url': , question:
it returns an error in the System message:
SYSTEM: Command browse_website returned: Error: Message: unknown error: cannot find Chrome binary Stacktrace: #0 0x55ac2df50fe3 <unknown> #1 0x55ac2dc8fd36 <unknown> #2 0x55ac2dcb6f4a <unknown> #3 0x55ac2dcb4a9b <unknown> #4 0x55ac2dcf6af7 <unknown> #5 0x55ac2dcf611f <unknown> #6 0x55ac2dced693 <unknown> #7 0x55ac2dcc003a <unknown> #8 0x55ac2dcc117e <unknown> #9 0x55ac2df12dbd <unknown> #10 0x55ac2df16c6c <unknown> #11 0x55ac2df204b0 <unknown> #12 0x55ac2df17d63 <unknown> #13 0x55ac2deeac35 <unknown> #14 0x55ac2df3b138 <unknown> #15 0x55ac2df3b2c7 <unknown> #16 0x55ac2df49093 <unknown> #17 0x7f0432fd2ea7 start_threadExpected behavior 🤔
It should be able to run the browse_website command normally and browse the website with the question.
Your prompt 📝
# Paste your prompt hereThe text was updated successfully, but these errors were encountered: