EcoGuardian initial commit

README.md

@@ -0,0 +1,39 @@
+# EcoGuardian: Monitor endangered wildlife and assess potential threats using autonomous drones
+
+## Inspiration
+
+Poaching is the greatest current threat to tigers, rhinos, elephants, gorillas and other African and Asian species. This is a problem that typically goes unnoticed because few countries provide statistics related to poaching. Specifically, from 2022 to 2023, rhino fatalities from poaching in South Africa increased by over 10%. While helicopters and night vision equipment are being used in South Africa's Kruger National Park to successfully decrease poaching, activity is just being pushed to other provinces. Likewise, an estimated 2 anti-poaching rangers are killed each week, but that number could be higher.
+
+This calls for a solution that can successfully deter poachers from more areas and make situations safer for anti-poachers with surveillance: EcoGuardian. 
+
+## What it does
+
+We have built software for autonomous drones that can be controlled with natural language for:
+
+Autonomous patrols of endangered species
+Using thermal imaging cameras, our autonomous drones can patrol large swaths of land at any time of day. Open-ended CV will identify and monitor species that are endangered or at risk of poaching.
+Detecting poachers and notifying law enforcement
+When a drone detects potential poachers, it notifies nearby rangers or law enforcement with critical textual and visual data, enabling rapid, effective responses.
+High-level reasoning
+By leveraging LLMs, our drones assess the environment for any potential hazards such as poachers or fire precursors. They can also understand and execute complex instructions, even from untrained operators.
+
+## How we built it
+
+We use AirSim, a drone simulator built on the Unreal Engine and we add MLLM integration with GPT-4 and Claude-Opus to interpret natural language instructions and call functions to control the drone. Through a custom programming sandbox, we create an environment that maximizes the reasoning capabilities of modern LLM agents to control a drone and yet still avoid any dangerous activity. With lightweight computation powering real-time decision making and long-term planning, we minimize the need of any human operator and make each drone operator 50x more powerful.
+
+## Challenges we ran into
+
+AirSim has been deprecated for over 2 years, so we ran into some issues with building it on macOS. Thankfully, we had some Windows laptops to run it on. We also ran into some issues with accuracy of computer vision models in the Unreal Engine environments but we were able to use the GPT-4 API instead to significantly improve performance.
+
+## Accomplishments that we're proud of
+One of the biggest milestones was getting the infrared detector to identify humans (poachers) much better than the human eye.
+
+## What we learned
+
+It is incredible getting to watch LLMs to reason through codes and interact with each other. Having an LLM reason through the code and interact with another LLM that answers questions for it was an aha moment for getting better results for reasoning about scenes.
+
+## What's next for EcoGuardian
+
+Next steps are to test this software in the field and add stronger collision detection to improve drone routes. AirSim software is easily transferable to be run on real drones. Our simulation is in Africa and focused on rhinos, but this software can be extended to protecting endangered species from poachers in any country.
+
+This software can also be extended to enable faster responses to disaster scenarios like earthquakes, falling bridges, floods, and/or wildfires. Specifically, EcoGuardian can be deployed quicker than humans and detect missing people with its infrared cameras.

environment.yml

@@ -0,0 +1,102 @@
+name: chatgpt
+channels:
+  - defaults
+dependencies:
+  - ca-certificates=2024.3.11=haa95532_0
+  - openssl=3.0.13=h2bbff1b_0
+  - pip=23.3.1=py39haa95532_0
+  - python=3.9.19=h1aa4202_0
+  - setuptools=68.2.2=py39haa95532_0
+  - sqlite=3.41.2=h2bbff1b_0
+  - tzdata=2024a=h04d1e81_0
+  - vc=14.2=h21ff451_1
+  - vs2015_runtime=14.27.29016=h5e58377_2
+  - wheel=0.41.2=py39haa95532_0
+  - pip:
+      - aiohttp==3.9.3
+      - aiosignal==1.3.1
+      - aiostream==0.5.2
+      - airsim==1.8.1
+      - annotated-types==0.6.0
+      - anthropic==0.21.3
+      - anyio==4.3.0
+      - async-timeout==4.0.3
+      - attrs==23.2.0
+      - beautifulsoup4==4.12.3
+      - cachetools==5.3.3
+      - certifi==2022.12.7
+      - charset-normalizer==3.0.1
+      - click==8.1.7
+      - colorama==0.4.6
+      - contourpy==1.2.0
+      - cycler==0.12.1
+      - distro==1.9.0
+      - exceptiongroup==1.2.0
+      - fastapi==0.110.0
+      - filelock==3.13.3
+      - fonttools==4.50.0
+      - frozenlist==1.4.1
+      - fsspec==2024.3.1
+      - google==3.0.0
+      - google-api-core==2.18.0
+      - google-auth==2.29.0
+      - google-cloud==0.34.0
+      - google-cloud-vision==3.7.2
+      - googleapis-common-protos==1.63.0
+      - grpcio==1.62.1
+      - grpcio-status==1.62.1
+      - grpclib==0.4.7
+      - h11==0.14.0
+      - h2==4.1.0
+      - hpack==4.0.0
+      - httpcore==1.0.5
+      - httpx==0.27.0
+      - huggingface-hub==0.22.2
+      - hyperframe==6.0.1
+      - idna==3.4
+      - importlib-resources==6.4.0
+      - kiwisolver==1.4.5
+      - markdown-it-py==3.0.0
+      - matplotlib==3.8.3
+      - mdurl==0.1.2
+      - modal==0.62.21
+      - msgpack-python==0.5.6
+      - msgpack-rpc-python==0.4.1
+      - multidict==6.0.5
+      - numpy==1.24.2
+      - openai==0.27.2
+      - opencv-contrib-python==4.7.0.68
+      - packaging==24.0
+      - pillow==10.2.0
+      - proto-plus==1.23.0
+      - protobuf==4.25.3
+      - pyasn1==0.6.0
+      - pyasn1-modules==0.4.0
+      - pydantic==2.6.4
+      - pydantic-core==2.16.3
+      - pygments==2.17.2
+      - pyparsing==3.1.2
+      - python-dateutil==2.9.0.post0
+      - pyyaml==6.0.1
+      - requests==2.28.2
+      - rich==13.7.1
+      - rsa==4.9
+      - sigtools==4.0.1
+      - six==1.16.0
+      - sniffio==1.3.1
+      - soupsieve==2.5
+      - starlette==0.36.3
+      - synchronicity==0.6.5
+      - tokenizers==0.15.2
+      - toml==0.10.2
+      - tornado==4.5.3
+      - tqdm==4.66.2
+      - typer==0.9.4
+      - types-certifi==2021.10.8.3
+      - types-toml==0.10.8.20240310
+      - typing-extensions==4.10.0
+      - urllib3==1.26.14
+      - watchfiles==0.21.0
+      - yarl==1.9.4
+      - zipp==3.18.1
+

prompts/airsim_basic.txt

@@ -0,0 +1,43 @@
+Here are some functions you can use to command the drone.
+
+aw.VQA(self, prompt) -> string: Answers visual questions by returning a string. Has powerful GPT4V Question answering capabilities. VQA always returns a string. ALWAYS PRINT THE RESULT OF VQA. 
+aw.VQA_int(self, prompt, lower_bound, upper_bound) -> int: Answers visual questions with an integer between lower_bound and upper_bound that can be safely cast. Powerful when visual questions will then be used for decision making or counting.
+aw.get_drone_position() -> list[3]: returns the current position of the drone as a list of 3 floats corresponding to XYZ coordinates.
+aw.fly_to([x, y, z]) -> None: flies the drone to the position specified as a list of three arguments corresponding to X, Y, Z coordinates.
+aw.fly_path(points) -> None: flies the drone along the path specified by the list of points. Each point is again a list of 3 floats corresponding to X, Y, Z coordinates.
+aw.set_yaw(yaw) -> None: sets the yaw of the drone to the specified value in degrees.
+aw.get_yaw() -> int: returns the current yaw of the drone in degrees.
+aw.flutter() -> None: the drone keeps moving in a 'random' way within a confined radius. 
+aw.generate_circular_path(center, radius, height, segments=12) -> points: generates a circular path around a center point with a specified radius and height
+aw.take_photo(display) -> None: Always call with display=True. The drone takes a photo of what it sees in FPV. There is no need to display anything
+aw.search(self, object_name) -> None: moves the drone to a specified object in the set ("rhino", "poacher", "hippo")
+aw.takeoff() -> None: takes off the drone.
+aw.land() -> None: lands the drone.
+aw.patrol_and_report() -> None: performs an involved report of the nearby area to check for hazards
+
+A few useful things: 
+When told to fly to an object, go a few units short by interpolating between your current location and its location. Face the direction of the objection you are photographing 
+The Z-axis is inverted. That is, to fly up 10 units, you must decrement your Z-position by 10.
+You can use VQA_int for navigation or turning by using integers as possible directions. 
+
+I.e. User Question: "Face towards the sun"
+
+Code Response: 
+direction_to_sun = aw.VQA_int("In which direction is the sun? Associate 1: Forward, 2: Backward, 3: Left, 4: Right", 1, 4)
+
+# Adjust the drone's yaw to face the sun
+if direction_to_sun == 1:
+    # Forward, no need to adjust yaw
+    pass
+elif direction_to_sun == 2:
+    # Backward, turn 180 degrees
+    aw.set_yaw((aw.get_yaw() + 180) % 360)
+elif direction_to_sun == 3:
+    # Left, turn 90 degrees to the left
+    aw.set_yaw((aw.get_yaw() - 90) % 360)
+elif direction_to_sun == 4:
+    # Right, turn 90 degrees to the right
+    aw.set_yaw((aw.get_yaw() + 90) % 360)
+
+In terms of axis conventions, forward means positive X axis. Right means positive Y axis. Up means negative Z axis.
+When flying to an object, stop a few units short to avoid colliding.

prompts/airsim_system.txt

@@ -0,0 +1,6 @@
+You are an assistant helping me with the AirSim simulator for drones.
+When I ask you to do something, you are supposed to give me Python code that is needed to achieve that task using AirSim.
+You are only allowed to use the functions I have defined for you.
+You are not to use any other hypothetical functions that you think might exist.
+You can use simple Python functions from libraries such as math and numpy.
+You will be able to solve every task with the functions we give you.