Demonstrating the "Reading the Robot Mind" system using a bird audio recognition AI system.
Presenting Internal Data Flow Within an AI for Classification of Bird Sounds in a Format Familiar to Subject Matter Experts
- The term “Reading the Robot Mind” is one that I use to describe a system that allows inspection of the internal workings of an Artificial Intelligence (AI) process, where…
-
- …the AI process is designed to automatically receive input data and come up with some desired result that mimics what would normally be a manual result generated by a human subject matter expert (SME).
-
- …the “Reading the Robot Mind” system provides a way to convert the inner workings of the AI into a format that is useful and common to SME’s (as opposed to just rows of numbers, or other format only useful to a software programmer).
- This presentation discusses a specific example of a “Reading the Robot Mind” system, to demonstrate how it is done. Specifically…
-
- An interactive Jupyter notebook is built for the purpose of training and deploying a deep learning neural network artificial intelligence (AI) to automatically identify birds from recorded audio.
-
- The notebook allows the user to modify parameters along the training and classification (inference) pipeline and observe the results. As with traditional observation methods, the notebook lets users view visual representations (spectrograms, etc.) of input vectors for similar and different birds.
-
- In addition to traditional methods, this notebook also presents data in its original format (audio recordings of birds).
- Presenting data in its original format (audio recordings of birds) is common practice for a field researcher or SME testing a microphone and recording system.
- The SME will want to listen to the recordings to see if they contain valid and sufficient information.
- A system is therefore required to take the large amounts of digital (numerical) data and convert it into sounds so the SME can tell if the microphone and recording system are working properly.
- The notebook extends this intuitive and useful technique to individual neural network layers - working backwards towards a best estimate of the original input (referred to as "reading the robot mind").
- The user can even provide just the "answer" (select a bird at the final output layer), and the reading the robot mind system will work backwards through the entire automated process and AI layers to let the SME hear a best approximation of what the AI has learned that bird sounds like.
- The BirdCLEF 2023 challenge is part of 2023 LifeCLEF and involves identifying Eastern African bird species by sound.
- Specifically, the task is to develop computational solutions to process continuous audio data and recognize the species by their calls. The best entries to this contest challenge will be able to train reliable classifiers with limited training data. The training data consists of short recordings of individual bird calls generously uploaded by users of xenocanto.org.
- The ability of AI systems to explain their classification conclusions (also called “explain ability” or “explainability”) is gaining increased research focus, both to provide support for decisions , as well as for “possibilities of exposing complex AI models to human users/operators in an interpretable and understandable ways.”
- Towards the goals of the latter, “Reading the Robot Mind” focuses on the information flow through a trained AI system, including ability to observe where important information may be discarded or corrupted in some way.
- Note that due to the time and compute limitations imposed by the computing environment and contest rules provided, the aforementioned Jupyter notebook is divided into four public notebooks:
- https://www.kaggle.com/code/pnussbaum/v15h-birdclef2023-mindreader - This notebook focuses on the Segmentation and Feature Extraction aspects of the AI solution, allowing users to make modifications and see and hear how much information is retained.
- https://www.kaggle.com/code/pnussbaum/v15h-all-birdclef2023-mindreader - This notebook allows the user to use their final decision related to segmentation and feature extraction, and convert and save all the BirdClef2023 data into this format.
- https://www.kaggle.com/code/pnussbaum/v16e-gpu-all-birdclef2023-mindreader - This notebook uses the final decisions noted above, and trains the entire AI for a longer period of time, achieving better accuracy, and saving the trained AI system.
- https://www.kaggle.com/code/pnussbaum/v17b-all-birdclef2023-mindreader - This notebook brings all of this together for the sake of the contest submission and scoring.
