We are thrilled to present PillKaBoo, an app thoughtfully crafted for blind and visually impaired(VI) parents.
PillKaBoo specifically aims to assist blind/VI parents in safely and independently administering medication to their children.
- PillKaBoo is now available in South Korea as "내약알기" in App Store & Google Play Store
- Currently, our service is only available in Korean and (in the case of medication recognition feature) medicine products registered in South Korea. We plan to expand the service internationally. In the meantime, send us an email if you wish the service in your country sooner!
- Currently, our service is only available in Korean and (in the case of medication recognition feature) medicine products registered in South Korea. We plan to expand the service internationally. In the meantime, send us an email if you wish the service in your country sooner!
- Visit our cool 🏠 Homepage: https://parkourapp.dev/
Refer to the following for our code:
- 🏠 Homepage: https://github.com/becooq81/PillKaBoo-Homepage
- 📱 Mobile: https://github.com/becooq81/PillKaBoo
- 🌐 Serverless API: https://github.com/becooq81/PillKaBoo-Backend
- 🔍 Problem Statement
- 🎯 UN SDGs & Targets
- 📼 Demo Video
- ✨ Key Functions
- ⚙️ Technology
- 🗂️ Directory Organization
- ⚡ Getting Started
- 🏃🏻♀️ Team Parkour
Administering medication to children poses unique challenges for blind parents, encompassing potential safety risks and emotional stress, underscoring the need for accessible medication management solutions and supportive resources.
Most Common Medication Challenges for the Blind & Visually Impaired(VI)
1. Identifying Medication 2. Understanding Medication Information 3. Using Liquid Medication
We learned that bilnd/VI parents struggle with giving medication to their children, especially with identifying and understanding the medication and adjusting the right dosage of liquid medication. They also expressed the desire to independently manage their child’s medication without relying on assistance. They feel anxious when giving medications to their child because of the risk of giving the wrong/expired medication and exceeding the required liquid dosage.
The Google Solution Challenge is designed to encourage participants to develop solutions that address one or more of the United Nations Sustainable Development Goals (SDGs). These 17 goals cover a broad range of social and economic development issues.
As part of the GDSC Google Solution Challenge, PillKaBoo(Team Parkour) tackles the following UN SDGs.
Our service directly contributes to improving the quality of life for visually impaired individuals, thereby aligning with the broader objectives of the UNSDGs to leave no one behind and to ensure inclusivity and equality. We not only tackle the challenges of medication management for the VI but also contribute to greater goals of health coverage, quality education, and reduced inequalities, fostering a more inclusive society for all.
PillKaBoo improves accessibility to information about medication and liquid medication intake so that blind/VI parents can reduce the risks of giving the wrong or expired medication and exceeding the required dosage. PillKaBoo contributes to the good health and well-being of VI parents and their children.
PillKaBoo provides information in audio guidance through screen readers (
Google Talk Back) helping blind/VI parents make educated choices for their children’s medication.
PillKaBoo reduces inequalities of inconveniences regard to medication due to visual impairments and empowers blind/VI parents with the independence to care for their children by themselves, regardless of their visual abilities.
Video Link: YouTube @parkour.official
Introducing a practical service tailored specifically for blind or visually impaired parents, aimed at simplifying the task of giving medications to their child. With accessibility at its core, our service ensures that anyone, regardless of visual ability, can manage the medications for their children safely and independently.
PillKaBoo presents in a user-friendly interface for blind and VI parents by utilizing semantic labels, imitating the user interactions of screen readers in the application interface, and providing customizable and noticeable interface components.
Important
📢 All features will work appropriately with Talk Back. Please turn on this accessibility feature!
1️⃣ Medication Recognition & Information
: Assists blind/VI users by identifying medication(s) through visual input, and provides essential details.
- In real time, Pillkaboo’s built-in camera recognizes the packaging of the medication.
- PillKaBoo provides various informations such as precautions for your child, potential allergic reactions you registered in setting, expiration date, and of course all the basics as well.
- Just swipe to learn accurate medication information!
2️⃣ Recognition of Prescription Labels
: Assists blind/VI users to optimize the effectiveness of the prescribed treatment by promoting timely and consistent prescription medication intake.
- In real time, PillKaBoo's built-in camera recognizes the text labels printed on the prescription packet.
- PillKaBoo will let you know which part of day (morning, noon, or evening) for which this packet is assigned, and the date of prescription.
3️⃣ Accurate Liquid Medication Intake
: Assists the process of pouring the desired amount of liquid medication in real time.
- To select the amount, adjust the slider by swiping up and down the screen.
- While you pour the liquid medication into the container, PillKaBoo's built-in camera will let you know if you are close to, or done pouring with varying audio feedback.
4️⃣ Remaining Liquid Medication
: Checks the remaining amount of liquid medication.
- Double check whether you poured the right amount,or keep track of how much medicine you have left for future usage.
- PillKaBoo tells you how much liquid medication is left in the container.
5️⃣ Settings
: Personalizes the settings for users' convenience.
- Register child’s allergies that will be used to check potential risks of allergic reactions when scanning medicine.
- Customize the colors that better suit personal color visibility.
| Medication Recognition | Medication Information | Recognition of Prescription Time Label |
|---|---|---|
 |
 |
 |
| Accurate Liquid Medication Intake | Remaining Liquid Medication |
|---|---|
 |
 |
| Settings - Register Allergies | Settings - Customize Colors | Settings - Guide |
|---|---|---|
 |
 |
 |
Flutter | Android | Go | TensorFlow Lite | Google Cloud Function | SQLite
Users of PillKaBoo directly interact with Mobile Layer through Google Talk Back (screen reader). Mobile Layer built with Flutter building the application’s user interface and handling logical operations. The user interface implements accessibility features using Google Talk Back, enhancing the app’s usability for VI users, and is responsible for providing a better user experience.
For Machine Learning, TensorFlow Lite and Google’s ML Kit for Flutter are integrated into the Flutter mobile application. Using TensorFlow Lite for a YOLO model (trained with hand-crafted dataset) for bottle detection used when identifying the liquid amount in the container while usingGoogle’s ML Kit for Flutter Text Recognition and Barcode Detection for identifing the medication packaging.
For more Mobile information 👉🏻 https://github.com/becooq81/PillKaBoo
All the data are managed in local storage with SQLite integration for providing software that doesn’t require an Internet connection. All the features of PillKaBoo can be used in anternet environment.
Connected to the frontend component through an HTTP request, the Google Cloud Function, written in Go, updates the latest database of medication information when Internet connection is available.
For more Back-End information 👉🏻 https://github.com/becooq81/PillKaBoo-Backend
For developers, PillKaBoo uses GitHub Action for the CI/CD pipeline to automate and streamline the process of building, testing, and deploying applications.
📂lib
├─ main.dart
├─ index.dart
└─ 📂src
├─ 📂app
│ ├─ app_lifecycle_reactor.dart
│ ├─ background_service.dart
│ └─ global_audio_player.dart
├─ 📂core
│ ├─ internationalization.dart
│ ├─ lat_lng.dart
│ ├─ pillkaboo_model.dart
│ ├─ pillkaboo_util.dart
│ ├─ place.dart
│ └─ uploaded_file.dart.dart
├─ 📂data
│ └─ 📂local
│ ├─ 📂database
│ │ ├─ barcode_db_helper.dart
│ │ ├─ children_db_helper.dart
│ │ ├─ ingredients_db_helper.dart
│ │ └─ processed_file_db_helper.dart
│ └─ 📂shared_preference
│ └─ app_state.dart
├─ 📂nav
│ ├─ nav.dart
│ └─ serialization_util.dart
├─ 📂network
│ ├─ connectivity.dart
│ └─ download_file.dart
├─ 📂ui
│ ├─ 📂pages
│ │ ├─ 📂main_menu_page
│ │ │ ├─ main_menu_page_model.dart
│ │ │ └─ main_menu_page_widget.dart
│ │ ├─ 📂med_recognition
│ │ │ ├─ 📂med_info_recognition
│ │ │ │ ├─ 📂med_recognition_page
│ │ │ │ │ ├─ med_recognition_page_model.dart
│ │ │ │ │ └─ med_recognition_page_widget.dart
│ │ │ │ └─ 📂med_info_page
│ │ │ │ ├─ med_info_page_model.dart
│ │ │ │ └─ med_info_page_widget.dart
│ │ │ ├─ 📂med_submenu_page
│ │ │ │ ├─ med_submenu_page_model.dart
│ │ │ │ └─ med_submenu_page_widget.dart
│ │ │ └─ 📂prescribed_med
│ │ │ ├─ 📂prescribed_med_recognition_page
│ │ │ │ ├─ prescribed_med_recognition_page_model.dart
│ │ │ │ └─ prescribed_med_recognition_page_widget.dart
│ │ │ └─ 📂prescribed_med_result_page
│ │ │ ├─ prescribed_med_result_page_model.dart
│ │ │ └─ prescribed_med_result_page_widget.dart
│ │ └─ 📂settings
│ │ ├─ 📂allergy
│ │ │ ├─ 📂allergy_add_page
│ │ │ │ ├─ allergy_add_page_model.dart
│ │ │ │ └─ allergy_add_page_widget.dart
│ │ │ └─ 📂allergy_list_page
│ │ │ ├─ allergy_list_page_model.dart
│ │ │ └─ allergy_list_page_widget.dart
│ │ ├─ 📂help_page
│ │ │ ├─ help_page_model.dart
│ │ │ └─ help_page_widget.dart
│ │ ├─ 📂pick_color_page
│ │ │ ├─ pick_color_page_model.dart
│ │ │ └─ pick_color_page_widget.dart
│ │ └─ 📂settings_menu_page
│ │ ├─ settings_menu_page_model.dart
│ │ └─ settings_menu_page_widget.dart
│ ├─ 📂styles
│ │ ├─ pillkaboo_icon_button.dart
│ │ ├─ pillkaboo_theme.dart
│ │ └─ pillkaboo_widgets.dart
│ └─ 📂widgets
│ ├─ 📂components
│ │ ├─ gesture_slider.dart
│ │ └─ home_button_widget.dart
│ ├─ 📂features
│ │ ├─ med_recognizer_widget.dart
│ │ └─ prescribed_med_recognizer_widget.dart
│ ├─ 📂views
│ │ ├─ barcode_detector_view.dart
│ │ ├─ camera_view.dart
│ │ ├─ detector_view.dart
│ │ └─ text_detector_view.dart
│ └─ pillkaboo_widgets.dart
└─ 📂utils
├─ coordinates_translator.dart
└─ date_parser.dart
Kang Heejin
Mobile BE |
Kim Jihoon
ML |
Yu Yeong Hyeon
Design |