Skip to content

Latest commit

 

History

History
296 lines (175 loc) · 9.29 KB

README.md

File metadata and controls

296 lines (175 loc) · 9.29 KB
Raw

GPT Web Guide

React GPT Web Guide - Interactable AI 3D Avatar that have controll over your frontend website

Try the demo React GPT Web Guide

Ask something? discord: albirrkarim


Features:



Docs React GPT Web Guide v1.4.3

This is the Documentation for private repo demo website

Table Of Contents


A. Introduction

What i want?

Recently, I worked on AI accessibility tools The idea is it's like conversational AI that can guide the user with a disability maybe a motoric, or visual problem. So the tools must be able to read the page, understand the user's voice, and do some action programmatically.

Not just benfiting for the user with disability, but also for the normal user.

I want it to be more interactive, more fun, and more engaging. I want to make it like a 3D avatar that can control the website.

Here what i got from the search on the internet:

Accesibility tools

There are many tools, mostly is subscription-based, and the price is expensive. I want to make it more affordable and more flexible. The practical solution one. Mostly just 2D interface, and doesn't have integrated with 3D Avatar. which enhance the user experience and make it more fun.

Conversational AI 3D Avatar

I found the Talking Head project. its a js library that can make 3D avatar and do conversation with user with also lip sync and have animation. But they don't have the ability to control the website.

3D AI avatar using React js

Since my project using react js so it would be easier to me to use react js. I found project from wawa sensei about Virtual Girlfriend but again they don't have the ability to control the website and text highlight when the AI talk.

Solutions

Making some frontend-package (npm package) that can doing various thing.

What position is this package in your production apps?

Imagine the human body, They have a mouth (text-to-speech), ears (speech-to-text), a brain (chat GPT API), hands (programmatically calling function to do some action), eyes (the function that searches the web section, button, form, etc).

How It Works

GPT Web Guide Overview

The Interaction with user (Mouth and Ears)

Using React / Vanilla Speech Highlight (text-to-speech with highlight) so the ai can speak and using React Speech Recognition for Speech Recognition (speech to text)

The Brain

I designing some engines that combine the LLM capabilities and efficient algorithm to save the cost. Featuring understanding user command with some initial knowledge (like what is your website about, etc).

The Hands

The hands is the function that can do some action programmatically. like search, click, etc.

The Eyes

The eyes is js function that can extract all section of your webpage that maybe inside it contain button, form, etc. then using LLM (Open AI API) make some literal text and after that the Mouth will speak it. (Human like informer)

The 3D Avatar

Using React Three Drei and the readyplayer.me and mixamo for the avatar animation. Like in the wawa sensei project virtual girlfriend


Use Cases

Conversational AI

Imagine the user can ask the AI about anything on your website. The AI can answer the user question, doing some action, etc.

As AI Web Accessibility Tools

Can be used as interactable web accessibility.

Screen Reader (all menus, buttons, form). Understand the user say and do the actions.

As Metaverse NPC

You can set programatic action. so the package can integrate into your system.

[
  {
    name: "Search",
    description: "Searching with some keyword",
    type: ACTION_TYPE.SEARCH,
    format: "search {keywords}",
    onAction: (params) => {
      console.log("Triggering onAction 2 with params: ");
      console.log(params);
      // params:
      //   {
      //     keywords: "elephant",
      //   }

      // this function will called if the user want to search something
    },
  },
];

B. Todo

  • Let me know what you want from this package, please write it on issues tab
  • Make Youtube Video Tutorial
  • Typescript, Eslint
  • API docs, JS Doc
  • Automate the testing (Prompt Test, Unit Test)


C. API & Example Code

See API.md

The full example code and implementation example using source code from demo website. the source code of demo website is included when you buy this package.


D. Changelog

Changelog contains information about new feature, fix bug, and what you should do when the version is update.

See CHANGELOG.md


E. Disclaimer & Warranty

There's no refund. I love feedback from my customers. You can write on the issue tab so when i have time i can try to solve that and deliver for the next update.



F. FAQ

Why it's expensive? Why it's not opensource package?

Try yourself to make this package. you will be grateful I am selling it cheap.

Well, i need money to funding the research, you know that making package is cost a lot of time and of course money.


Is Chat GPT api sometime give wrong output?

Yes, the ai not always 100% accurate, thats why i make prompt testing (which call real API not mocking) and unit test using jest library.


Voice Recognition is good enough?

For now i use react speech recognition package. the accuracy and the supported device is also based on that package.


Is the open ai API cost expensive?

I try to do efficiency for each prompt so the cost is minimum and keep accurate.


How can I provide information for my chatbot?

This package have something like initial knowledge. its some JSON that tell about:

What is your website about? -> so when the User ask the AI they can answer correctly.


Is it well documented and well crafted?

You can see the docs in this repo, and this package is written with typescript, and tested using jest to make sure the quality.

You don't have to read all the docs in here, because this package now support VS Code IntelliSense what is that? simply its when you hover your mouse into some variable or function VS Code will show some popup (simple tutorial) what is the function about, examples, params, etc...

Just use the source code from demo website, you can literally just understand the package.



G. Payment


The price is only $140 USD.

React GPT Web Guide ($100) + React Speech Highlight ($94)($40) = $140

What you got

After you pay you will be invited inside my private repo and stay inside for 1 year to receive any updates.


Payment method

I accept various payment method:



If you are in indonesia (my country) you can easily transfer through bank and e wallet (gopay, shopee pay, jenius)



Keywords

So this package is the answer for you who looking for:

  • Web Accessibility Tools
  • web guide ai
  • AI Web Accessibility
  • react accessibility library
  • Conversational AI for Websites
  • 3d avatar that have control over website
  • can ai controll over web
  • Using AI to Enhance Web Accessibility
  • Web Metaverse NPC