AR-supported guitar

AR(Augmented Reality) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory.

Users play the guitar with Hololens that displays some support information; (1)proper 3D hand models, (2)positions that must be held with hingers, (3)a mirror of the guitar neck tracked based on structural features and (4)evaluations and advices considering frequency features.

Demo

00:00-00:10 C code proper model and positions are displayed, play it correctly.

00:10-00:20 Advices are shown 'cause of played incorrectly.

00:20-00:33 Correctly again, then switched to E code thanks to red indicator become full.

Requirement

Note my environment details.

Software

• Unity v2017.3.1f1 (main frame)
• Holographic Remoting (for remoting to HMD)
• Visual Studio 2017 v15.6.7 (sub frame; for building to UWP)
• opencv3.4.4 (C++ native)

Hardware

• Computer: HP Probook 470 G5(Intel Core i7-8550U, 8GB RAM, NVIDEO GeForce 930MX)
• HMD: Microsoft HoloLens 1st
• Mic: Sony ECM-PC60

Papers

• Shun Miura, Toshihiko Ando: "A Suggestion of an AR-supported System for Learning Guitar Playing Using HMD", The Institute of Electronics, Information and Communication Engineers Society Conference 2020, D-15-10, p.124(2020).
• Shun Miura, Toshihiko Ando: "A Suggestion of an AR-supported System for Learning Guitar Playing", Information Processing Society of Japan Interaction 2020, 1P-77, pp.461-463(2020).
• Shun Miura, Toshihiko Ando: "Improvement of an AR-supported System for Learning Guitar Playing", The 82nd National Convention of Information Processing Society of Japan, 2ZF-06, pp.4-529-4-530(2020).
• Shun Miura, Toshihiko Ando: "Improvement of an AR-supported System for Learning Guitar Playing", The 82nd National Convention of Information Processing Society of Japan, 2ZF-06, pp.4-529-4-530(2020).
• Shun Miura, Takuo Suganuma, Toshihiko Ando: "Design of an AR-supported System for Learning Guitar Playing", The 81st National Convention of Information Processing Society of Japan, 2ZJ-08, pp.4-803-4-804(2019).