TUAS integrates Hand Recognition and Finger Tracking technology in the MarSEVR

TUAS integrates Hand Recognition and Finger Tracking technology in the MarSEVR

Prof. Dr. Evangelos Markopoulos and Prof. Dr. Mika Luimula of the Game Lab of the Turku University of Applied Sciences (TUAS) Research Group of Futuristic Interactive Technologies, Panagiotis Markopoulos, PhD student in the doctorate program of Mathematics and Computer Science at the University of Turku, and Niko Laivuori, TUAS Game Lab engineer, achieved to integrate Hand Recognition and Finger Tracking on the MarSEVR (Maritime Safety Education with VR Technology) Command Bridge VR Technology.

The MarSEVR (Maritime Safety Education with VR Technology) is the first MarISOT technology developed at TUAS with the support of Aboa Mare maritime specialists. The technology provides practical situational awareness and decision making by replicating a ship command bridge environment to be used for virtual education in maritime safety training. Committed to follow the VR technology trends and challenges from the game design, the neuro cognitive science perspective, and the user requirements, the evolution of the MarISOT MarSEVR technology has been directed towards increasing impressiveness to obtain more effective user interfaces (UI) and immersive user experiences (UX).

The technology was initiated with the development of the first prototype using the HTC Vive Pro 2.0 controller. Today two research initiatives run in parallel integrating through the Varjo VR-2 Pro headsets finger tracking, hand recognition and eye tracking features. In this stage hand recognition and finger tracking has been achieved with the integration of the Varjo VR-2 Pro VR headsets and the Ultraleap’s Motion Leap Controller which offers various gestures to track and trace hand and finger movement replacing even more effectively the normal controllers. This light controller, of 32 grams, is an optical hand tracking module with 150×120 degrees field of view, and 80cm range for user’s arm length and has two infrared cameras with resolution of 640×240 pixels with 120 frames per second.

The integration of the finger tracking and hand recognition technologies of VR in the MarISOT technology was approached from a multidisciplinary perspective. The cognitive neuroscience dimension indicated the pedagogical and behavioral impact of such technologies on the trainee. This can be supported furthermore with behavioral biometrics since every person has a unique way to be expressed with individual behavioral characteristics (body movement, coordination, usage, etc). To increase the engagement of the trainee, challenges in game design and user experience in VR have been studied. The maritime sector and the shipping industry rely significantly on the updated training and readiness of the seafarers. Today the MarSEVR technology users are able to point, touch, press, grab, hold and move the command bridges instruments hands free, as they would normally do.

The results of this research have been submitted for publication under the following academic paper:

Evangelos Markopoulos, Panagiotis Markopoulos., Niko Laivuori, Christos Moridis, Mika Luimula (2020). Finger tracking and hand recognition technologies in virtual reality maritime safety training applications. Submitted to CogInfoCom 2020. 11th IEEE International Conference on Cognitive Infocommunications. On-Line Conference. Sept. 23-25, 2020.