Muted - Content4All @Fraunhofer HHI / Berlin

Oct 7, 2019

The research institute Fraunhofer HHI member of consortium Content4All aims to make more content accessible to the deaf community by developing a photorealistic 3D human avatar.

They use a volumetric studio to capture a performer, this innovative capturing technology allow to reproduce fine details (e.g. finger, hands for sign language animation) with limited modeling effort.

Volumetric studio @Fraunhofer HHI

Volumetric studio @Fraunhofer HHI

A 3D point cloud of the performer is built from the volumetric studio capture.

From all these points in space, the challenge for me is to bring out movements, emotions more intensely than if it had been told to us with words.

Tests - 3D character process

1- Captation in studio 2- 3D Pointcloud avatar 3- Realtime 3D Avatar, 1M particles from movement (90FPS 3k stereo)

Between vision and sound: vibration

Hear with the eyes, see with the sound.

Tests - 3D character / 1M particles following waves oscillations / Realtime VR 90 FPS

Part of the work will be oriented on waves as a transfer of sensation between sight and hearing. The visual creation of the 3D character will be associated with sound thanks to the wave phenomenon.

The wave passes from one world to another by changing its frequency and length, like a channel for movement, sound, colour....

The particles composing the 3D character follow the fluctuations of waves that can resonate. (just incredible inside the VR headset !!)

One million interactive particles

To be able to emit and control 1 million particles interactively (realtime) in 3K at a frequency of 90 frames per second, it is impossible to use the computer's CPU alone.

The technique I use allows me to store the spatial coordinates x,y,z of each particle emitted inside an image. The texture thus created can be directly used by the GPU (Graphic card) to place each particle in space. This way of storing point coordinates is up to 10 times faster. For each frame (90 per second) an image is built.

Principle :

A square image of 1024 pixels on the side therefore contains 1,048,576 pixels Each pixel contains 4 channels: Red, Green, Blue and Alpha

For on particule position Px is stored in the red channel Py is stored in the green channel Px is stored in the blue channel (Alpha channel is left vacant)

one pixel = one point in space

1 million pixels = 1 million points

Space coordinates stored inside 1,048,576 pixels : Same 1,048,576 particles in space :

Created by Christophe Monchalin

with the support of the

VERTIGO / STARTS Residencies project as part of the STARTS program of the European Commission, based on technological elements from Content4All.

Fraunhofer HHI - Vision and Imaging Technologies

Chroniques 2018 – International digital art biennal

Created as part of the production platform, supported by the Région Sud

Coordination : Seconde Nature et ZINC.

With the support of Fédération Wallonie-Bruxelles

and Wallonie-Bruxelles international

With the help of :

La Maison des Cultures et de la Cohésion Sociale de Molenbeek-Saint-Jean

Le BRASS – Centre Culturel de Forest

La Maison du peuple, Commune de Saint-Gilles

Le Centre culturel Bruxelles Nord - Maison de la création

La Raffinerie / Charleroi Danses

Muted v1 Prototype

IT Dev and Graphics : Christophe Monchalin, Gauthier Roumagne - Interpretation : Léonore Guy, Maud Chapoutier - Illustration : Tomoko Yoshida, Christophe Monchalin - 2D animation : Evelien de Roeck - Music : Yann Deval - Voice : Léonore Guy

Search tags

VERTIGO is funded under the H2020 European STARTS initiative, innovation at the nexus of Science, Technology, and the ARTS. STARTS promotes the arts as catalysts for efficient conversion of science and technology knowledge into products, services, and processes.