Research seminar: Beskow, Mesch, Alexanderson, Klezovich, Malmberg (Stockholm University/KTH)

Abstract

We are going to present to you the work we are doing in the ongoing project SignBot, dedicated to generative AI for sign languages at KTH, Speech, Music, and Hearing department.

In our studies so far we have been exploring representation learning for Swedish Sign Language (SSL) with an aim to use the learned representations for SSL generative modeling onto a 3D avatar.

The learned representations, or ”motion primitives”, were trained on sign language video data and evaluated against the phonological annotation from the SSL lexicon.

We also describe recording of our own motion capture dataset of SSL including detailed face and finger capture, how these recordings are mapped to high-quality sign language animation, and how this can be evaluated with native signers.

This data will later be used to train generative models, allowing the representations learnt from 2D to drive 3D sign language animation. In this talk we will discuss the recent advances in generative AI for sign language, our ongoing work and our future goals.

On the project SIGNBOT