thanks for reading this question!

We are trying to create simple animations of, for example, a talking cartoon character. The challenge is to synchronize several images that correspond with certain 'mouth shapes', (closed lips for a P or B or M sound for example), with some unknown audio speaking potentially any words.

The audio can be in any language, though we will know what language it is as that is identified by the end user. Thus, the challenge requires some kind of syncing to the audio without being specifically stuck to English or another specific language.

I was thinking it might be possible to map an International Phonetic Alphabet (IPA) string to the images, but then the challenge is still to figure out how to somehow map the audio + language identifier to the right IPA string, and then to synchronize the right parts of the string to the right times in the audio.

If it helps, we do also have exact transcripts of the text that the audio is speaking.

If third party libraries are available for this kind of thing, they would be welcome, though our code base is in JS/TS only.

Thanks!