MuscleVAE: Model-Based Controllers of Muscle-Actuated Characters

Peking University, China1, National Key Lab of General AI, China2


In this paper, we present a simulation and control framework for generating biomechanically plausible motion for muscle-actuated characters. We incorporate a fatigue dynamics model, the 3CC-r model, into the widely-adopted Hill-type muscle model to simulate the development and recovery of fatigue in muscles, which creates a natural evolution of motion style caused by the accumulation of fatigue from prolonged activities. To address the challenging problem of controlling a musculoskeletal system with high degrees of freedom, we propose a novel muscle-space control strategy based on PD control. Our simulation and control framework facilitates the training of a generative model for muscle-based motion control, which we refer to as MuscleVAE. By leveraging the variational autoencoders (VAEs), MuscleVAE is capable of learning a rich and flexible latent representation of skills from a large unstructured motion dataset, encoding not only motion features but also muscle control and fatigue properties. We demonstrate that the MuscleVAE model can be efficiently trained using a model-based approach, resulting in the production of high-fidelity motions and enabling a variety of downstream tasks.



Our character can reproduce the motions accurately using muscles. The character is capable of imitating highly dynamic movements. Even for those are not seen during training.



Spin Jump

Jump Spin Kick

Horse Stance

Random Motion

Unseen Locomotion

Unseen Dance

Random Sampling

Our character is controlled by the prior network for random sampling in latent space. The character is able to perform diverse set of skills. While generating natural transitions between them.

Velocity Control

Using high level policy, direction and speed control can be achieved. We can see here that the character can follows the control accurately and can even handle sharp turns smoothly.

Fatigue Results

Our character is instructed to run forward and then to recover by adopting slow walk. The character can regain its force by resting and run again. MuscleVAE can generate fatigue data from motions that does not initially include fatigue characteristics.