Animation from a sentence: NVIDIA Kimodo

What Kimodo is, and why we use it

Kimodo is an open model from NVIDIA. The code is released under the Apache 2.0 licence, and the models we use are under the NVIDIA Open Model License, which allows commercial use, with NVIDIA claiming no ownership of what the model produces. One of the published models is under a research-only licence instead, so we leave it alone. What ships in Second Chance is the animation, never the model.

Before Kimodo, a new movement meant James downloading a motion pack and hoping the move was in it. Knocking on a door and walking downstairs were. A fist bump and a pat on the shoulder for the man who sells you your stock were not, and in the dart throw we had, the arm finished moving before the dart left the hand.

The bigger reason is walking. Every resident walks with a clip recorded at one speed and is moved through the town at another, so most of them marched, legs going about three times faster than they travelled. Kimodo can be told where the body goes as well as what it does, which means a walk can be made at the speed that resident actually moves. On one batch of walks, a path constraint asked for 140 cm a second and got between 139.6 and 141.2.

The pipeline, step by step

1. Write the movement as one sentence. A first draft for the darts read: "A person stands side-on, aims and throws a dart overarm with the right hand." Why it has to be one sentence is further down.
2. Generate three or four takes. Take quality varies. Of four dart takes, one walked 51 cm during the throw and one let go too early. Kimodo saves each take as motion data and as a BVH file, a plain-text skeleton animation in centimetres at 30 frames a second.
3. Measure the takes rather than eyeballing them. For Simon's greeting we scored how far the hand reached forward of the hips, whether it came back to rest and whether the feet drifted. A contact sheet of stick figures, side and front, makes the differences visible.
4. Convert in Blender, headless. A script imports the BVH, checks Kimodo's 77 bones are all there, builds a stand-in body of one thin box per bone (Unreal will not make a skeleton from a bare armature) and exports an FBX with the animation baked in.
5. Import into Unreal Engine 5.8 onto one shared Kimodo skeleton, then retarget onto the MetaHuman who will use it. Kimodo's skeleton uses the same bone names as the Motifect motion-capture pack the project already used, so the existing retarget chains bound it first time, fingers included.
6. Check the result on the target body: hand travel for one-handed clips, feet on the floor, walk speed against a second, different measurement, and finally watching it play.

Installing it on Windows

Two things cost time. The first is a C++ part of the package, a motion clean-up step that stops feet skating, which needs CMake to build. The install stops with CMake must be installed to build this package. Installing CMake into the Python environment was not enough on its own, because pip builds the package in an isolated environment that cannot see it. Installing CMake, setuptools and wheel into the environment and then installing Kimodo with pip's --no-build-isolation flag worked, and the build found the Visual Studio C++ tools by itself. Skipping the clean-up step would have been quicker, and every clip would have skated.

The second is the graphics card. Run on the card, Kimodo wants about 17 GB of video memory. Ours has 8, and the Unreal editor needs all of it. So we run Kimodo on the processor, which is slow but needs no card at all, as long as the program cannot find one. The usual way to hide a card is to set CUDA_VISIBLE_DEVICES to an empty value. On Windows an empty environment variable is dropped before the program starts, so the program saw the card anyway. Setting it to -1 works, and a one-line check that has to print False proves it.

Kimodo reads the sentence with Meta's Llama 3 8B Instruct model. Meta's own copy needs a manual approval, so James chose an ungated mirror, and we checked it against Meta's before trusting it: all four weight files, the tokenizer and the config had the same checksums. One configuration value has to keep Meta's original model name exactly, because the prompt template is chosen from that string.

On the processor, one run holds about 19 GB of memory and takes minutes, not seconds: four three-second takes took about five minutes of generation once the language model had loaded, and Simon's two greeting clips took about eleven minutes in all. The PC has 64 GB. Loading the language model beside the open town took free memory from 49 GB to 12, so Kimodo now runs with the editor closed or on an empty level, behind a guard script that refuses to start below 40 GB free. We also tried the card with the editor shut. Generation fell from about seven and a half minutes to 22 seconds, but each run still took about nine, because the language model loads on the processor every time.

Five things that went wrong

A full stop splits the prompt. Each sentence becomes its own segment at the full requested length, so a three-second throw with a second sentence about planted feet came back as six seconds: the throw, then three seconds of a man standing still. We write one sentence, or trim the clip afterwards.

The default rest pose lies flat. Kimodo's default rest lays every bone along one axis at hip height. The motion is the same either way, but Unreal's retargeter lines two skeletons up by their rest poses, and a man lying flat in mid-air lines up with very little. Kimodo can re-export a take with a standing T-pose rest (kimodo_convert take.npz take.bvh --bvh_standard_tpose) without running the model again, which takes seconds.

Blender only ever made the clip longer. Our conversion script exported 250 frames for a 90-frame clip, because the Blender call it used to fit the scene only extends the range. It now fits the scene to the animation.

The same take number gave the same walk. The roadmen on the estate each got a walk and an idle in words of their own. Laid out as stick figures, take two of one lad and take two of another were within one or two centimetres of each other, against four to thirteen for walks that really differed. The randomness was seeded by take number, not by the sentence. Now each character gets a seed of his own, or the set is picked together with a check on how different the walks are.

Lying down sinks. Retargeting onto a smaller body scales the pelvis height, which is right for someone standing and pushes someone lying into the floor or the mattress. The injured clips for the ambulance needed the pelvis lifting until the ankles matched the source: seven centimetres for lying, a little over nine for the beds.

Where it falls short

Fingers. Every Kimodo clip has the same relaxed curl in the hands; the model does not animate fingers. That is fine for a fist bump and wrong for a barbell. When James saw the first gym stills, the hands held the bar like a mug somebody had said was hot. The fix was made in Blender: the fingers now close into a power grip, and on the rower both wrists are pinned to one handle line, 32 centimetres apart. Wozza drumming his fingers had to be keyed by hand.

Machines. For the gym, Kimodo's first round squatted with the hands hanging and no bar, held a leg press still with the legs out, and rowed in the air. A second round of prompts still could not work the machines. So three of the seven workout clips, the back squat, the leg press and the rower, are a Kimodo base with the arms and legs placed by inverse kinematics in Blender. The treadmill run, the chest press and the two rests between sets came from Kimodo directly.

Hand to mouth while walking. Eight takes of a man smoking or vaping as he walks all kept the wrist at the face for the whole clip. For the smokers we splice the drag in from an existing clip instead. Taking shoes off failed twice, too.

Where you will see it

• Edward's dart throw at the Slag and Slurry, which replaced a throw where the arm had finished moving before the dart left. Measured on the output, his hand cocks at 175 cm and comes forward 51 cm in four frames.
• Simon, the supplier on the crime path, who bumps fists when he arrives and pats the player on the shoulder every twenty seconds or so while they talk. Both clips worked first time and both reached into thin air, because he stopped 130 cm away. At 95 cm both hands land.
• The roadmen out on the estate after eleven, each with his own walk and idle: hands in the hoodie pouch, a hand up on the bag strap, arms crossed, one who blows into his hands because it is cold.
• The hotel staff at the Bumoral: 22 clips in one batch, among them a receptionist at her desk, a bellboy with a trolley and security walking a guest out.
• The injured on the way to A&E: seated, lying and in bed, and a doctor's examination.
• The cab drivers: idle at the wheel, steer left, steer right, brake, and waiting at a rank checking a phone. The brake came back with the right thigh barely moving and was generated again.
• The gym at the leisure centre: seven clips, the workouts and the rests between sets, each retargeted onto a man's body and a woman's.