JointSelector question

[kelvinqin]

Dear Taras,
Today I want to modify your code to not only select upper body (15 joints) but also select legs and foots (addition 10 joints). A problem comes out:

Your original JointSelector:
('jtsel', JointSelector(['Spine','Spine1','Spine2','Spine3','Neck','Neck1','Head','RightShoulder', 'RightArm', 'RightForeArm', 'RightHand', 'LeftShoulder', 'LeftArm', 'LeftForeArm', 'LeftHand'], include_root=True)),
Will return expmap in 45 dims

My revised JointSelector:
('jtsel', JointSelector(['Spine','Spine1','Spine2','Spine3','Neck','Neck1','Head','RightShoulder', 'RightArm', 'RightForeArm', 'RightHand', 'LeftShoulder', 'LeftArm', 'LeftForeArm', 'LeftHand', 'RightUpLeg', 'RightLeg', 'RightFoot', 'RightForeFoot', 'RightToeBase', 'LeftUpLeg', 'LeftLeg', 'LeftFoot', 'LeftForeFoot', 'LeftToeBase'], include_root=True)),
Unfortunately return expmap in 65 dims (I expect 75 dims)

Can you please take a look whether preprocessing.py has a bug please? Thanks!

BTW, a side question is why you did not include figures in your original JointSelector of upperbody?

Kelvin

[Svito-zar]

Dear Kelvin,

Did you confirm that all of the following joints are present in the BVH file?

 'RightUpLeg', 'RightLeg', 'RightFoot', 'RightForeFoot', 'RightToeBase', 'LeftUpLeg', 'LeftLeg', 'LeftFoot', 'LeftForeFoot', 'LeftToeBase'

As for the side question - I am sorry, I did not understand it. Can you please clarify what do you mean?

Best regards,
Taras

[kelvinqin]

Dear Taras,
Yes, I printed out the BVH file, those joints indeed present. Thanks if you can take a look at your side also. For example, if you print out "GENEA_Challenge_2020_data_release/Training_data/Motion/Recording_001/, you will find all the above joints. For example:
JOINT RightUpLeg
{
OFFSET -9.98441 0 0
CHANNELS 3 Zrotation Xrotation Yrotation
JOINT RightLeg
{
OFFSET 0 -42.1179 0
CHANNELS 3 Zrotation Xrotation Yrotation
JOINT RightFoot
{
OFFSET 0 -45.0002 -1.07276e-005
CHANNELS 3 Zrotation Xrotation Yrotation
JOINT RightForeFoot
{
OFFSET 0 -3.7482 0
CHANNELS 3 Zrotation Xrotation Yrotation
JOINT RightToeBase
{
OFFSET 0 0.0632497 14.4312
CHANNELS 3 Zrotation Xrotation Yrotation
End Site
{
OFFSET 0 0 0
}
}
}
}
}
}

JOINT LeftUpLeg
{
OFFSET 9.98441 0 0
CHANNELS 3 Zrotation Xrotation Yrotation
JOINT LeftLeg
{
OFFSET 0 -42.118 0
CHANNELS 3 Zrotation Xrotation Yrotation
JOINT LeftFoot
{
OFFSET 0 -45.0007 0
CHANNELS 3 Zrotation Xrotation Yrotation
JOINT LeftForeFoot
{
OFFSET 0 -3.74824 0
CHANNELS 3 Zrotation Xrotation Yrotation
JOINT LeftToeBase
{
OFFSET 0 0.0632505 14.4311
CHANNELS 3 Zrotation Xrotation Yrotation
End Site
{
OFFSET 0 0 0
}
}
}
}
}
}

Sorry for my typo in my side question, the correct question is: why you did not include fingers (Thumb, Index, Middle, Ring, Pinky) related joints in your training?

Kelvin

[Svito-zar]

Hm ... I am not sure what is the problem here, to be honest.
Maybe @simonalexanderson can answer your question since he wrote this code.

As for the side question - we did not use fingers, because the fingers data in this dataset has very low quality and high noise-to-signal ratio. If you visualize some training sequences - you will see that fingers are completely broken. Unfortunately.

Taras

[kelvinqin]

Dear Taras,
Thanks so much for your answer, now I understood why you did not include fingers :-)

Dear Simon,
When I run bvh2feature.py by setting JointSelector to include leg and foot, I got expmap with dim 65 which I expect to be 75 (25*3), not sure if you have chance to take a look at the related code of pymo package please? Thanks!

Here is my JointSelector:
('jtsel', JointSelector(['Spine','Spine1','Spine2','Spine3','Neck','Neck1','Head','RightShoulder', 'RightArm', 'RightForeArm', 'RightHand', 'LeftShoulder', 'LeftArm', 'LeftForeArm', 'LeftHand', 'RightUpLeg', 'RightLeg', 'RightFoot', 'RightForeFoot', 'RightToeBase', 'LeftUpLeg', 'LeftLeg', 'LeftFoot', 'LeftForeFoot', 'LeftToeBase'], include_root=True)),

Kelvin

[kelvinqin]

Dear Taras,
Since I don't know whether Simon can see your comment, so sbout JointSelector question, I just sent an email to @simonalexanderson,
have a nice day,
Kelvin

[simonalexanderson]

Hi Kelvin,
My guess is that some of the rotations are constants and filtered out by the ConstanstRemover() further down in the feature-extraction pipeline. These would usuallly be hinge-joints such as knee, and elbow joints. You can check the output chanels by ommitting the last Numpyfier() step in the pipeline and printing the chanel-names: print(output_data[0].values.columns).

[kelvinqin]

Dear Simon,
Thanks so much, I am checking that. :-)
Kelvin

[kelvinqin]

Dear Simon,
With your help, I found the reason ---

This is the output channels:
out_data[0].values.columns = Index(['Spine_Zrotation', 'Spine_Xrotation', 'Spine_Yrotation',
'Spine1_Zrotation', 'Spine1_Xrotation', 'Spine1_Yrotation',
'Spine2_Zrotation', 'Spine2_Xrotation', 'Spine2_Yrotation',
'Spine3_Zrotation', 'Spine3_Xrotation', 'Spine3_Yrotation',
'Neck_Zrotation', 'Neck_Xrotation', 'Neck_Yrotation', 'Neck1_Zrotation',
'Neck1_Xrotation', 'Neck1_Yrotation', 'Head_Zrotation',
'Head_Xrotation', 'Head_Yrotation', 'RightShoulder_Zrotation',
'RightShoulder_Xrotation', 'RightShoulder_Yrotation',
'RightArm_Zrotation', 'RightArm_Xrotation', 'RightArm_Yrotation',
'RightForeArm_Zrotation', 'RightForeArm_Xrotation',
'RightForeArm_Yrotation', 'RightHand_Zrotation', 'RightHand_Xrotation',
'RightHand_Yrotation', 'LeftShoulder_Zrotation',
'LeftShoulder_Xrotation', 'LeftShoulder_Yrotation', 'LeftArm_Zrotation',
'LeftArm_Xrotation', 'LeftArm_Yrotation', 'LeftForeArm_Zrotation',
'LeftForeArm_Xrotation', 'LeftForeArm_Yrotation', 'LeftHand_Zrotation',
'LeftHand_Xrotation', 'LeftHand_Yrotation', 'RightUpLeg_Zrotation',
'RightUpLeg_Xrotation', 'RightUpLeg_Yrotation', 'RightLeg_Zrotation',
'RightLeg_Xrotation', 'RightLeg_Yrotation', 'RightFoot_Zrotation',
'RightFoot_Xrotation', 'RightFoot_Yrotation', 'RightToeBase_Xrotation',
'RightToeBase_Yrotation', 'LeftUpLeg_Zrotation', 'LeftUpLeg_Xrotation',
'LeftUpLeg_Yrotation', 'LeftLeg_Zrotation', 'LeftLeg_Xrotation',
'LeftLeg_Yrotation', 'LeftFoot_Zrotation', 'LeftFoot_Xrotation',
'LeftFoot_Yrotation', 'LeftToeBase_Xrotation', 'LeftToeBase_Yrotation'],
dtype='object')

Seems the following channels were filtered out due to the nature:
LeftForeFoot (Z,X,Y), RightForeFoot (Z,X,Y), LeftToeBase (Z) and RightToeBase(Z)

I have a solution to keep them now by disabling ConstantsRemover :-)

Thanks so much,
Kelvin

[simonalexanderson]

Why don't you just un-select the foot-joints or keep the ConstantsRemover? The ConstantsRemover automatically puts all constant values back when inverting the pipeline, so you won't loose anything. Training a model on constant features is usually a bad idea...

[kelvinqin]

Why don't you just un-select the foot-joints or keep the ConstantsRemover? The ConstantsRemover automatically puts all constant values back when inverting the pipeline, so you won't loose anything. Training a model on constant features is usually a bad idea...

Dear Simon, and Taras,

I am not so skillful to modify the code :-) , so it is just for a quick and dirty experiment.

I just build a model to generate the "full body" skeleton, attached pls find the generated video in mp4 format. As you noticed that there is an issue --- the body shaking left and right or back and forth. I guess it is because I am treating Hip as the center, as you two are the only experts I knew on BVH, not sure if you can guide me to resolve this issue pls? Thanks a lot.

Meanwhile, I will try your suggestion soon by keep ConstantRemover, I agree with you it is a bad idea to learn constant features. Can you elaborate it a little on "Why don't you just un-select the foot-joints or keep the ConstantsRemover?" What do you mean by "un-select" pls? How to do it?

Thanks in advance!

Output.mp4

Kelvinn

[kelvinqin]

Dear Simon,
I guess what you mean by "un-select foot joints" is to move out those "constant" joints by this way:

data_pipe = Pipeline([
   ('dwnsampl', DownSampler(tgt_fps=fps,  keep_all=False)),
   ('root', RootTransformer('hip_centric')),
   ('mir', Mirror(axis='X', append=True)),
   ('jtsel', JointSelector(['Spine','Spine1','Spine2','Spine3','Neck','Neck1','Head','RightShoulder', 'RightArm', 'RightForeArm', 'RightHand', 'LeftShoulder', 'LeftArm', 'LeftForeArm', 'LeftHand', 'RightUpLeg', 'RightLeg', 'RightFoot', 'LeftUpLeg', 'LeftLeg', 'LeftFoot'], include_root=True)),
   # ('jtsel', JointSelector(['Spine','Spine1','Spine2','Spine3','Neck','Neck1','Head','RightShoulder', 'RightArm', 'RightForeArm', 'RightHand', 'LeftShoulder', 'LeftArm', 'LeftForeArm', 'LeftHand'], include_root=True)),
   ('exp', MocapParameterizer('expmap')), 
   ('cnst', ConstantsRemover()),
   ('np', Numpyfier())
])

Thanks for your help,

Kelvin

[Svito-zar]

Yes, I think what @simonalexanderson meant was to remove those joints from the JointSelector list of joints.
Hope it helps.

If it solves your original issue - please close this issue.
Otherwise - let us know.

If you have some other issues - please open a new issue :)

[kelvinqin]

Taras,
Thanks so much for all the help, yes, please close it :-)
Kelvin