Send Human Image with airSlate SignNow

Send human image

in this work we are given a single image of a human and our goal is to estimate the 3d pose and shape for the full body the face as well as the hands in a unified manner previous methods have addressed each part of the body in an independent manner or required sophisticated setups with a large number of cameras to enable holistic capture in this work we estimate the pose and shape of the body hands and face together while requiring only a single RGB image as input to achieve this goal we first need a realistic 3d model of the body that is able to represent the complexity of human faces hands and body poses current models include only the body include body and hands without a deformable face or model all parts together but are a result of stitching the individual parts leading to non fully realistic results with artifacts to address this we use a large corpus of 3d scans to learn a new holistic body model with a deformable face and articulated hands we call our model simplex standing for a simple expressive simplex is based on simple retaining all the benefits of the original model compatibility with graphic software simple parameterization small size efficient and differentiable given our expressive body model we formulate a method to estimate the model parameters directly from RGB images our method is based on simplify and we call it simplify X similar to simplify we first detect 2d image features bottom-up and then fit simplex to them in a top-down manner simplify X improve simplify in all directions our more expressive simplex model pushes us to go beyond body only 2d key points used by simplifying to this end we rely on an updated off-the-shelf to Depot's detector that can detect body joints as well as hand and face key points that enable expressive body capture with a level of detail that was not possible before moreover we introduced a better body post-fire which we learn from millions of body pose examples our prior is based on a neural network and more specifically a variational auto encoder the learn representation of this V AE is a low dimensional late space of valid body poses optimizing over this latent space and using a quadratic penalty on the latent representation forces the recovered poses to remain in the manifold of valid poses even with a good pose prior the recovered poses can still include self collisions and inter penetrations of body parts that are physically impossible to avoid these problems we employ an inner penetration penalty based on a detailed collision based model for meshes our formulation is explicitly differentiable and more accurate than the capsule approximation of simplifying finally we also train a gender detector to automatically estimate the gender of the person in the image fitting a generous Pacific model to 2d evidence leads to more accurate and realistic 3d reconstructions than using a gender neutral model so the final optimization objective includes a reprojection term ensuring that the projection of the 3d joints of the model agrees the detected 2d key points we use pose priors for the body the hand and the face priors for the shape and expression parameters and finally the inner penetration term to avoid collisions on the mesh level the complete fitting pipeline is implemented in pi torch achieving an eight-time speed-up over the original chump hai implementation besides the new simplex model and the corresponding fitting method simplify X we also contribute a new data set to enable evaluation of our expressive capture we create the Stata set by aligning simplex to detailed 3d body scans and carefully curating to select a hundred accurate alignments with diverse body poses hand pose and facial expressions we call our data set eh F standing for expressive hands and face to the best of our knowledge this is the first data set including pose and shaped ground truth for body pose hand pose and facial expressions to demonstrate the importance of more expressive models we fit different variants of simplex that is without expressive hands and/or face as expected more expressive models lead to more accurate and detailed reconstruction x' both qualitatively and quantitatively holistic capture can also improve part specific approaches that model the hands and the face independently we compare with the hands only approach of patent Alerus at all which also uses hand key point detection in case of good detection x' both methods perform well but the hand only approach is typically failing when the hand key point detections are noisy in contrast to that the holistic fitting can benefit from the context of the human body and improve reconstructions for these challenging cases this finding extends also to head only models here fitting the flame head model to facial landmark can be inaccurate under occlusions or failed completely as in the case in the last row while simplify X can still recover reasonable reconstructions make our simplex model the simplify X code in the EHF data set publicly available thank you