XHand: Real-time Expressive Hand Avatar (2025)

II-A Parametric Model-based Method

3D animatable human models[10, 9, 5] enable shape deformation and animation by decoding the low-dimensional parameters into a high-dimensional space. Loperet al.[10] introduce a linear model to explicitly represent the human body through adjusting shape and pose parameters. MANO hand model[9] utilizes a rigged hand mesh with fixed-topology that can be easily deformed according to the parameters. The low resolution of the template mesh hinders its application in scenarios requiring higher precision. To address this limitation, Liet al.[7] integrate muscle groups with shape registration, which results in an optimized mesh with finer appearance. Furthermore,parametric model-based methods[43, 44, 45, 11, 1, 2, 46, 47, 48] have shown the promising results in accurately recovering hand poses from input images, however, they have difficulty in effectively capturing textures and geometric details for the resulting meshes. In this paper, our proposed XHand approach is able to capture the fine details of both appearance and geometry by taking advantages of Lambertian reflectance model[49].

II-B Model-free Approach

Parametric models have proven to be valuable in incorporating prior knowledge of pose and shape in hand geometry reconstruction[9], while their representation capability is restricted due to the low resolution of the template mesh. To address this issue, Choiet al.[17] introduce a network based on graph convolutional neural networks (GCN) that directly estimates the 3D coordinates of human mesh from 2D human pose. Chenet al.[18] present a UV-based representation of 3D hand mesh to estimate hand vertex positions. Mobrecon[50] predicts hand mesh in real-time through a 2D encoder and a 3D decoder. Despite the encouraging results, the above methods still cannot capture the geometric details of hand. Moonet al.[19] propose an encoder-decoder framework that employs a template mesh to learn corrective parameters for pose and appearance. Although having achieved the improved geometry and articulated deformation, it has difficulty in rendering photo-realistic hand images. Gan et al.[51] introduce an optimized pipeline that utilizes multi-view images to reconstruct a static hand mesh. Unfortunately, it overlooks the variations due to joint movements. Karunratanakulet al.[6] design a shadow-aware differentiable rendering scheme that optimizes the abledo and normal map to represent hand avatar. However, its geometry remains smoothing. In contrast to the above methods, our proposed XHand approach is able to simultaneously synthesize the detailed geometry and photo-realistic images for drivable hands.

II-C Neural Hand Representation

There are various alternatives available for neural hand representations, such as HandAvatar[27], HandNeRF[26], LISA[16] and LiveHand[8]. In order to achieve high fidelity rendering of human hands, Chenet al.[27] propose HandAvatar to generate photo-realistic hand images with arbitrary poses, which take into account both occupancy and illumination fields. LISA[16] is a neural implicit model with hand textures, which focuses on signed distance functions (SDFs) and volumetric rendering. Mundraet al.[8] propose LiveHand that makes use of a low-resolution NeRF representation to describe dynamic hands and a CNN-based super-resolution module to facilitate high-quality rendering. Despite the efficiency in rendering hand images, it is hard for those approaches to capture the details of hand mesh geometry. Luanet al.[52] introduce a frequency decomposition loss to estimate the personalized hand shape from a single image, which effectively address the challenge of data scarcity. Chenet al. introduce a spatially varying linear lighting model as a neural renderer to preserve personalized fidelity and sharp details under natural illumination. Zhenget al. facilitate the creation of detailed hand avatars from a single image by learning and utilizing data-driven hand priors. In this work, our presented XHand method focuses on synthesizing the hand avatars with fine-grained geometry in real-time.

II-D Generic Animatable Objects

In addition to the aforementioned methods on hand modeling, there have been some studies reconstructing animatable whole or partial human bodies or animals[28, 29, 30]. Face models primarily pay their attention to facial expressions, appearance, and texture, rather than handling large-scale deformations[32, 33, 34, 35]. Zheng at al.[32] bridge the gap between explicit mesh and implicit representations by a deformable point-based model that incorporates intrinsic albedo and normal shading. To build human body model[5, 23, 53, 36, 37, 25, 38], numerous challenges arise from the intricate deformations, which make it arduous to precisely capture intricate details, such as textures and scan-like appearances, especially in smaller areas like the hands and face. Previous works[22, 39, 40] have explored to establish the correspondences between pose space and template space through linear blend skinning and inverse skinning weights. Alldiecket al.[13] employ learning-based implicit representations to model human bodies via SDFs. Chenet al.[23] propose a forward skinning model that finds all canonical correspondences of deformed points. Shenet al.[54] introduce XAvatar to achieve high fidelity of rigged human bodies, which employ part-aware sampling and initialization strategies to learn neural shapes and deformation fields.

III-A Detailed Hand Representation

The original MANO mesh[9], consisting of 778 vertices and 1538 faces, has limited capacity to accurately represent fine-grained details[27]. To overcome this challenge by enhancing the mesh resolution to capture intricate features, we employ an uniform subdivision strategy on the MANO template mesh, as shown in Fig.3. By adding new vertices at midpoint of each edge for three times, we obtain a refined mesh with 49,281 vertices and 98,432 faces. To associate skinning weights with these additional vertices, we compute the average weights assigned to the endpoints of the corresponding edges.

Feature Embedding Module. Generally, it is challenging to learn the distinctive hand features in different poses. To better separate between the deformation caused by changes in posture and the inherent characteristics of the hand, we present an efficient feature embedding module in this paper. It relies on the average features of hand mesh and computes offsets of features in different poses, as illustrated in Fig.4.

Remark. The feature embedding modules allows for the interpretable acquisition of hand features $f_{\mathcal{M}}$ corresponding to the pose $\theta_{t}$. The average mesh features are stored in $\bar{f}_{\mathcal{M}}$, while the features offsets are affected by the pose $\theta$. More importantly, the training objectives are greatly simplified by taking into account of the average features constraints, which leads to the faster convergence and improved accuracy.

III-B Mesh Rendering

Mesh-based Neural Rendering.The NeRF-based methods usually employ volumetric rendering along its corresponding camera ray $\mathbf{d}$ to acquire pixel color[26, 8], which usually require a large amount of training time. Instead, we aim to minimize the sampling time and enhance the rendering quality by making use of a mesh-based neural rendering method that is able to take advantage of the consistent topology of our refined mesh.

The mesh is explicitly represented by triangular facets so that the intersection points between rays and meshes are located within the facets. The features that describe meshes, such as position, color, and normal, are associated with their respective vertices. Consequently, the attributes of intersection points can be calculated by interpolating the three vertices of triangular facet to its intersection point. The efficient differentiable rasterization[56] ensures the feasibility of inverse rendering and mesh-based neural rendering.

Given a camera view $\pi^{i}$, our mesh-based neural render $\mathcal{C}(\pi^{i})$ synthesizes the image with respect to the position $\mathbf{x}$, normal $\mathcal{N}$, feature vector $\mathbf{h}$ and ray direction $\mathbf{d}$, where $\mathbf{x}$, $\mathbf{h}$ and $\mathcal{N}$ are obtained through interpolating with $\mathcal{M}_{fine}$. $\mathbf{h}$ in neural render $\mathcal{C}(\pi^{i})$ contains the latent codes $Q_{D}$ and $Q_{\rho}$ detached from $\Psi_{D}$ and $\Psi_{\rho}$, and feature vector $Q_{render}$[51]. $Q_{render}$ is utilized to represent the latent code of vertices during rendering. As in[20], the neural network $\mathcal{C}$ comprises 8 fully-connected layers with ReLU activations and 256 channels per layer, excluding the output layer. Furthermore, it includes a skip connection that concatenates the input to the fifth layer, which is depicted in Fig.5.

III-C Training Process

To obtain a personalized hand representation, the parameters of the three feature embedding modules $\Psi_{D}$, $\Psi_{lbs}$, and $\Psi_{\rho}$, as well as the neural render $\mathcal{C}$, require to be optimized based on multi-view image sequences. Our training process consists of three steps, including initialization, training feature embedding modules, and training the mesh-based neural render.

Initialization of XHand.To train our proposed XHand model, the average features $\bar{f}_{\mathcal{M}}$ of mesh in feature embedding significantly affect training efficiency and results. Random initialization has great impact on training due to estimation errors in $\Psi_{lbs}$ and $\Psi_{D}$, which m ay lead to the failure of inverse rendering. Therefore, it is crucial to initialize the neural hand representation. To this end, the reconstruction result of the first frame ($t=1$) is treated as the initial model.

IV-A Datasets

InterHand2.6M. The InterHand2.6M dataset[41] is a large collection of images, each with a resolution of $512\times 334$ pixels, accompanied by MANO annotations. It includes multi-view temporal sequences of both single and interacting hands. The experiments primarily utilize the 5 FPS version of this dataset.

DeepHandMesh. The DeepHandMesh dataset[19] features images captured from five different viewpoints, matching the resolution of those in InterHand2.6M. It also provides corresponding 3D hand scans, facilitating the validation of mesh reconstruction quality against 3D ground truth data.

IV-B Experimental Setup

Implementation Details. In the experiments, our proposed XHand model is mainly trained and evaluated on the 5FPS version of Interhand2.6M dataset[41], which is made of large-scale multi-view sequences capturing a wide range of hand poses. Each sequence has dozens of images with the size of $512\times 334$. As in[27, 26], XHand model is trained on the InterHand2.6M dataset with 20 views across 50 frames for each sequence. The remaining frames are used for evaluation. To assess the quality of mesh reconstruction, we conduct experiments on DeepHandMesh dataset[19], which consists of 3D hand scans along with images captured from five different views. The images are with the same size as those in InterHand2.6M dataset. We conducted all the experiments on a PC with NVIDIA RTX 3090 GPU having 24GB GPU memory.

We employ PyTorch and Adam Optimizer with a learning rate of $5e^{-4}$. To facilitate differentiable rasterization, we make use of the off-the-shelf renderer nvdiffrast[56]. As in[57], positional encoding is performed on $\mathbf{d}$ and $\mathbf{x}$ before feeding them into the rendering network. In our training process, the feature embedding modules are firstly trained for 500 epochs using inverse rendering. Then, feature embedding modules and neural render are jointly trained for 500 epochs, where the average features $\bar{f}_{\mathcal{M}}$ in feature embedding modules are updated every 50 epochs. We empirically found that the best performance is achieved in case of $\lambda=\omega=0.8$, $\alpha_{1}=10$, $\alpha_{2}=1e^{5}$, and $\alpha_{3}=1e^{4}$. To avoid the excessive displacements and color variations, in $\mathcal{L}_{pLap}(\rho)$, $p_{1}$ is set to the first quartile of $A_{\rho}$, $\gamma_{1}$ is set to $0.1$, and $\gamma_{2}$ is $1$. Similarly, in $\mathcal{L}_{pLap}(D)$, $p_{1}$ is the median of $A_{D}$, and $\gamma_{1}=0.1$, $\gamma_{2}=20$. The lengths of latent codes $Q_{lbs}$, $Q_{D}$, $Q_{\rho}$ and $Q_{render}$ are set to 10, 10, 10 and 20, respectively.

Evaluation Metrics. In the experiments, we fit the hand mesh representations to multi-view images sequence for single scene. For fair comparison, we employ the same evaluation metrics as in[8, 27, 26], which measure the synthesized results with peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and learned perceptual image patch similarity (LPIPS). We calculate the average point-to-surface Euclidean distance (P2S) to assess the accuracy of the reconstructed hand mesh, which is measured in millimeters since the Chamfer distance metric is considered unsuitable due to scale variations between MANO and 3D scans.

IV-C Experimental Results

To investigate the efficacy of our proposed XHand, we treat the subdivided MANO model[9] with vertex albedo as our baseline, which has the merits of the efficient explicit representation. Moreover, we compare our model against several rigged hand expression methods, including LISA[16], HandAvatar[27], HandNeRF[26], and LiveHand[8]. For fair comparison, LiveHand is re-trained with the same setting and LISA is reproduced by[8].

We firstly perform the quantitative evaluation on rendering quality, as shown in TableI. The evaluation metrics of LISA[16] are adopted from LiveHand[8] and the results of HandNeRF[26] are obtained from their original paper. It can be seen that our proposed XHand approach achieves the best results with a PSNR of 34.3dB. Our baseline drives a textured MANO model through LBS weights. Due to lacking the ability to handle illumination changes across different scenes and poses, there exist some artifacts with a PSNR of 28.6dB. NeRF-based methods[16, 27, 26, 8] present the competitive PSNR results, which rely on MANO mesh without fine-grained geometry during rendering. By taking advantage of fine-grained meshes estimated by XHand, our method outperforms the previous approaches using volumetric representation in terms of the rendering quality. Benefiting from our design, XHand achieves 56 frames per second (FPS) on inference. Specifically, the feature embedding modules require 0.7 milliseconds, inverse rendering requires 15 milliseconds and the neural rendering module needs 0.1 milliseconds.

TableII shows the results on DeepHandMesh dataset. Our method outperforms the annotated MANO mesh[9] and DHM[19] by 3.3 mm and 1.2 mm on P2S. This indicates that our proposed feature embedding module can accurately capture the underlying hand mesh deformation comparing to the encoder-decoder scheme in DHM.More experimental results conduct on the DeepHandMesh[19] dataset are visualized in Fig.7.

For better illustration, Fig.6 shows the more detailed comparisons of rendering and geometry on InterHand2.6M test split. Due to the limited expressive capability, it is hard for the baseline MANO model[9] to capture muscle details varying across different poses. Although the hand meshes generated by HandAvatar[27] have more details than MANO, they are still smoothing compared to ours. In terms of geometry, our method exhibits more prominent skin wrinkles based on different poses. The NeRF-based method HandNeRF[26] and LiveHand[8] yield the competitive render results, while they still rely on the MANO model and cannot obtain fine-grained hand geometry. On the contrary, our approach effectively presents an accurate hand representation by taking advantage of the feature embedding module and the topological consistent mesh model, resulting in enhanced rendering quality and geometry quality. Fig.8 visualizes the results of different identities animated using reference poses.

The proposed method efficiently drives personalized hand expressions from arbitrary hand gesture inputs. To demonstrate its performance, in-the-wild data serve as a reference for hand poses, as illustrated in Fig.9. The pose parameters of in-the-wild videos are extracted from HaMeR[42].It is worth noting that we can enhance the vividness of the images by using different spherical harmonic coefficients for relighting.

IV-D Ablation Study

We perform extensive ablation experiments on the InterHand2.6M dataset test set to validate the contributions of various modules and settings within our framework. First, we aim to demonstrate the performance improvements achieved by our proposed feature embedding module and part-aware Laplace smoothing strategy, consistent with our design intentions for the fusion modules. Second, we intend to showcase the robust performance of our XHand model across different numbers of views, highlighting its effectiveness even with limited viewpoints. Furthermore, we conduct a comparative analysis of various neural rendering networks. Based on this evaluation, we have chosen MLPs to enhance both the inference speed and the rendering quality, ensuring efficient and high-fidelity output. The following sections detail these ablation experiments and analyze the results comprehensively.

Ablation Study on Different Components. In the first row of Fig.10, it can be seen that our method significantly highlights skeletal movements and skin changes. Moreover, our design resolves the issue of lighting variations. Our proposed part-aware Laplacian regularization effectively reduces the surface artifacts without sacrificing the details. The feature embedding modules are able to guide the learning of hand avatars by distinguishing average features and pose features, which enhance the reconstruction accuracy.

TableIII shows that the level of mesh detail significantly affects image quality. The rendering results are substantially enhanced through feature embedding. The part-aware Laplacian regularization yields more realistic geometric results, indirectly improving the accuracy of the neural render. Furthermore, the Position Encoder in neural rendering leads to better image quality.

Ablation Study on Number of Views. Typically, the performance of each model is improved along with the increasing number of input images, particularly for the NeRF-based methods. Also, insufficient training data may lead to the reconstruction failure. We conducted ablation experiments using different numbers of views as inputs. As shown in TableIV, we trained the model on sequences of 1, 5, 10, 20 and 30 views to demonstrate the impact of views. Despite being trained with a limited number of viewpoints, including as few as a single viewpoint, our method effectively captures the hand articulations. Furthermore, we achieve the competitive results in case of more than 10 input views.

Choices of Neural Rendering. Traditional neural radiance fields[20] typically employ 8-layer MLPs as the renderer. In contrast, our mesh-based network eliminates the necessity for point cloud sampling, which is able to render through vertex features. Benefiting from topology consistency, our neural renderer can make use of UNet[62] which leads to promising performance. To explore this, we conduct ablation experiments on both network architectures, as detailed in TableV. These experimental results demonstrate that a UNet with 4 layers achieves superior rendering quality, albeit at the expense of inference speed. In comparison to UNet, MLPs can enhance performance by 20% with only a marginal loss in accuracy. Therefore, we have chosen to employ MLPs as our neural renderer. Furthermore, our investigation into a well-designed image generation network, EG3D[61], reveals its unsuitability for neural rendering.