MagicFace: Training-free Universal-Style Human Image Customized Synthesis

Current state-of-the-art methods for human image customized synthesis typically require tedious training on large-scale datasets. In such cases, they are prone to overfitting and struggle to personalize individuals of unseen styles. Moreover, these methods extensively focus on single-concept human image synthesis and lack the flexibility needed for customizing individuals with multiple given concepts, thereby impeding their broader practical application. To this end, we propose MagicFace, a novel training-free method for universal-style human image personalized synthesis, enabling multi-concept customization by accurately integrating reference concept features into their latent generated region at the pixel level. Specifically, MagicFace introduces a coarse-to-fine generation pipeline, involving two sequential stages: semantic layout construction and concept feature injection. This is achieved by our Reference-aware Self-Attention (RSA) and Region-grouped Blend Attention (RBA) mechanisms. In the first stage, RSA enables the latent image to query features from all reference concepts simultaneously, extracting the overall semantic understanding to facilitate the initial semantic layout establishment. In the second stage, we employ an attention-based semantic segmentation method to pinpoint the latent generated regions of all concepts at each step. Following this, RBA divides the pixels of the latent image into semantic groups, with each group querying fine-grained features from the corresponding reference concept, which ensures precise attribute alignment and feature injection. Throughout the generation process, a weighted mask strategy is employed to ensure the model focuses more on the reference concepts. Extensive experiments demonstrate the superiority of MagicFace in both human-centric subject-to-image synthesis and multi-concept human image customization. It also can be applied to texture transfer, further enhancing its versatility and applicability.

Given reference images, their segmentation masks, and text prompts, we generate personalized image z₀ aligned to the target prompt $P$. The sampling pipeline consists of two paths: (a) the reference path and (b) the customization path. In (a), we first employ a diffusion forward process on the reference images. Then, the noised reference latents are input into vanilla U-Net. In (b), we first sample a Gaussian noise z_T and introduce a coarse-to-fine generation process involving two stages: semantic layout construction and concept feature injection. At each step t, we pass latent z_t to our modified U-Net: (1) in the first stage, we employ RSA to integrate the features from the reference path to facilitate the initial semantic scene construction; (2) in the second stage, we first obtain the latent semantic map of z_t via attention-based segmentation method. Based on this, RBA divides the latent image and ensures fine-grained feature injection for each generated concept. A weighted mask strategy is adopted to ensure the model focuses more on given concepts.