JorgeAskur/garment-uv-ip-adapter-v2

Garment UV IP-Adapter (v2)

Fine-tuned IP-Adapter weights that, combined with the garment UV ControlNet, generate UV-space texture atlases for 3D garment meshes from a photo of a real garment.

This is a delta on top of h94/IP-Adapter (sdxl_models/ip-adapter_sdxl.bin):

• Image projection layers (4 tensors)
• IP K/V cross-attention projections across all UNet cross-attention layers (140 tensors)
• Total: ~351 M parameters

Trained on (rendered garment view → ground-truth UV atlas) pairs — the views are the per-sample meshes textured with their target atlases and rendered from 3 angles (front, +45°, -45°).

Usage

import torch
from diffusers import (
    AutoencoderKL, ControlNetModel,
    StableDiffusionXLControlNetPipeline, UniPCMultistepScheduler,
)
from PIL import Image
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

# Load the geometry-aware ControlNet
controlnet = ControlNetModel.from_pretrained(
    "JorgeAskur/garment-uv-controlnet-v3", torch_dtype=torch.float16
)
vae = AutoencoderKL.from_pretrained(
    "madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16
)
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    controlnet=controlnet, vae=vae, torch_dtype=torch.float16,
)
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)

# Load base IP-Adapter, then apply the fine-tuned delta
pipe.load_ip_adapter("h94/IP-Adapter",
                     subfolder="sdxl_models",
                     weight_name="ip-adapter_sdxl.bin")
ft_path = hf_hub_download(repo_id="JorgeAskur/garment-uv-ip-adapter-v2", filename="ip_adapter.safetensors")
pipe.unet.load_state_dict(load_file(ft_path), strict=False)

pipe.set_ip_adapter_scale(0.7)
pipe = pipe.to("cuda")

photo = Image.open("clothing_photo.jpg").convert("RGB").resize((1024, 1024))
normal = Image.open("normal.png").convert("RGB").resize((1024, 1024))

atlas = pipe(
    prompt="garment panels unfolded on canvas, UV texture atlas",
    negative_prompt="3d render, photograph, person wearing, blurry",
    image=normal,
    ip_adapter_image=[photo],
    num_inference_steps=40,
    guidance_scale=5.0,
    controlnet_conditioning_scale=1.0,
    height=1024, width=1024,
).images[0]
atlas.save("atlas.png")

Inputs

• image= (ControlNet): UV-space tangent normal map, encoded (N * 0.5 + 0.5) * 255 per channel, black background.
• ip_adapter_image= (this adapter): photo or render of the garment. Flat-lay product shots and renders of the garment on its mesh work best; in-the-wild photos of people wearing clothes are out-of-distribution.

Tuning

• pipe.set_ip_adapter_scale(0.7) is a balanced default. Push to 0.9 for stronger appearance match (may dominate text guidance), drop to 0.4 to weight the prompt more.
• controlnet_conditioning_scale=1.0 keeps UV layout strict; lower it if the model is forcing the photo into the wrong spatial regions.

Limitations

• CLIP image encoder is 224×224 — fine repeating textures (knit weave, dense plaid) get smoothed in the encoder and may not transfer faithfully.
• Five garment categories from the companion ControlNet (long-shirt, short-shirt, long-dress, pants, shorts).

License

OpenRAIL++ (inherits from SDXL base + IP-Adapter base).