AI Captures the Soul of Manga Through Performance

In the world of manga, a character's soul is conveyed through subtle facial expressions—a knowing glance, a flicker of suspicion, or a smile tinged with regret. Current text-to-image AI models, while capable of generating aesthetically striking manga-style characters, consistently fail to capture these nuanced emotional shifts, resulting in compositionally sound but emotionally vacant panels. This "nuance gap" forces manga artists into laborious fine-tuning, manually redrawing faces to ensure the story flows correctly, breaking the creative flow state crucial for narrative work under tight deadlines. stems from a fundamental cognitive-linguistic gap: our rich mental imagery of expressions is inherently resistant to complete verbal encoding, making it difficult to articulate desired emotions with precision through text prompts alone.

To address this, researchers have developed a novel, dual-hybrid pipeline that integrates performative input from artists into AI-assisted manga creation. The system allows manga artists to use their own facial expressions, captured via video, to directly control character emotions in panels, offering a more intuitive and direct means of "infusing souls" into their work. This approach builds on the LivePortrait engine for high-fidelity facial reenactment, blending intuitive performance with fine-grained numerical control through sliders for adjustments like gaze correction or lip curvature. The primary contribution is not a new generative model but an interactive workflow designed to bridge the gap between artistic intent and AI execution, empowering artists as collaborative partners rather than rigid tools.

Ology involves a three-stage pipeline tailored for manga artists' workflows. First, in the face preparation stage, a hybrid approach combines landmark-based auto-detection using the insightface library with a manual framing tool for handling complex hairstyles or accessories, ensuring robustness and artist-centric control. Second, the interactive expression mapping stage leverages video input from a webcam or pre-recorded reference, allowing artists to scrub through a timeline to select keyframes and then fine-tune expressions with sliders. Third, the composition and refinement stage re-integrates the modified faces into the original panel, though it leaves predictable artifacts like geometric misalignments for artists to polish with their own tools, reinforcing the system's role as an assistant rather than a replacement.

From an end-to-end case study, illustrated in Figure 1, demonstrate the pipeline's effectiveness and limitations. In Stage 1, the auto-detector successfully framed primary faces but failed on a small, distant character, highlighting the need for manual override and justifying the hybrid approach. Stage 2 revealed a crucial temporal offset: the most aesthetically pleasing reenacted expression often appeared a few frames before or after the artist's perceived best performance, underscoring the importance of the interactive timeline slider for collaborative exploration. Stage 3 showed successful re-integration of expressive faces, but persistent artifacts from the underlying LivePortrait model, such as static hair during head rotation and style mismatches introducing photorealistic features, were observed, as detailed in Figure 1(e).

Of this work extend beyond technical innovation to address broader concerns in the manga community. An expert interview with a professional manga artist noted that current digital tools have led to stylistic homogenization, paralleling the expressive homogenization from text-to-image models. This system offers a counter-movement by re-centering the artist, allowing them to capture personal performative intent and maintain creative control, thus serving as a constructive model for human-AI co-creation. It streamlines the tedious task of redrawing faces for emotional shifts, potentially transforming artists into digital sculptors of character performance, with applications in other visual storytelling domains where nuanced expression is critical.

However, the system has clear limitations, as identified in the paper. The lack of holistic, 3D-aware understanding in artifacts where hair and ears remain static during head rotations, and facial reenactment is sensitive to head poses beyond 45 degrees from the camera. Style mismatches can break aesthetic cohesion, suggesting the need for fine-tuned models specific to manga art. Additionally, the landmark-based auto-detector is not designed for non-human characters common in manga, and the workflow inherits risks from facial reenactment technologies, though the focus remains on constructive applications. Future work could integrate 3D-aware models for better geometric control and conduct formal user studies with professional manga artists to quantitatively assess workflow efficiency and expressive control.