Controllable multi-domain semantic artwork synthesis

Yuantian Huang*, Satoshi Iizuka, Edgar Simo-Serra, Kazuhiro Fukui

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

We present a novel framework for the multi-domain synthesis of artworks from semantic layouts. One of the main limitations of this challenging task is the lack of publicly available segmentation datasets for art synthesis. To address this problem, we propose a dataset called ArtSem that contains 40,000 images of artwork from four different domains, with their corresponding semantic label maps. We first extracted semantic maps from landscape photography and used a conditional generative adversarial network (GAN)-based approach for generating high-quality artwork from semantic maps without requiring paired training data. Furthermore, we propose an artwork-synthesis model using domain-dependent variational encoders for high-quality multi-domain synthesis. Subsequently, the model was improved and complemented with a simple but effective normalization method based on jointly normalizing semantics and style, which we call spatially style-adaptive normalization (SSTAN). Compared to the previous methods, which only take semantic layout as the input, our model jointly learns style and semantic information representation, improving the generation quality of artistic images. These results indicate that our model learned to separate the domains in the latent space. Thus, we can perform fine-grained control of the synthesized artwork by identifying hyperplanes that separate the different domains. Moreover, by combining the proposed dataset and approach, we generated user-controllable artworks of higher quality than that of existing approaches, as corroborated by quantitative metrics and a user study.[Figure not available: see fulltext.]

Original languageEnglish
Pages (from-to)355-373
Number of pages19
JournalComputational Visual Media
Volume10
Issue number2
DOIs
Publication statusPublished - 2024 Apr

Keywords

  • datasets
  • generative adversarial network (GAN)
  • non-photorealistic rendering
  • semantic artwork synthesis

ASJC Scopus subject areas

  • Computer Vision and Pattern Recognition
  • Computer Graphics and Computer-Aided Design
  • Artificial Intelligence

Fingerprint

Dive into the research topics of 'Controllable multi-domain semantic artwork synthesis'. Together they form a unique fingerprint.

Cite this