TCTS_FAS

Text-Conditioned Sampling Framework for Text-to-Image Generation with Masked Generative Models

Jaewoong Lee^*,1, Sangwon Jang^*,2, Jaehyeong Jo¹, Jaehong Yoon¹, Yunji Kim³, Jin-Hwa Kim³, Jung-Woo Ha³, Sung Ju Hwang¹

¹KAIST, ²Yonsei University, ³NAVER AI Lab
ICCV 2023
^*Indicates Equal Contribution

Abstract

Token-based masked generative models are gaining popularity for their fast inference time with parallel decoding. While recent token-based approaches achieve competitive performance to diffusion-based models, their generation performance is still suboptimal as they sample multiple tokens simultaneously without considering the dependence among them. We empirically investigate this problem and propose a learnable sampling model, Text-Conditioned Token Selection (TCTS), to select optimal tokens via localized supervision with text information. TCTS improves not only the image quality but also the semantic alignment of the generated images with the given texts. To further improve the image quality, we introduce a cohesive sampling strategy, Frequency Adaptive Sampling (FAS), to each group of tokens divided according to the self-attention maps. We validate the efficacy of TCTS combined with FAS with various generative tasks, demonstrating that it significantly outperforms the baselines in image-text alignment and image quality. Our text-conditioned sampling framework further reduces the original inference time by more than 50% without modifying the original generative model.

Motivation

Reconstructed image using x̂₀^(t) each step during diffusion using the fixed and revocable method.
Only the revocable method is able to edit the tokens to generate a plane according to the text “A view of the end of an airplane in the sky over mountains”.

Reconstructed images using x̂₀^(t) each step during diffusion using the revocable method with and without FAS.
The backgrounds of the images in the top row are over-simplified while our proposed FAS prevents this, as shown in the bottom row. The text is “This small bird is greyish in color with flecks of yellow on the back and breast, and a bit of white on the belly”.

Method

Overall generation framework of proposed TCTS (Text-Conditioned Token Selection) and FAS (Frequency Adaptive Sampling).
After the generator predicts the tokens, TCTS exploits the text condition to detect misaligned tokens and outputs the score map. Meanwhile, FAS splits the tokens according to the frequency using the self-attention map from the generator, performing revocable sampling to high-frequency split and persistent sampling to low-frequency split. The adaptive sampling predicts x̂₀ and decide a few of the locations to mask according to x_t. The token maps produced by FAS is combined with the score map to predict x_t-1.

Generated samples on MS-COCO dataset and evaluation graph of various sampling methods showing their trade-off.
Uniform sampling is a fixed strategy with notably poor text alignment compared to other methods. Random revoke sampling is a revocable strategy with improved text alignment. Ours is TCTS combined with FAS, where both the image quality and the text alignment are significantly better compared to those of baselines. Metrics are measured on all 40K images with their corresponding single caption. The classifier-free guidance scale was fixed at 5 for all sampling methods.

Experiments

Examples of mask-free editing samples with cross-attention map.
Motivated by the operation of self-attention maps in frequency adaptive sampling, we leverage a cross-attention map corresponding to the word of the object that is to be changed, giving weights to resample tokens so that the corresponding locations can be resampled. The cross-attention map is multiplied to the score map of TCTS to perform weighted sampling.

Comparing mask-free object editing samples with and without cross-attention map weighted sampling.
Starting from the image on the left, the result images every 20 steps of editing with 30% masking ratio. Top: Failure case without weighted sampling, Bottom: Results with weighted sampling.

Results of image refinement using TCTS.
Top: Original samples, Bottom: Refined images for 8-steps with TCTS.

BibTeX

@misc{lee2023textconditioned, title={Text-Conditioned Sampling Framework for Text-to-Image Generation with Masked Generative Models}, author={Jaewoong Lee and Sangwon Jang and Jaehyeong Jo and Jaehong Yoon and Yunji Kim and Jin-Hwa Kim and Jung-Woo Ha and Sung Ju Hwang}, year={2023}, eprint={2304.01515}, archivePrefix={arXiv}, primaryClass={cs.LG} }

Text-Conditioned Sampling Framework for Text-to-Image Generation with Masked Generative Models

Abstract

Motivation

Reconstructed image using x̂₀^(t) each step during diffusion using the fixed and revocable method.
Only the revocable method is able to edit the tokens to generate a plane according to the text “A view of the end of an airplane in the sky over mountains”.

Method

Experiments

Comparing mask-free object editing samples with and without cross-attention map weighted sampling.
Starting from the image on the left, the result images every 20 steps of editing with 30% masking ratio. Top: Failure case without weighted sampling, Bottom: Results with weighted sampling.

Results of image refinement using TCTS.
Top: Original samples, Bottom: Refined images for 8-steps with TCTS.

Performance comparison of each method at different steps.
In our experiments, we fixed classifier-free guidance to 5. When we use FAS method, it was possible to lower the FID score while maintaining text alignment.

Comparison of our model and the baseline in performance over generation time.
In our experiments, we fixed classifier-free guidance to 5.

Poster

BibTeX

Text-Conditioned Sampling Framework for Text-to-Image Generation with Masked Generative Models

Abstract

Motivation

Reconstructed image using x̂0(t) each step during diffusion using the fixed and revocable method. Only the revocable method is able to edit the tokens to generate a plane according to the text “A view of the end of an airplane in the sky over mountains”.

Method

Experiments

Comparing mask-free object editing samples with and without cross-attention map weighted sampling. Starting from the image on the left, the result images every 20 steps of editing with 30% masking ratio. Top: Failure case without weighted sampling, Bottom: Results with weighted sampling.

Results of image refinement using TCTS. Top: Original samples, Bottom: Refined images for 8-steps with TCTS.

Performance comparison of each method at different steps. In our experiments, we fixed classifier-free guidance to 5. When we use FAS method, it was possible to lower the FID score while maintaining text alignment.

Comparison of our model and the baseline in performance over generation time. In our experiments, we fixed classifier-free guidance to 5.

Poster

BibTeX

Reconstructed image using x̂₀^(t) each step during diffusion using the fixed and revocable method.
Only the revocable method is able to edit the tokens to generate a plane according to the text “A view of the end of an airplane in the sky over mountains”.

Comparing mask-free object editing samples with and without cross-attention map weighted sampling.
Starting from the image on the left, the result images every 20 steps of editing with 30% masking ratio. Top: Failure case without weighted sampling, Bottom: Results with weighted sampling.

Results of image refinement using TCTS.
Top: Original samples, Bottom: Refined images for 8-steps with TCTS.

Performance comparison of each method at different steps.
In our experiments, we fixed classifier-free guidance to 5. When we use FAS method, it was possible to lower the FID score while maintaining text alignment.

Comparison of our model and the baseline in performance over generation time.
In our experiments, we fixed classifier-free guidance to 5.