[Paper Review] Emojich -- zero-shot emoji generation using Russian language: a technical report

Alex Shonenkov, Daria Bakshandaeva+2 more|arXiv (Cornell University)|Dec 4, 2021

Hand Gesture Recognition Systems4 citations

TL;DR

This paper presents Emojich, a zero-shot text-to-image model fine-tuned from ruDALL-E Malevich (XL) to generate stylized Russian-language emoji from textual descriptions. Using 8-bit Adam optimization and a weighted cross-entropy loss on codebook vectors, the model achieves high emoji stylization after 40 epochs of fine-tuning, with results available via a public Telegram bot for custom sticker generation.

ABSTRACT

This technical report presents a text-to-image neural network "Emojich" that generates emojis using captions in Russian language as a condition. We aim to keep the generalization ability of a pretrained big model ruDALL-E Malevich (XL) 1.3B parameters at the fine-tuning stage, while giving special style to the images generated. Here are presented some engineering methods, code realization, all hyper-parameters for reproducing results and a Telegram bot where everyone can create their own customized sets of stickers. Also, some newly generated emojis obtained by "Emojich" model are demonstrated.

Motivation & Objective

• To develop a zero-shot text-to-image model that generates stylized emojis from Russian-language captions without requiring new emoji proposals to Unicode.
• To preserve the world knowledge of the pretrained ruDALL-E Malevich (XL) model while adapting its output style to match emoji aesthetics.
• To enable users to generate personalized emoji stickers via a public Telegram bot using fine-tuned generation and automated segmentation.
• To address overfitting on limited caption variations through a high-weighted cross-entropy loss on image codebook vectors.
• To develop a robust, scalable pipeline for converting generated emoji images into RGBA format with transparent backgrounds using U-Net and contour-based segmentation.

Proposed method

• Fine-tuned the 1.3B-parameter ruDALL-E Malevich (XL) model by freezing self-attention and feedforward layers to prevent catastrophic forgetting.
• Applied a weighted cross-entropy loss with coefficient 10³ on image codebook vectors to emphasize style adaptation while preserving semantic content.
• Used 8-bit Adam optimizer with fp16 precision and a one-cycle learning rate scheduler (initial 4e-7, peak 1e-5, final 2e-8) over 40 epochs with batch size 2.
• Employed nucleus sampling (top-p = 0.995) and top-k = 2048 with temperature = 1.0 for autoregressive decoding of latent vectors in the VQ-VAE decoder.
• Enhanced image resolution from 256×256 to 512×512 using an inverse discrete wavelet transform (IDWT) module adapted from MobileStyleGAN.
• Trained a U-Net with EfficientNetB7 encoder for binary semantic segmentation of generated emojis, using pseudo-labeling and manual correction to expand the training set to 246,089 images.

Experimental results

Research questions

• RQ1 Can a large-scale multimodal pretrained model like ruDALL-E Malevich (XL) be effectively fine-tuned to generate stylized, culturally appropriate emojis in Russian without catastrophic forgetting?
• RQ2 How can the model maintain high-quality image generation while adapting to the distinct visual style of emojis, especially given limited and ambiguous caption data?
• RQ3 What training and optimization strategies are most effective for fine-tuning a large vision-language model on a low-resource, highly stylized dataset like Russian emojis?
• RQ4 How can automated and reliable background removal be achieved for generated emoji images to enable direct use in messaging platforms?
• RQ5 To what extent does the model generalize to novel, out-of-distribution prompts while preserving both identity and emoji aesthetics?

Key findings

• After 40 epochs of fine-tuning on 2,749 Russian-captioned emoji images, the model successfully transitioned from generating photorealistic images to stylized emoji-like outputs.
• The model began producing recognizable emoji-style images after approximately 12,000 iterations, with consistent stylization observed by 56,000 iterations.
• By 68,000 iterations, the model started losing semantic fidelity, generating uniform, impersonal images (e.g., 'Einstein' transformed into an unknown man with a moustache), indicating overfitting or mode collapse.
• The U-Net-based segmentation pipeline achieved high reliability, with confidence thresholds (≥0.99) enabling automated processing of 99% of generated images, while remaining cases were resolved via manual contour-based methods.
• The final model, when prompted with seed=42 and 16 samples per prompt, generated high-quality emoji images suitable for direct use as stickers, with further quality gains possible through 512-sample sampling and manual selection.
• A public Telegram bot was successfully deployed, enabling users to generate and export custom emoji sticker sets using the fine-tuned Emojich model and segmentation pipeline.