Overview

The goal of this workshop is to showcase the latest developments of generative AI for creating, editing, restoring, and compressing rich media, such as images, videos, neural radiance fields, and 3D scene properties. Generative AI models, such as GAN and diffusion models have enabled remarkable achievements in rich media from both academia research and industrial applications. For instance, cloud-based video gaming is a booming industry with an expected global market value of over $12 billion by 2025. Generative AI transforms the gaming industry by enabling anyone to build and design games without professional artistic and technical skills, empowering immeasurable market growth.

With the success of the 1st RichMediaGAI Workshop@WACV 2024, we expand the 2nd RichMediaGAI workshop@ACCV2024 by organizing competitions with industry-level data, soliciting paper submissions, and continuing to invite top-tier speakers from both industry and academia to fuse the synergy.

Important Dates + Author Guidelines

Author Guidelines: Formatting, Page Limits, Author Kits, and Submission Policies follow the ACCV 2024 Author Guidelines

Challenges Data Available at:	August 6, 2024, 11:59 PM PST
Regular Paper Submission Deadline:	Extented to September 27, 2024, 11:59 PM PST
Challenges Results and Reports Submission Deadline:	Extended to September 27, 2024, 11:59 PM PST
Submission Site:	CMT Submission Site
Paper Review Back and Decision Notification:	October 4, 2024, 11:59 PM PST
Challenges Results and Decision Notification:	October 4, 2024, 11:59 PM PST
Camera-Ready Deadline:	October 10, 2024, 11:59 PM PST

1. Regular Paper Submissions

Papers addressing topics related to image/video restoration, compression, enhancement, and manipulation, using generative AI technologies are welcome to submit. The topics include but are not limited to:

• Restoration and enhancement of rich media with generative AI
• Editing and manipulation of rich media with generative AI
• Compression and codec design with generative AI
• Modeling and rendering rich media with generative AI
• Neural radiance fields with generative AI
• Rich media creation and editing with large language models
• Acceleration of generative AI models on edge devices

Author Guidelines: Formatting, Page Limits, Author Kits, and Submission Policies follow the ACCV 2024 Author Guidelines All papers must be uploaded to the submission site by the deadline. There is no rebuttal for this call. Reviews and paper decisions will be sent back to the authors on the date specified above.

2. Challenges

Call for Participation

Cloud gaming poses tremendous challenges for compression and transmission. To avoid delay and bandwidth overload, high-quality frames need to be heavily compressed with very low latency. Traditional codecs like H.264/H.265/H.266 or recent neural video coding targeting natural videos generally do not perform well.

Generative AI technologies, e.g., super-resolution, image synthesis and rendering, can largely alleviate the transmission issues. Server-side computation and transmission can be reduced by leveraging the computation power of client de- vices. For example, the server can render low-resolution (LR) frames to transmit, and high-resolution (HR) frames can be computed on client side. In multiview gaming, the server can render part of views to transmit, and the remaining views can be computed by client devices. Nvidia's Deep Learning Super Sampling (DLSS) has commercialized this idea, and one key factor of its success is the large-scale ground-truth LR-HR or multiview gaming data used for training.

In comparison, the research community uses pseudo training data for many restoration tasks. For example, for super-resolution, the LR data is generated from the HR data by downsampling and adding degradation like noises and blurs. Such pseudo data do not match real gaming data. True LR gaming frames are high-quality, sharp and clear without noises or blurs. There are unnatural visual effects and object movements, but with limited motion blur, different from captured natural videos. We need ground-truth gaming data for effective training.

In this competition, a large computer-synthesized ground-truth dataset is provided, targeting two different applications:

• Track 1: Super Resolution in Deferred Rendering

This track aims to restore HR images from LR images along with additional GBuffers during the deferred rendering stage (i.e., segmentation map, depth map), supporting the gaming solution of transferring LR images with assistive information using reduced bits and restoring HR images on client side.

The dataset has 320 LR-HR paired sequences at 1440p and 720p. Each sequence has 60 frames, totalling 19200 LR-HR paired frames. The sequences are rendered by the open source CARLA simulator with the Unreal Engine. The paired sequences capture 3D scenes from 8 different towns (20 static scenes and 20 dynamic scenes for each town). The corresponding paired segmentation maps and depth maps synchronize with the RGB images are also provided. Data collected from 7 towns form the training set, and data from the remaining 1 town form the test set.

Given the LR RGB images (720p) and additional GBuffers as input, the task is to develop algorithms to recover the HR RGB images (1440p).

Algorithms will be evaluated based on 4 objective quality metrics: PSNR and SSIM to measure pixel-level distortion, LPIPS and FID to measure perceptual quality. In detail, assume there are N methods, they will be ranked according to each metric, and a ranking score between [1,2N-1] will be given to each method (1st place 2N-1, 2nd place 2N-3, ..., last place 1). The average ranking score of all 4 metrics will be used as the overall score to rank the N methods. The amount of additional GBuffers used will also be considered (the more GBuffers used, the more bits consumed). For example, if two methods have similar overall score, the less bits consumed the better.

huggingface Download includes the following:

• Mini training set: For each paired video sequence, 5 paired frames are randomly selected to form the mini dataset.
• Full training set
• Code scripts to read data and compute evaluation metrics

Challenge result submission: participants must submit the recovered HR RGB images for the test set to be evaluated. The download link of the results should be provided by the deadline.

• Track 2: Multiview Video Frame Synthesis

This track aims to synthesize intermediate frames from a sparse set of input frames in multiview videos, along with camera intrinsic and extrinsic parameters and additional segmentation maps and depth maps, supporting the multiview gaming solution of transferring part of multiview frames with assistive information using reduced bits and generating the remaining frames on client side.

The dataset contains 160 sets of sequences rendered by CARLA simulator. Each set of sequences consists of static 3D scenes captured by six cameras mounted on the top of a car moving within one of the 8 towns. Each town has 20 sets of sequences and each sequence has 60 frames, totaling 57600 frames. The corresponding segmentation maps and depth maps synchronize with the RGB images are also provided.

For each set of sequences, a subset of multiview frames will be randomly selected as inputs, and the task is to synthesis the remaining frames.

Algorithms will be evaluated based on 4 objective metrics: PSNR and SSIM to measure pixel-level distortion, LPIPS and FID to measure perceptual quality. In detail, assume there are N methods, they will be ranked according to each metric, and a ranking score between [1,2N-1] will be given to each method (1st place 2N-1, 2nd place 2N-3, ..., last place 1). The average ranking score of all 4 metrics will be used as the overall score to rank the N methods. The amount of additional GBuffers used will also be considered (the more GBuffers used, the more bits consumed). For example, if two methods have similar overall score, the less bits consumed the better.

huggingface Download includes the following:

• Mini training set: For each paired video sequence, 5 paired frames are randomly selected to form the mini dataset.
• Full training set
• Code scripts to read data and compute evaluation metrics

Challenge result submission: participants must submit the synthesized RGB frames for the test set to be evaluated. The download link of the results should be provided by the deadline.

The winners will be announced at the RichMediaGAI workshop, and the top 3 non-corporate winners of each track will be rewarded with 1st $2000, 2nd $1000, 3rd $500. The winners are invited to submit a paper to the RichMediaGAI workshop through the paper submission system. For the paper to be accepted, each paper must be a self-contained description of the method, and be detailed enough to reproduce the results. The paper submission must follow the ACCV 2024 Author Guidelines .