ImmerseSketch: Transforming Creative Prompts into Vivid 3D Environments in VR

Posted: Jul 28th 2024
Abstract

We propose ImmerseSketch, a framework designed to transform creative prompts into vivid and detailed 3D content within a Virtual Reality (VR) environment. Our aim is to inspire creative ideation and to provide creators with 3D reference content for painting. We focus on generating initial panoramic images through diffusion models and then converting these images into rich 3D environments with the aid of depth estimation

ImmerseSketch: Transforming Creative Prompts into Vivid 3D Environments in VR

DL ACM

Brief Summary

ImmerseSketch transforms creative prompts into detailed 3D content within VR environments, designed to enhance artistic ideation and provide creators with immersive 3D references for painting. Unlike traditional 2D photo references that lack depth information, ImmerseSketch leverages VR to deliver spatial understanding and three-dimensional structure through depth, physicality, and presence.

The system operates through three core modules:

  • Diffusion Panorama Generation: Processes text or voice prompts using a Stable-Diffusion-based framework (Diffusion360) to generate multiple 360-degree panoramic images.
  • Environment Mesh Conversion: Converts selected 2D panoramas into 3D environment meshes using DepthAnything for zero-shot depth estimation. Each pixel is projected outward from a central point, creating 3D meshes based on estimated depth values.
  • Segmentation and Object Generation: Enables independent object observation by employing One-2-3-45 to transform user-selected panorama objects into standalone 3D models. These generated objects can be imported into the virtual environment for rotation and detailed examination.

Citation

@inproceedings{10.1145/3641234.3671078,
author = {Lan, Alfred and Tsai, Tai-Chen and Huang, Chih-Chuan and Ching, Pu and Pan, Tse-Yu and Hu, Min-Chun},
title = {ImmerseSketch: Transforming Creative Prompts into Vivid 3D Environments in VR},
year = {2024},
isbn = {9798400705168},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3641234.3671078},
doi = {10.1145/3641234.3671078},
booktitle = {ACM SIGGRAPH 2024 Posters},
articleno = {56},
numpages = {2},
keywords = {Generative Environment, VR Creation, Virtual Reality},
location = {Denver, CO, USA},
series = {SIGGRAPH '24}
}