*/}\n
\n Aided by text-to-image and text-to-video diffusion models, existing 4D content creation pipelines utilize score distillation sampling to optimize entire dynamic 3D scene. However, as these pipelines generate 4D content from text or image inputs, they incur significant time and effort in prompt engineering through trial and error.\nThis work introduces 4DGen, a novel, holistic framework for grounded 4D content generation that decomposes the 4D generation task into multiple stages. We identify static 3D assets and monocular video sequences as key components in constructing the 4D content.\nOur pipeline facilitates conditional 4D generation, enabling users to specify geometry (3D assets) and motion (monocular videos), thus offering superior control over content creation.\nFurthermore, we construct our 4D representation using dynamic 3D Gaussians, which permits efficient, high-resolution supervision through rendering during training, thereby facilitating high-quality 4D generation.\nAdditionally, we employ spatial-temporal pseudo labels on anchor frames, along with seamless consistency priors implemented through 3D-aware score distillation sampling and smoothness regularizations.\nCompared to existing baselines, our approach yields competitive results in faithfully reconstructing input signals and realistically inferring renderings from novel viewpoints and timesteps.\nMost importantly, our method supports grounded generation, offering users enhanced control, a feature difficult to achieve with previous methods.\n
\n\n TL;DR: We introduce grounded 4D content generation. We identify static 3D assets and monocular video sequences as key components in constructing the 4D content. Our pipeline facilitates conditional 4D generation, enabling users to specify geometry (3D assets) and motion (monocular videos), thus offering superior control over content creation.\n
\n {/*\n Previous work generates 4D content in one click. Our work introduces Grounded 4D Content Generation, which employs a video sequence and an optional 3D asset to specify the appearance and motion.\n
\n\n We conduct 4D generation grounded by a static 3D asset and a monocular video sequence. Our 4D scene is implemented by deforming a static set of 3D Gaussians. Besides reconstructing the input signals, we supervise our framework with spatial-temporal pseudo labels on anchor frames by a multi-view diffusion model. Moreover, we adopt seamless consistency priors implemented with score distillation sampling and unsupervised smoothness regularizations. \n
\n\n Our method is evaluated against per-frame baselines and concurrent method Consistent4D. We also provide image-to-4D results by using Stable Video Diffusion to generate a video from an image.\n
\n
\n If you want to cite our work, please kindly use:\n
\n \n {`@article{yin20234dgen,\n title={4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency},\n author={},\n journal={arXiv preprint},\n year={2023}\n}\\}`}\n \n \n \n