Notes on How A.I. Will Change the 3D Industry
My notes from Andrew Price's talk at Blender Conference 2018 on how A.I. will change the 3D industry.
Overview
I went back and watched the talk Andrew Price (Blender Guru) gave at Blender Conference 2018 on how A.I. will change the 3D industry. This time, I decided to take some notes.
Question to consider: What is not going to change in the next 10 years?
Assets are unreasonably expensive
Creating a building Asset:
- Modeling: 12 hours
- Texturing: 10 hours
- First Pass total: 22 hours
- Revisions: x2-3
Problem: static workflows
Solution: procedural workflows
-
Practical Procedural Generation for Everyone (GDC 2017)
-
Procedural Modeling Example: Procedural Lake Village by Anastasia Opera
-
Procedural Texturing Example: Poliigon Substance Designer
- Procedural Texturing Example: Substance Painter
- Bake → Smart Materials → Smart Masks
-
Procedural Level Design: Houdini
Procedural World Generation of Ubisoft’s Far Cry 5
- Create an ecosystem
- Set rules to define where certain trees and plants would live
- Other factors
- Occlusion
- Flow
- Slope
- Curvature
- Illumination
- Altitude
- Latitude
- Longitude
- Wind
- Tools for customization like roads and buildings
- Create an ecosystem
Machine Creep
Traditional Software: input (e.g. photo) → action (filter) → output
Machine Learning: input (e.g. photo) → assess → appropriate action → compare → is it good? → output
- Needs huge datasets and fast hardware
Machine Learning Use Cases
- De-noising
- Super resolution
- Motion Capture
- Animation
- Mode-adaptive Neural Networks for motion control
Machine-Assisted Creativity
-
Problem: experimentation takes up 50%-70% of time
-
BicycleGAN
- Model input: outline of an object in an image and the ground truth image
- Model output: generate variations of image
- GitHub Repository
- Toward Multi-modal Image-to-image translation
- DivCo: Diverse Conditional Image Synthesis via Contrastive Generative Adversarial Network
- Sketch to Image
- StyleGAN2
- Progressive Growing of GANS (PGAN)
- Text to Image
- StackGAN V2 (2017)
- text2image (April 2021)
- TediGAN (March 2021)
- DF-GAN
- Style Transfer
- A Style-Aware Content Loss for Real-time HD Style Transfer
- Adaptive Style Transfer (TensorFlow 2018)
- color-transform (PyTorch 2019)
- A Style-Aware Content Loss for Real-time HD Style Transfer
Not in Presentation
Related Works
-
GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds
- NeRS: Neural Reflectance Surfaces for Sparse-View 3D Reconstruction in the Wild (October 2021)
- PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization (2020)
-
3DStyleNet: Creating 3D Shapes with Geometric and Texture Style Variations
https://nv-tlabs.github.io/3DStyleNet/assets/animal-new.mp4
3DStyleNet: Creating 3D Shapes with Geometric and Texture Style Variations
-
DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction
- GitHub Repository (Tensorflow)
- GitHub Repository (PyTorch)
- Learning Linear Transformations for Fast Arbitrary Style Transfer
- Neural Cages for Detail-Preserving 3D Deformations
- Taming Transformers for High-Resolution Image Synthesis
[Overview] Taming Transformers for High-Resolution Image Synthesis
- Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes
- GitHub Repository (TensorFlow)
- GitHub Repository (PyTorch)
- Network-to-Network Translation with Conditional Invertible Neural Networks
- Artistic Style Transfer with Internal-external Learning and Contrastive Learning
- GitHub Repository
- Based on: SANET
-
Synthetic Silviculture: Multi-scale Modeling of Plant Ecosystems
Makowski.etal-2019-Synthetic-Silviculture.pdf
Makowski.etal-2019-Synthetic-SilvicultureSup.pdf
Synthetic Silviculture: Multi-scale Modeling of Plant Ecosystems
- DualConvMesh-Net: Joint Geodesic and Euclidean Convolutions on 3D Meshes
Other Applications
How A.I will affect the art industry
Takeaway: AI will handle more and more of the tedious manual work that humans don’t like doing (or is extremely time consuming)
This will reduce the cost of production, enabling more productions overall
-
Rotoscoping
What is rotoscoping animation and how to do it
- segmentaion
-
Retopology
What is Retopology? (A Complete Intro Guide For Beginners)
- Appearance-Driven Automatic 3D Model Simplification (2021)
- Human provides general outline/concept and a model fills in technical details
-
NVIDIA GauGAN2
-
NVIDIA Canvas
-
- Facial animations
References: