Making a NeRF movie with NVIDIA's Instant NGP

NOTE! I made a newer video for the updated instant-ngp software: https://www.youtube.com/watch?v=3TWxO1PftMc

These notes include the command lines you'll need.

Search on "Dyanna Skunk Somerville" to find more information about the robot sculpture captured. See https://rafiproperties.com/news-and-updates/life-size-sculpture-of-powerful-female-icon-dyanna-unveiled-in-somerville/ as a start.

Instant NGP software and build instructions: https://github.com/NVlabs/instant-ngp
*Problems?* For help, see the Discussion area on the site https://github.com/NVlabs/instant-ngp/discussions and search on _all_ issues at https://github.com/NVlabs/instant-ngp/issues?q=is%3Aissue

Complementary blog post and video with more (slightly dated) info: https://developer.nvidia.com/blog/getting-started-with-nvidia-instant-nerfs

Need more help? Try the NeRF Discord server: https://discord.gg/2nEDC52hs8

The commands I used in the conda command window ("ngp" is an environment I earlier created for instant-ngp; see below):

cd C:\Users\ehaines\Documents\_documents\Github\instant-ngp
conda activate ngp
cd data/nerf/robot

(COLMAP must be installed and add COLMAP-3.7-windows-cuda to the $PATH.)

(OPTIONAL: if you took a video instead of a set of images, do this next command to pull images out of your video and put them in the "images" subdirectory:)

python C:\Users\ehaines\Documents\_documents\Github\instant-ngp\scripts\colmap2nerf.py --video_in YOUR_VIDEO.MOV --video_fps 2 --run_colmap --aabb_scale 16

(Perform feature matching among your set of images:)

python C:\Users\ehaines\Documents\_documents\Github\instant-ngp\scripts\colmap2nerf.py --colmap_matcher exhaustive --run_colmap --aabb_scale 16

(Sometimes COLMAP will fail, in which case you need to take more photos.)

(Generate NeRF and set camera path:)

cd C:\Users\ehaines\Documents\_documents\Github\instant-ngp
.\build\testbed --scene data/nerf/robot

(FFMPEG must be installed first before generating a video - test first!)

(Save camera path and NeRF solution:)

python scripts/run.py --scene data/nerf/robot --load_snapshot data/nerf/robot/base.msgpack --video_n_seconds 8 --video_fps 60 --width 1920 --height 1080 --mode nerf --video_camera_path data/nerf/robot/base_cam.json

(Generates video.mp4.)

*Initial installation notes:*
You need to install COLMAP (and Anaconda, if you want to control Python that way). Search the internet; you'll easily find them. You may need to add "...\COLMAP-3.7-windows-cuda" to your PATH to get colmap2nerf.py to run.

The anaconda environment setup commands I personally use to install the packages needed (just need to do these once for the project, other than the "activate"):

conda create -n ngp python=3.10
conda activate ngp
cd C:\Users\ehaines\Documents\_documents\Github\instant-ngp
pip install -r requirements.txt
conda install -c conda-forge ffmpeg

The above creates a conda "environment" called ngp, makes it the active one, then installs the software you need. When an environment is active, you can "conda install" from anywhere.

Also, if you run into "ModuleNotFoundError: No module named 'pyngp'" when running run.py, first run in your conda window:

python -V

to see the version number, e.g., "Python 3.10.4". Then, change CMakeLists.txt in your topmost directory instant-ngp from:

find_package(Python 3.7 COMPONENTS Interpreter Development)

to

find_package(Python 3.10 EXACT COMPONENTS Interpreter Development)

- note you don't need to put the ".4" in "3.10.4". Delete the "build" directory and remake everything. That should solve it.

One more detail: you won't see the DLSS option if you don't have an RTX card.