Updated: Making a NeRF movie with NVIDIA's Instant NGP
視聴回数 65,006 回
0
0
NOTE: there's a faster way to extract the images, shown at 1:20 - read these notes!
Video made is here: https://youtu.be/27AbBbl38Is
Watch it loop here: https://looptube.io/?videoId=27AbBbl38Is
The model was filmed at Chesterwood, in Stockbridge, MA, USA. A great place to visit! https://www.chesterwood.org/
The final NeRF created from this model is here: https://en.wikipedia.org/wiki/Equestrian_statue_of_Charles_Devens
Instant NGP software and build instructions: https://github.com/NVlabs/instant-ngp
Instructions on making your own NeRFs - extremely useful: https://github.com/NVlabs/instant-ngp/blob/master/docs/nerf_dataset_tips.md
Locking the exposure when you make a video can definitely help, e.g., https://camerajabber.com/iphone-video-tips-how-to-focus-improve-exposure/ - short version for iPhone: before recording, hold your finger on the focus point for a few seconds, then press the yellow "AE/AF lock" button that appears above.
Problems? For help, see the Discussion area on the site https://github.com/NVlabs/instant-ngp/discussions and search on all issues at https://github.com/NVlabs/instant-ngp/issues?q=is%3Aissue
Complementary blog post and video with more (somewhat dated) info: https://developer.nvidia.com/blog/getting-started-with-nvidia-instant-nerfs/
Need more help? Try the NeRF Discord server: https://discord.gg/2nEDC52hs8 or perhaps the reddit group https://www.reddit.com/r/NeuralRadianceFields/ (there is also https://www.reddit.com/r/NeRF3D, but that's for making 3D models, e.g., meshes, from NeRFs).
The commands I used in the conda command window ("ngp" is an environment I earlier created for instant-ngp; see it at the bottom of these notes):
# Set environment in Anaconda
conda activate ngp
# Pull images from movie; I've put movie directory "chesterwood" in the instant-ngp directory for simplicity. Change "fps 2" to whatever is needed to give you around 100 images.
cd C:\Users\(your path here)\Github\instant-ngp
cd chesterwood
python ..\scripts\colmap2nerf.py --video_in IMG_9471.MOV --video_fps 2 --run_colmap --overwrite
# NOTE! This line is a bit different than shown in the video, as advice on aabb_scale's use has changed. Also, I usually want to delete a few images after extracting them, so I don't do an exhaustive match at this point. In fact, I usually hit break (Control-C) when I see "Feature extraction" starting, as the images have all been extracted at that point.
# After you delete any blurry or useless frames, continue below to match cameras.
# Camera match given set of images. Do for any set of images. Run from directory containing your "images" directory.
python C:\Users\(your path here)\Github\instant-ngp\scripts\colmap2nerf.py --colmap_matcher exhaustive --run_colmap --aabb_scale 16 --overwrite
# For videos or closely related sets of shots, you can take out the "--colmap_matcher exhaustive" from the line above, since your images are in order. This saves a few minutes. You could also leave off "--aabb_scale 16" or put 64, the new default; the docs say it is worth playing with this number, see nerf_dataset_tips.md for how (short version: edit it in transforms.json). In my limited testing, I personally have not seen a difference.
# run interactive instant-ngp - run from the main directory "instant-ngp"
cd ..
instant-ngp chesterwood
(NOTE! If you don't seen the "Camera path" menu, drag the "instant-ngp" menu off of it - the two menus start in the same space, oddly enough. And, if you don't see the "Render video" button in this menu, resize the menu to view it. The menu starts very small. You can also close the "Path manipulation" submenu - click on the arrow to its left - to see the other Camera path options.)
------------
The anaconda environment setup commands I personally use to install the packages needed (just need to do these once for the project, other than the "activate"):
conda create -n ngp python=3.10
conda activate ngp
cd C:\Users\ehaines\Documents\_documents\Github\instant-ngp
pip install -r requirements.txt
The above creates a conda "environment" called ngp, makes it the active one, then installs the software you need. After creating an environment is created, you can open a new Anaconda window and "conda activate ngp" from anywhere to use it.
If you get stuck, you might want to check my previous video's notes: https://www.youtube.com/watch?v=8GbENSmdVeE
Video made is here: https://youtu.be/27AbBbl38Is
Watch it loop here: https://looptube.io/?videoId=27AbBbl38Is
The model was filmed at Chesterwood, in Stockbridge, MA, USA. A great place to visit! https://www.chesterwood.org/
The final NeRF created from this model is here: https://en.wikipedia.org/wiki/Equestrian_statue_of_Charles_Devens
Instant NGP software and build instructions: https://github.com/NVlabs/instant-ngp
Instructions on making your own NeRFs - extremely useful: https://github.com/NVlabs/instant-ngp/blob/master/docs/nerf_dataset_tips.md
Locking the exposure when you make a video can definitely help, e.g., https://camerajabber.com/iphone-video-tips-how-to-focus-improve-exposure/ - short version for iPhone: before recording, hold your finger on the focus point for a few seconds, then press the yellow "AE/AF lock" button that appears above.
Problems? For help, see the Discussion area on the site https://github.com/NVlabs/instant-ngp/discussions and search on all issues at https://github.com/NVlabs/instant-ngp/issues?q=is%3Aissue
Complementary blog post and video with more (somewhat dated) info: https://developer.nvidia.com/blog/getting-started-with-nvidia-instant-nerfs/
Need more help? Try the NeRF Discord server: https://discord.gg/2nEDC52hs8 or perhaps the reddit group https://www.reddit.com/r/NeuralRadianceFields/ (there is also https://www.reddit.com/r/NeRF3D, but that's for making 3D models, e.g., meshes, from NeRFs).
The commands I used in the conda command window ("ngp" is an environment I earlier created for instant-ngp; see it at the bottom of these notes):
# Set environment in Anaconda
conda activate ngp
# Pull images from movie; I've put movie directory "chesterwood" in the instant-ngp directory for simplicity. Change "fps 2" to whatever is needed to give you around 100 images.
cd C:\Users\(your path here)\Github\instant-ngp
cd chesterwood
python ..\scripts\colmap2nerf.py --video_in IMG_9471.MOV --video_fps 2 --run_colmap --overwrite
# NOTE! This line is a bit different than shown in the video, as advice on aabb_scale's use has changed. Also, I usually want to delete a few images after extracting them, so I don't do an exhaustive match at this point. In fact, I usually hit break (Control-C) when I see "Feature extraction" starting, as the images have all been extracted at that point.
# After you delete any blurry or useless frames, continue below to match cameras.
# Camera match given set of images. Do for any set of images. Run from directory containing your "images" directory.
python C:\Users\(your path here)\Github\instant-ngp\scripts\colmap2nerf.py --colmap_matcher exhaustive --run_colmap --aabb_scale 16 --overwrite
# For videos or closely related sets of shots, you can take out the "--colmap_matcher exhaustive" from the line above, since your images are in order. This saves a few minutes. You could also leave off "--aabb_scale 16" or put 64, the new default; the docs say it is worth playing with this number, see nerf_dataset_tips.md for how (short version: edit it in transforms.json). In my limited testing, I personally have not seen a difference.
# run interactive instant-ngp - run from the main directory "instant-ngp"
cd ..
instant-ngp chesterwood
(NOTE! If you don't seen the "Camera path" menu, drag the "instant-ngp" menu off of it - the two menus start in the same space, oddly enough. And, if you don't see the "Render video" button in this menu, resize the menu to view it. The menu starts very small. You can also close the "Path manipulation" submenu - click on the arrow to its left - to see the other Camera path options.)
------------
The anaconda environment setup commands I personally use to install the packages needed (just need to do these once for the project, other than the "activate"):
conda create -n ngp python=3.10
conda activate ngp
cd C:\Users\ehaines\Documents\_documents\Github\instant-ngp
pip install -r requirements.txt
The above creates a conda "environment" called ngp, makes it the active one, then installs the software you need. After creating an environment is created, you can open a new Anaconda window and "conda activate ngp" from anywhere to use it.
If you get stuck, you might want to check my previous video's notes: https://www.youtube.com/watch?v=8GbENSmdVeE
