This is the official implementation for LERF.

LERF follows the integration guidelines described here for custom methods within Nerfstudio. Update to nerfstudio==1.0.3.

Follow these instructions up to and including "tinycudann" to install dependencies and create an environment

git clone https://github.com/jayhsu0627/Event3DGS

Navigate to this folder and run python -m pip install -e .

pip install git+https://github.com/nerfstudio-project/[email protected]

Run ns-train -h: you should see a list of "subcommands" with lerf, lerf-big, and lerf-lite included among them.

Now that ESplat is installed you can play with it!

• Launch training with ns-train esplatfacto --data <data_folder>. This specifies a data folder to use. For more details, see Nerfstudio documentation.

ns-train esplatfacto --data C:\Users\sjxu\3_Event_3DGS\Data\nerfstudio\sewing

ns-train esplatfacto-big --data C:\Users\sjxu\3_Event_3DGS\Data\nerfstudio\sewing --pipeline.model.use_scale_regularization True --pipeline.model.cull_alpha_thresh=0.005 --pipeline.model.continue_cull_post_densification=False

• Connect to the viewer by forwarding the viewer port (we use VSCode to do this), and click the link to viewer.nerf.studio provided in the output of the train script. Use the viewer running locally at: http:https://localhost:7007

TODO: edit class ExportGaussianSplat(Exporter) in exporter.py

• Output *.ply ns-export gaussian-splat --load-config outputs\plane\esplatfacto\2024-04-22_201709\config.yml --output-dir exports/ply

  File "C:\Users\sjxu\AppData\Local\miniconda3\envs\event3dgs\lib\site-packages\nerfstudio\scripts\exporter.py", line 614, in entrypoint
    tyro.cli(Commands).main()
  File "C:\Users\sjxu\AppData\Local\miniconda3\envs\event3dgs\lib\site-packages\nerfstudio\scripts\exporter.py", line 536, in main
    assert isinstance(pipeline.model, SplatfactoModel)
AssertionError

1. Split RGB channels independent 3dgs
2. I'm now a pure 3dgs with no 159 assumption
3. Add t0 and t estimation?

ray_samplers = load_event_data_split(args.datadir, args.scene, camera_mgr=camera_mgr, split=args.train_split,
                                         skip=args.trainskip, max_winsize=args.winsize,
                                         use_ray_jitter=args.use_ray_jitter, is_colored=args.is_colored,
                                         polarity_offset=args.polarity_offset, cycle=args.is_cycled,
                                         is_rgb_only=args.is_rgb_only, randomize_winlen=args.use_random_window_len,
                                         win_constant_count=args.use_window_constant_count)

To see how "load_event_data_split" determine

prev_file = img_files[(i-winsize+len(img_files))%len(img_files)]
curr_file = img_files[i]

If you find this useful, please cite the paper!

@inproceedings{lerf2023,
 author = {Kerr, Justin and Kim, Chung Min and Goldberg, Ken and Kanazawa, Angjoo and Tancik, Matthew},
 title = {LERF: Language Embedded Radiance Fields},
 booktitle = {International Conference on Computer Vision (ICCV)},
 year = {2023},
}