MrNeRF/LichtFeld-Studio

AI Summary: This issue aims to implement proper splat transformations within the CUDA-based inference rasterizer. Currently, splats rely on a placeholder view matrix transform, and the task is to replace this with a correct and performant transformation directly in the CUDA rasterization code.

AI Summary: Evaluate the performance and suitability of integrating the DeGauss.io 3D reconstruction tool into the existing 3D Gaussian Splatting project. Consider its performance, licensing (GPLv3), and potential benefits/drawbacks compared to the current implementation. A recommendation on whether to integrate it should be provided.

AI Summary: Implement the Histogram of Oriented Gradients (HOG) feature descriptor from the provided paper into an existing C++/CUDA 3D Gaussian Splatting renderer, supporting both Markov Chain Monte Carlo (MCMC) and Adaptive Density Control (ADC) methods. This involves integrating the HOG implementation into the existing codebase, ensuring compatibility with the current rendering pipeline, and potentially optimizing for performance on GPUs.

AI Summary: The task is to debug and fix a UI issue in a C++/CUDA application for 3D Gaussian Splatting. Dragging rendering setting windows causes unintended mouse events, and the fix requires preventing other events while dragging. This likely involves modifying event handling within the application's UI framework.

AI Summary: The task involves upgrading the C++ codebase of a high-performance 3D Gaussian Splatting renderer from C++17 to C++23. This will enable the use of modern C++ features like ranges, coroutines, and newer STL components for potential performance improvements and code simplification.

AI Summary: Port a sparsification technique from a specific branch of the gsplat repository (https://github.com/MrNeRF/gsplat/tree/sparsity) into the main Gaussian Splatting CUDA project. This technique, demonstrated in a Sutro Tower example (https://vincentwoo.com/3d/sutro_tower/), aims to reduce the number of splats after training, improving splat fidelity and potentially speeding up rendering. The goal is to integrate this functionality to allow users to apply the sparsification after training their own models.

AI Summary: Verify the functionality of batched image processing within the existing 3D Gaussian Splatting C++/CUDA implementation. This involves modifying the training loop to handle multiple images simultaneously (batch_size > 1) and ensuring correct functionality and performance improvements. Testing should confirm that batched processing improves geometry and doesn't introduce errors.

AI Summary: Integrate the fisheye undistortion fix from the linked gsplat pull request (#736) into the existing 3D Gaussian Splatting C++ and CUDA implementation. This likely involves reviewing the changes in the pull request, adapting them to the current codebase, testing the integration, and potentially resolving any conflicts or bugs that arise.

AI Summary: Implement a default densification strategy for 3D Gaussian Splatting in C++ and CUDA, porting functionality from a Python implementation in the gsplat repository. This involves understanding the original densification algorithm (adc) and translating it into a performant C++/CUDA version.

AI Summary: The task involves modifying the C++/CUDA code of a 3D Gaussian Splatting renderer to compute the backward pass after each loss calculation instead of accumulating losses before backpropagation. This change aims to improve memory efficiency and requires benchmarking and memory consumption verification. The modification should be made around line 220 of `trainer.cpp`.

AI Summary: The task requires enabling optional pose optimization in a C++ and CUDA implementation of 3D Gaussian Splatting. This involves connecting existing, but unused, gradient computations for camera extrinsics (view matrices) in the rasterizer's backward pass, likely requiring modifications to the C++ code and potentially the CUDA kernels to correctly propagate gradients. Reference implementation details are provided from a Python example.

AI Summary: Remove the unnecessary `abs()` function calls from the scale and opacity regularization terms within the `trainer.cpp` file of the 3D Gaussian Splatting C++/CUDA project. This involves modifying two lines of code to improve efficiency.

AI Summary: Implement a flexible cluster-based level-of-detail (LOD) system for real-time rendering of composed scenes within the existing 3D Gaussian Splatting C++ and CUDA framework. This involves designing and integrating a system that dynamically adjusts the detail level of rendered scenes based on factors like distance from the camera and available resources, potentially leveraging the existing Gaussian splatting structure for efficient LOD management. The goal is to improve rendering performance, particularly for complex scenes, while maintaining visual fidelity.

AI Summary: Integrate RmlUi into an existing C++/CUDA project (3D Gaussian Splatting renderer) to create a Blender-style user interface, alongside maintaining ImGui for debugging. This involves designing and implementing the UI elements (tool panels, property editors, etc.) using RmlUi, and integrating it seamlessly with the existing rendering pipeline.

AI Summary: Implement Visual SLAM (Simultaneous Localization and Mapping) to replace the Colmap dependency in the 3D Gaussian Splatting project. This involves creating a system to estimate camera poses and build a 3D map without relying on pre-existing structure from motion data. The initial goal is to generate pose priors for global bundle adjustment.

AI Summary: Implement a visualization tool for the 3D Gaussian Splatting project to display COLMAP camera positions and dataset images before training. This involves creating a 3D viewer showing camera frustums, integrating an image browser for dataset images, and adding GUI controls for navigation and validation. The goal is to improve scene understanding and allow users to verify dataset integrity before starting the computationally expensive training process.

AI Summary: Implement key editing features from the Splatshop project (brush selection, deletion, basic transformations, undo/redo) into the existing 3D Gaussian Splatting C++/CUDA renderer. Future phases involve adding a layer system and advanced features like painting and asset management.

AI Summary: Implement a viewer-only mode for the 3D Gaussian Splatting project, allowing users to load and visualize pre-trained models without the training pipeline. This involves loading splat data from a .ply file, setting up a camera (either from COLMAP data or a default), integrating the viewer with existing rendering capabilities, and ensuring efficient performance.

AI Summary: Refactor the C++/CUDA 3D Gaussian Splatting trainer to run in a separate thread when a GUI is enabled, allowing the GUI to be the main entry point. Implement GUI controls (stop/resume, save, exit) to manage the training process, ensuring thread-safe communication and graceful handling of training state.

AI Summary: Implement enhanced camera controls for a 3D Gaussian Splatting viewer, including orbit and first-person modes, and a GUI control widget to switch between them. This involves modifying existing viewer code to add new camera movement logic and a user interface element.

AI Summary: Integrate the existing 3DGUT kernels and the implemented MCMC strategy into the project's trainer to enable 3DGUT functionality. This involves connecting the kernels and strategy to the existing training pipeline.