GSOC 2024: 3D Gaussian Splat for point cloud generation with MPM simulations

[kks32]

3D Gaussian Splat for Point Cloud Generation in Physics-Driven MPM Simulations

Abstract

This project aims to bridge the gap between real-world phenomena and computational physics simulations by converting video data of dynamic events, such as granular column collapses or dam breaks, into point clouds in real time using Gaussian splatting techniques. The generated point clouds will then be utilized in Material Point Method (MPM) simulations to model and understand these complex scenarios' underlying physics accurately. This approach will integrate with the CB-Geo MPM project, focusing on rendering natural hazards like landslides with an in-situ visualization interface. The project holds high priority due to its potential to enhance predictive modeling and digital twin reconstructions of natural disasters, contributing significantly to computational geotechnics and disaster management.

Intensity	Priority	Involves	Mentors
Moderate Duration (350 hours)	High	Integrating Gaussian splatting with existing rendering systems in CB-Geo MPM to generate 3D point clouds of natural hazards such as landslides.	Krishna Kumar and Justin Bonus and Pedro Arduino

Benefits of working on this project

Students engaging in this project will enhance their skills in:

• Neural rendering frameworks and text-to-mesh generation
• Application of machine learning techniques in physics-based models.
• Working with advanced rendering and simulation tools in natural hazard engineering.

Motivation

Current techniques for modeling natural hazards and other dynamic events often rely on static datasets that do not capture the full scope of real-world variability. This project aims to create more dynamic and accurate digital twins of natural hazards by leveraging real-time video data and converting it into point clouds for MPM simulations. This method will enable the prediction and analysis of complex phenomena with unprecedented detail and fidelity.

Technical Details

• Gaussian Splatting: A technique used for converting video data into point clouds by projecting video pixels into 3D space, using a Gaussian function to manage the distribution of points.
• Material Point Method (MPM) Simulations: A physics-based computational method used for capturing the behavior of materials under various conditions, ideal for simulating natural disasters and hazards.
• The project will integrate these technologies to convert real-world video footage into detailed simulations, aiming to minimize the loss between observed and simulated data to identify accurate physical models.

Benefits to Project/Community

This project will significantly contribute to the fields of robotics, natural hazard prediction, and digital twin reconstruction by:

• Offering a novel method for real-time data conversion and simulation.
• Enhancing the accuracy and applicability of predictive models for natural disasters.
• Providing a robust framework for integrating video data with physical simulations.

Helpful Experience

Candidates interested in this project should ideally:

• Have a working knowledge of C++ and Python.
• Be experienced or keen on rendering or visualization tools like OpenUSD, Houdini, or Blender.

First Steps

Prospective participants should:

• Review the foundational paper describing the Gaussian splatting process and its application in MPM simulations.
• Acquaint themselves with existing rendering engines like Blender, the OpenUSD framework, and CB-Geo MPM's current simulation and visualization infrastructure.
• Gain familiarity with conducting petascale simulations on high-performance computing resources like the Texas Advanced Computing Center (TACC) supercomputer.

[Ishneet0710]

Hi, I am Ishneet a high school graduate from Singapore awaiting university admission. I have extensive experience in AI and robotics, having published multiple papers in journals like nature. I am very keen to work on this project! Would I be able to hop onto a call or discuss the project details further!

https://www.linkedin.com/in/ishneet-singh-a51697210

[Raghav323]

Hi,
I am highly interested in working on this project since it involves 3d reconstruction and differentiable rendering, especially the method of Gaussian splatting which is the current state-of-the-art in this domain. I have previously worked with various types of NeRFs (vanilla,instant-ngp,shadow-nerf). I am familiar with the theoretical knowledge of computer graphics, including rendering and rasterization pipelines.

I have gone through and understood the Gaussian splatting paper, read through the OpenUSD doc for Python API, and read through the theory behind mpm via the theoretical manual of CB-Geo MPM and other sources. I know the basics of Blender and have created a TACC account. Are there any resources for working with TACC? Also are there any other tasks that I could complete?

[kks32]

Hi @Raghav323 and @Ishneet0710

Thank you for your interests! Would you mind generating a point cloud using NeRF with this video: https://utexas.box.com/s/brba43gjw5dpav7xx7bamuv83i2lkaag

This could be a good preliminary result to show in your app.

[Raghav323]

Hi @kks32, I have generated a point cloud using NeRF (with multi-resolution hash encoding) for the given video. First, I extracted 4 frames per second from the video to create an image dataset of 248 images representing the scene, then I used colmap to extract the camera poses for images and then used instant-ngp (nerf with hash encoding) to train a NeRF within 3-4 minutes. I have cropped the final NeRF output to only the table and the granular soil. Most of the scripts I used were directly available in the instant-ngp repository. You can find the polygon file for the obtained point cloud here. The resolution is limited because of my GPU RAM. I can also try obtaining the point cloud via Mip-NeRF or Vanilla NeRF. I also have my own implementation of vanilla NeRF in Python which I could use to get the point cloud. Please let me know of any further improvements or tasks.
Thanks

[Raghav323]

Thanks @kks32 and @JustinBonus. I have started writing a proposal. I was also thinking about point cloud segmentation for the automatic creation of particle entity sets for the user. I was able to fit a plane on the point cloud using the ransac algorithm which almost segments the entire table. Perhaps if we could fit something like a cone, we could automate the process of calculating the angle of repose.

Possibly there is a way to segment everything into classes using semantic segmentation of point clouds through DL methods or just simply color segmentation for our specific case.

[JustinBonus]

Segmentation into geometry primitives, materials, and/or general classes (e.g. "car", "sand") would be great functionalities to look into for real-world cases. Namely, natural disaster reconnaissance after tsunamis, hurricanes, and earthquakes require researchers to quantify piles of debris (trees, parts of houses, cars, etc.) so that they can improve hazard mitigation guidelines. 3DGS / NeRF + classification could have benefits for the rapid execution of this sort of work. Good work!

[Raghav323]

Hi, I would like to know if we can chat sometime whenever any of you find time since I had doubts regarding the specific implementation details of the project. This would also help me improve my proposal and add more details. Thanks

[kks32]

Could you email me [email protected] and we can chat tomorrow morning (Texas time). Will be evening for you?

[Raghav323]

Hi @kks32, I have mailed you regarding the meeting. Thanks.

[AagashramNeelakandan]

Hi,
I am Aagashram Neelakandan. I am currently doing my Masters in Computational Material Science in Freiberg, Germany, where we deal with the FEM and Atomistic simulations. I saw this topic in Google Summer of Code. For the past 5 months, I have been working (and playing) on the topics of 3D Gaussian Splatting and testing many open-source versions of them. I created splats in various conditions and tested their fidelity and limitations. I also have certain expertise in 3D Art (using Blender, Cinema4D) and Graphics and always wanted to work in the field of Material Point Method for simulation of real-life phenomena like fracture, tearing, Fluid-Solid interaction, etc. I am looking forward to getting input from you.

Linkedin : https://www.linkedin.com/in/aagashram-neelakandan
Behance : https://www.behance.net/aagashramn

[kks32]

Thanks @AagashramNeelakandan Your work is very relevant to what we do. Please reach out, contact info https://www.caee.utexas.edu/people/faculty/faculty-directory/krishna-kumar

[AagashramNeelakandan]

Thanks @kks32 for your quick reply. I sent mail with my info to you.