I did work on this as part of my thesis quite a few years back at the university. One other optimization would be to process the points in parallel.

Regarding the coloring of each 3d point, it might be feasible to not use one camera image, but a weighted sum of all camera images that can see the same point in the scene. Each pixel color is then weighted with the scalar product of the points normal and the viewing direction of the camera. This would also regard for noise and specular reflections (which can mess up the original color).

Yes, I am working on using numpy to do the projection using matrices so we dont have to loop over each point and project it individually. That should be a big boost.

The way I handle the different camera images is to simply see which one provides a lower depth and use - with the idea that if the camera is closer, it would provide better information. But what you are suggesting is pretty interestint. I'm going to try that as well.

Lidars are pretty powerful, but one big disadvantage of using point clouds for perception is that they are not colored. This makes identifying objects more difficult compared to camera images. However, by combining camera images with lidar data, we can enhance the point cloud by assigning colors to the points based on the corresponding camera image pixels. This makes visualizing and processing the point cloud much easier.

Shameless plug, but I work on a multispectral lidar that does produce “colored” point clouds in the SWIR [0].

It is pretty cool, we use it for detection of humidity degree or for species discrimination (e.g. plants, minerals, chemicals…).

[0]: https://www.iridesense.tech/

I'm not sure why you've just restated the first paragraph of the article.

Likely for the engagement... any bets as to if they are a bot or not?

Their (human) reply makes me suspect it's a human using an llm without realizing how obvious it is that they're using one.

Edit: Just realized they're probably the author too of the post

Makes sense given English is likely not a strong primary language for them.

LLM's are getting heavy use from the ESL (and even English as a third or fourth language) crowd.

> one big disadvantage of using point clouds for perception is that they are not colored

That depends entirely on the capture device.

True. But my reference point is Lidars which mostly produce non-colored points.

Lidars are expensive, if you want spare point clouds, that are not quite real time you might want to check out colmap https://colmap.github.io/

Thanks, been trying to look into AI tools to generate point clouds from photos for a hobby robot. Crazy that a mediocre LIDAR costs more than every other part of the robot combined, maybe times 10.

You might like this:

https://github.com/leggedrobotics/open3d_slam

Its not AI, but it is simple and you can re-use a point cloud to re-localise against (ie once the map has been generated you can just localise rather than have to map the same time.)

Some places use ML to make a more robust descriptor (ie the thing that identifies the point in a point cloud) which mostly practical. I've not yet seen a practical "deep" SLAM pipeline. (but I'd not looked recently. )

*sparse

Have been doing something similar to this using image to image translation (XYZ rendered images to RGB space domain). Most of the information is contained in the Z-axis which gives you the height information, e.g. helps to distinguish the grass and buildings color. However I am skeptical if the X and Y is noise and how much spatial information it provides during Conv blocks. Anyone who had previous experience on this?

https://github.com/f0ti/thesis

As pointed out in my other comment, using a single image for point coloring is prone to errors due to noise, specular reflection and occlusion. I'd consider using a (normalized) cross-correlation approach with several images.

Check out https://m.youtube.com/watch?v=OjyxFGmcu74

Isn't there some math which crosses over between what Lidar is showing vs what photogrammetry provides from overlapping photograph images -> providing depth corrected/adjusted/ground-truthing of images?

Would an accurate ELI5 of this be :

* Mathematically align the photograph and the lidar point cloud.

* For each photograph pixel, colour whichever aligned lidar point is closest to the camera.

So you end up with one coloured lidar point per photograph pixel?

Mostly true. But the last part is incorrect. There are far more pixels in an image compared to the number of points in the point cloud area covered by that image. So you get 1 pixel per point. In addition there can be multiple points that map to the same pixel.

There are quite a few apps for iPhone Pro devices that achieve this. I don’t know why they aren’t more popular.