Time of Flight (ToF) VL53L5CX - 8x8 pixel sensor

Posted by: @robotbuilder

@davee

I confess I haven't worked through that yet, but I am curious to know the origin of this information and where/how is it implemented in the system?

As I wrote in my post, it is implemented in my code for a 3d view in a computer game.

https://forum.dronebotworkshop.com/user-robot-projects/inqling-junior-robot-mapping-vision-autonomy/paged/7/#post-31321

As @inq didn't comment I can only assume it is not implemented in the VL53L5CX to correct a display for the different distances from the sensor travelled by the light to the wall where the distance from the horizontal would be the same all along the wall. If you want a complete explanation I can give it to you in a private post so as not to hijack this thread with material irrelevant to the workings of the VL53L5CX sensor.

 

I did do a search of the library code base they give out and there is no explicit trigonometry or use of Pythagorean theorem.  However, they mention that at startup the begin() statement has to upload a bunch of code to the sensor (2.5 seconds worth).  They don't say what's in there, but the sensor does have its own MPU.  I would assume our compiler isn't capable of compiling for that MPU so it would have to be a binary already compile... and thus not visible to us.  (My guess only)

I've written several return posts to their forum and I tried to be nice, but really I found two cases where their concept of fudging the numbers is totally... well... I sugar coated it!  I'll include them here... if you wish to see them or just want more of my anal-retentive rhetoric.  🤣 

VBR,

Inq

Hi @robotbuilder,

Thanks for your reply and kind offer just above... apologies, but I think I misunderstood ... I thought you had found some evidence of the 'correction' method in the VL53L5CX documentation or software.

Presently, I think we are all trying to imagine what the sensor and its associated support software, etc. must be doing to achieve what it claims, but little concrete evidence to support.

Best wishes my friend, Dave

OK... I think we have their ear now... things are happening in hours instead of weeks.  Here's the response...

I'll incorporate the Trigonometry in the client so the numbers will be displayed in the images above and pop out some more thrilling data charts.  🤣  And... calculate some new error numbers.  It will also require adjusting all my oblique testing.  But... hopefully to the better!

VBR,

Inq

@inq 

We test by putting our sensors perpendicular to a wall at some distance N, and the returned distances should all be N.

Maybe it is like the display problem I had converting distances into a visual display?

I wonder if that is used for the TOF camera as well.

To enlarge an image, right click image and choose Open link in new window.

Posted by: @robotbuilder

@inq 

We test by putting our sensors perpendicular to a wall at some distance N, and the returned distances should all be N.

Maybe it is like the display problem I had converting distances into a visual display?

I wonder if that is used for the TOF camera as well.

To enlarge an image, right click image and choose Open link in new window.

 

Your picture on the right is what should be observed in the sensor data given in the tables of all those tests I did.  The distance to a flat wall gets further away as the angle is off-center.  All that Geometry lesson above was the Mathematics to prove what we were supposed to see in the tables.  That is not what we have been seeing and after getting on the ST forum, we got our answer... https://forum.dronebotworkshop.com/sensors-modules/time-of-flight-tof-vl53l5cx-8x8-pixel-sensor/paged/15/#post-34260 ... that they in fact do use the Trig to adjust the data so that it gives a flat line.  As John (from ST) points out it was done for two reasons.

1. So QC can test it against a flat wall at any distance and all data is supposed to be the same for a passed sensor within the +/- 5% tolerance.
2. The main reason... the bazillions they sell to the cell-phone people.  They want it that way!  😆 

I since asked... "Why don't you have that configurable?"  They have many settings that have to be changed in #define statements and compiled.  I asked could they add one of those.  I also asked, Why is there no documentation?  I haven't received a reply yet.

This blows my mind... that no one else has ever noticed it.

It does change the Math for converting to the point cloud, but it should not be that major of a change.  

VBR,

Inq

@inq 

It does change the Math for converting to the point cloud, but it should not be that major of a change.

But doesn't that save converting from distances (red lines) from a single point to an object (if that is what the tof software is doing) to distances (orange lines) to an object from a line drawn through the sensor perpendicular to the direction in which it is looking?

The blue squares are the objects in 2d space but the logic is the same with extra math for 3d space.

I wasn't sure if the returning photons go through a lens and are focused on an array of sensors like an ordinary camera.

To enlarge image, right click image and choose Open link in new window.

Posted by: @robotbuilder

@inq 

It does change the Math for converting to the point cloud, but it should not be that major of a change.

But doesn't that save converting from distances (red lines) from a single point to an object (if that is what the tof software is doing) to distances (orange lines) to an object from a line drawn through the sensor perpendicular to the direction in which it is looking?

The blue squares are the objects in 2d space but the logic is the same with extra math for 3d space.

I wasn't sure if the returning photons go through a lens and are focused on an array of sensors like an ordinary camera.

To enlarge image, right click image and choose Open link in new window.

 

You got it!  👍 That is exactly what they are doing.  They return the orange lines.  And your drawing is very similar to the ones I posted on the ST forum.

The red lines are what the sensor actually measures... and coincidentally what any single laser type device, including the spinning Lidar returns.  I questioned John, why they didn't mention this massive change without documenting it... AT ALL.  

Posted by: @robotbuilder

But doesn't that save converting from distances (red lines) from a single point

It certainly does.  And that is why I said it won't be that much of a change.

If we consider only the local coordinate system of the sensor.  X,Y,Z=0 being the sensor, with Z being the direction it is pointed, thus X,Y is the plane of the sensor's surface.  In your drawing...

They have given us Z, so we don't have to calculate it anymore. 

But we must have all three coordinates. 

In the original thinking

1. The red lines were the hypotenuse of a right triangle
2. We know the angle that each pixel uses on the sensor
3. We calculate X,Y = sin(pixel) * hypotenuse
4. We calculate Z = cos(pixel) * hypotenuse

In this ST thinking

1. The orange line is Z
2. We calculate X,Y = tan(pixel) * Z

Note - The pixel angle is dependent for each pixel and on which axis we're using.  It is different for all three components.

So yes... they have saved us a third of the Trigonometry coding we need to perform.  Unfortunately, I just benchmarked tan(angle)... It takes precisely 2.0x times longer to run than sin(angle)... so they've actually added to our overhead if running on the MPU.

VBR,

Inq

Posted by: @inq

In this ST thinking

1. The orange line is Z
2. We calculate X,Y = tan(pixel) * Z

Note - The pixel angle is dependent for each pixel and on which axis we're using.  It is different for all three components.

So yes... they have saved us a third of the Trigonometry coding we need to perform.  Unfortunately, I just benchmarked tan(angle)... It takes precisely 2.0x times longer to run than sin(angle)... so they've actually added to our overhead if running on the MPU.

Perhaps I'm confused, but it seems to me that the 8x8 array always has the same relative angles away from the direction that the sensor is pointed.

That being the case, the angles should always be the same, so you should be able to calculate the tangents of the 8x8 matrix only one time and use that lookup table for on the go calculations. That would reduce your  time accessing the tangent from the full calculation to a single indexed lookup.

Posted by: @will

Perhaps I'm confused, but it seems to me that the 8x8 array always has the same relative angles away from the direction that the sensor is pointed.

That being the case, the angles should always be the same, so you should be able to calculate the tangents of the 8x8 matrix only one time and use that lookup table for on the go calculations. That would reduce your  time accessing the tangent from the full calculation to a single indexed lookup.

Good catch.  You are totally right.  Angles are constant so tangent of angle is constant.  

Anyway, I'll be doing that on the client side on a Windows i7 machine (to start with).  The ESP8266 will just pump that same set of data that those charts show above.  IOW the Sketch code won't change any.  

The client side initialization code I figured would startup with the angle between  pixels.  But, we've already seen the angles are not very consistent, so I'd like those to be configurable and persisted.  It would start up and create two different 8x8 arrays using Tangent so 128 tangent calculations.  (1) for x-component, (2) y-component.

VBR,

Inq

Hi @inq and @robotbuilder,

  Sorry, I haven't any direct experience of ray tracing or similar software, so this may be a totally stupid question or two .. and I am only asking out of curiosity on my part at present, so please do not go to any trouble on my behalf to answer it.

[NB The question does not directly relate to the recent confusion between the 'hypotenuse' length, that we were expecting and 'adjacent' (perpendicular) length that the device apparently reports. I am assuming that can be dealt with using usual trigonometry as required...  now we 'know' what the sensor is doing😀.]

---------------

The 'pixel' in the VL53L5CX returns a single distance value from a "large" (roughly?) rectangular area in space.

i.e. Assuming the sensor is stationary, when a sensor pixel (consistently) says a photon is being returned for most flashes, from an object 2 metres away, there is no way of knowing if it is a small (possibly very reflective) object anywhere in a much larger rectangular area, or an object the size of the whole rectangle.

Of course, if all of the surrounding pixels are reporting an object at the same distance, then it is probably a single 'very' large object, such as a wall ... but that is the 'easy' case to interpret. The more interesting case is when the pixel is 'looking' at or near the edge of an object, so that surrounding pixels are reporting a different distance or maybe no new data report, when the first object is out of range and no photon is reflected back in the allowed time to be detected.

----------------

I can understand that with a 'normal' camera, an approach could be to look for the edges of object, and possibly extend this by taking two pictures at slightly different angles ... but a 'normal' camera will be relatively high resolution, and the 'edges' will actually be edges of some type, even if they are only the border between two different paint colours.

This sensor is very low resolution, and hence poor at distinguishing edges from narrow objects. Furthermore, the area covered by one pixel becomes very significant as the perpendicular distance grows.

Plus, it is not clear how well defined the pixel imaging is ... the documentation mentions lens (or lenses), but the focussing might be rather poor.

---------------

Hence, does any software trying to 'map' what the sensor observes need a model that relates to the sensor's viewpoint? 

One possibility could consider each pixel as describing a rectangle ... but this rectangle changes size depending upon the perpendicular distance. (It also changes depending on which pixel is considered, but this is a smaller effect.)

In a 2 dimensional world, an alternative coordinate system to (x,y) is radius and angle (r, theta).

In this 3 dimensional world, this may be extended to (r, theta, phi)

From a programming viewpoint, it might make sense to 'quantise' the angles to the pixel size .. e.g. 1 unit is 45/8 degrees (about 5.6 degrees).

I am not aware of the availability of software to cover such a view of the world, or whether it will be a useful model for robot navigation..

I wondered if you had any comments or thoughts.

Best wishes, Dave

@davee - I am always open to better ideas and just about everyone that has been on this thread or one of my others has swayed me to a different or more refined view point of how to do something.

Posted by: @davee

The 'pixel' in the VL53L5CX returns a single distance value from a "large" (roughly?) rectangular area in space.

i.e. Assuming the sensor is stationary, when a sensor pixel (consistently) says a photon is being returned for most flashes, from an object 2 metres away, there is no way of knowing if it is a small (possibly very reflective) object anywhere in a much larger rectangular area, or an object the size of the whole rectangle.

Of course, if all of the surrounding pixels are reporting an object at the same distance, then it is probably a single 'very' large object, such as a wall ... but that is the 'easy' case to interpret. The more interesting case is when the pixel is 'looking' at or near the edge of an object, so that surrounding pixels are reporting a different distance or maybe no new data report, when the first object is out of range and no photon is reflected back in the allowed time to be detected.

Here are some thoughts, that I plan on pursuing until a better idea comes from the forum or from some nightmare I have.

• I currently plan to do more of the oblique tests on this sensor with my new found knowledge of what it's really returning and try to identify the location of the "pixel".  This would be the x and y angle from the centerline as well as it being the nearest hit point within the pixel square or an average distance in the square.  However all squares will be looking at a uniform flat wall when doing these tests.
• I would like to explore their supposed ability to return more than one hit from each pixel.  It is configurable to support up to 4 targets in each of 64 pixels.  Talk about smoke and mirrors going on.  It will supposedly tell you there are 1-4 targets in the pixel and the distance to all of them.  This would be an instant awakening that there is something special going on here.

(more to come)

VBR,

Inq

Posted by: @davee

I can understand that with a 'normal' camera, an approach could be to look for the edges of object, and possibly extend this by taking two pictures at slightly different angles ... but a 'normal' camera will be relatively high resolution, and the 'edges' will actually be edges of some type, even if they are only the border between two different paint colours.

This sensor is very low resolution, and hence poor at distinguishing edges from narrow objects. Furthermore, the area covered by one pixel becomes very significant as the perpendicular distance grows.

The last post was about what to do with the sensor itself in a more stand alone concept.  This post is about my current strategies about what to do with the robot itself while using the sensor.  As you pointed out... the sensor is for-crap resolution wise.  At 4 meters the pixels are almost 400 mm square.  

Test Site

My main test site is the old school that in in the Inqling Jr thread videos.  A long hall with class rooms on both sides.  It is one floor with no stairways.  At the center of the long hall is the auditorium and the main entrance foyer.  It is relatively symmetric.  

"Current" Mapping Strategies

These are my current thoughts...

Survey Mode

1. Have it do a survey mode - this would hopefully be a scanning and mapping mode while on the move.  Start in the hallway and circumnavigate the hallway only... ignoring the rooms.
2. Map walls and door opening in a pretty course resolution, but not entering the rooms
3. Gets an overview to prioritize and plan a strategy for getting more details.
4. The SLAM technique talks about the error build up of making lots of turns and having to adjust all the data based on the end not actually returning to the start point.  That SLAM video somewhere talks about this toward the end.  It appears to me from the calibration tests, that turns are likely to induce the most error.  Not going into each and every room to map eliminates 90% of the turns.
5. Although the resolution is pretty crappy, extend the thought to including the movement.  Say the robot is traveling parallel to the hall wall about half a meter away from it.  Wall is on the robot's right. 
6. I'll turn the head to the right (say 15°) such that I have the left pixel column covering the forward progress of the robot and use those pixels to detect obstacles.
7. That says the right column is pointing about 37° off-center.  So it is scanning the wall about 0.66 meters ahead of the bot and the scan distances will be about 0.83 meters away.  In the sweet spot of the sensor range.
8. Assume I get it up to 4 kph (good walking pace) - It'll be taking a snapshot (15Hz) every 7.5 cm.  
9. With multiple frames there is plenty of overlap so data smoothing can be used.
10. With the spacing, I won't miss any doorway larger than 8cm.  Basically a dimension that is too small for the bot to explore "inside".

Detail Mode

1. Now that it has an overview of the hallway, I can have it start somewhere... probably at the beginning assuming it returned to the beginning anyway.  
2. It can go to the first door that it remembered on the right.  It can now move in as close as we determine is necessary through experimentation.  It can move to the door jam say 25 cm away where the resolution is only 2.5 cm.  
3. If that is not enough it can slowly turn its head and get resolution down in the sub centimeter range.  

Divide And Conquer

1. Then it becomes like a recursive subroutine - Enter a room and go back into Survey mode and go around the perimeter of the room.
2. Back to Detail Mode and note obstructions details down to the sub-centimeter range.  
3. If the room is larger than the bot can see the opposite wall, it'll start exploring the interior of the room.

Bazillion lines of code down the road and we're done. 🤣 

VBR,

Inq

@davee

This sensor is very low resolution, and hence poor at distinguishing edges from narrow objects. Furthermore, the area covered by one pixel becomes very significant as the perpendicular distance grows.

This is what the edges of a hand would look like.

The wall doesn't have to be perpendicular to the sensor in the experiments. We are calculating the distance from the x,y position of the object to a line drawn through the sensor which is perpendicular to the direction in which the sensor is pointed. The distance d1 returned by the ray is converted to distance d1' and the distance d2 is converted to distance d2'.

As least as I understand it IF it behaving like ray casting in a computer game. The distance returned by the ray is converted to a distance from an axis perpendicular to the direction the sensor is pointing (horizontal screen axis) and the new distance value is the distance along the other axis (in games the screens vertical axis).

If the viewer is looking directly at a wall all the distances from the horizontal axis will be the same. If you just plot distance along the vertical axis with the returned distance of the ray you will get a curved wall.

To enlarge an image, right click image and choose Open link in new window.

Posted by: @robotbuilder

IF it behaving like ray casting in a computer game.

John at ST simply said...

• it aids the QC all values are the same when looking at a perpendicular wall.  That seems reasonable for QC purposes, but not to make it the foundation for how it works.
• Then he said that the camera people want it that way.  Again from ST standpoint... sure buy ten million, we'll paint it Chartreuse for you if you want.

In your drawing... where the lines come down and intersect the sensor's plane... those points don't stand out (to me at least) as having any significance and the distance to that plane doesn't seem to have as much value... don't people want to know how far things are away from the sensor... not some plane out in space???

It sounds like you have some experience where that is useful?  To gaming... to camera?   To the price of beans in China... anything??  Is it some optimization/simplification that allowed Doom to run on 8086 back in '85?  Can you elaborate?  

It's really bothering me that I don't understand the usefulness of that.

Thanks!

VBR,

Inq