XIAO ESP32S3 Sense - Tiny ESP32 Camera

Thank you Bill for another excellent video.  I'm glad you do these and I don't mind that you get them later.  That way we know you're not getting them for free and have to say something nice.  I'm getting all the RasPi 5 vids in my news feeds and I don't even watch them.  Yours is the only one I trust to give the strait skinny on what we need here on the forum.  

1. How easy/hard is it to actually use those pads on the bottom?  They look awfully tiny, close together and without holes, making it difficult to solder a wire to them.
2. At 23:50, you're presenting the touch1detected variable.  This being a dual core and the interrupt changes this value... is there a chance of some collisions?  I've done loads of multi-threaded stuff in Windows, but never on a micro.  Should this protected by a Mutex, Semaphore, Critical Section or some other means like using volatile?  
3. At 25:10, you're presenting the interrupt itself.  My ignorance on the ESP32 begs a question.  On the ESP2866, we have to prefix the function with ICACHE_RAM_ATTR to make sure it stays in ram versus being in flash.  Not doing this (on the ESP8266) is a very tenacious bug.  Most of the time, it'll fail immediately, but sometimes it may run for a while giving too much hope. 😊 Does the ESP32 have/need something equivalent?
4. At 42:35, you're presenting the questionable auto focus feature.  When you leave the card forward and you moved back, it appeared you were out of focus.  I would think that with its limited (compared to DSLR) it would have only one focus zone (in the center).  I would think this would indicate the autofocus is working.  It focused on the card, but the limited depth of field showed you in the distance slightly out of focus.  Maybe @zander can weigh in here.
5. Not in video.  With the 5MP sensor.  You explored it at 800x600.  Do you think it can handle the full 1080P/30fps you mention in the lead up?  
6. Is there some provisions in the libraries to support making MP4 or similar with the integrated sound?  At 1080P/30fps?
7. Again, my limited use of the ESP32-CAM.  It seemed if you used everything (Video streaming, to SD card) that most of the exposed pins were unusable.  How is this one setup if you use everything (Video/Audio/SD streaming)?  Are there enough pins left to say PWM to robot motors and maybe a few extra pins to do I2C for some sensors?  IOW... can we make a DroneBot? 😉 

Thanks again.

VBR,

Inq

P.S.  I bet you have to beat the women off with a stick when out in public.  😉 

@inq I have not seen the video yet, but I can answer one question. All of the serialization methods mentioned in points 2 & 3 are available on the ESP32 as well as the 8266.

As far as auto focus, it does NOT mean everything will be in focus, the focal length will determine the depth of field. The term autofocus simply means that the lens will (attempt) to focus on whatever spot it has been instructed to and that is dependent on the amount of light and again focal length.

I have some chores to do but hope to look at the video later today. Looks interesting, but the Arduca solution is better glass albeit at 3x, 4x, 5x, the price. I don't care about that amount of $, a similar commercial unit is 20x.

Posted by: @zander

↑

As far as auto focus, it does NOT mean everything will be in focus, the focal length will determine the depth of field. The term autofocus simply means that the lens will (attempt) to focus on whatever spot it has been instructed to and that is dependent on the amount of light and again focal length.

I understand what autofocus does and I know depth of field is also a function of how close the focal distance is.  In my early days ('70s, '80s) using film based Nikons had ONE focal area in the center that was about 5% of the image.  Modern cell phone cameras use AI to find all the faces in the view and does some magic to try to get most of them in focus.  I'd imagine whatever this 5MP camera has, its closer to the former than the latter.

The point being... the best way to test for autofocus, is to put a subject (test card)  up close where it is thought the autofocus portion is working (the center).  Then move it as close to the camera that still stays in focus.  Say... can it get within 1 foot?  I doubt is has any macro ability.  At the closest location the depth of field will be at it's minimum... only about twice as far (say... 2 feet).  When doing that, it should be quite noticeable that Bill at the outer edge of the frame at say... 20x distance is very blurry.  If so, we know autofocus is working because it focused on the card... not his face.

It appears in the video... that that might be going on.  If the card is moved closer to the camera, Bill, at a distance, should become more proportionally blurry.  If everything is still sharp, then the camera just has exceptional depth of field (but I doubt it).  😉

Hi @inq,

 Apologies I haven't seen Bill's video yet, so I hope this doesn't cause too much embarassment, and an Instamatic is my photography level, but I did look for a data sheet...

-----------------

The product brief for the camera is at https://digilent.com/reference/_media/reference/add-ons/pcam-5c/ov5640_product_brief.pdf says:

1/4-inch, 5-Megapixel SOC Image Sensor Optimized for High-Volume Mobile Markets
OV5640 5-megapixel product brief lead free available in a lead-free
package The OV5640 delivers a complete 5-megapixel camera solution on a
single chip, aimed at offering cost efficiencies that serve the high-
volume autofocus (AF) camera phone market. The system-on-a-
chip (SOC) sensor features OmniVision's 1.4 micron OmniBSI™
backside illumination architecture to deliver excellent pixel
performance and best-in-class low-light sensitivity, while enabling
ultra compact camera module designs of 8.5 mm x 8.5 mm with
<6 mm z-height. The OV5640 provides the full functionality of a
complete camera, including anti-shake technology, AF control, and
MIPI while being easier to tune then two-chip solutions, making it
an ideal choice in terms of cost, time-to-market and ease of
platform integration.

and

auto focus control (AFC) with embedded AF VCM driver

============

Whilst the "full" data sheet at https://cdn.sparkfun.com/datasheets/Sensors/LightImaging/OV5640_datasheet.pdf

Page 19 includes:

The OV5640 has an embedded microcontroller, which can be combined with an internal autofocus engine and programmable general purpose I/O modules (GPIO) for external autofocus control. It also provides an anti-shake function with an internal anti-shake engine. For identification and storage purposes, the OV5640 also includes a one-time programmable (OTP) memory.

plus some autofocus mention starting on page 74:

5.13 draw window
The draw window module is used to display a window on top of live video. It is usually used by autofocus to display a focus window.

immediately followed by over a page of Draw Window commands.

----------

-------------------------------------

I hope this means more to you, than it does to me, but I get a rough feeling that it is expecting an external microcontroller to do all the user interface, and it in turn will assist by manipulating images, as per instructed, as well as providing the interface for the focusing unit.

If my feeling is remotely related to reality, I guess that means that a large chunk of the control is down to the programme in the external microcontroller, plus possibly programming the internal microcontroller, all of which is probably hidden in another data sheet, that might be harder to find ... the one here only has 179 pages!

Best wishes, Dave

Posted by: @davee

↑

the one here only has 179 pages!

Best wishes, Dave

Well jeeze... only 179 pages? 🤣 

Bill's videos makes it sound like a worth while upgrade.  I have about 6 ESP32-CAM gathering dust that I hope to add to a robot.  I'll probably have to upgrade to this ESP32-S3 model just so I can get that camera.  Although... it sounds like most of what you found was marketing hype.  And it still wasn't clear whether you have to do something with the AF, or the library already does, or the camera just does it by itself.  Also... it's tiny... but as Bill found... you got to put a huge heat sync on it making it larger than the whole rest of the rig and you would have to have it out in the air to be useful.  IOW... it'll never be used in a phone or web cam.  So... bad design on their part making it basically only useful to DIY people like us.  

@inq I just had a quick look at the video and immediately noticed that the camera moved whenever Bill took a picture. This is easily fixed by moving the touch sensor OFF the camera breadboard.

I have to be careful which Ron is speaking, if it is Ron the electronics hobbyist then these little cameras are quite miraculous but if it's Ron the former professional photographer the 5MP camera is a throwback to my very first digital camera. The 2MP camera is ok for certain use cases, none of which are 'photography'.

Somewhere I do have the code to drive the autofocus motors and if/when I find it again will post a link in this topic. As the boards ship using the libraries Bill used the autofocus is not implemented.

@inq IIRC, the 5MP camera pushes the AI-Thinker board too far and the facial recognition code was crippled at least for a while. There are several versions of the library and some work but no facial code. IOW, do a small test before committing anything to a robot base.

Again, as far as AF is concerned, additional code is needed.

Hi @inq,

  The first doc was marketing, but the other one isn't. However, it is more like an instruction set list, but next to nothing about how to actually use it. Sadly, a lot more info needed to make sense of it seems to be proprietary.

I have the impression the 5640 has the same or similar interface as the 2640 ... I seem to recall during a search I did sometime ago that there are just 2 or 3 camera devices that use this interface .. more recent ones have a higher performance alternate.

How useful buying the camera alone will be to you I don't know. The 5640 camera is widely advertised on AliExpress ... e.g. just the first of many adverts for the same range of items

Best wishes, Dave

Posted by: @zander

↑

I have to be careful which Ron is speaking, if it is Ron the electronics hobbyist...

I just adore you even when you're out of body... 🤣 

Posted by: @zander

↑

Somewhere I do have the code to drive the autofocus motors

So... are you saying there is a physical "motor" inside that little thing????

Posted by: @zander

↑

@inq IIRC, the 5MP camera pushes the AI-Thinker board too far and the facial recognition code was crippled at least for a while. There are several versions of the library and some work but no facial code. IOW, do a small test before committing anything to a robot base.

WOW!  I certainly rely on you for your expertise around these camera things even if they're...

Posted by: @zander

↑

The 2MP camera is ok for certain use cases, none of which are 'photography'.

Posted by: @zander

↑

There are several versions of the library and some work but no facial code. IOW, do a small test before committing anything to a robot base.

I don't care if they recognize humans... we'll run them down with just as much impunity.

Don't wrap around a wheel Ron... I'm being ironical. 😆 

Posted by: @davee

↑

How useful buying the camera alone will be to you I don't know. The 5640 camera is widely advertised on AliExpress ... e.g. just the first of many adverts for the same range of items

I would hope, they didn't swap wires... but at that price, I might need to do a long lead time purchase with Ali.  At my age, I'm willing to pay more to get it in one or two days versus... four weeks.  I might forget why I ordered it otherwise.  🤣 

VBR,

Inq

@inq  

So... are you saying there is a physical "motor" inside that little thing????

Posted by: @zander

https://github.com/0015/ESP32-OV5640-AF

One of many

Also see Arduino docs at https://www.arduino.cc/reference/en/libraries/ov5640-auto-focus-for-esp32-camera/

To get the entire scope, just google "esp32cam autofocus"

EDIT: Almost forgot

Hi @inq,

  I only quote AliExpress because their search engine is easy to use, and links, copy/paste etc usually work, whilst the obvious competitor makes life hard ... please don't regard as a purchasing recommendation .. just 'proof' the bits are available from someone.

In practice, a lot of AliExpress stuff is arriving (to me in UK) in about 2 weeks, a few in 1 week. Of course, other bits, especially those that are not amenable air freight take forever!

Best wishes, Dave

Posted by: @davee

↑

Of course, other bits, especially those that are not amenable air freight take forever!

So... you're saying... the saying, "Slow boat from China is real?"  

I did a 280AHr LiFePO4, 4x build... it can't fly and it was right in the container scarcity time.  I want to say it took about four months to get my cells.  I thought I had made a bad mistake!

https://www.macgregorsailors.com/forum/viewtopic.php?t=28149  

Hi @inq, 

   To the best of my knowledge, lithium based batteries can only fly in very specfic circumstances, like hand baggage, and then usually must be inside the phone or whatever they are fitted in. The idea being, if they start to overheat, the aircraft staff can dunk the whole thing in the nearest pool of water, (champagne bucket even?) and encourage it to cool down.

Lithium batteries in the cargo hold have caused a few major problems, especially with the air delivery fleet aircraft, and hence are really not welcomed.

-----------------

I note some other things, like magnets (only small ones, nothing extreme) and solder paste can also be 2-3 months. For obvious scrooge-like reasons, I haven't bought any larger items that would just be expensive to fly.

So the slow container boat from China is probably real!

But AliExpress, Banggood and others are starting to have 'local warehouse' stocks in UK, US and elsewhere, so some stuff is only using a local cheap lorry delivery company, much the same as Amazon, etc., and also the Royal Mail.

I now look to see the items marked something like "12-day delivery on £9". (The exact wording changes daily). You only pay the standard postage, sometimes less, but if the total cost (before postage and tax) of items so marked reaches the "£9", then all the small packets from different suppliers get put in a larger bag, and airfreighted, usually in the time they claim.

Of course, there is a fair bit of 'faff' finding the best pick and mix, so you might prefer to ignore it!

Best wishes, Dave