Raspberry Pi 5, Hailo AI Chip 13 & 26 tops

@axe

I have not got the Hailio AI Hat for the rpi5 but I have got the rpi AI camera as shown in the @dronebot-workshop video of a few weeks ago and I had a quick play.

It does come with some example programs that demonstrate the camera detecting various objects, but the programs are not documented, contain no helpful comments, and theres no clear tutorial on the whole process for getting the camera to recognise one's own objects to be found in the rpi documents.  I expect it's the same with the AI Hat.  It's quite disappointing.   A dig around the internet does garner some more info but right now i decided I do not have the time so the AI camera has been put to one side for another day. 

In my short play with the example programs I see I have stored a small snippet on my computer, though at this point I'm not sure if its a straight copy of an example program or one I bodged but it recognises a 'cup' object and when it sees this object it triggers a print statement, but of course this could trigger what you like.  Of course it's for the AI camera and probably of no use for you Hailio AI hat, but I show it below just in case.

Looking at the snippet, whilst a 'cup' object is in the IMX500Detector, so is a bunch of other stuff like computer screen, persons, etc. and of course the doc's don't exactly make it clear what objects it contains, though I did find a list somewhere, and its probably in the rpi5 to which the camera is attached.  As you see the snippet is very crude and whilst it waits for the cup to be recognised for 4 frames theres no attempt to ensure they are consecutive frames etc. and it was just a quick go I had to try to understand what one could do with the AI camera.

For getting your own object into your own list there are some pointers in this video 

and I see in those video's there in one on the AI Hat so that may be of use to you. 

Well good luck with getting your objects recognised with your Hat, and sorry not to have been more help at this time. 

from ai_camera import IMX500Detector
import time

camera = IMX500Detector()

# Start the detector with preview window
camera.start(show_preview=True)

# Main loop
cup_detect = 0
cup_sightings = 1

while True:
    # Get the latest detections
    detections = camera.get_detections()
    
    # Get the labels for reference
    labels = camera.get_labels()
    
    # Process each detection
    for detection in detections:
        label = labels[int(detection.category)]
        confidence = detection.conf
        #print(label,confidence)
        """        
        # Example: Print when a person is detected with high confidence
        if label == "person" and confidence > 0.4:
            print(f"Person detected with {confidence:.2f} confidence!")
        """
        # Print when a cup is detected with high confidence
        if label == "cup" and confidence > 0.4:
            cup_detect +=1
            
        if cup_detect > 4:
            print(f"A cup is detected with {confidence:.2f} confidence! after 4 frames")
            print(f"cup sightings = {cup_sightings}")
            cup_detect = 0
            cup_sightings +=1
     
    
    # Small delay to prevent overwhelming the system
    time.sleep(0.1)

I believe that the object IDing dataset is the Coco dataset https://cocodataset.org/#home
the website says: COCO is a large-scale object detection, segmentation, and captioning dataset. COCO has several features:

• Object segmentation
• Recognition in context
• Superpixel stuff segmentation
• 330K images (>200K labeled)
• 1.5 million object instances
• 80 object categories
• 91 stuff categories
• 5 captions per image
• 250,000 people with keypoints

  So I'm quite excited to get all 80 objects triggering for me with a customized Language Model.
  This is the video I'm working it. Gonna have another go at it. NOt to sure why there is so little info on this..

Thanks

@axe

Motivated a bit by putting up that bit of noddy code I posted and thinking I should at least tidy it up and make it work properly as intended I got out the rpi5 with its attached rpi AI camera to have another play.  

First up I did not have a yolov8n_pp processing model in the list, but I found it and downloaded it into the list of available models, and its the last in the list below.  I would expect the rpi Halio Hat has a similar list.  But to note that all the models  I have tried from the list seem to work.

In the demo program I posted it imports: from ai_camera import IMX500Detect  and in this module the desired model to use is identified.

Turning my attention to the demo program I see it was too much of a bodge in many ways including reloading the object label every time in the loop.   I amended it to just load them at the start and also to print out all the objects that will be detected, which is determined by the import of NetworkIntrainsics as per the ai_camera module just referred to in the pic above where its imported from the picamera2 module.

So In my example I find the list of objects are:

['person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light', 'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard', 'tennis racket', 'bottle', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'potted plant', 'bed', 'dining table', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddy bear', 'hair drier', 'toothbrush']

Undoubtedly from the coco dataset but not as large as the whole dataset you refer to. (and a peculiar mix I must say)

I changed the demo program example to report on recognising a mouse (of the computer pointing variety not a living squeaky one 😀 ) as my cup of coffee was not on my desk, and made it only report if the mouse is continuously found in 5 detection loops.   Also when a mouse detection is confirmed it also sends an MQTT message.  For what its worth I post my snippet.

from ai_camera import IMX500Detector
import mqtt_pub as MQTT
import time

camera = IMX500Detector()

# Start the detector with preview window
camera.start(show_preview=True)

# object detection counters
mouse_detect = 0
mouse_sightings = 1
detection_frame = 0

# Get the labels for reference
labels = camera.get_labels()
print(labels)

while True:
    # Get the latest detections
    detections = camera.get_detections()
    detection_frame += 1
    print(f'Process new detections___________________{detection_frame}')
    
    # Assing labels to each object detected for this detection loop
    for detection in detections:
        label = labels[int(detection.category)]
        confidence = detection.conf
        # uncomment below to print all objects detected in the last grab from detections.
        #print(label,confidence)

        # Find if there is a mouse object detedted with high confidence
        if label == "mouse" and confidence > 0.4:
            mouse_detect +=1
            if mouse_detect > 1: # so must have been detected before
                if detection_frame != frame + 1:  # but it was not detected in next sequential frame
                    frame = 0  # it was not the next in sequence so abort
                    mouse_detect = 0
                elif mouse_detect > 4: # yes - it must be a mouse OK
                    print(f"A mouse is detected with {confidence:.2f} confidence! and in 5 frames")
                    print(f"The number of mouse sightings = {mouse_sightings}")
                    # and publish a MQTT message
                    MQTT.publish('AI_Cam/Detect/Mouse', 'Mouse Detected')
                    mouse_detect = 0
                    frame = 0
                    mouse_sightings +=1
                else:
                    frame = detection_frame
            else:
                frame = detection_frame
            print(f'was that a mouse? {mouse_detect}')
            

    
    # Small delay in the detection loop to prevent overwhelming the system
    time.sleep(0.1)

Now that little play with my AI camera took all morning and at the end of that time, I've got my demo nicely tidied up, which is good, but I have not really progress much in properly using the rpi AI camera at all.   It again reminds me of the amount of time one would need to devote to this AI stuff with the lack of supporting material to be found, so it's time to pack it all away again for now.   

I got the AI camera with the thought of using it perched on a wheeled robot but this sort of use is now very much on the back burner.  I do see a use of it to be a better motion detector where instead of just detecting movement it could just report if a person was in the frame.   It would be good to use it to report of persons walking up the drive to my house, but ignoring a passing rabbit. 😎  On such a detection it could send a MQTT message that my home monitor program would pick up and issue an alert signal to all its monitor nodes.

But if you progress further with your rpi AI Hat then do let us know of what you've found as I would be most interested.  Now its time to find that coffee cup and fill it with a decent brew. 

@byron

I won't be giving up anytime soon..

If you find any more info please share!

Thank you for your efforts so far!

Enjoy that coffee!

@axe    FYI @byron

Amazing how painting the dog black makes it look more menacing. Police do the same thing with black uniforms.

Although I don't have an AI camera or the Hailo AI Hat for Raspi5 I am always interested in what people can do with them and how they compare for any particular task with an hand coded solution and follow with interest anyone one experimenting with or using them.

Regarding the people detection example in the above atttached video my first person detector worked like the vision system of biological systems that detects movement (or change).  When humans lose the ability to see motion (akinetopsia) due to lesions in a particular part of the visual system  they see something like a moving car as a series of static images. Insects use visual motion (optic flow) to avoid objects and determine their velocity.

If you have a shape such as a square made up of random dots on a background of random dots it will be invisible but the moment it moves you can see the square (or whatever shape) instantly. But I digress, sorry.

The black happens to be the colour of the 12 capacitive touch areas. When I learn how to electroplate it will be mostly chrome. 🙂