Raspberry Pi End-Cloud Collaboration --- Intelligent AI Glasses

Update System Software:

# Find the detection closest to center
if bbox_area < min_area_in_center:
min_area_in_center = bbox_area
center_detection = {
'detection': detection,
'bbox': bbox,
'confidence': confidence,
'distance': bbox_area
}
if center_detection is not None:
center_bbox = center_detection['bbox']

# 转换归一化坐标到像素坐标
xmin = int(center_bbox.xmin() * frame.shape[1])
ymin = int(center_bbox.ymin() * frame.shape[0])
xmax = int(center_bbox.xmax() * frame.shape[1])
ymax = int(center_bbox.ymax() * frame.shape[0])

# 确保坐标在有效范围内
xmin = max(0, xmin)
ymin = max(0, ymin)
xmax = min(frame.shape[1] - 1, xmax)
ymax = min(frame.shape[0] - 1, ymax)

if xmax > xmin and ymax > ymin:
# 裁剪图像
cropped_frame = frame[ymin:ymax, xmin:xmax]
# 转换为BGR并保存
cropped_bgr = cv2.cvtColor(cropped_frame, cv2.COLOR_RGB2BGR)
cv2.imwrite("test.jpg?x-oss-process=image/watermark,g_center,image_YXJ0aWNsZS9wdWJsaWMvd2F0ZXJtYXJrLnBuZz94LW9zcy1wcm9jZXNzPWltYWdlL3Jlc2l6ZSxQXzQwCg,t_20", cropped_bgr)
trig_ai_vl_event.set()

Call the cloud-based large model for in-depth analysis through the intercepted target of interest image. The key code is as follows:

def ai_vl_thread():
# 初始化OpenAI客户端
client = OpenAI(
api_key="sk-xxxxxxxxxxxxxx",
base_url="https://dashscope.aliyuncs.com/compatible-mode/v1"
)
while True:
trig_ai_vl_event.wait()
trig_ai_vl_event.clear()

# 将本地图片转换为base64
base64_image = encode_image("test.jpg?x-oss-process=image/watermark,g_center,image_YXJ0aWNsZS9wdWJsaWMvd2F0ZXJtYXJrLnBuZz94LW9zcy1wcm9jZXNzPWltYWdlL3Jlc2l6ZSxQXzQwCg,t_20")

# 创建聊天完成请求
completion = client.chat.completions.create(
model="qwen3-vl-plus",
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": f"data:image/jpeg;base64,{base64_image}"
},
},
{"type": "text", "text": "解释这张图片，100个字以内"},
],
},
],
stream=True
)

#print("AI回复：")
ai_vl_reply = ""
for chunk in completion:
if chunk.choices:
delta = chunk.choices[0].delta
if hasattr(delta, 'content') and delta.content:
ai_vl_reply += delta.content
display_text_pipeline.set_property("text", ai_vl_reply)
#print(delta.content, end='', flush=True)

1. How to further optimize the compatibility between the AI HAT and the Raspberry Pi main board to avoid the problem of unstable NPU operation caused by firmware version differences?
2. In the actual flight scenario of the drone, how to reduce the delay of cloud model calls to ensure that the analysis results can be fed back in real time to assist flight decision-making?
3. How to balance the performance of on-device AI inference and the power consumption of the device, so as to adapt to the long-time operation needs of the drone’s outdoor flight?
4. For the privacy protection of drone flight data, what more targeted optimization measures can be taken to avoid the leakage of sensitive information?