Notifications
Clear all

Need Help with GPIO display and Camera Preview

26 Posts
3 Users
0 Likes
1,731 Views
Robo Pi
(@robo-pi)
Robotics Engineer
Joined: 5 years ago
Posts: 1669
Topic starter  

I'm trying to build a custom camera using an Arducam on a Raspberry Pi zero with a 3.5" GPIO touchscreen display.

Preliminary info:

I have a similar set up already working on a RPI 4 that has an HDMI monitor attached. 

I also have the Arducam working on the Raspberry Pi Zero.

I can run the camera using raspivid and via picamera.   I can run the camera from python using both of these methods.

I also have the 3.5" touchscreen working on the Pi Zero.  The display works perfectly for everything except the camera preview.

The issue:

The 3.5" display plugs into the GPIO header, rather than into the HDMI port.  Special software had to be installed to get the display to work.  But once that is done the display works normally for everything except the camera preview.

What I have done thus far:

I currently have desktop icons that I can tap on with the touchscreen pen to launch my camera programs.  When I do that the camera comes on and records the video as I had instructed it to do.  And after the video has been recorded I can play it back, or save it to my main computer for video editing.

What I can't do:

I can't see a preview of what the camera is seeing.  It simply won't show up on this GPIO touchscreen.  Apparently both raspivid and picamera send their preview video out to the HDMI connector.  And I can't figure out how to force them to change that.  So while they are sending the preview out to the HDMI connector, I can't see it.  I have no way of knowing what's in the field of view, or whether anything is in focus.  Right now, all I can do is record video blindly and hope that the picture is centered in the field of view and is in focus.  That's really unacceptable.

What I have learned thus far:

Apparently the GPIO display that I am currently using doesn't use the GPU in the RPI processor.  Instead it uses it's own software to direct most standard outputs to the touchscreen.  It seems to work for everything except the camera preview.

So I'm basically at an impasse right now where I can't seem to make any progress.

raspivid does have a -dn option (display number) that will supposedly allow you to chose which display the preview should be displayed on.   But I've tried using different display numbers for that option and nothing seems to work.

So this entire system is fairly useless until I'm able to see a preview of what the camera is pointing at and whether or not it is in focus.

It's frustrating.

One thing I have noticed is that there are 3.5" touchscreen displays that actually plug into the HDMI connector instead of onto the GPIO pins.  I would imagine if I got one of those it should work as a normal HDMI monitor.  They are however, a bit more expensive.  Plus I already bought of couple of these GPIO displays so I would really like to get them to work.

What I need:

I either need a way to force either raspivid or picamera to redirect their preview output to the GPIO display.

I've even been thinking that there might be a way to do that via some sort of system command or system setting?  I haven't really looked into that too deeply yet.

OR

I need to find some other camera software that will automatically send a preview of the video to the GPIO display instead of to the GPU or HDMI connector.

OR

I need to break down and buy the more expensive HDMI 3.5" displays.  😮 

That should do it, but I hate to spend more money if I can solve the problem I'm having with these GPIO displays I already have.  They seem to work just fine for everything but the video preview.

Any ideas?

DroneBot Workshop Robotics Engineer
James


   
Quote
byron
(@byron)
No Title
Joined: 5 years ago
Posts: 1112
 
Posted by: @robo-pi

Any ideas?

Just a thought.  I usually use my rpis headless and view their output on my mac or pc computer via VNC.   As one sees the same output as if I had plugged an HTMI monitor into the rpi, maybe you could view the output on one of your computers instead of your mini screen which is left to do its touchscreen stuff.


   
ReplyQuote
Robo Pi
(@robo-pi)
Robotics Engineer
Joined: 5 years ago
Posts: 1669
Topic starter  

@byron

 

That's an interesting suggestion.  It won't solve my problem though because I need for this camera to be self-contained and portable.  So I won't have my computer available to see the preview.

Although I'm still interested in your solution.  I'm currently running the RPi headless using Remote Desktop.  However, I have the same problem there.  Even though I can see the RPi screen using Remote Desktop, the camera preview does not show up for some reason.

You mention VNC?  I've heard that mentioned before.  Not sure exactly how to implement that.  What software do I need to install to use VNC?

As I say, this won't solve my long term problem because I still won't be able to see the preview on the portable 3.5" display.   None the less, it would be some progress to be able to see the preview via a headless connection.   So it would probably be worth learning how to do that anyway.

Apparently Remote Desktop won't do it.   It shows the standard desktop, but not the camera preview.

So how do I set up for VNC? I'm already connected to the pi headless with Remote Desktop.  What do I need to incorporate VNC with that?

~~~~

Just for further information:

I have realized that I can use raspistill to take a still jpg.  I can then view the jpg on the 3.5" display. I could try to use that to check what is in the view of the camera.  I could also use that to try to adjust the focus.  But it would be a bit awkward.  For example, if the still picture is out of focus I'd have to try to adjust the focus blindly.  Then take another still pic to see if I'm better focused.  I'd probably need to do that several times before finding a focus that was suitable.

It's not a practical method since it wouldn't be in real-time.  It would be a real pain to try to set the focus this way.  I only mention it because it does at least serve as some level of preview.  It's just not a live real-time video preview.   So that's not going to be suitable in the long term.  I really need to be able to get a live video preview on the 3.5" display somehow.

I'm guessing that if I buy a 3.5" display that actually plugs into the HDMI connector that should solve the problem.  So that's one solution that is currently on the table.  But that solution costs more money.  So I can't help but want to try to solve the problem with what I already have.

 

DroneBot Workshop Robotics Engineer
James


   
ReplyQuote
byron
(@byron)
No Title
Joined: 5 years ago
Posts: 1112
 
Posted by: @robo-pi

You mention VNC?  I've heard that mentioned before.

The VNC server comes installed on the pi by default and you need to activate it in the in interfacing options.  You then need to download a VNC client for you desktop from https://www.realvnc.com/en/connect/download/viewer/

Here is a pic of the VNC viewer on my mac showing the the output from a headless pi.  I also had a terminal session connected to the pi via ssh into which I was inputting the camera commands but I could have as easily pulled up a terminal session within the VNC viewer.

Screenshot 2020 11 20 at 00.04.49

   
ReplyQuote
byron
(@byron)
No Title
Joined: 5 years ago
Posts: 1112
 
Posted by: @robo-pi

I'm guessing that if I buy a 3.5" display that actually plugs into the HDMI connector that should solve the problem. 

Probably 😀 though this certainly works with an RPI display that attaches to the display port on the pi board.  You could probably make a really neat solution with the official RPI touchscreen, but maybe an old laptop running linux could be packaged into a good portable solution for you?


   
ReplyQuote
Robo Pi
(@robo-pi)
Robotics Engineer
Joined: 5 years ago
Posts: 1669
Topic starter  

@byron

You gave me an idea with your post on the VNC server and Viewer.

Would it be possible to have the raspberry pi be both the server and the viewer?  I see they have VNC viewer software listed for the Raspberry Pi.

Could I have it serve the image back to itself?   This way I could view the image on the 3.5" GPIO screen.

I'm willing to bet this might actually work.   I'll see if I can give it a try.

If this works I'll have to go publish it on other forums where I've been seeking solutions.  Thus far no one has been able to come up with anything.   Although the topic of using VNC and a laptop has come up.  But if the pi can serve the image to itself then the laptop won't be needed.

I'll see if I can give this a shot and have it send the video to itself via VNC. 🤣 

Most people wouldn't think of doing this because it seems ridiculous to send an image to the same computer that's generating the image.   After all, most computers should be able to see images that they themselves generate.   But in this special case with the GPIO connector, sending an image to itself may be the only way to view it.

Owl check it out.

Thanks for the idea. 👍 

DroneBot Workshop Robotics Engineer
James


   
ReplyQuote
byron
(@byron)
No Title
Joined: 5 years ago
Posts: 1112
 
Posted by: @robo-pi

You gave me an idea with your post on the VNC server and Viewer.

Good to hear that the post maybe spawned an alternative idea, as I fear I have led you up the garden path on the use of VNC as its now sunk in that you want a continual live feed and this does not work with VNC when one programs a python program to do live feed. 😯 

You may find some better inspiration if googling using the pi camera as a PC webcam.  I saw something on 

https://www.tomshardware.com/uk/how-to/use-raspberry-pi-as-pc-webcam

But its too late in the day for me to read it to see if its any good for you.


   
ReplyQuote
Robo Pi
(@robo-pi)
Robotics Engineer
Joined: 5 years ago
Posts: 1669
Topic starter  

Unfortunately VNC doesn't appear to solve the problem.

I enabled VNC server on the Pi.

Then I installed the VNC viewer on my laptop.

Then I tried previewing a video and the preview still doesn't show up.

Apparently the VNC viewer is getting the same information as Remote Desktop gets.

I didn't try loading a VNC viewer on the actual pi, but if it doesn't show the preview on the Windows machine it's probably not going to show it on the Pi either.

Plus this whole scheme requires being in range of a router where the VNC viewer can recognize the VNC server as being 'online'.   I really didn't even want this camera to be dependent on WiFi access.  Although, that solution might have been better than no solution at all.

In any case, the video preview doesn't appear to come across on the VNC viewer anyway.  So unfortunately this didn't help.

Nice idea though. 👍  It was worth a shot.

DroneBot Workshop Robotics Engineer
James


   
ReplyQuote
byron
(@byron)
No Title
Joined: 5 years ago
Posts: 1112
 
Posted by: @robo-pi

In any case, the video preview doesn't appear to come across on the VNC viewer anyway.

Oh yes it can 😀 

Ha, our post crossed just as I was telling you a python program using the preview did not work over VNC. you find likewise.

But ... Yes it can.  I was wondering why it would not stream wereas one can happily stream youtube vids over VNC so I did a little digging.

You need to amend one of the options in the Rpi VNC server.  Go to the square VNC icon you should find in the RPI taskbar. Up should pop a window and towards the top right you see a square menu option to click.  Choose options, troubleshooting and tick the box 'Enable direct capture mode'.   Your python preview program will then be seen on your VNC client.

Of course this is not really a solution you seek.  I was thinking that as you can display a jpeg OK on your gpio screen, and if the preview is just for checking the framing and the focus, although not as good as a live stream, maybe you could offer up a still jpeg every second or so in your python script something like, take pic, display pic, in the background take a new pic, when written to disc then display new pic, in background remove old pic from disk and take new pic etc.  

By golly I'm in proper kludge mode all right, I'd better go and eat my morning bowl of fruit and get the brain cells properly charged up 😀 


   
ReplyQuote
Robo Pi
(@robo-pi)
Robotics Engineer
Joined: 5 years ago
Posts: 1669
Topic starter  
Posted by: @byron

You need to amend one of the options in the Rpi VNC server.  Go to the square VNC icon you should find in the RPI taskbar. Up should pop a window and towards the top right you see a square menu option to click.  Choose options, troubleshooting and tick the box 'Enable direct capture mode'.

 I can't get that to work.

When I go to the VNC icon and the menu pops up, I choose "options" and a huge window pops up that I can only see part of.   There's no way to scroll around in the window to even see what options are being offered.   I also cannot even close the window after opening it.  All I can do is power down the pi and reboot to regain control.

Not sure why that is.  It no doubt has something to do with this 3.5" GPIO display.

Although, when I try this using the VNC connection from the Windows computer I get precisely the same result.  It just opens a huge window that I cannot navigate in.  (i tried arrow keys, pg up and pg down, etc.) nothing works.   So I can't even see what's on the options page.

I don't currently have a mini HDMI connector for this pi zero, so I can't even plug this into a regular HDMI monitor right now.  I have some HDMI mini connectors ordered, but they haven't come in yet.

This is frustrating.  If it wasn't for this video preview problem I would have been up and running a long time ago.  Everything else work.  It records videos just fine.  You just can't see the preview to set things up before you start recording.   Got to have a preview.  Especially for the focus.  But also to make sure everything is in the frame the way I want it.   A still jpg will serve for the latter.  But it's going to be hard to manually focus with updated jpegs.  Even with automatically updated jpegs.  They still take a while between stills.  It's doable, but a bit of a pain that way.

DroneBot Workshop Robotics Engineer
James


   
ReplyQuote
byron
(@byron)
No Title
Joined: 5 years ago
Posts: 1112
 
Posted by: @robo-pi

 I can't get that to work.

Well thats a pain in the proverbial.  It worked for me when I accessed my rpi through a VNC client on my mac where I could set the local VNC server options without any problem and the subsequent python program to preview the camera was showing nicely.  I know the VNC viewer cannot co-exist with the MS remote desktop on a rpi so could it be some residual MRD left on your pi?  Have you got another pi or linux computer to see if you can get a remote vnc client working with your pi.


   
ReplyQuote
byron
(@byron)
No Title
Joined: 5 years ago
Posts: 1112
 
Posted by: @robo-pi

a huge window pops up

just another thought - maybe you need to start the vnc server with a larger screen resolution - I use:

vncserver -randr=1920x1080


   
ReplyQuote
Robo Pi
(@robo-pi)
Robotics Engineer
Joined: 5 years ago
Posts: 1669
Topic starter  

I just went through the whole shebang.  I doesn't work.

Since the 3.5" GPIO display is software driven I am able to turn it on and off via the command terminal.

So what I did was turn off the GPIO display.  Then I logged onto the pi using the VNC viewer.  By doing that I was able to see the VNC options menu and enable the direct capture.

However, after setting that up, I tried going back over to using the GPIO display again.  The GPIO display showed up on the actual display, but now the VNC viewer only shows the last page of the boot-up info.  It's like it's stuck in the HDIM mode, and the last page of the boot up log is all that's being displayed there.

So I had to go back through the whole process to disable direct capture again just to get back to being able to see what's going on via the VNC viewer when the pi zero is using the GPIO display.

It looks like the only solution is going to end up with having to buy an actual 3.5" HDMI display.  I should bought those in the first place.  I never thought this GPIO displays were going to be limited in this way.

What a bummer!

~~~~~~

Just for the record, these little 3.5" GPIO displays are pretty nice!   As long as you don't need to use a camera preview they seem to work fairly good for everything else.  Although it did have problems displaying the VNC server menu properly too.   But everything else seems to be showing up pretty well.

I'll be working with writing some Tkinter window programs for this little 3.5" touchscreen display.  You can write python programs that can be launched and run using just the touchscreen pen.  So you won't even need a keyboard unless you need to type in text.   Although you could probably set up a touchscreen keyboard and poke at it with the touchscreen pen.

I won't need to enter text for my camera program.  I'll have a drop-down list I can use to chose video file names, and then just have the program automatically append numbers to them so I can have several videos of each topic I'll be filming.

I just go with using the still jpegs for previews for now.  It won't be as nice as a live video preview.  But it will get me by for now.

It doesn't look like there's going to be a way to actually view a video preview on these touchscreens.  That's a shame.  What a bummer!  I'll do the sill jpegs.

Thanks, @byron!  Good suggestions!  Too bad they didn't pan out.  It was still worth trying.  Who knew it wouldn't work?

~~~~

By the way, there is a difference between the VNC viewer and Remote Desktop.

When using Remote Desktop I can run the raspberry pi and see all the programs and displays on the laptop computer.  However, the display on the raspberry pi shows no activity.  In other words, if I launch a program using Remote Desktop it launches on the Remote Desktop, but it doesn't launch it on the Pi display.  Strangely I can actually open programs on the Pi Display at the same time.  So they act like two entirely separate desktops.

However, when using the VNC viewer, anything I do on the laptop also happens on the pi display.  If I move the mouse on the laptop the cursor moves on both the VNC view AND on the pi display.  Not so with Remote Desktop.

I don't think one system is better than the other.  But they are different in how they work. 

Of course, Remote Desktop doesn't use the pi VNC server.  Instead you need to install xrdp on the pi.  So it is a different system entirely.

Anyway, that's an aside that doesn't have anything to do with the GPIO display.  Just an observation that VNC viewer and Remote Desktop are two entirely different systems.

DroneBot Workshop Robotics Engineer
James


   
ReplyQuote
byron
(@byron)
No Title
Joined: 5 years ago
Posts: 1112
 
Posted by: @robo-pi

However, when using the VNC viewer, anything I do on the laptop also happens on the pi display

Actually that depends.  If my pi is on 10.0.1.20 then if I connect my VNC client to that address I mirror the screen attached to the pi.  If I connect with 10.0.1.20:1 (or :2 or :3 etc) then I get a new screen.  The mirror screen has the same resolution as the attached screen, the additional screens can be at different resolutions.

I find this especially useful when amending programs on a remote RPI with a display attached as I can program in one larger sized screen and also see how the display looks on a separate window that mirrors the remote display. 

Another difference worthy of note is that the Ms Remote Desktop restricts the user from performing some functions like rpi configuration or shutdown etc. whereas no problem with VNC.

A really nice display for your project would be the official RPi touch screen where you could have all your clicky menu items, but also see the preview in a 7in display.  Not cheep though as you will need to add some sort of screen holder, but they usually accommodate both the screen and the rpi in a nice unit.  You could also incorporate the camera into the unit with a little bit of glue 😎 


   
ReplyQuote
frogandtoad
(@frogandtoad)
Member
Joined: 5 years ago
Posts: 1458
 

@robo-pi

Posted by: @robo-pi

It looks like the only solution is going to end up with having to buy an actual 3.5" HDMI display.  I should bought those in the first place.  I never thought this GPIO displays were going to be limited in this way.

What a bummer!

A quick search led me to the following, which may be of interest to you:

DIY WiFi Raspberry Pi Touchscreen Camera

Posted by: @robo-pi

By the way, there is a difference between the VNC viewer and Remote Desktop.

When using Remote Desktop I can run the raspberry pi and see all the programs and displays on the laptop computer.  However, the display on the raspberry pi shows no activity.  In other words, if I launch a program using Remote Desktop it launches on the Remote Desktop, but it doesn't launch it on the Pi display.  Strangely I can actually open programs on the Pi Display at the same time.  So they act like two entirely separate desktops.

However, when using the VNC viewer, anything I do on the laptop also happens on the pi display.  If I move the mouse on the laptop the cursor moves on both the VNC view AND on the pi display.  Not so with Remote Desktop.

I don't think one system is better than the other.  But they are different in how they work. 

Actually, they work very similar, but have different defaults as far as sessions go.  I've been using both for many years in the IT field, and after a while you happen to learn a few things.

If you type the following into your Windows command shell:

C:\>mstsc /?

... it will provide you with a list of switches that aren't readily available from the usual menu.

To connect via RDP client to perform administrative duties (like session 0, as if though your sitting in front of the machine), look to use the /admin or /console switch depending on the version of your client.

So, launch RDP from your command shell prompt instead, providing the necessary switches.

*RDP does not restrict you if you log on as an administrator using the switches provided.

Cheers.


   
ReplyQuote
Page 1 / 2