Raspberry Pi System Image Backup Questions.

Following. I have the same issue. You would think getting a larger (faster?) sd card would be a good thing. NOPE! I suffer from the same problems you do. I wish the win32diskimager would allow forjust writing the space needed without writing an entire 16gb. It's a waste of time and space (and bandwidth if you store to the cloud).

Have you tried compressing your images with something like gzip after you have copied them from the SD card to your hard drive? I can generally compress a 16GB image down to 4-5GB with that method.

If you want to be more aggressive, there is a set of instruction at http://www.aoakley.com/articles/2015-10-09-resizing-sd-images.php . I have found the process to be more trouble then it is worth for me.

As a workaround, you might want to consider getting a home 2bay NAS. Storaging backup images is a pain.

If you are interested, I can play with the script listed above to see how effectively it works.

I also have dozens of backup images of RPi installs, and I've tried a number of various tasks to make the process less painful, including the linux DD, but, at the end of the day, the easiest method I personally found was to use the win32disk imager like you do, and store them all on a NAS drive that I connected to a RPi running Samba

This was before I got my new router, which has a USB port on the back for exactly this purpose, but, an added bonus for the Pi was the fact that I could install a PLEX server on the Pi and can then share all my backed up movies to any device in my house, or even my phone if I'm away from home. Theoretically I could also give a code to my friends so that they could also have access to my movies, but, so far, nobody has taken me up on that offer

While I don't have a lot of different images, I also use Win32DiskImager.  Mac OS X is supposed to have an easy command line way to do this to, however, I have not tried it yet.

I started making backups after a well meaning person pulled the MicroSD card from one of my 3B's while it was powered on.  The card has not been readable or recoverable, having tried several methods.  

I have a 2-drive docking station that I use for backups.  I also have some the old drives I from my notebook computers as I had replaced them before they died just because they were getting older.  So I'm thinking of using one of those old drives dedicated to just SD backups.  The only pain now is the time it takes to back up an entire image.   And then double that time if want to write it back out to a new SD card.   I suppose I can live with those times, it's just pain knowing that the vast majority of the SD card image is just empty formatted space. 

I tried compressing the files, but when I started the process I could see that compressing them would take even longer than writing them.  So I don't see any real advantage there.  It might reduce the file size significantly but it would also increase the amount of processing time significantly too.  So not sure it's worth the trade-off.

Looks like we're stuck with having to back up the whole SD card.  That being the case I'm just going to buy more 16GB SD cards instead of the 32GB.  I don't yet need the extra space, and I'm not sure if I ever will.  If I need more space it will probably just be space for data storage, and I could do that on a different storage unit instead of on the OS SD card.    In fact, I could probably get by with 8GB SD card for the OS.  But since I already have 16GB cards I may as well stick with that size to keep everything uniform.   Had I known this when first starting out I would have gone with 8GB cards from the get go.   That's more than enough for the OS.  At least for my purposes.   In fact, I could probably get by with a 4GB OS card actually. 

I might actually try using a 4GB card for the Raspberry Pi OS and see how that goes.   If that works for me I might just end up switching all my pies over to a 4GB OS card and using and external SD card for data.  That way I can back up system images a lot faster.

I think it really depends on the application. When I started out with my 3D printers, 4G was plenty big enough for the OctoPrint, with it barely taking up 2G of those cards, but, since the 4G cards are getting harder to find, I've moved the Octo up to 8G, and then the Raspbian Strech full is almost 8G without even installing anything on it. Then of course, I wanted bigger cards for Mycroft and Picroft, and since SNIPS runs mostly locally, I needed bigger cards there too, and the Jetbot image for the Nano is almost 64 gigs right off the bat, so I needed to move that one up to 128

So, it all depends on the application, and like @Robo Pi said, the space you save during compression is time you waste compressing it.

Personally, I'd rather save the time than the space

Yes, It either comes down to saving time or saving space. The only way I can think of the save both is to spend a lot on a high-speed processor.

No silver bullets that I know of on this one.

Posted by: @twobits

Yes, It either comes down to saving time or saving space. The only way I can think of the save both is to spend a lot on a high-speed processor.

 

No silver bullets that I know of on this one.

That's actually only mostly true. I found a few videos a few days ago comparing high speed transfer rates which I found interesting

The problem with this information is that it doesn't cover the problem of creating the image, and then the additional problem of local network speed, the NAS itself, or the RPi that I'm personally using to run the NAS. All of these will obviously effect the time involved in our backup process. Still, as you said, no silver bullet, but, any increase in speed is still a saving in time

This tells me that I would personally potentially save time by upgrading my NAS-Pi to a model 4 with the USB3.0, and upgrading all the wiring in my house to CAT6a, but, that still leaves the question of creating the image in the first place. See, the youtube videos that I showed only show data transfer rates, not image creation, which, in my experience, is the weakest link in this chain, complicated by the fact that I don't have USB3.0 on the computer that's creating the image

Now that I'm writing it down, I see where I could save more time.

I need a USB3.0 card for my computer

Accessing over WiFi from PC to Rpi3

With host name and ROS master URI

NOT Able to connect to rpi3,  always says not able to connect to the gateway 22....... with ssh...

Can you suggest what is wrong

@sdey76

See the Turtlebot thread

@robo-pi which are the best ways for backup the whole SD card of the Pi ? (And also restore it)

Does is the remote backup better over backup locally on the Pi ?

I seen the most popular solution is using an USB to a portable SD. However I like the idea of saving at least on an USB, as I don't have to buy an extra adapter.

The most preferable solution would be a backup remotely from the PC, and stored on the PC / an extra hard drive.

Do you know some good websites or video tutorials for it ? 

You can back up an SD card on Linux using the "dd" command, but I don't know if the card has to be unmounted for it to work. Do you need to backup the whole card, or could you just backup specific files?