Notifications
Clear all

Raspberry Pi System Image Backup Questions.

13 Posts
8 Users
0 Likes
3,324 Views
Robo Pi
(@robo-pi)
Robotics Engineer
Joined: 5 years ago
Posts: 1669
Topic starter  

Any Pi experts out there?

I currently have 5 Raspberry Pies.  Four A+ models and one B+ model.   Use the B+ as my development prototyping machine, and the four A+ models as the brains of four different robot projects.   Each of my Raspberries runs the same Raspbian OS.    However they each have different configurations and various software packages unique to them.   To make a long story short, I'd like to back up all their SD cards.

They currently all have a 16GB SD card.  I was recently thinking about upgrading them to 32GB cards, but not because I need the space.  Simply because I can now purchase 32GB SD cards for basically the same prices as the 16GB cards.

However there are two problems with this.  First is that the 32GB cards obviously take up twice the space to back up on my hard drive.  The second problem is that they would also take twice as long to back up.   The 16GB cards already take quite a while to backup.  I'm using Win 32 Disk Imager to back up and restore or make copies of the system SD cards.

My question:

The actual Raspbian OS is anywhere from 0.5GB to 2GB depending on which version I choose to install.  Is there any way that I can just copy and image of the OS alone and restore it without having to copy the entire SD card image?

It would sure go a lot quicker if I only had to backup and restore 2GB instead of 16GB.  The rest of the disk is mostly data stories, or contains programs that I wrote for my robots.  I can back up and restore those programs without having to make a whole SD card image.

So I'm just wondering if there is a way to back up the Raspbian OS and all of its configuration data without having to back up the entire SD card.  Anyone know if that's possible?

Thanks.

DroneBot Workshop Robotics Engineer
James


   
Quote
scottplude
(@scottplude)
Member
Joined: 5 years ago
Posts: 3
 

Following. I have the same issue. You would think getting a larger (faster?) sd card would be a good thing. NOPE! I suffer from the same problems you do. I wish the win32diskimager would allow forjust writing the space needed without writing an entire 16gb. It's a waste of time and space (and bandwidth if you store to the cloud).


   
ReplyQuote
(@twobits)
Member
Joined: 5 years ago
Posts: 113
 

Have you tried compressing your images with something like gzip after you have copied them from the SD card to your hard drive? I can generally compress a 16GB image down to 4-5GB with that method.

If you want to be more aggressive, there is a set of instruction at http://www.aoakley.com/articles/2015-10-09-resizing-sd-images.php . I have found the process to be more trouble then it is worth for me.

As a workaround, you might want to consider getting a home 2bay NAS. Storaging backup images is a pain.

If you are interested, I can play with the script listed above to see how effectively it works.


   
ReplyQuote
Spyder
(@spyder)
Member
Joined: 5 years ago
Posts: 846
 

I also have dozens of backup images of RPi installs, and I've tried a number of various tasks to make the process less painful, including the linux DD, but, at the end of the day, the easiest method I personally found was to use the win32disk imager like you do, and store them all on a NAS drive that I connected to a RPi running Samba

This was before I got my new router, which has a USB port on the back for exactly this purpose, but, an added bonus for the Pi was the fact that I could install a PLEX server on the Pi and can then share all my backed up movies to any device in my house, or even my phone if I'm away from home. Theoretically I could also give a code to my friends so that they could also have access to my movies, but, so far, nobody has taken me up on that offer


   
ReplyQuote
(@airbrushed)
Member
Joined: 5 years ago
Posts: 9
 

While I don't have a lot of different images, I also use Win32DiskImager.  Mac OS X is supposed to have an easy command line way to do this to, however, I have not tried it yet.

I started making backups after a well meaning person pulled the MicroSD card from one of my 3B's while it was powered on.  The card has not been readable or recoverable, having tried several methods.  


   
ReplyQuote
Robo Pi
(@robo-pi)
Robotics Engineer
Joined: 5 years ago
Posts: 1669
Topic starter  

I have a 2-drive docking station that I use for backups.  I also have some the old drives I from my notebook computers as I had replaced them before they died just because they were getting older.  So I'm thinking of using one of those old drives dedicated to just SD backups.  The only pain now is the time it takes to back up an entire image.   And then double that time if want to write it back out to a new SD card.   I suppose I can live with those times, it's just pain knowing that the vast majority of the SD card image is just empty formatted space. 

I tried compressing the files, but when I started the process I could see that compressing them would take even longer than writing them.  So I don't see any real advantage there.  It might reduce the file size significantly but it would also increase the amount of processing time significantly too.  So not sure it's worth the trade-off.

Looks like we're stuck with having to back up the whole SD card.  That being the case I'm just going to buy more 16GB SD cards instead of the 32GB.  I don't yet need the extra space, and I'm not sure if I ever will.  If I need more space it will probably just be space for data storage, and I could do that on a different storage unit instead of on the OS SD card.    In fact, I could probably get by with 8GB SD card for the OS.  But since I already have 16GB cards I may as well stick with that size to keep everything uniform.   Had I known this when first starting out I would have gone with 8GB cards from the get go.   That's more than enough for the OS.  At least for my purposes.   In fact, I could probably get by with a 4GB OS card actually. 

I might actually try using a 4GB card for the Raspberry Pi OS and see how that goes.   If that works for me I might just end up switching all my pies over to a 4GB OS card and using and external SD card for data.  That way I can back up system images a lot faster.

DroneBot Workshop Robotics Engineer
James


   
ReplyQuote
Spyder
(@spyder)
Member
Joined: 5 years ago
Posts: 846
 

I think it really depends on the application. When I started out with my 3D printers, 4G was plenty big enough for the OctoPrint, with it barely taking up 2G of those cards, but, since the 4G cards are getting harder to find, I've moved the Octo up to 8G, and then the Raspbian Strech full is almost 8G without even installing anything on it. Then of course, I wanted bigger cards for Mycroft and Picroft, and since SNIPS runs mostly locally, I needed bigger cards there too, and the Jetbot image for the Nano is almost 64 gigs right off the bat, so I needed to move that one up to 128

So, it all depends on the application, and like @Robo Pi said, the space you save during compression is time you waste compressing it.

Personally, I'd rather save the time than the space


   
ReplyQuote
(@twobits)
Member
Joined: 5 years ago
Posts: 113
 

Yes, It either comes down to saving time or saving space. The only way I can think of the save both is to spend a lot on a high-speed processor.

 

No silver bullets that I know of on this one.


   
ReplyQuote
Spyder
(@spyder)
Member
Joined: 5 years ago
Posts: 846
 
Posted by: @twobits

Yes, It either comes down to saving time or saving space. The only way I can think of the save both is to spend a lot on a high-speed processor.

 

No silver bullets that I know of on this one.

That's actually only mostly true. I found a few videos a few days ago comparing high speed transfer rates which I found interesting

The problem with this information is that it doesn't cover the problem of creating the image, and then the additional problem of local network speed, the NAS itself, or the RPi that I'm personally using to run the NAS. All of these will obviously effect the time involved in our backup process. Still, as you said, no silver bullet, but, any increase in speed is still a saving in time

This tells me that I would personally potentially save time by upgrading my NAS-Pi to a model 4 with the USB3.0, and upgrading all the wiring in my house to CAT6a, but, that still leaves the question of creating the image in the first place. See, the youtube videos that I showed only show data transfer rates, not image creation, which, in my experience, is the weakest link in this chain, complicated by the fact that I don't have USB3.0 on the computer that's creating the image

Now that I'm writing it down, I see where I could save more time.

I need a USB3.0 card for my computer


   
ReplyQuote
soumitra
(@soumitra)
Member
Joined: 5 years ago
Posts: 61
 

Accessing over WiFi from PC to Rpi3

With host name and ROS master URI

NOT Able to connect to rpi3,  always says not able to connect to the gateway 22....... with ssh...

Can you suggest what is wrong

 


   
ReplyQuote
Spyder
(@spyder)
Member
Joined: 5 years ago
Posts: 846
 

@sdey76

See the Turtlebot thread


   
ReplyQuote
(@cristishor201)
Member
Joined: 3 years ago
Posts: 23
 

@robo-pi which are the best ways for backup the whole SD card of the Pi ? (And also restore it)

Does is the remote backup better over backup locally on the Pi ?

I seen the most popular solution is using an USB to a portable SD. However I like the idea of saving at least on an USB, as I don't have to buy an extra adapter.

The most preferable solution would be a backup remotely from the PC, and stored on the PC / an extra hard drive.

Do you know some good websites or video tutorials for it ? 


   
ReplyQuote
(@yurkshirelad)
Member
Joined: 3 years ago
Posts: 493
 

You can back up an SD card on Linux using the "dd" command, but I don't know if the card has to be unmounted for it to work. Do you need to backup the whole card, or could you just backup specific files?


   
ReplyQuote