Talk:OSE Linux Persistence
https://en.m.wikipedia.org/wiki/UNetbootin claims to have a persistence option for ubuntu distros. I couldn't get it to work. --Dorkmo (talk) 19:33, 14 July 2018 (UTC)
I've used persistence on USB for over 10 years now as my primary system (4+ hours/day) and will share some thoughts.
It has been great, I can move my programs, data, and OS from workstation to workstation with a breeze. From a desktop to a laptop, a library computer, or computer at work at times even. I'll never going back to installing Linux on a HDD. I use HDDs on workstations for storing data, or even making a ~8GB swap file if I need more system memory for some big task (e.g. big FreeCAD model).
I don't use unetbootin, ISOs, or any other setups, just Debian's 'persistence' partition. On Ubuntu the labeling of this partition may be 'casper-rw' instead of 'persistence'. It is very simple, you have the second partition labeled 'persistence' with a configuration file, and Debian's live-boot scripts handle everything.
live-boot Squashfs/Persistence
- 'live' partition contains the squashfs, initrd and vmlinuz in /boot. Can only be mounted ro unless 'toram' kernel parameter used.
- 'persistence' partition is mounted rw.
Benefits:
- With a squashfs/persistence install instead of a full install to USB, you'll see a much faster boot, because the 'live' partition is compressed (squashfs) and is thus sequentially read into memory faster compared to random reads of various uncompressed files from the USB as they're needed during the boot process. Mine boots within 20 or so seconds.
- When running the system with day to day tasks, the performance and responsiveness of a squashfs/persistence setup is better than a full install, because the squashfs may be loaded to memory (toram) occupying around 600-800MB RAM, and is thus not a bottleneck on reads/writes to the USB slowing the system down like you'd see in a full install to USB. This translates into better framerate playing games like Starcraft 2, which I've run from this flash drive, and a more responsive system in general (to mouse clicks, key presses, etc).
Limitations:
- A 64-bit install won't work on a 32-bit machine
- A USB with /etc/X11/xorg.conf set to use an NVIDIA GPU driver won't be as mobile:
- e.g. Will not boot graphically on an Intel GPU system but to a virtual console.
- This can be mitigated by editing xorg.conf while on the Intel system's virtual console and rebooting to get the graphical interface. Or possibily without rebooting with some wizardry like rmmod and then startx, but I don't recall off the top of my head. And you have to redo this in reverse going back to the NVIDIA system. Maybe this is just a limitation of proprietary GPU drivers.
- Copy-on-write aufs/overlayfs (whetever distro has chosen) has some limitations:
- Read-only 'live' partition needs to be updated whenever libc or linux-image are updated, or the system won't boot. So it needs to be mounted read-write and in order to do so 'toram' kernal boot parameter must be used. I work around this problem manually via inspecting apt output before installing anything, but this chore can be mitigated via an apt hook which I've not written yet.
- 'live' partition containing the squashfs should only have packages which are not updated often, being core packages, and not something like firefox which is updated often. This will minimize the size of the squashfs.
- When Debian stable is updated to the next revision, the linux-image must be updated, and the squashfs recompressed. To do this, all non-core packages must be removed, the distro upgraded, the squashfs recompressed, and then non-core packages reinstalled. This can also be scripted and I have some non-robust helper scripts I've written.
- Flash memory has limitations, where sequential reads/writes are ~10000x faster than random reads/writes. Due to this, there should always be at least 3GB free on the 'persistence' filesystem, or else files become fragmented and the any application blocking on reads/writes will hang from very slow reading and writing. In this circumstance it may be a number of minutes before such applications respond again, but you're free to use this time to do other things with already opened applications, like use the web browser or read opened documents.
I personally use a lightweight Debian distro (not Ubuntu!) to minimize RAM footprint, minimize squashfs size, and maximize responsiveness. It is all instantaneous on USB3 (was a bit sluggish on USB2 on an older 4GB stick when I first started). I recommend at least a 32GB stick.
Once the apt hook is implemented and distro-upgrade script polished, this should be a very nice option for interested users.
There are tons of persistence guides, which unfairly dismiss this live-boot squashfs/persistence setup because they don't know how to mitigate/handle the upgrade problem as I do, because making a squashfs is not something well known which requires a lot of plumbing. Or they suggest other unnecessarily complex setups or make suggestions without having investigated performance implications (e.g. suggesting a full install to USB).
I do not recommend modern Samsung, Sandisk, or any other cheap off-brands for this "heavy" use. Good chance they'll fail within a year and be too hot to touch. Buy from Toshiba/Kioxia, the inventors of flash memory. Yes their performance as shown in benchmarks isn't as good, nor are they inexpensive, but they are reliable and will last years. Do not skimp to save a few $ buying a Sandisk/Samsung on a sale for this use-case, I'm speaking from experience!
Encryption
Pros:
- Sure it may be useful if for example, you forget the drive in an internet cafe or library. Or have something you don't want to turn up in investigation.
Cons:
- It costs additional CPU & energy usage.
- Encryption offers zero "online"/mounted protection, e.g. encryption does not protect against a web browser vulnerability allowing disk access. Mental energy would be better spent on 'hardening' the system.
- Harder to recover data when things break. Big headache.
--Andrewusu (talk) 20:40, 4 November 2020 (UTC)
I will write more thoughts, I see this is an ongoing need/interest OSE_Linux_Log. Today I have ordered two Toshiba USB flash drives. I will write a guide here for preparing any Debian distro for USB persistence with optimal performance (squashfs & persistence partition) for interested readers.
GUIDE
Prepare squashfs
Download ISO of desired debian distro (e.g. OSE Linux). Put it at /tmp/
Mount the ISO, and get filesystem.squashfs, initrd.img, vmlinuz off of it. Also config, System.map and filesystem.packages.xz if you want to follow convention.
# Temp directories sudo mkdir /media/iso.squashfs /media/iso /media/myadditions /media/overlay sudo mount -o loop /tmp/debian-distro-i386.iso /media/iso # copy filesystem.squashfs, initrd.img, and vmlinuz off of there. cd /media/iso cp live/filesystem.squashfs /tmp #I forget what folder these are in, and they may have slight naming differences cp boot/vmlinuz /tmp cp boot/initrd.img /tmp sudo umount /media/iso
Now we mount the squashfs we have taken from the ISO in a way to make a new smaller squashfs:
sudo mount -t squashfs /tmp/filesystem.squashfs /media/iso.squashfs # rw branch must be before ro, if out of order you will see error and /media/overlay will be ro instead of rw # We do this to make new smaller squashfs (with stuff removed). sudo mount -t aufs -o br=/media/myadditions=rw:/media/iso.squashfs=ro -o udba=reval none /media/overlay/
Now we chroot into the squashfs, so that we can run apt within it and remove packages:
# resolv.conf and mount binds are to remove error/warning messages when removing packages with apt sudo cp /etc/resolv.conf /media/overlay/etc/resolv.conf sudo mount --bind /dev/ /media/overlay/dev/ sudo mount --bind /dev/pts /media/overlay/dev/pts sudo mount --bind /proc /media/overlay/proc # Now chroot into the squashfs so we can use apt within it to remove packages sudo chroot --userspec root:root /media/overlay
Now we gut the iso (filesystem.squashfs on the iso) of packages we don't want, are frequently updated (e.g. web browser), with particular scrutiny of packages that are large. All of those can they can go onto persistence partition. We remove them now, and install them after the squashfs has been made, or as needed.
# Run this to see packages installed ordered by size, then apt-get remove or purge stuff. dpkg-query -Wf 'Size\t\n' | sort -n | less # I think I installed deborphan and localepurge to help clean stuff. apt-get install deborphan localepurge # A subset of stuff I removed from a different Debian distro, but I removed some firmware packages so that reduces portability: apt-get purge vlc-l10n firefox-esr filezilla apt-get purge p7zip-full libgoffice-0.10-10-common firmware-ti-connectivity firmware-cavium firmware-qlogic gnumeric-common libgsf-1-114 libgsf-1-common p7zip pxlib1 apt-get purge firmware-brcm80211 aptitude-common aptitude gstreamer1.0-plugins-bad gstreamer1.0-plugins-good synaptic apt-get purge transmission-gtk hexchat-common libevent-2.0-5 libminiupnpc10 libnatpmp1 transmission-common apt-get purge samba-libs libavahi-glib1 libcdio-cdda1 libcdio-paranoia1 libgd3 libgdata-common libgdata22 libgoa-1.0-0b libgoa-1.0-common libgphoto2-6 libgphoto2-port12 libldb1 libmtp-common libmtp9 libnfs8 liboauth0 libpython2.7 libtalloc2 libtevent0 libwbclient0 python-talloc apt-get purge libflite1 libass5 libatomic1 libavformat57 libbs2b0 libchromaprint1 libebur128-1 libfftw3-double3 libgme0 libopencv-core2.4v5 libopencv-imgproc2.4v5 libopenmpt0 libpgm-5.2-0 libpostproc54 librubberband2 libsodium18 libssh-gcrypt-4 libswscale4 libtbb2 libzmq5 apt-get purge gstreamer1.0-plugins-ugly liba52-0.7.4 libcdio13 libdvdnav4 libdvdread4 libmad0 libmpeg2-4 libmpg123-0 libopencore-amrnb0 libopencore-amrwb0 libsidplay1v5
Some tips:
# After you're done pruning things, run deborphan a couple times removing things it tells you to. # Deborphan helps you find "orphan" packages not needed by anything else, which take up space. # Such orphaned packages may have been necessary by a previously installed package. deborphan apt-get autoremove apt-get clean # I recommend vim.tiny over vim, vim.tiny much smaller, also install whatever else your heart desires that is small and infrequently updated. # These will be available in 'live' mode (if installed when in 'persistence' mode, it won't be available in plain 'live' mode): apt-get install vim-tiny ln -s /usr/bin/vi /usr/local/bin/vim # On a Debian distro I've done this before on, the live scripts may have a heart attack if the user named 'user' doesn't exist... I guess this setup does things out of the order the live scripts expect before they setup the 'user' account, so to fix that problem we make it here within the chroot: #adduser user # If you boot up and it prompts you to log in and says 'Authentication Failure' and virtual terminals don't let you log in either, this is the problem, 'user' account was not set up. # There might be other trivial bugs I forgot about fixing. Like apt may complain warnings about /etc/default/locale being not set up. That problem is only within the chroot and won't exist after booting up. # Within the chroot into the squashfs, remove stuff in /boot, we already have copies. vmlinuz and initrd.img will go into the live partition, if they are duplicated within the squashfs it does nothing but waste space. rm -r /boot
Now that we have a set of "core" packages to go in the squashfs, of minimal size, on the 'live' partition, we must make our new squashfs!
#Print out a list of "core" packages we have, we'll need them for reference later. Lets put them in filesystem.packages.xz:
dpkg-query -W --showformat='${Package}\t${Version}\n' | xz > /tmp/filesystem.packages.xz
#TODO, I have messy scripts and it is late night
exit #exit the chroot
#preview: https://gist.github.com/AndrewSmart/90eb186aea08db8f1426
Hopefully the squashfs you made is around 800MB or so. Let us say your flash drive has 100MB/s sequential read speed. That is 8 seconds. How long do you think yours will take to boot?
Cleanup!
sudo umount /media/overlay/dev/pts sudo umount /media/overlay/dev/ sudo umount /media/overlay/proc sudo umount /media/overlay sudo umount /media/iso.squashfs sudo umount /media/iso sudo rm -r /media/myadditions sudo rmdir /media/iso /media/iso.squashfs /media/overlay
Now we copy the vmlinuz, initrd.img we got from the ISO, and the squashfs we made onto the live partition!
#mount your live partition to /media/live mkdir /media/live/live/ cp /tmp/vmlinuz /media/live/live/ cp /tmp/initrd /media/live/live/ cp /tmp/filesystem.squashfs /media/live/live
Make sure to fill in this template /media/live/boot/grub/grub.cfg on 'live' partition with your appropriate UUIDs (unique to the partitions YOU made):
menuentry "DEBIAN 64bits - by UUID - Persistence" {
#Enter the UUID of your boot partition (this is where grub and your kernel reside)
set uuid_grub_boot=ab3920b1-2d3f-4bf8-9da1-27bc2b5c9e2f
#Enter the UUID of the persistence partition.
set uuid_os_root=aedab356-05bc-4e30-ba77-6317b0210c7e
#Here we set the grub "root" variable by locating the uuid of the root partition identified above
search --no-floppy --fs-uuid $uuid_os_root --set=root
#Here we set a custom variable grub_boot by locating the uuid of the boot partition identified above
search --no-floppy --fs-uuid $uuid_grub_boot --set=grub_boot
#Here's the magic. We test to see if the boot and root partitions have the same UUID.
#If they do we append /boot to the $grub_boot variable. For ex. (hd0,1) becomes (hd0,1)/boot.
if [ $uuid_grub_boot == $uuid_os_root ] ; then
set grub_boot=$grub_boot/boot
fi
linux ($grub_boot)/live/vmlinuz boot=live config quiet noeject toram persistence persistence-storage=filesystem usbcore.autosuspend=-1 live-media=/dev/disk/by-uuid/$uuid_grub_boot elevator=as
initrd ($grub_boot)/live/initrd.img
}
menuentry "DEBIAN 64bits - by UUID - Live" {
#Enter the UUID of your boot partition (this is where grub and your kernel reside)
set uuid_grub_boot=ab3920b1-2d3f-4bf8-9da1-27bc2b5c9e2f
#Enter the UUID of the persistence partition.
set uuid_os_root=aedab356-05bc-4e30-ba77-6317b0210c7e
#Here we set the grub "root" variable by locating the uuid of the root partition identified above
search --no-floppy --fs-uuid $uuid_os_root --set=root
#Here we set a custom variable grub_boot by locating the uuid of the boot partition identified above
search --no-floppy --fs-uuid $uuid_grub_boot --set=grub_boot
#Here's the magic. We test to see if the boot and root partitions have the same UUID.
#If they do we append /boot to the $grub_boot variable. For ex. (hd0,1) becomes (hd0,1)/boot.
if [ $uuid_grub_boot == $uuid_os_root ] ; then
set grub_boot=$grub_boot/boot
fi
linux ($grub_boot)/live/vmlinuz boot=live config quiet noeject toram usbcore.autosuspend=-1 live-media=/dev/disk/by-uuid/$uuid_grub_boot
initrd ($grub_boot)/live/initrd.img
}
And don't forget to make the persistence.conf file '/ union' on the persistence partition:
#cd to wherever you have mounted your persistence partition, e.g. /media/persistence cd /media/persistence echo "/ union" > persistence.conf
I don't recall the grub install command to the live partition. I'll run through this guide with OSE Linux and a USB drive when it arrives and post complete details of getting it to work.
Congratulations! You now have an Linux OS on your USB with persistence that will boot within seconds and reliably outperforms the competition!
Upgrading
Q: What do we do when linux-image is upgraded? A: apt will want to update /boot/vmlinuz and/or /boot/initrd.img (I forget). So we need to do things in order to allow this to happen. We must boot with 'toram' and also remount 'live' as rw.
#TODO, I wrote this elsewhere need to copy here
In this upgrade process apt might rename vmlinuz to say vmlinuz-3.16.0-4-amd64, and initrd similarly. It didn't always do this. But, you must match the names of those files with what is listed in grub.cfg! Either use 'vmlinuz' or whatever name the apt script renamed them to.
Q: What do we do when libc is upgraded? Be it a security update, or when Debian Stable advances every couple years. A: We will want to do a resquash:
- We need to to the same as above, 'toram' and remount 'live' as rw.
- dpkq-query our currently installed packages, and compare to /boot/filesystem.packages.xz we made earlier.
- Remove all non "core" packages, they should be cached in /var/cache/apt/archives/ so they don't need to be downloaded again, just reinstalled after we do the resquash.
#TODO #preview script #1 https://gist.github.com/AndrewSmart/90eb186aea08db8f1426 #preview script #2, cleanup https://gist.github.com/AndrewSmart/2f67f79f6f1922c4556f
Troubleshooting
Keep in mind with the USB we can still boot either in 'live' mode, or 'persistence' mode depending on the GRUB entry we choose. This is useful in case something breaks in 'persistence' mode, like libc was upgraded in 'persistence' but libc on 'live' is still old! This will result in segmentation faults... a kernel panic, or whatever, it won't boot. We can access all of the software in the squashfs in 'live mode'. Eventually I'd like to write an apt hook to prevent the update of libc unless the system is prepared ('toram', and 'live' partition is mounted rw, the hook won't trigger unless 'persistence' kernel parameter is used).