Talk:OSE Linux Persistence
https://en.m.wikipedia.org/wiki/UNetbootin claims to have a persistence option for ubuntu distros. I couldn't get it to work. --Dorkmo (talk) 19:33, 14 July 2018 (UTC)
Andrewusu's Thoughts
I've used persistence on USB for over 10 years now as my primary system (4+ hours/day) and will share some thoughts. Way back with USB2 I spent a lot of time investigating performance, ways to speed up boot time and system responsiveness. Keep in mind benchmarks of any flash memory device, where sequential reading is 10-20x faster than non-sequential reads (possibly more depending on the device). By following this setup I describe below you take advantage of those fast reads for a quick boot and a very responsive OS.
It has been great, I can move my programs, data, and OS from workstation to workstation with a breeze. From a desktop to a laptop, a library computer, or computer at work at times even. I'll never going back to installing Linux on a HDD. I use HDDs on workstations for storing data, or even making a ~8GB swap file if I need more system memory for some big task (e.g. big FreeCAD model).
I don't use unetbootin, ISOs, or any other setups, just Debian's 'persistence' partition. On Ubuntu the labeling of this partition may be 'casper-rw' instead of 'persistence'. It is very simple, you have the second partition labeled 'persistence' with a configuration file, and Debian's live-boot scripts handle everything.
live-boot Squashfs/Persistence
- 'live' partition contains the squashfs, initrd and vmlinuz in /live. Can only be mounted ro unless 'toram' kernel parameter used.
- 'persistence' partition is mounted rw.
Benefits:
- With a squashfs/persistence install instead of a full install to USB, you'll see a much faster boot, because the 'live' partition is compressed (squashfs) and is thus sequentially read into memory faster compared to random reads of various uncompressed files from the USB as they're needed during the boot process. Mine boots within 20 or so seconds.
- When running the system with day to day tasks, the performance and responsiveness of a squashfs/persistence setup is better than a full install, because the squashfs may be loaded to memory (toram) occupying around 600-800MB RAM, and is thus not a bottleneck on reads/writes to the USB slowing the system down like you'd see in a full install to USB. This translates into better framerate playing games like Starcraft 2, which I've run from this flash drive, and a more responsive system in general (to mouse clicks, key presses, etc).
Limitations:
- A 64-bit install won't work on a 32-bit machine
- A USB with /etc/X11/xorg.conf set to use an NVIDIA GPU driver won't be as mobile:
- e.g. Will not boot graphically on an Intel GPU system but to a virtual console.
- This can be mitigated by editing xorg.conf while on the Intel system's virtual console and rebooting to get the graphical interface. Or possibily without rebooting with some wizardry like rmmod and then startx, but I don't recall off the top of my head. And you have to redo this in reverse going back to the NVIDIA system. Maybe this is just a limitation of proprietary GPU drivers.
- Copy-on-write aufs/overlayfs (whetever distro has chosen) has some limitations:
- Read-only 'live' partition needs to be updated whenever libc or linux-image are updated, or the system won't boot. So it needs to be mounted read-write and in order to do so 'toram' kernal boot parameter must be used. I work around this problem manually via inspecting apt output before installing anything, but this chore can be mitigated via an apt hook which I've not written yet.
- 'live' partition containing the squashfs should only have packages which are not updated often, being core packages, and not something like firefox which is updated often. This will minimize the size of the squashfs.
- When Debian stable is updated to the next revision, the linux-image must be updated, and the squashfs recompressed. To do this, all non-core packages must be removed, the distro upgraded, the squashfs recompressed, and then non-core packages reinstalled. This can also be scripted and I have some non-robust helper scripts I've written.
- Flash memory has limitations, where sequential writes are 5000x-10000x faster than random writes. Due to this, there should always be at least 2GB free on the 'persistence' filesystem, or else risk files becoming fragmented and the any application blocking on reads/writes will hang from very slow reading and writing. In this circumstance it may be a number of minutes before such applications respond again, but you're free to use this time to do other things with already opened applications, like use the web browser or read opened documents.
I personally use a lightweight Debian distro (not Ubuntu!) to minimize RAM footprint, minimize squashfs size, and maximize responsiveness. It is all instantaneous on USB3 (was a bit sluggish on USB2 on an older 4GB stick when I first started). I recommend at least a 32GB stick.
Once the apt hook is implemented and distro-upgrade script polished, this should be a very nice option for interested users.
There are tons of persistence guides, which unfairly dismiss this live-boot squashfs/persistence setup because they don't know how to mitigate/handle the upgrade problem as I do, because making a squashfs is not something well known which requires a lot of plumbing. Or they suggest other unnecessarily complex setups or make suggestions without having investigated performance implications (e.g. suggesting a full install to USB).
I do not recommend modern Samsung, Sandisk, or any other cheap off-brands for this "heavy" use. Good chance they'll fail within a year and be too hot to touch. Buy from Toshiba/Kioxia, the inventors of flash memory. Yes their performance as shown in benchmarks isn't as good, nor are they inexpensive, but they are reliable and will last years. Do not skimp to save a few $ buying a Sandisk/Samsung on a sale for this use-case, I'm speaking from experience!
Encryption
Pros:
- Sure it may be useful if for example, you forget the drive in an internet cafe or library. Or have something you don't want to turn up in investigation.
Cons:
- It costs additional CPU & energy usage.
- Encryption offers zero "online"/mounted protection, e.g. encryption does not protect against a web browser vulnerability allowing disk access. Mental energy would be better spent on 'hardening' the system.
- Harder to recover data when things break. Big headache.
--Andrewusu (talk) 20:40, 4 November 2020 (UTC)
I will write more thoughts, I see this is an ongoing need/interest OSE_Linux_Log. Today I have ordered two Toshiba USB flash drives. I will write a guide here for preparing any Debian distro for USB persistence with optimal performance (squashfs & persistence partition) for interested readers.
GUIDE
Prepare squashfs
Download ISO of desired debian distro (e.g. OSE Linux). Put it at /tmp/
Mount the ISO, and look for filesystem.squashfs, initrd.img, vmlinuz on it. Also note the location of config, System.map and filesystem.packages.xz, we'll copy or make these to follow convention.
Make temporary directories:
sudo mkdir /media/filesystem.squashfs /media/iso /media/myadditions /media/overlay
Mount the iso:
sudo mount -o loop /tmp/debian-distro-i386.iso /media/iso
Mount filesystem.squashfs on the iso:
sudo mount -t squashfs /media/iso/live/filesystem.squashfs /media/filesystem.squashfs
Prepare the union filesystem (so that we're able to edit the squashfs on the iso). aufs on one system I tried this on wasn't happy with /media/myadditions not being a separate device, so I made it tmpfs I think this may be due to a sanity check aufs does to ensure it doesn't end up with a recursive problem.
sudo mount -t tmpfs -o size=1024m none /media/myadditions
We use aufs here in this guide instead of overlayfs as overlayfs hasn't been widely available until kernel 3.18, and aufs is available on earlier kernels (from kernel 3.9 late 2014 aufs supports xattr) via aufs-tools. Hopefully aufs supports extended attributes (xattr) good enough to not loose said attributes, which are important for hardening the system (extended attributes are I believe where Linux capabilities are stored/configured). Given this concern you're welcome to mount with overlayfs instead if you're able (lower would be /media/filesystem.squashfs, upper would be /media/myadditions, and workdir can be /media/myadditions/workdir/).
sudo mount -t aufs -o br=/media/myadditions=rw -o br=/media/filesystem.squashfs=ro -o udba=reval none /media/overlay/
chroot
Prepare the chroot, so that we can run apt within it and remove packages. resolv.conf and mount binds are to remove error/warning messages when installing packages with apt.
sudo cp /etc/resolv.conf /media/overlay/etc/resolv.conf sudo mount --bind /dev/ /media/overlay/dev/ sudo mount --bind /dev/pts /media/overlay/dev/pts #https://wiki.debian.org/chroot#A.2Fdev.2Fpts sudo mount --bind /proc /media/overlay/proc sudo mount --bind /sys /media/overlay/sys
Now chroot into the squashfs so we can use apt within it to remove packages:
sudo chroot --userspec root:root /media/overlay
Configure the locale, you'll get many warning messages in apt otherwise. You probably want to select UTF-8 for your country/language, for example I picked en_US.UTF-8. Press space bar to toggle each locale you want generated, then press enter to proceed. Then it asks if you want the locales you chose to be applied systemwide, forcing your selection on each user, which may be desired for desktop software which starts up before the user-specific ~/.profile is read:
dpkg-reconfigure locales
When following this guide on a persistent system (as I am) I need /lib/live/mount/medium/live to exist on the chroot so that update-initramfs will function, but you probably don't need to do the following. update-initramfs has logic/assumption problems regarding a read/write test that this works around:
mkdir -p /lib/live/mount/medium/live #If you ever run into problems with update-initramfs, this is what you run once the problem is fixed: dpkg --configure initramfs-tools
As we remove packages, such at btrfs-progs, update-initramfs will be triggered which should update /boot/vmlinuz and /boot/initrd, we need to make sure to copy these files out before we're done, as they're not compressed into the filesystem.squashfs and we need those files to boot.
`man live-boot` is a useful reference, ensure live-boot-doc is installed. Also the Debian live-manual is a useful reference.
apt-get install live-boot-doc
Remove non-core packages
Now we gut the filesystem.squashfs from the iso of packages that:
- We don't want.
- Are frequently updated (e.g. web browser).
- With particular scrutiny of packages that are large.
- Are not necessary to boot to a desktop.
All of those can they can go onto persistence partition. We remove them now, and install them after the squashfs has been made, or as needed. Examples of what should be considered "core" packages necessary to boot are: linux-image (this is the kernel), firmware, GPU driver, and X11 server. This way the USB will:
- Boot fast,
- The OS running from it will be maximally responsive,
- Will have minimal memory footprint,
- And be persistent!
Run this to see packages installed ordered by size, then apt-get remove or purge stuff, and ignore anything beginning with "lib":
dpkg-query -W -f '${Installed-Size}\t${Package}\n' | sort -n | less
To understand what a package is, you can search packages.debian.org. If the description there isn't informative enough, you can try clicking the links on the right hand side such as a project homepage, or looking through the files of the package. Ignore or don't pay much attention to any package beginning with "lib", with the exception of things like libc6, but apt will probably warn you about doing something silly like that with the prompt:
You are about to do something potentially harmful. To continue type in the phrase 'Yes, do as I say!'
An example of stuff I removed from a different Debian distro (BunsenLabs), but I removed some firmware packages so that reduces portability:
apt-get remove -y firefox-esr libjsoncpp1 apt-get remove -y libreoffice-core libreoffice-common coinor-libcbc3 coinor-libcgl1 coinor-libclp1 coinor-libcoinmp1v5 coinor-libcoinutils3v5 coinor-libosi1v5 fonts-opensymbol libabw-0.1-1 libboost-atomic1.67.0 libboost-chrono1.67.0 libboost-date-time1.67.0 libboost-filesystem1.67.0 libboost-locale1.67.0 libboost-thread1.67.0 libclucene-contribs1v5 libclucene-core1v5 libcmis-0.5-5v5 libcolamd2 libe-book-0.1-1 libeot0 libepubgen-0.1-1 libetonyek-0.1-1 libexttextcat-2.0-0 libexttextcat-data libgpgmepp6 liblangtag-common liblangtag1 libmhash2 libmwaw-0.3-3 libmythes-1.2-0 libneon27-gnutls libnumbertext-1.0-0 libnumbertext-data libodfgen-0.1-1 liborcus-0.14-0 libraptor2-0 librasqal3 librdf0 librevenge-0.0-0 libstaroffice-0.0-0 libsuitesparseconfig5 libwpd-0.10-10 libwpg-0.3-3 libwps-0.4-4 libxmlsec1 libxmlsec1-nss libyajl2 lp-solve uno-libs3 ure apt-get remove -y firmware-atheros firmware-netronome firmware-iwlwifi firmware-brcm80211 firmware-qlogic firmware-ti-connectivity firmware-myricom firmware-intelwimax firmware-libertas dahdi-firmware-nonfree atmel-firmware firmware-cavium firmware-bnx2x firmware-qcom-media firmware-netxen firmware-samsung firmware-ipw2x00 firmware-siano firmware-ivtv firmware-bnx2 apt-get remove -y samba-libs libavahi-glib1 libcdio-cdda2 libcdio-paranoia2 libgd3 libgdata-common libgdata22 libgoa-1.0-0b libgoa-1.0-common libgphoto2-6 libgphoto2-port12 libldb1 liblmdb0 liboauth0 libpython2.7 libtalloc2 libtevent0 libwbclient0 python-talloc apt-get remove -y fonts-noto-core fonts-noto-cjk papirus-icon-theme bunsen-papirus-icon-theme gnome-icon-theme gdebi-core gir1.2-vte-2.91 python3-apt python3-chardet python3-debian python3-pkg-resources rm -r /usr/share/gdebi/GDebi /usr/lib/python3/dist-packages/aptsources /usr/lib/python3/dist-packages/apt/progress /usr/lib/python3/dist-packages/debian_bundle /usr/lib/python3/dist-packages/debian /usr/lib/python3/dist-packages/chardet/cli /usr/lib/python3/dist-packages/pkg_resources/extern /usr/lib/python3/dist-packages/pkg_resources/_vendor/packaging apt-get remove -y evince synaptic bubblewrap evince-common gnome-desktop3-data libbrotli1 libdjvulibre-text libdjvulibre21 libept1.5.0 libevdocument3-4 libevview3-3 libgnome-desktop-3-17 libgspell-1-1 libgspell-1-common libgxps2 libharfbuzz-icu0 libhyphen0 libjavascriptcoregtk-4.0-18 libkpathsea6 libspectre1 libsynctex2 libwebkit2gtk-4.0-37 libwebpdemux2 libwoff1 xdg-dbus-proxy zenity zenity-common apt-get remove -y vlc libaribb24-0 libbasicusageenvironment1 libcddb2 libdvbpsi10 libebml4v5 libgles2 libgroupsock8 libixml10 liblirc-client0 liblivemedia64 liblua5.2-0 libmad0 libmatroska6v5 libmtp-common libmtp9 libnfs12 libopenmpt-modplug1 libplacebo7 libprotobuf-lite17 libqt5x11extras5 libresid-builder0c2a libsdl-image1.2 libsdl1.2debian libsidplay2 libspatialaudio0 libupnp13 libusageenvironment3 libva-wayland2 libvlc-bin libvlc5 libxcb-xv0 vlc-bin vlc-data vlc-plugin-base vlc-plugin-qt vlc-plugin-video-output vlc-plugin-notify libvlccore9 apt-get remove -y file-roller p7zip-full p7zip libarchive13 libnautilus-extension1a xfburn libburn4 libexo-1-0 libisofs6 libjte1 unar gnustep-base-common gnustep-base-runtime gnustep-common libgnustep-base1.26 libobjc4 apt-get remove -y transmission-gtk libevent-2.1-6 libminiupnpc17 libnatpmp1 transmission-common filezilla filezilla-common libfilezilla0 libpugixml1v5 libwxbase3.0-0v5 libwxgtk3.0-0v5 yudit-common feh apt-get remove -y libpython2.7-stdlib libblas3 libgfortran5 libkeybinder0 liblapack3 libpython2.7-minimal libsodium23 lua-bit32 lua-expat lua-penlight lua-posix lua-socket lua5.2 python-apt-common python-minimal python2-minimal python2.7-minimal libavformat58 libmysofa0 libnorm1 libpgm-5.2-0 libpostproc55 librubberband2 libssh-gcrypt-4 libswscale5 libvidstab1.1 apt-get remove -y aptitude aptitude-common libboost-iostreams1.67.0 libboost-system1.67.0 libcwidget3v5 libxapian30 ristretto file libmagic-mgc libmagic1 hexchat-common apt-get install wpasupplicant network-manager --no-install-recommends #modem support apt-get remove -y modemmanager libmbim-glib4 libmbim-proxy libqmi-glib5 libqmi-proxy openssh-client libxatracker2 nitrogen libgtkmm-2.4-1v5 lvm2 dmeventd libaio1 libdevmapper-event1.02.1 liblvm2cmd2.03 libreadline5 nano geany geany-common ghostscript poppler-data libcupsimage2 libgs9-common libijs-0.35 libjbig2dec0 libpaper1 pcmciautils vdpau-va-driver arandr apt-get remove -y intel-microcode iucode-tool va-driver-all i965-va-driver xserver-xorg-video-intel libxvmc1 intel-media-va-driver libigdgmm5 xserver-xorg-video-qxl btrfs-progs cryptsetup-initramfs cryptsetup-bin cryptsetup-run apt-get remove -y python3.7-minimal dconf-cli distro-info-data gir1.2-atk-1.0 gir1.2-freedesktop gir1.2-gdkpixbuf-2.0 gir1.2-glib-2.0 gir1.2-gtk-3.0 gir1.2-pango-1.0 libfm-extra4 libgirepository-1.0-1 libmenu-cache-bin libmenu-cache3 libmpdec2 libpython3-stdlib libpython3.7-minimal libpython3.7-stdlib libxslt1.1 mime-support wmctrl rm -r /usr/lib/python3 apt-get remove -y iso-codes liba52-0.7.4 libaa1 libass9 libatomic1 libavc1394-0 libavfilter7 libavformat58 libbs2b0 libcaca0 libcdio18 libcdparanoia0 libchromaprint1 libdc1394-22 libdca0 libde265-0 libdv4 libdvdnav4 libdvdread4 libfaad2 libfftw3-double3 libflite1 libfluidsynth1 libgme0 libgssdp-1.0-3 libgstreamer1.0-0 libgupnp-1.0-4 libgupnp-igd-1.0-4 libiec61883-0 libilmbase23 libkate1 liblilv-0-0 libmjpegutils-2.1-0 libmms0 libmodplug1 libmpcdec6 libmpeg2-4 libmpeg2encpp-2.1-0 libmpg123-0 libmplex2-2.1-0 libmysofa0 libnice10 libnorm1 libofa0 libopenal-data libopenal1 libopencore-amrnb0 libopencore-amrwb0 libopenexr23 libopenmpt0 libpgm-5.2-0 libpoppler-glib8 libpoppler82 libpostproc55 libraw1394-11 librubberband2 libsbc1 libserd-0-0 libshout3 libsidplay1v5 libsndio7.0 libsord-0-0 libsoundtouch1 libspandsp2 libsratom-0-0 libsrtp2-1 libssh-gcrypt-4 libswscale5 libtumbler-1-0 libv4l-0 libv4lconvert0 libvidstab1.1 libvisual-0.4-0 libvo-aacenc0 libvo-amrwbenc0 libvulkan1 libwildmidi2 libzbar0 libzmq5 tumbler-common
Again, much of what was removed can be installed again after persistence is set up. These packages are conveniently listed as removed within dpkg utilities, I'll illustrate how to conveniently reinstall them later.
Misc tips
On a Debian distro I've done this before on, the live scripts had errors if the user named 'user' doesn't exist... I guess this setup does things out of the order the live scripts expect before they setup the 'user' account, so to fix that problem we add the user named 'user' here within the chroot. But on an older different distro I didn't have this problem, so I'm not sure if this will be necessary. If you boot up and it prompts you to log in and says 'Authentication Failure' and virtual terminals don't let you log in either, this is the problem, 'user' account was not set up.
adduser user
These options to live-boot will minimize the generated initramfs size:
cat<<-EOF > /etc/live/config.conf.d/40config.conf #man live-boot DISABLE_CDROM=true DISABLE_FAT=true DISABLE_FUSE=true DISABLE_NTFS=true EOF
Pruning tips
Install deborphan and localepurge to help clean stuff:
apt-get install deborphan localepurge
After you're done pruning things, run deborphan and remove things it tells you to as you desire. Deborphan helps you find "orphan" packages not needed by anything else, which take up space. Such orphaned packages may have been necessary by a previously installed package.
deborphan
Next run localepurge to remove space consuming files for locales you'll never use, I removed 160MB from my system:
localepurge
And finally, clear out apt temporary files:
apt-get autoremove apt-get clean
I recommend vim.tiny over vim, vim.tiny much smaller, also install whatever else your heart desires that is small and infrequently updated. Such small utilities will be available in 'live' mode (if installed when in 'persistence' mode, it won't be available in plain 'live' mode):
apt-get install vim-tiny
Hardening tips
I think it is best for /tmp to be mounted noexec, but many applications have bugs regarding this Double check the /etc/fstab file with a text editor after editing /etc/fstab:
echo "tmpfs /tmp tmpfs nosuid,nodev,noexec 0 0" >> /etc/fstab mount /tmp
Often apt packages during installation will require the ability to execute temporary files on /tmp, this is how we allow this:
echo "DPkg::Pre-Install-Pkgs {"mount -o remount,exec /tmp";};" >> /etc/apt/apt.conf.d/50remount
echo "DPkg::Post-Invoke {"mount -o remount /tmp";};" >> /etc/apt/apt.conf.d/50remount
A common thing people do, is charge their android phone, or other USB device via a USB port on the computer. Realize that this is physical access, which a compromised device may use to log in to Linux via root. I suspect I had fallen victim to in the past, a keylogger following a pattern I'd seen exhibited in the news, which I believe came after I'd plugged in a new android I had run questionable rooting software on, to my computer. I suggest both setting a root password, and disallowing this access, by editing /etc/securetty and commenting out root access via UART serial ports:
# UART serial ports #ttyS0 #ttyS1 #ttyS2 #ttyS3 #ttyS4 #ttyS5
Set the root password:
passwd
What has piqued my interest lately is Linux capabilities. Debian distributions should include both libcap2 and libcap-ng0 (RedHat's fork). And I'm looking for a more convenient open-source means of administration. What has me interested in the idea of a proactive defense against root-kits by removing module loading capabilities (CAP_SYS_MODULE) even for the root user, after the system had booted. It appears that lcap is no longer functional, it and many capabilities articles detail a system-wide capability bounding set in pre-2.6.25 kernels, but since kernel 2.6.25 capability bounding sets are now per-thread. The images here may help visually explain capabilities and their inheritance. What I have in mind may be within a custom /etc/init.d/ script calling setcap on init (pid 1) to remove the CAP_SYS_MODULE capability from the bounding set system-wide. The capabilities on any process can be viewed with `cat /proc/<pid>/status`.
Make filesystem.packages and filesystem.squashfs
Now that we have a set of "core" packages to go in the squashfs, of minimal size, on the 'live' partition, we must make our new squashfs!
Print out a list of "core" packages we have, we'll need them for reference later. Lets put them in filesystem.packages.xz:
dpkg-query -W --showformat='${Package}\t${Version}\n' | xz > /tmp/filesystem.packages.xz
Now let's make the squashfs:
cd /lib/live rm mount/medium/live/initrd.img rm mount/medium/live/vmlinuz rmdir -p mount/medium/live apt-get -y install squashfs-tools
Exclude stuff we don't want in the squashfs:
EXCLUDES='.wh* boot dev live-persistence.conf live lost+found media mnt proc .pulse* run home selinux sys tmp* var/cache var/lib/apt/lists var/tmp'
Directives to squashfs-tools to make pseudofiles, necessary to boot. Linux will make many pseudofiles in /dev when it boots, but not all, and through experimentation and observations I've come up with this list:
cat<<-EOF > /tmp/mksquashfs.pf boot d 755 root root dev d 755 root root dev/console c 662 root tty 5 1 dev/full c 666 root root 1 7 dev/kmem c 640 root kmem 1 2 dev/kmsg c 644 root root 1 11 dev/loop0 b 660 root disk 7 0 dev/loop1 b 660 root disk 7 1 dev/loop2 b 660 root disk 7 2 dev/loop3 b 660 root disk 7 3 dev/loop4 b 660 root disk 7 4 dev/loop5 b 660 root disk 7 5 dev/loop6 b 660 root disk 7 6 dev/loop7 b 660 root disk 7 7 dev/mem c 640 root kmem 1 1 dev/null c 666 root root 1 3 dev/port c 640 root kmem 1 4 dev/ptmx c 666 root tty 5 2 dev/pts d 755 root root dev/ram0 b 640 root disk 1 0 dev/ram1 b 640 root disk 1 1 dev/ram2 b 640 root disk 1 2 dev/ram3 b 640 root disk 1 3 dev/ram4 b 640 root disk 1 4 dev/ram5 b 640 root disk 1 5 dev/ram6 b 640 root disk 1 6 dev/ram7 b 640 root disk 1 7 dev/ram8 b 640 root disk 1 8 dev/ram9 b 640 root disk 1 9 dev/ram10 b 640 root disk 1 10 dev/ram11 b 640 root disk 1 11 dev/ram12 b 640 root disk 1 12 dev/ram13 b 640 root disk 1 13 dev/ram14 b 640 root disk 1 14 dev/ram15 b 640 root disk 1 15 dev/ram16 b 640 root disk 1 16 dev/random c 644 root root 1 8 dev/shm d 755 root root dev/tty c 662 root tty 5 0 dev/tty0 c 600 root tty 4 0 dev/urandom c 644 root root 1 9 dev/zero c 644 root root 1 5 home d 755 root root media d 755 root root mnt d 755 root root proc d 755 root root run d 755 root root run/lock d 1777 root root sys d 755 root root tmp d 1777 root root var/cache d 755 root root var/cache/apt d 755 root root var/cache/apt/partial d 755 root root var/cache/apt/lock f 640 root root echo "" var/lib/apt/lists d 755 root root var/lib/apt/lists/partial d 755 root root var/tmp d 1777 root root EOF
Now make the squashfs, it is compressed with xz compression:
mksquashfs / /tmp/filesystem.squashfs -comp xz -info -always-use-fragments -noappend -wildcards -pf /tmp/mksquashfs.pf -e ${EXCLUDES} > /tmp/mksquashfs.log
/tmp/mksquashfs.log will contain interesting details, such as the filesystem is now roughly 30% of its uncompressed size.
Hopefully the squashfs you made is around 800MB or less, mine for a different Debian distribution was about 300MB. Let us say your flash drive has 100MB/s sequential read speed, and your "core" filesystem was squashed down to 300MB, which will take 3 seconds to read into memory. How long do you think yours will take to boot?
Prepare the live partition
I don't recall the grub install command to the live partition. I'll run through this guide with OSE Linux and a USB drive when it arrives and post complete details of getting it to work. Here is my work in progress.
Here is my example command of how I made the ext4 filesystem for the live partition. The option ^has_journal disables the journal, important for flash memory longevity. Run from within the chroot:
sudo mkfs.ext4 -m 0 -b 4096 -N 2048 -L u365-live -E stride=4,stripe-width=2048 -O ^has_journal /dev/sdd1
Now from outside the chroot, we move the files in /media/overlay/boot/ (namely vmlinuz & initrd.img), and the squashfs we made onto our new live partition!
# live partition mounted to /media/live cd /media/live mkdir live/ mv /media/overlay/boot/vmlinuz* live/ mv /media/overlay/boot/initrd* live/ mv /media/overlay/tmp/filesystem.squashfs live/ mv /media/overlay/tmp/filesystem.packages.xz live/
Also copy files like System.map-4.19.0-9-amd64 and config-4.19.0-9-amd64 to /media/live/live, they should be in /media/overlay/boot or /media/iso/live/.
The file named config is a nice reference, it says what flags the kernel was built with, there are a vast number of options. This System.map file, stored on the read-only live partition may be useful, for example, finding if your system had been compromised with a rootkit.
Prepare GRUB on the live partition and USB device
The introduction of UEFI hardware complicates things. You're welcome to modify the instructions below if you desire to support UEFI functionality on such consumer hardware. The instructions below handles computers with traditional BIOS, with a traditional MBR, as well as UEFI hardware which supports such "legacy" standards.
From within the chroot, install non-uefi grub (grub-pc), and --no-install-recommends will result in os-prober not being installed. I think os-prober is workstation-specific and not useful to configure a USB device which may move between workstations. grub-pc will ask two questions, don't select anything and enter on Ok on the first, and press enter on Yes on the next to continue without --exclude.
apt-get install grub-pc --no-install-recommends
Install grub, to both the USB device and the live partition on the USB device. Where /dev/sdd below is the USB device (not a partition), and /media/live is where the live partition is mounted, yours will be named differently:
grub-install --target=i386-pc --root-directory=/media/live /dev/sdd
Now grub needs a configuration file. We can either craft one by hand or use grub-mkconfig, but grub-mkconfig doesn't really support this use-case.
Either (1) Handcraft grub.cfg
What follows is my legacy grub.cfg, which should work but I hope to make something workable within the newer /etc/grub.d/
Make sure to fill in this template /media/live/boot/grub/grub.cfg on 'live' partition with your appropriate UUIDs. The partitions you made will have unique UUIDs. You can determine which is which on your system by comparing `ls -l /dev/disk/by-uuid/` and ls -l `/dev/disk/by-label/`.
Also fill in the template with how your vmlinuz and initrd.img are named, which may not have a suffix of "-4.19.0-9-amd64".
Make note of the kernel parameters used here, such as 'toram' and 'persistence'. I'm not sure if elevator=as is the optimal choice, feel free to read about it.
You can also customize the locale, language, and keyboard layout here via kernel parameters here such as `locales=fr_FR.UTF-8 keyboard-layouts=fr keyboard-variants=bepo keyboard-model=tm2030usb`, but I think the console-setup package may be required, and perhaps these things are fine configured elsewhere.
menuentry "DEBIAN 64bits - by UUID - Persistence" {
#Enter the UUID of your boot partition (this is where grub and your kernel reside)
set uuid_grub_boot=ab3920b1-2d3f-4bf8-9da1-27bc2b5c9e2f
#Enter the UUID of the persistence partition.
set uuid_os_root=aedab356-05bc-4e30-ba77-6317b0210c7e
#Here we set the grub "root" variable by locating the uuid of the root partition identified above
search --no-floppy --fs-uuid $uuid_os_root --set=root
#Here we set a custom variable grub_boot by locating the uuid of the boot partition identified above
search --no-floppy --fs-uuid $uuid_grub_boot --set=grub_boot
#Here's the magic. We test to see if the boot and root partitions have the same UUID.
#If they do we append /boot to the $grub_boot variable. For ex. (hd0,1) becomes (hd0,1)/boot.
if [ $uuid_grub_boot == $uuid_os_root ] ; then
set grub_boot=$grub_boot/boot
fi
linux ($grub_boot)/live/vmlinuz-4.19.0-9-amd64 boot=live config quiet noeject toram persistence persistence-storage=filesystem usbcore.autosuspend=-1 live-media=/dev/disk/by-uuid/$uuid_grub_boot elevator=as
initrd ($grub_boot)/live/initrd.img-4.19.0-9-amd64
}
menuentry "DEBIAN 64bits - by UUID - Live" {
#Enter the UUID of your boot partition (this is where grub and your kernel reside)
set uuid_grub_boot=ab3920b1-2d3f-4bf8-9da1-27bc2b5c9e2f
#Enter the UUID of the persistence partition.
set uuid_os_root=aedab356-05bc-4e30-ba77-6317b0210c7e
#Here we set the grub "root" variable by locating the uuid of the root partition identified above
search --no-floppy --fs-uuid $uuid_os_root --set=root
#Here we set a custom variable grub_boot by locating the uuid of the boot partition identified above
search --no-floppy --fs-uuid $uuid_grub_boot --set=grub_boot
#Here's the magic. We test to see if the boot and root partitions have the same UUID.
#If they do we append /boot to the $grub_boot variable. For ex. (hd0,1) becomes (hd0,1)/boot.
if [ $uuid_grub_boot == $uuid_os_root ] ; then
set grub_boot=$grub_boot/boot
fi
linux ($grub_boot)/live/vmlinuz-4.19.0-9-amd64 boot=live config quiet noeject toram usbcore.autosuspend=-1 live-media=/dev/disk/by-uuid/$uuid_grub_boot
initrd ($grub_boot)/live/initrd.img-4.19.0-9-amd64
}
#With an unsafe shutdown or powerloss the persistence filesystem will have errors,
#since it has no journaling, we want to run fsck on it once
menuentry "DEBIAN 64bits - by UUID - Persistence fsck" {
#Enter the UUID of your boot partition (this is where grub and your kernel reside)
set uuid_grub_boot=ab3920b1-2d3f-4bf8-9da1-27bc2b5c9e2f
#Enter the UUID of the persistence partition.
set uuid_os_root=aedab356-05bc-4e30-ba77-6317b0210c7e
#Here we set the grub "root" variable by locating the uuid of the root partition identified above
search --no-floppy --fs-uuid $uuid_os_root --set=root
#Here we set a custom variable grub_boot by locating the uuid of the boot partition identified above
search --no-floppy --fs-uuid $uuid_grub_boot --set=grub_boot
#Here's the magic. We test to see if the boot and root partitions have the same UUID.
#If they do we append /boot to the $grub_boot variable. For ex. (hd0,1) becomes (hd0,1)/boot.
if [ $uuid_grub_boot == $uuid_os_root ] ; then
set grub_boot=$grub_boot/boot
fi
linux ($grub_boot)/live/vmlinuz-4.19.0-9-amd64 boot=live config quiet noeject toram persistence persistence-storage=filesystem usbcore.autosuspend=-1 live-media=/dev/disk/by-uuid/$uuid_grub_boot elevator=as forcefsck
initrd ($grub_boot)/live/initrd.img-4.19.0-9-amd64
}
Or (2) Generate grub.cfg with grub-mkconfig
Again, using the handcrafted grub.cfg above is probably fine, but here are details on how to get grub-mkconfig to work, which geterates grub.cfg from directives in /etc/grub.d/
Now, /usr/sbin/grub-mkconfig is not robust line 142 & 146 in our context of chroot usage, assumes we're installing to / device and /boot, and offers no way to configure around the problem, so we change line 142 & 146:
# Line 142:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# Change to:
GRUB_DEVICE="`${grub_probe} --target=device /media/live`"
# Line 146:
GRUB_DEVICE="`${grub_probe} --target=device /boot`"
# Change to:
GRUB_DEVICE="`${grub_probe} --target=device /media/live/boot`"
Mount your live partition at /media/live within the chroot, so that grub-probe can find it:
mkdir /media/live mount /dev/disk/by-label/live /media/live
Copy by hand the legacy grub.cfg menu entries I have, with your disk UUIDs into /etc/grub.d/40_custom.
Now run grub-mkconfig, from within the chroot:
grub-mkconfig -o /boot/grub/grub.cfg
Then move the grub.cfg generated to the live partition:
mv /boot/grub/grub.cfg /media/live/boot/grub/
Unmount /media/live:
unmount /media/live
Prepare the persistence partition
Here is the formatting command I used, best if done within the chroot:
sudo mkfs.ext4 -m 1 -b 4096 -L u365-persistence -E stride=4,stripe-width=2048 -O ^has_journal /dev/sdc2
And don't forget to make the persistence.conf file '/ union' on the persistence partition:
#cd to wherever you have mounted your persistence partition, e.g. /media/persistence cd /media/persistence echo "/ union" > persistence.conf
Also, we must make a link so that when future apt-get updates will update something in /boot, it will update the 'live' partition's /boot:
cd /media/persistence ln -s /lib/live/mount/medium/live live #or ln -s /lib/live/mount/medium/boot boot #wherever your distro expects vmlinuz, initrd.img to be
From /media/overlay/home grab the OSE Linux stuff in there, and put in /media/persistence/home:
cp -a /media/overlay/home /media/persistence/home
There may also be other OSE Linux specific things removed by prior purges of non-"core" packages, e.g. in /etc/ I suppose there may be a way to "diff" those out, and re-add them to the persistence partition after the packages are installed anew. Maybe there is a convenient dpkg way of doing that... diffing and saving the configuration file changes from stock packages instead of purging them? Or is that the difference between apt-get remove and apt-get purge where apt-get remove will only remove stock configuration files leaving modified ones intact? Yes I think that may be right.
Cleanup
Cleanup!
sudo umount /media/overlay/dev/pts sudo umount /media/overlay/dev/ sudo umount /media/overlay/proc sudo umount /media/overlay/sys sudo umount /media/overlay/tmp sudo umount /media/overlay sudo umount /media/filesystem.squashfs sudo umount /media/iso sudo umount /media/myadditions sudo rmdir /media/iso /media/filesystem.squashfs /media/overlay
Congratulations! You now have an Linux OS on your USB with persistence that will boot within seconds and reliably outperforms the competition!
Further Details
Upgrading
Q: What do we do when linux-image (the package containing the kernel) is upgraded?
A: apt will want to update /boot/vmlinuz and /boot/initrd.img. So we need to do things in order to allow this to happen. We must boot with 'toram' and also remount 'live' as rw before running apt-get update.
#TODO, I wrote this elsewhere need to copy here, something along these lines: mount -o remount,rw /lib/live/mount/medium /dev/disk/by-label/live sudo apt-get update
In this upgrade process apt might rename vmlinuz to say vmlinuz-3.16.0-4-amd64, and initrd similarly. It didn't always do this. But, you must match the names of those files with what is listed in grub.cfg! Either use 'vmlinuz' or whatever name the apt script renamed them to.
Q: What do we do when libc is upgraded? Be it a security update, or when Debian Stable advances every couple years.
A: We will want to do a resquash:
- We need to to the same as above, 'toram' and remount 'live' as rw.
- dpkq-query our currently installed packages, and compare to /boot/filesystem.packages.xz we made earlier.
- Remove all non "core" packages, they should be cached in /var/cache/apt/archives/ so they don't need to be downloaded again, just reinstalled after we do the resquash.
- apt-get update
- Resquash, reboot to verify things working, in 'live' mode remove squashed stuff from persistance partition.
- squash script https://gist.github.com/AndrewSmart/90eb186aea08db8f1426
- cleanup script https://gist.github.com/AndrewSmart/2f67f79f6f1922c4556f
- Reinstall packages removed prior to resquash from /var/cache/apt/archives
Troubleshooting
Keep in mind with the USB we can still boot either in 'live' mode, or 'persistence' mode depending on the GRUB entry we choose. This is useful in case something breaks in 'persistence' mode, like libc was upgraded in 'persistence' but libc on 'live' is still old! This will result in segmentation faults... a kernel panic, or whatever, it won't boot. We can access all of the software in the squashfs in 'live mode'. Eventually I'd like to write an apt hook to prevent the update of libc unless the system is prepared ('toram', and 'live' partition is mounted rw, the hook won't trigger unless 'persistence' kernel parameter is used).
Debian's live-boot scripts will put stuff in /lib/live/, and /lib/live/mount is of particularly good use in troubleshooting.
Ubuntu is a bit of a can of worms from my perspective as a fan of vanilla Debian. So... hopefully things will go smoothly... I highly recommend a lighter-weight environment for USB persistence, but I understand everyone has their preferences.
Since 'toram' puts the squashfs into memory, lets say you have 8GB RAM and an 800MB squashfs, that is fine, but say you move to a workstation with only 2GB RAM, you will probably want to remove the 'toram' kernel parameter, things will be slower but the memory footprint used by the OS will be less, and only use 'toram' when updating the 'live' partition as detailed above. Perhaps consider 2 additional GRUB entries, in addition to the two I have listed above, where the 2 additional entries are live and persistence mode without 'toram' kernel parameter, and when booting select the entry depending on the amount of RAM the system has.
In the event of a power outage, kernel panic, unsafe shutdown or whatever, the filesystem may have problems you're warned about on next boot. In this setup, the 'live' partition was mounted ro, so it is not possible to have filesystem errors there, but with the 'persistence' partition there will likely be errors. So, on next boot we select the 'live' mode in grub, and run fsck.ext4 -p /dev/disk/by-label/persistence, and all problems are fixed!