Talk:OSE Linux Persistence
https://en.m.wikipedia.org/wiki/UNetbootin claims to have a persistence option for ubuntu distros. I couldn't get it to work. --Dorkmo (talk) 19:33, 14 July 2018 (UTC)
Contents
Andrewusu's Thoughts
I've used persistence on USB for over 10 years now as my primary system (4+ hours/day) and will share some thoughts. Way back with USB2 I spent a lot of time investigating performance, ways to speed up boot time and system responsiveness. Keep in mind benchmarks of any flash memory device, where sequential reading is 10-20x faster than non-sequential reads (possibly more depending on the device). By following this setup I describe below you take advantage of those fast reads for a quick boot and a very responsive OS.
Here is a video of a minimal Debian distro on a live-boot persistence USB booting. By 35s system uptime I have launched both the terminal and filesystem browser.
It has been great, I can move my programs, data, and OS from workstation to workstation with a breeze. From a desktop to a laptop, a library computer, or computer at work at times even. I'll never going back to installing Linux on a HDD. I use HDDs on workstations for storing data, or even making a ~8GB swap file if I need more system memory for some big task (e.g. big FreeCAD model).
I don't use unetbootin, ISOs, or any other setups, just Debian's live-boot and a 'persistence' partition.
On Ubuntu an alternative to live-boot is casper, which supports more options I believe. The 'boot=live' kernel parameter results in live-boot being used, while the 'boot=casper' kernel parameter results in casper being used. With casper the default labeling of the persistence partition is 'casper-rw' instead of 'persistence' with live-boot.
It is very simple, you have the second partition labeled 'persistence' with a configuration file, and Debian's live-boot scripts handle everything.
live-boot Squashfs/Persistence
- 'live' partition contains the squashfs, initrd and vmlinuz in /live. Can only be mounted ro unless 'toram' kernel parameter used.
- 'persistence' partition is mounted rw.
Benefits:
- With a squashfs/persistence install instead of a full install to USB, you'll see a much faster boot, because the 'live' partition is compressed (squashfs) and is thus sequentially read into memory faster compared to random reads of various uncompressed files from the USB as they're needed during the boot process. Mine boots within 20 or so seconds.
- When running the system with day to day tasks, the performance and responsiveness of a squashfs/persistence setup is better than a full install, because the squashfs may be loaded to memory (toram) occupying around 600-800MB RAM, and is thus not a bottleneck on reads/writes to the USB slowing the system down like you'd see in a full install to USB. This translates into better framerate playing games like Starcraft 2, which I've run from this flash drive, and a more responsive system in general (to mouse clicks, key presses, etc).
Limitations:
- A 64-bit install won't work on a 32-bit machine
- A USB with /etc/X11/xorg.conf set to use an NVIDIA GPU driver won't be as mobile:
- e.g. Will not boot graphically on an Intel GPU system but to a virtual console.
- This can be mitigated by editing xorg.conf while on the Intel system's virtual console and rebooting to get the graphical interface. Or possibily without rebooting with some wizardry like rmmod and then startx, but I don't recall off the top of my head. And you have to redo this in reverse going back to the NVIDIA system. Maybe this is just a limitation of proprietary GPU drivers.
- Copy-on-write aufs/overlayfs (whetever distro has chosen) has some limitations:
- Read-only 'live' partition needs to be updated whenever libc or linux-image are updated, or the system won't boot. So it needs to be mounted read-write and in order to do so 'toram' kernal boot parameter must be used. I work around this problem manually via inspecting apt output before installing anything, but this chore can be mitigated via an apt hook which I've not written yet.
- 'live' partition containing the squashfs should only have packages which are not updated often, being core packages, and not something like firefox which is updated often. This will minimize the size of the squashfs.
- When Debian stable is updated to the next revision, the linux-image must be updated, and the squashfs recompressed. To do this, all non-core packages must be removed, the distro upgraded, the squashfs recompressed, and then non-core packages reinstalled. This can also be scripted and I have some non-robust helper scripts I've written.
- Flash memory has limitations, where sequential writes are 5000x-10000x faster than random writes. Due to this, there should always be at least 2GB free on the 'persistence' filesystem, or else risk files becoming fragmented and the any application blocking on reads/writes will hang from very slow reading and writing. In this circumstance it may be a number of minutes before such applications respond again, but you're free to use this time to do other things with already opened applications, like use the web browser or read opened documents.
I personally use a lightweight Debian distro (not Ubuntu!) to minimize RAM footprint, minimize squashfs size, and maximize responsiveness. It is all instantaneous on USB3 (was a bit sluggish on USB2 on an older 4GB stick when I first started). I recommend at least a 32GB stick.
Once the apt hook is implemented and distro-upgrade script polished, this should be a very nice option for interested users.
There are tons of persistence guides, which unfairly dismiss this live-boot squashfs/persistence setup because they don't know how to mitigate/handle the upgrade problem as I do, because making a squashfs is not something well known which requires a lot of plumbing. Or they suggest other unnecessarily complex setups or make suggestions without having investigated performance implications (e.g. suggesting a full install to USB).
I do not recommend modern Samsung, Sandisk, or any other cheap off-brands for this "heavy" use. Good chance they'll fail within a year and be too hot to touch. Buy from Toshiba/Kioxia, the inventors of flash memory. Yes their performance as shown in benchmarks isn't as good, nor are they inexpensive, but they are reliable and will last years. Do not skimp to save a few $ buying a Sandisk/Samsung on a sale for this use-case, I'm speaking from experience!
Encryption
Pros:
- Sure it may be useful if for example, you forget the drive in an internet cafe or library. Or have something you don't want to turn up in investigation.
Cons:
- It costs additional CPU & energy usage.
- Encryption offers zero "online"/mounted protection, e.g. encryption does not protect against a web browser vulnerability allowing disk access. Mental energy would be better spent on 'hardening' the system.
- Harder to recover data when things break. Big headache.
--Andrewusu (talk) 20:40, 4 November 2020 (UTC)
I will write more thoughts, I see this is an ongoing need/interest OSE_Linux_Log. Today I have ordered two Toshiba USB flash drives. I will write a guide here for preparing any Debian distro for USB persistence with optimal performance (squashfs & persistence partition) for interested readers.
GUIDE
This guide will first make a live USB stick persistent.
In the appendix will be additional knowledge and procedures to make it perform better.
Brief steps:
- Make live USB
- Partition the USB.
- Mount the ISO, and copy select files to the USB's live partition.
- Install grub.
- Make the live USB a persistent USB
- Boot to live USB & Setup persistence.
Make live USB
You can follow a different guide and then run the command `live-persistence`. Then when it comes time to upgrade libc or the kernel you can run `live-toram`, then follow the squash guide below.
But here is the way I'd do it.
Partition the USB
Use gparted. Make a GPT partition, with 16MB unused.
Make live partition 1GB is fine. Big enough to fit the squashfs, vmlinuz, and initrd files from the ISO. Label it 'live', or whatever you want.
Make persistence partition filling up the remainder. Label it 'peristence', or whatever you want, but you'll have to use the label you use in the grub.cfg later.
Now make a small partition at least 1MB big into the 16MB unused space at the start. Don't format it with a filesystem, leave it unformatted. This is where GRUB installs stuff into.
Mount the ISO, and copy select files to the USB's live partition
Yep, copy the squashfs, vmlinuz, and initrd to /live on the live partition, e.g. /media/live/live Easy peasy.
Install grub
Should go to /media/live/grub
See appendix: Talk:OSE_Linux_Persistence#Prepare_GRUB_on_the_live_partition_and_USB_device
Boot to the USB! It is live!
Make the live USB a persistent USB
While you're booted to the live USB, mount the persistence partition, and put a peristence.conf file on it.
echo "/ union" | sudo tee /media/peristence/persistence.conf
Now, read the short manual on live-tools. And type:
man live-tools live-peristence
Neat, all your files you're using in the live USB, are now saved to the persistence partition. Though, we have to write a few changes to grub.cfg in order to load that stuff on next boot.
Edit grub.cfg
See appendix: Talk:OSE_Linux_Persistence#Either_.281.29_Handcraft_grub.cfg
Appendix
This guide seeks a USB stick with optimal performance, which requires a minimal squashfs.
If USB space and performance are not concerns for you, you're welcome to modify an existing live USB, name a 2nd partition "persistence", add suitable kernel parameters including "persistence", and write a persistence.conf file containing "/ union" in the persistence partition. You can understand necessary details by understanding the guide below. Boot time may take a minute or longer, and applications may be slow and hang due to limitations of flash media which this guide seeks to minimize.
Let's proceed with making the USB with optimal performance.
Some notes on flash media
The first partition should start after some multiple of the medium's erase block. The erase block size might be published by the manufacturer, or reasoned about via benchmarking. However SSDs and modern USB sticks use more sophisticated controllers so flashbench won't provide the necessary insights. So starting the first partition at a 16MiB or 32MiB offset would be a reasonable choice, being some multiple of the erase block.
When considering the stride and stripe-width parameters I'm not sure it makes a difference anymore with more sophisticated controllers.
On the flash media, writing is slow, so modern ones typically have a faster volatile cache, which is then written to the non-volatile flash memory. So when you write a large file to the medium, and it appears to finish via the command completing, but in reality it won't be done for some time. This is why writing speed appears to slow down in benchmarking the longer the write goes, from say 20MB/s to 4MB/s, this is due to the volatile cache filling up. ext4 when writing the journal uses a "barrier" somehow implemented, and usually implemented by the USB driver/device where the command blocks until the write is actually written to non-volatile memory, but I think this feature isn't exposed by ext4 for use in benchmarking. This is something to be cautious about before shutting down the system, leave the system idle for about 30-60 seconds after shutting down the browser or other write activity before powering down the system, as it is possible the write is still transferring from the volatile cache to non-volatile memory, and the OS doesn't know better, as we'd disabled ext4 journalling for performance and longevity reasons, and it has been my experience that linux doesn't wait for the non-volatile write to finish anyway on shutdown, printing write errors on shutdown if this is the case. If this happens, then run a fsck on next boot, we'll make the Grub entry later to make running fsck convenient to best fix the consistency problems.
Partitioning
A GPT partition table is fine, as is an MBR partition table.
If doing GPT then make a small unformatted partition in the 16MiB or 32MiB offset before the live partition. It only needs to be 1MiB, but can be larger. Set the 'boot_grub' flag on it in gparted. Grub will install to it later in this guide.
Here is my example command of how I made the ext4 filesystem for the live partition. The option ^has_journal disables the journal, important for flash memory longevity. Determine which partition to run mkfs.ext4 on, mine is /dev/sdd1 and /dev/sdd2.
sudo mkfs.ext4 -m 5 -b 4096 -N 2048 -L live -E stride=4,stripe-width=2048 -O ^has_journal <yourpartition>
If you want to do an EFI system partition (ESP), then have this also be the live partition, which we use as our "/boot" partition. This must be FAT32 instead of ext4 unfortunately. Set the 'esp' flag on the live partition in gparted. When we install GRUB later, it will install EFI files to the live partition.
And for the persistence partition:
sudo mkfs.ext4 -m 5 -b 4096 -L persistence -E stride=4,stripe-width=2048 -O ^has_journal <your partition>
Prepare squashfs
Download ISO of desired debian distro (e.g. OSE Linux). Put it at /tmp/
chroot just uses your current Kernel. That's just how it works. Because of this it is best if the kernel of the system you're running to be the most recent available for the distro release of the ISO (e.g. for me linux-image-4.19.0-13-amd64 as of this writing). If this isn't possible, boot to the ISO and skip to the Talk:OSE_Linux_Persistence#Remove_non-core_packages instructions without doing the chroot. You can do this all on a single USB stick if you do it correctly. Or you could use [1] or virtualization, lots of possibilities outside the scope of this guide.
Actually now that I think about it writing the guide following the live USB installation without doing the chroot may be best, being more generalized and perhaps easier. I will rewrite this guide that way in the future, but for now, here are the chroot directions.
prepare chroot filesystem
Let us continue with the chroot assuming we're running the same distro/kernel. Mount the ISO, and look for filesystem.squashfs, initrd.img, vmlinuz on it. Also note the location of config, System.map and filesystem.packages.xz, we'll copy or make these to follow convention.
Make temporary directories:
sudo mkdir /media/filesystem.squashfs /media/iso /media/overlay /media/myadditions /media/myadditions/rw /media/myadditions/work
Mount the iso:
sudo mount -o loop /tmp/debian-distro-i386.iso /media/iso
Mount filesystem.squashfs on the iso:
sudo mount -t squashfs /media/iso/live/filesystem.squashfs /media/filesystem.squashfs
Mount an in-memory filesystem to hold temporary files:
sudo mount -t tmpfs -o size=1024m none /media/myadditions
Prepare the union filesystem, so that we're able to edit the squashfs on the iso and compress the result. We can use either overlayfs or aufs. Here is the overlayfs command:
sudo mount -t overlay overlay -o lowerdir=/media/filesystem.squashfs,upperdir=/media/myadditions/rw,workdir=/media/myadditions/work /media/overlay
Or afus. overlayfs hasn't been widely available until kernel 3.18, and aufs is available on earlier kernels (from kernel 3.9 late 2014 aufs supports xattr) via aufs-tools, and their may be some differences I'm not aware of but I think the concensus is that overlayfs is better. Hopefully aufs supports extended attributes (xattr) good enough to not loose said attributes, which are important for hardening the system (extended attributes are I believe where Linux capabilities are stored/configured).
sudo mount -t aufs -o br=/media/myadditions=rw -o br=/media/filesystem.squashfs=ro -o udba=reval none /media/overlay/
chroot
Prepare the chroot, so that we can run apt within it and remove packages. resolv.conf and mount binds are to remove error/warning messages when installing packages with apt.
sudo cp /etc/resolv.conf /media/overlay/etc/resolv.conf sudo mount --bind /dev/ /media/overlay/dev/ sudo mount --bind /dev/pts /media/overlay/dev/pts #https://wiki.debian.org/chroot#A.2Fdev.2Fpts sudo mount --bind /proc /media/overlay/proc sudo mount --bind /sys /media/overlay/sys
Now chroot into the squashfs so we can use apt within it to remove packages:
sudo chroot --userspec root:root /media/overlay
Configure the locale, you'll get many warning messages in apt otherwise. You probably want to select UTF-8 for your country/language, for example I picked en_US.UTF-8. Press space bar to toggle each locale you want generated, then press enter to proceed. Then it asks if you want the locales you chose to be applied systemwide, forcing your selection on each user, which may be desired for desktop software which starts up before the user-specific ~/.profile is read:
dpkg-reconfigure locales
And set the timezone:
dpkg-reconfigure tzdata
If /boot exists, then back it up, we're replacing /boot with a symlink:
mv /boot /boot.bak
We must make a symlink so that as we work with apt-get, it may call `live-update-initramfs -u`, and we need those updates to /boot to go to the proper location, the 'live' partition's /live:
ln -s /lib/live/mount/medium/live /boot
If you've not yet made the live partition, then:
mkdir -p /lib/live/mount/medium/live
If already have the live partition ready, or like me are updating an existing live partition's squashfs, then:
mkdir -p /lib/live/mount/medium mount /dev/disk/by-label/live /lib/live/mount/medium mkdir /lib/live/mount/medium/live
We have to do the above steps to work around update-initramfs's logic and assumptions regarding a read/write test that this works around. If you ever run into problems with update-initramfs while using apt-get, this is what you run once the problem is fixed:
dpkg --configure initramfs-tools
As we remove packages, such at btrfs-progs, update-initramfs will be triggered which should update /boot/vmlinuz and /boot/initrd, these files are not compressed into the filesystem.squashfs and we need those files within /live of the live partition to boot.
`man live-boot` is a useful reference, ensure live-boot-doc is installed. Also the Debian live-manual is a useful reference.
apt-get install live-boot-doc
Remove non-core packages
Now we gut the filesystem.squashfs from the iso of packages that:
- We don't want.
- Are frequently updated (e.g. web browser).
- With particular scrutiny of packages that are large.
- Are not necessary to boot to a desktop.
All of those can they can go onto persistence partition. We remove them now, and install them after the squashfs has been made, or as needed. Examples of what should be considered "core" packages necessary to boot are: linux-image (this is the kernel), firmware, GPU driver, and X11 server. This way the USB will:
- Boot fast,
- The OS running from it will be maximally responsive,
- Will have minimal memory footprint,
- And be persistent!
Run this to see packages installed ordered by size, then apt-get remove or purge stuff, and ignore anything beginning with "lib":
dpkg-query -W -f '${Installed-Size}\t${Package}\n' | sort -n | less
To understand what a package is, you can search packages.debian.org. If the description there isn't informative enough, you can try clicking the links on the right hand side such as a project homepage, or looking through the files of the package. Ignore or don't pay much attention to any package beginning with "lib", with the exception of things like libc6, but apt will probably warn you about doing something silly like that with the prompt:
You are about to do something potentially harmful. To continue type in the phrase 'Yes, do as I say!'
An example of stuff I removed from a different Debian distro (BunsenLabs), but I removed some firmware packages so that reduces portability:
apt-get remove -y firefox-esr libjsoncpp1 apt-get remove -y libreoffice-core libreoffice-common coinor-libcbc3 coinor-libcgl1 coinor-libclp1 coinor-libcoinmp1v5 coinor-libcoinutils3v5 coinor-libosi1v5 fonts-opensymbol libabw-0.1-1 libboost-atomic1.67.0 libboost-chrono1.67.0 libboost-date-time1.67.0 libboost-filesystem1.67.0 libboost-locale1.67.0 libboost-thread1.67.0 libclucene-contribs1v5 libclucene-core1v5 libcmis-0.5-5v5 libcolamd2 libe-book-0.1-1 libeot0 libepubgen-0.1-1 libetonyek-0.1-1 libexttextcat-2.0-0 libexttextcat-data libgpgmepp6 liblangtag-common liblangtag1 libmhash2 libmwaw-0.3-3 libmythes-1.2-0 libneon27-gnutls libnumbertext-1.0-0 libnumbertext-data libodfgen-0.1-1 liborcus-0.14-0 libraptor2-0 librasqal3 librdf0 librevenge-0.0-0 libstaroffice-0.0-0 libsuitesparseconfig5 libwpd-0.10-10 libwpg-0.3-3 libwps-0.4-4 libxmlsec1 libxmlsec1-nss libyajl2 lp-solve uno-libs3 ure apt-get remove -y firmware-atheros firmware-netronome firmware-iwlwifi firmware-brcm80211 firmware-qlogic firmware-ti-connectivity firmware-myricom firmware-intelwimax firmware-libertas dahdi-firmware-nonfree atmel-firmware firmware-cavium firmware-bnx2x firmware-qcom-media firmware-netxen firmware-samsung firmware-ipw2x00 firmware-siano firmware-ivtv firmware-bnx2 apt-get remove -y samba-libs libavahi-glib1 libcdio-cdda2 libcdio-paranoia2 libgd3 libgdata-common libgdata22 libgoa-1.0-0b libgoa-1.0-common libgphoto2-6 libgphoto2-port12 libldb1 liblmdb0 liboauth0 libpython2.7 libtalloc2 libtevent0 libwbclient0 python-talloc apt-get remove -y fonts-noto-core fonts-noto-cjk papirus-icon-theme bunsen-papirus-icon-theme gnome-icon-theme gdebi-core gir1.2-vte-2.91 python3-apt python3-chardet python3-debian python3-pkg-resources rm -r /usr/share/gdebi/GDebi /usr/lib/python3/dist-packages/aptsources /usr/lib/python3/dist-packages/apt/progress /usr/lib/python3/dist-packages/debian_bundle /usr/lib/python3/dist-packages/debian /usr/lib/python3/dist-packages/chardet/cli /usr/lib/python3/dist-packages/pkg_resources/extern /usr/lib/python3/dist-packages/pkg_resources/_vendor/packaging apt-get remove -y evince synaptic bubblewrap evince-common gnome-desktop3-data libbrotli1 libdjvulibre-text libdjvulibre21 libept1.5.0 libevdocument3-4 libevview3-3 libgnome-desktop-3-17 libgspell-1-1 libgspell-1-common libgxps2 libharfbuzz-icu0 libhyphen0 libjavascriptcoregtk-4.0-18 libkpathsea6 libspectre1 libsynctex2 libwebkit2gtk-4.0-37 libwebpdemux2 libwoff1 xdg-dbus-proxy zenity zenity-common apt-get remove -y vlc libaribb24-0 libbasicusageenvironment1 libcddb2 libdvbpsi10 libebml4v5 libgles2 libgroupsock8 libixml10 liblirc-client0 liblivemedia64 liblua5.2-0 libmad0 libmatroska6v5 libmtp-common libmtp9 libnfs12 libopenmpt-modplug1 libplacebo7 libprotobuf-lite17 libqt5x11extras5 libresid-builder0c2a libsdl-image1.2 libsdl1.2debian libsidplay2 libspatialaudio0 libupnp13 libusageenvironment3 libva-wayland2 libvlc-bin libvlc5 libxcb-xv0 vlc-bin vlc-data vlc-plugin-base vlc-plugin-qt vlc-plugin-video-output vlc-plugin-notify libvlccore9 apt-get remove -y file-roller p7zip-full p7zip libarchive13 libnautilus-extension1a xfburn libburn4 libexo-1-0 libisofs6 libjte1 unar gnustep-base-common gnustep-base-runtime gnustep-common libgnustep-base1.26 libobjc4 apt-get remove -y transmission-gtk libevent-2.1-6 libminiupnpc17 libnatpmp1 transmission-common filezilla filezilla-common libfilezilla0 libpugixml1v5 libwxbase3.0-0v5 libwxgtk3.0-0v5 yudit-common feh apt-get remove -y libpython2.7-stdlib libblas3 libgfortran5 libkeybinder0 liblapack3 libpython2.7-minimal libsodium23 lua-bit32 lua-expat lua-penlight lua-posix lua-socket lua5.2 python-apt-common python-minimal python2-minimal python2.7-minimal libavformat58 libmysofa0 libnorm1 libpgm-5.2-0 libpostproc55 librubberband2 libssh-gcrypt-4 libswscale5 libvidstab1.1 apt-get remove -y aptitude aptitude-common libboost-iostreams1.67.0 libboost-system1.67.0 libcwidget3v5 libxapian30 ristretto file libmagic-mgc libmagic1 hexchat-common apt-get install wpasupplicant network-manager --no-install-recommends #modem support apt-get remove -y modemmanager libmbim-glib4 libmbim-proxy libqmi-glib5 libqmi-proxy openssh-client libxatracker2 nitrogen libgtkmm-2.4-1v5 lvm2 dmeventd libaio1 libdevmapper-event1.02.1 liblvm2cmd2.03 libreadline5 nano geany geany-common ghostscript poppler-data libcupsimage2 libgs9-common libijs-0.35 libjbig2dec0 libpaper1 pcmciautils vdpau-va-driver arandr apt-get remove -y intel-microcode iucode-tool va-driver-all i965-va-driver xserver-xorg-video-intel libxvmc1 intel-media-va-driver libigdgmm5 xserver-xorg-video-qxl btrfs-progs cryptsetup-initramfs cryptsetup-bin cryptsetup-run apt-get remove -y python3.7-minimal dconf-cli distro-info-data gir1.2-atk-1.0 gir1.2-freedesktop gir1.2-gdkpixbuf-2.0 gir1.2-glib-2.0 gir1.2-gtk-3.0 gir1.2-pango-1.0 libfm-extra4 libgirepository-1.0-1 libmenu-cache-bin libmenu-cache3 libmpdec2 libpython3-stdlib libpython3.7-minimal libpython3.7-stdlib libxslt1.1 mime-support wmctrl rm -r /usr/lib/python3 apt-get remove -y iso-codes liba52-0.7.4 libaa1 libass9 libatomic1 libavc1394-0 libavfilter7 libavformat58 libbs2b0 libcaca0 libcdio18 libcdparanoia0 libchromaprint1 libdc1394-22 libdca0 libde265-0 libdv4 libdvdnav4 libdvdread4 libfaad2 libfftw3-double3 libflite1 libfluidsynth1 libgme0 libgssdp-1.0-3 libgstreamer1.0-0 libgupnp-1.0-4 libgupnp-igd-1.0-4 libiec61883-0 libilmbase23 libkate1 liblilv-0-0 libmjpegutils-2.1-0 libmms0 libmodplug1 libmpcdec6 libmpeg2-4 libmpeg2encpp-2.1-0 libmpg123-0 libmplex2-2.1-0 libmysofa0 libnice10 libnorm1 libofa0 libopenal-data libopenal1 libopencore-amrnb0 libopencore-amrwb0 libopenexr23 libopenmpt0 libpgm-5.2-0 libpoppler-glib8 libpoppler82 libpostproc55 libraw1394-11 librubberband2 libsbc1 libserd-0-0 libshout3 libsidplay1v5 libsndio7.0 libsord-0-0 libsoundtouch1 libspandsp2 libsratom-0-0 libsrtp2-1 libssh-gcrypt-4 libswscale5 libtumbler-1-0 libv4l-0 libv4lconvert0 libvidstab1.1 libvisual-0.4-0 libvo-aacenc0 libvo-amrwbenc0 libvulkan1 libwildmidi2 libzbar0 libzmq5 tumbler-common
Again, much of what was removed can be installed again after persistence is set up. These packages are conveniently listed as removed within dpkg utilities, I'll illustrate how to conveniently reinstall them later.
Misc setup tips
One important consideration is that the live user is created by live-boot at boot time. We need to ensure the live user is set up. Inspect the contents of /etc/live/ for any configuration files. If there is none in your distribution then let's make it, and feel free to customize this:
cat<<-EOF > /etc/live/config.conf.d/10-user-setup.conf LIVE_HOSTNAME=debian-persistence LIVE_USERNAME=user LIVE_USER_FULLNAME="Debian LiveUser" LIVE_USER_DEFAULT_GROUPS="cdrom floppy audio dip video plugdev fuse bluetooth netdev scanner staff" EOF
These options to live-boot will minimize the generated initramfs size:
cat<<-EOF > /etc/live/boot.conf #man live-boot #see /usr/share/initramfs-tools/hooks/live, might make initrd smaller DISABLE_CDROM=true DISABLE_FAT=true DISABLE_FUSE=true DISABLE_NTFS=true EOF
Performance tips
USB sticks use flash memory, and to maximize performance with it we can change the IO scheduler for such devices.
BFQ looks like a nice scheduler. Quote: "Regardless of the actual background workload, BFQ guarantees that, for interactive tasks, the storage device is virtually as responsive as if it was idle. [...] As a comparison, with CFQ, NOOP or DEADLINE, and in the same conditions, applications experience high latencies, or even become unresponsive until the background workload terminates (also on SSDs)."
mq_deadline is fine while the system is booting. The scheduler in use can also be tuned, but I need to do more research before giving advice on tuning in this context. To setup/enable:
cat<<-EOF > /etc/modules-load.d/bfq.conf bfq EOF
Set bfq scheduler during startup. Alternatives include mq_deadline and none, either might be faster than bfq during startup when GUI application responsiveness isn't important, but I'd need to do some research/experiments.
cat<<-EOF > /etc/udev/rules.d/60-ssd-scheduler.rules
# set bfq scheduler for non-rotating disks during startup
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="bfq"
EOF
Use profile-sync-daemon to enhance performance of the browser, and minimize flash memory writes (maximize longevity).
apt-get install profile-sync-damon
For SD card longevity, consider mounting a tmpfs on /var/log. But perhaps only after you've booted to the system a few times and verify the system is stable, as putting log files in RAM means they can't be read offline to diagnose a problem. So disable this feature if needed.
echo "tmpfs /var/log tmpfs nosuid,nodev,noexec 0 0" >> /etc/fstab cat<<-EOF > /etc/tmpfiles.d/varlog.conf d /var/log/apt 755 root root d /var/log/lightdm 711 root root EOF
vlc's default lookahead is too short, leads to music cutting out. 'Tools->Preferences', toggle 'Show all settings', click 'Input/Codecs' in tree view, scroll down to 'Advanced' section and look at caching settings. Increase them to at least 600ms. This should largely mitigate problems with music/video stopping momentarily due to I/O delays.
Pruning tips
Install deborphan and localepurge to help clean stuff:
apt-get install deborphan localepurge
After you're done pruning things, run deborphan and remove things it tells you to as you desire. Deborphan helps you find "orphan" packages not needed by anything else, which take up space. Such orphaned packages may have been necessary by a previously installed package.
deborphan
Next run localepurge to remove space consuming files for locales you'll never use, I removed 160MB from my system:
localepurge
And finally, clear out apt temporary files:
apt-get autoremove apt-get clean
I recommend vim.tiny over vim, vim.tiny much smaller, also install whatever else your heart desires that is small and infrequently updated. Such small utilities will be available in 'live' mode (if installed when in 'persistence' mode, it won't be available in plain 'live' mode):
apt-get install vim-tiny echo "set nocp" >> /etc/vim/vimrc.tiny #Allows arrow keys to work, and vim.tiny to act like vim instead of vi
Hardening tips
I think it is best for /tmp to be mounted noexec, but many applications have bugs regarding this Double check the /etc/fstab file with a text editor after editing /etc/fstab:
echo "tmpfs /tmp tmpfs nosuid,nodev,noexec 0 0" >> /etc/fstab mount /tmp
Often apt packages during installation will require the ability to execute temporary files on /tmp, this is how we allow this:
cat<<-EOF > /etc/apt/apt.conf.d/50remount
DPkg::Pre-Install-Pkgs {"mount -o remount,exec /tmp";};
DPkg::Post-Invoke {"mount -o remount /tmp";};
EOF
A common thing people do, is charge their android phone, or other USB device via a USB port on the computer. Realize that this is physical access, which a compromised device may use to log in to Linux via root. I suspect I had fallen victim to in the past, a keylogger following a pattern I'd seen exhibited in the news, which I believe came after I'd plugged in a new android I had run questionable rooting software on, to my computer. I suggest both setting a root password, and disallowing this access, by editing /etc/securetty and commenting out root access via UART serial ports:
# UART serial ports #ttyS0 #ttyS1 #ttyS2 #ttyS3 #ttyS4 #ttyS5
Set the root password:
passwd
What has piqued my interest lately is Linux capabilities. Debian distributions should include both libcap2 and libcap-ng0 (RedHat's fork). And I'm looking for a more convenient open-source means of administration. What has me interested in the idea of a proactive defense against root-kits by removing module loading capabilities (CAP_SYS_MODULE) even for the root user, after the system had booted. It appears that lcap is no longer functional, it and many capabilities articles detail a system-wide capability bounding set in pre-2.6.25 kernels, but since kernel 2.6.25 capability bounding sets are now per-thread. The images here may help visually explain capabilities and their inheritance. What I have in mind may be within a custom /etc/init.d/ script calling setcap on init (pid 1) to remove the CAP_SYS_MODULE capability from the bounding set system-wide. The capabilities on any process can be viewed with `cat /proc/<pid>/status`.
Something else that can be done is use of more than one account, for example an account dedicated to web browsing with minimal privileges, and an account with administration privileges (sudo, member of staff,adm groups).
sudo adduser browser dm-tool switch-to-user browser #with LightDM desktop manager: https://wiki.ubuntu.com/LightDM
Files downloaded with the browser can be transfered to other users via /tmp, which is a good place to set as the default location to download files in the browser settings anyway.
With this separation of user privileges, if malware managed to gained the privileges of the browser account (no privilege escalation), and escaped the browser sandbox, the worst it could do is keylog, delete or encrypt/ransom the browser account's files. Since the browser account is running on a difference X11 server than an administration account, I think risks are reduced.
Switching between the administration account and browser account X11 servers can be done with ctrl+alt+F7 or F8 (or F1/F2 or whatever).
Or just have the desktop load with the browser account instead of the administrative account, and do administrative tasks on a virtual terminal (ctrl+alt+F1 through F6).
Set up the shell history file, .bash_history better. An easy way a bad actor can circumvent the stock configuration from logging his evil deeds is to proceed those evil things with a space, this guide will fix that!
lsattr /home/browser/.bash_history #See what attributes are set sudo chattr +a /home/browser/.bash_history #Only root user can chattr sudo chattr +a /home/user/.bash_history
Make filesystem.packages and filesystem.squashfs
Now that we have a set of "core" packages to go in the squashfs, of minimal size, on the 'live' partition, we must make our new squashfs!
Print out a list of "core" packages we have, we'll need them for reference later. Lets put them in filesystem.packages.xz:
dpkg-query -W --showformat='${Package}\t${Version}\n' | xz > /tmp/filesystem.packages.xz
Now let's make the squashfs:
apt-get -y install squashfs-tools
Exclude stuff we don't want in the squashfs:
EXCLUDES='.wh* dev live-persistence.conf live lost+found media mnt proc .pulse* run home selinux sys tmp* lib/live/mount usr/lib/live/mount var/cache/apt var/cache/apt-xapian-index var/lib/apt/lists var/tmp'
Directives to squashfs-tools to make pseudofiles, necessary to boot. Linux will make many pseudofiles in /dev when it boots, but not all, and through experimentation and observations I've come up with this list:
cat<<-EOF > /tmp/mksquashfs.pf dev d 755 root root dev/console c 662 root tty 5 1 dev/full c 666 root root 1 7 dev/kmem c 640 root kmem 1 2 dev/kmsg c 644 root root 1 11 dev/loop0 b 660 root disk 7 0 dev/loop1 b 660 root disk 7 1 dev/loop2 b 660 root disk 7 2 dev/loop3 b 660 root disk 7 3 dev/loop4 b 660 root disk 7 4 dev/loop5 b 660 root disk 7 5 dev/loop6 b 660 root disk 7 6 dev/loop7 b 660 root disk 7 7 dev/mem c 640 root kmem 1 1 dev/null c 666 root root 1 3 dev/port c 640 root kmem 1 4 dev/ptmx c 666 root tty 5 2 dev/pts d 755 root root dev/ram0 b 640 root disk 1 0 dev/ram1 b 640 root disk 1 1 dev/ram2 b 640 root disk 1 2 dev/ram3 b 640 root disk 1 3 dev/ram4 b 640 root disk 1 4 dev/ram5 b 640 root disk 1 5 dev/ram6 b 640 root disk 1 6 dev/ram7 b 640 root disk 1 7 dev/ram8 b 640 root disk 1 8 dev/ram9 b 640 root disk 1 9 dev/ram10 b 640 root disk 1 10 dev/ram11 b 640 root disk 1 11 dev/ram12 b 640 root disk 1 12 dev/ram13 b 640 root disk 1 13 dev/ram14 b 640 root disk 1 14 dev/ram15 b 640 root disk 1 15 dev/ram16 b 640 root disk 1 16 dev/random c 644 root root 1 8 dev/shm d 755 root root dev/tty c 662 root tty 5 0 dev/tty0 c 600 root tty 4 0 dev/urandom c 644 root root 1 9 dev/zero c 644 root root 1 5 home d 755 root root media d 755 root root mnt d 755 root root proc d 755 root root run d 755 root root run/lock d 1777 root root sys d 755 root root tmp d 1777 root root var/cache/apt d 755 root root var/cache/apt/partial d 755 root root var/cache/apt/lock f 640 root root echo "" var/lib/apt/lists d 755 root root var/lib/apt/lists/partial d 755 root root var/tmp d 1777 root root EOF
Now make the squashfs, it is compressed with xz compression:
mksquashfs / /tmp/filesystem.squashfs -comp xz -info -always-use-fragments -noappend -wildcards -pf /tmp/mksquashfs.pf -e ${EXCLUDES} > /tmp/mksquashfs.log
/tmp/mksquashfs.log will contain interesting details, such as the filesystem is now roughly 30% of its uncompressed size.
Hopefully the squashfs you made is around 800MB or less, mine for a different Debian distribution was about 300MB. Let us say your flash drive has 100MB/s sequential read speed, and your "core" filesystem was squashed down to 300MB, which will take 3 seconds to read into memory. How long do you think yours will take to boot?
Prepare the live partition
If you hadn't yet mounted the live partition to /lib/live/mount/medium earlier in the guide, we need to move files off, then mount it, and move vmlinuz, initrd, and etc onto it:
mv /lib/live/mount/medium /tmp/ mount /dev/disk/by-label/live /lib/live/mount/medium cd /lib/live/mount/medium mkdir live mv /tmp/medium/boot/vmlinuz* live/ mv /tmp/medium/boot/initrd* live/ mv /tmp/filesystem.squashfs live/ mv /tmp/filesystem.packages.xz live/ #update-initramfs or something else pointed at live/ incorrectly puts grub/unicode.pf2 here, this symlink fixes that: ln -s ../boot/grub/ live/grub
Also copy files like System.map-4.19.0-9-amd64 and config-4.19.0-9-amd64 to /lib/live/mount/medium/live, they should be in /media/overlay/boot.bak and/or /media/iso/live/ (outside the chroot). Or if the kernel is updated then you'll get these files, corresponding to the new kernel, automatically put in there anyway.
The file named config is a nice reference, it says what flags the kernel was built with, there are a vast number of options. This System.map file, stored on the read-only live partition may be useful, for example, finding if your system had been compromised with a rootkit.
Prepare GRUB on the live partition and USB device
The introduction of UEFI hardware complicates things. You're welcome to modify the instructions below if you desire to support UEFI functionality on such consumer hardware. The instructions below handles computers with traditional BIOS, with a traditional MBR, as well as UEFI hardware which supports such "legacy" standards.
From within the chroot, install non-uefi grub (grub-pc), and --no-install-recommends will result in os-prober not being installed. I think os-prober is workstation-specific and not useful to configure a USB device which may move between workstations. grub-pc will ask two questions, don't select anything and enter on Ok on the first, and press enter on Yes on the next to continue without --exclude.
apt-get install grub-pc --no-install-recommends
Find your live partition:
ls -l /dev/disk/by-label/
Let's say it is /dev/sdd1. Then we use the device /dev/sdd in the command below.
Install grub, to both the USB device and the live partition on the USB device. Where /dev/sdd below is the USB device (not a partition), and /lib/live/mount/medium is where the live partition is mounted:
grub-install --target=i386-pc --root-directory=/lib/live/mount/medium /dev/sdd
Now grub needs a configuration file. We can either craft one by hand or use grub-mkconfig, but grub-mkconfig doesn't really support this use-case.
Either (1) Handcraft grub.cfg
What follows is my legacy grub.cfg, which should work but I hope to make something workable within the newer /etc/grub.d/
Make sure to fill in this template /lib/live/mount/medium/boot/grub/grub.cfg on 'live' partition with your appropriate UUIDs. The partitions you made will have unique UUIDs. You can determine which is which on your system by comparing `ls -l /dev/disk/by-uuid/` and ls -l `/dev/disk/by-label/`.
Change "LABELorGPTname" to the label or GPT name of your persistence partition. I labeled mine "u365-persistence".
Fill in the template with how your vmlinuz and initrd.img are named, which may not have a suffix of "-4.19.0-9-amd64".
Make note of the kernel parameters used here, such as 'toram' and 'persistence'. I'm not sure if elevator=as is the optimal choice, feel free to read about it.
You can customize the locale, language, and keyboard layout here via kernel parameters here such as `locales=fr_FR.UTF-8 keyboard-layouts=fr keyboard-variants=bepo keyboard-model=tm2030usb`, but I think the console-setup package may be required (installed before the squashfs is made), and perhaps these things are fine configured elsewhere.
You can see other things configurable via kernel parameters such as tzdata, and initial username by looking at /lib/live/config/*. But be aware many things are only configured once via /var/lib/live/config/ (remove the relevant file to allow reconfiguration).
set timeout=3
set default=0
#Enter the UUID of your boot partition (this is where grub and your kernel reside)
set uuid_grub_boot=dc434800-22ac-4fd4-b6cb-603da325d5d0
#Enter the UUID of the persistence partition.
set uuid_os_root=858d963f-8718-4520-83dd-fb89a842e9bd
#Here we set the grub "root" variable by locating the uuid of the root partition identified above
search --no-floppy --fs-uuid $uuid_os_root --set=root
#Here we set a custom variable grub_boot by locating the uuid of the boot partition identified above
search --no-floppy --fs-uuid $uuid_grub_boot --set=grub_boot
#Here's the magic. We test to see if the boot and root partitions have the same UUID.
#If they do we append /boot to the $grub_boot variable. For ex. (hd0,1) becomes (hd0,1)/boot.
if [ $uuid_grub_boot == $uuid_os_root ] ; then
set grub_boot=$grub_boot/boot
fi
set kernel_parameters_common="boot=live config quiet noeject toram live-media=/dev/disk/by-uuid/$uuid_grub_boot usbcore.autosuspend=-1"
set kernel_parameters_persistence="persistence persistence-storage=filesystem persistence-label=LABELorGPTname"
set myversion=4.19.0-13-amd64
menuentry "DEBIAN 64bits - by UUID - Persistence" {
linux ($grub_boot)/live/vmlinuz-$myversion $kernel_parameters_common $kernel_parameters_persistence
initrd ($grub_boot)/live/initrd.img-$myversion
}
#With an unsafe shutdown or powerloss the persistence filesystem will have errors,
#since it has no journaling, we want to run fsck on it once
menuentry "DEBIAN 64bits - by UUID - Persistence fsck" {
linux ($grub_boot)/live/vmlinuz-$myversion $kernel_parameters_common $kernel_parameters_persistence forcefsck
initrd ($grub_boot)/live/initrd.img-$myversion
}
menuentry "DEBIAN 64bits - by UUID - Live" {
linux ($grub_boot)/live/vmlinuz-$myversion $kernel_parameters_common
initrd ($grub_boot)/live/initrd.img-$myversion
}
Or (2) Generate grub.cfg with grub-mkconfig
Again, using the handcrafted grub.cfg above is probably fine, but here are details on how to get grub-mkconfig to work, which geterates grub.cfg from directives in /etc/grub.d/
Now, /usr/sbin/grub-mkconfig is not robust line 142 & 146 in our context of chroot usage, assumes we're installing to / device and /boot, and offers no way to configure around the problem, so we change line 142 & 146:
# Line 142:
GRUB_DEVICE="`${grub_probe} --target=device /`"
# Change to:
GRUB_DEVICE="`${grub_probe} --target=device /lib/live/mount/medium`"
# Line 146:
GRUB_DEVICE="`${grub_probe} --target=device /boot`"
# Change to:
GRUB_DEVICE="`${grub_probe} --target=device /lib/live/mount/medium/boot`"
Copy by hand the legacy grub.cfg menu entries I have, with your disk UUIDs into /etc/grub.d/40_custom.
Now run grub-mkconfig, from within the chroot:
grub-mkconfig -o /lib/live/mount/medium/boot/grub/grub.cfg
Unmount /lib/live/mount/medium:
unmount /lib/live/mount/medium
Prepare the persistence partition
And don't forget to make the persistence.conf file '/ union' on the persistence partition:
#cd to wherever you have mounted your persistence partition, e.g. /media/persistence cd /media/persistence echo "/ union" > persistence.conf
From /media/overlay/home grab the OSE Linux stuff in there, and put in /media/persistence/home:
cp -a /media/overlay/home /media/persistence/home
There may also be other OSE Linux specific things removed by prior purges of non-"core" packages, e.g. in /etc/ I suppose there may be a way to "diff" those out, and re-add them to the persistence partition after the packages are installed anew. Maybe there is a convenient dpkg way of doing that... diffing and saving the configuration file changes from stock packages instead of purging them? Or is that the difference between apt-get remove and apt-get purge where apt-get remove will only remove stock configuration files leaving modified ones intact? Yes I think that may be right.
Cleanup
Cleanup!
sudo umount /media/overlay/dev/pts sudo umount /media/overlay/dev/ sudo umount /media/overlay/proc sudo umount /media/overlay/sys sudo umount /media/overlay/tmp sudo umount /media/overlay sudo umount /media/filesystem.squashfs sudo umount /media/iso sudo umount /media/myadditions sudo rmdir /media/iso /media/filesystem.squashfs /media/overlay
Congratulations! You now have an Linux OS on your USB with persistence that will boot within seconds and reliably outperforms the competition!
Further Details
Upgrading
Q: What do we do when linux-image (the package containing the kernel) is upgraded?
A: apt will want to update /boot/vmlinuz and /boot/initrd.img. So we need to do things in order to allow this to happen. We must boot with 'toram' and also remount 'live' as rw before running apt-get update.
#TODO, I wrote this elsewhere need to copy here, something along these lines: mount -o remount,rw /lib/live/mount/medium /dev/disk/by-label/live sudo apt-get update
In this upgrade process apt might rename vmlinuz to say vmlinuz-3.16.0-4-amd64, and initrd similarly. It didn't always do this. But, you must match the names of those files with what is listed in grub.cfg! Either use 'vmlinuz' or whatever name the apt script renamed them to.
Q: What do we do when libc is upgraded? Be it a security update, or when Debian Stable advances every couple years.
A: We will want to do a resquash:
- We need to to the same as above, 'toram' and remount 'live' as rw.
- dpkq-query our currently installed packages, and compare to /boot/filesystem.packages.xz we made earlier.
- Remove all non "core" packages, they should be cached in /var/cache/apt/archives/ so they don't need to be downloaded again, just reinstalled after we do the resquash.
- apt-get update
- Resquash, reboot to verify things working, in 'live' mode remove squashed stuff from persistance partition.
- squash script https://gist.github.com/AndrewSmart/90eb186aea08db8f1426
- cleanup script https://gist.github.com/AndrewSmart/2f67f79f6f1922c4556f
- Reinstall packages removed prior to resquash from /var/cache/apt/archives
Q: That looks like a lot to deal with.
A: I agree. I'm looking at using dpkg triggers or /etc/apt/apt.conf.d/ hooks to manage this upgrade process. I hope to have something sufficient soon.
I hope to also make a smaller guide that is not optimal in performance, but has way less steps, so the end user can test out the persistence and take the time to "upgrade" the performance later if so inclined.
Troubleshooting
Keep in mind with the USB we can still boot either in 'live' mode, or 'persistence' mode depending on the GRUB entry we choose. This is useful in case something breaks in 'persistence' mode, like libc was upgraded in 'persistence' but libc on 'live' is still old! This will result in segmentation faults... a kernel panic, or whatever, it won't boot. We can access all of the software in the squashfs in 'live mode'. Eventually I'd like to write an apt hook to prevent the update of libc unless the system is prepared ('toram', and 'live' partition is mounted rw, the hook won't trigger unless 'persistence' kernel parameter is used).
Debian's live-boot scripts will put stuff in /lib/live/, and /lib/live/mount is of particularly good use in troubleshooting.
Ubuntu is a bit of a can of worms from my perspective as a fan of vanilla Debian. So... hopefully things will go smoothly... I highly recommend a lighter-weight environment for USB persistence, but I understand everyone has their preferences.
Since 'toram' puts the squashfs into memory, lets say you have 8GB RAM and an 800MB squashfs, that is fine, but say you move to a workstation with only 2GB RAM, you will probably want to remove the 'toram' kernel parameter, things will be slower but the memory footprint used by the OS will be less, and only use 'toram' when updating the 'live' partition as detailed above. Perhaps consider 2 additional GRUB entries, in addition to the two I have listed above, where the 2 additional entries are live and persistence mode without 'toram' kernel parameter, and when booting select the entry depending on the amount of RAM the system has.
In the event of a power outage, kernel panic, unsafe shutdown or whatever, the filesystem may have problems you're warned about on next boot. In this setup, the 'live' partition was mounted ro, so it is not possible to have filesystem errors there, but with the 'persistence' partition there will likely be errors. So, on next boot we select the 'live' mode in grub, and run fsck.ext4 -p /dev/disk/by-label/persistence, and all problems are fixed!
--Andrewusu (talk) 06:43, 5 November 2020 (UTC)
Works well. I hope to make this more user friendly, to write an apt hook which handles live partition and resquashing where appropriate. Also a systemd service which cleans up that which was resquashed, which runs prior to the overlayfs mount at boot, intelligently regenerates /var/cache components, and handles tmpfs bells and whistles. I wrote part of the apt hook. With those things done then there will be no burden on the end-user, but until then this squashfs persistence technique is not production ready.
