Tag-Archive for » ubuntu «

Sunday, August 04th, 2013 | Author:

History

Much had changed since I last mentioned my personal server – it has grown by leaps and bounds (it now has a 7TB md RAID6) and it had recently been rebuilt with Ubuntu Server.

Arch was never a mistake. Arch Linux had already taught me so much about Linux (and will continue to do so on my other desktop). But Arch definitely requires more time and attention than I would like to spend on a server. Ideally I’d prefer to be able to forget about the server for a while until a reminder email says “um … there’s a couple updates you should look at, buddy.”

Space isn’t free – and neither is space

The opportunity to migrate to Ubuntu was the fact that I had run out of SATA ports, the ports required to connect hard drives to the rest of the computer – that 7TB RAID array uses a lot of ports! I had even given away my very old 200GB hard disk as it took up one of those ports. I also warned the recipient that the disk’s SMART monitoring indicated it was unreliable. As a temporary workaround to the lack of SATA ports, I had even migrated the server’s OS to a set of four USB sticks in an md RAID1. Crazy. I know. I wasn’t too happy about the speed. I decided to go out and buy a new reliable hard drive and a SATA expansion card to go with it.

The server’s primary Arch partition was using about 7GB of disk. A big chunk of that was a swap file, cached data and otherwise miscellaneous or unnecessary files. Overall the actual size of the OS, including the /home folder, was only about 2GB. This prompted me to look into a super-fast SSD drive, thinking perhaps a smaller one might not be so expensive. It turned out that the cheapest non-SSD drive I could find actually cost more than one of these relatively small SSDs. Yay for me. 🙂

Choice? Woah?!

In choosing the OS, I’d already decided it wouldn’t be Arch. Out of all the other popular distributions, I’m most familiar with Ubuntu and CentOS. Fedora was also a possibility – but I hadn’t seriously yet considered it for a server. Ubuntu won the round.

The next decision I had to make didn’t occur to me until Ubiquity (Ubuntu’s installation wizard) asked it of me: How to set up the partitions.

I was new to using SSDs in Linux – I’m well aware of the pitfalls of not using them correctly, mostly due to their risk of poor longevity if misused.

I didn’t want to use a dedicated swap partition. I plan on upgrading the server’s motherboard/CPU/memory not too far in the future. Based on that I decided I will put swap into a swap file on the existing md RAID. The swap won’t be particularly fast but its only purpose will be for that rare occasion when something’s gone wrong and the memory isn’t available.

This then left me to give the root path the full 60GB out of an Intel 330 SSD. I considered separating /home but it just seemed a little pointless, given how little was used in the past. I first set up the partition with LVM – something I’ve recently been doing whenever I set up a Linux box (really, there’s no excuse not to use LVM). When it got to the part where I would configure the filesystem, I clicked the drop-down and instinctively selected ext4. Then I noticed btrfs in the same list. Hang on!!

But a what?

Btrfs (“butter-eff-ess”, “better-eff-ess”, “bee-tree-eff-ess”, or whatever you fancy on the day) is a relatively new filesystem developed in order to bring Linux’ filesystem capabilities back on track with current filesystem tech. The existing King-of-the-Hill filesystem, “ext” (the current version called ext4) is pretty good – but it is limited, stuck in an old paradigm (think of a brand new F22 Raptor vs. an F4 Phantom with a half-jested attempt at an equivalency upgrade) and is unlikely to be able to compete for very long with newer Enterprise filesystems such as Oracle’s ZFS. Btrfs still has a long way to go and is still considered experimental (depending on who you ask and what features you need). Many consider it to be stable for basic use – but nobody is going to make any guarantees. And, of course, everyone is saying to make and test backups!

Mooooooo

The most fundamental difference between ext and btrfs is that btrfs is a “CoW” or “Copy on Write” filesystem. This means that data is never actually deliberately overwritten by the filesystem’s internals. If you write a change to a file, btrfs will write your changes to a new location on physical media and will update the internal pointers to refer to the new location. Btrfs goes a step further in that those internal pointers (referred to as metadata) are also CoW. Older versions of ext would have simply overwritten the data. Ext4 would use a Journal to ensure that corruption won’t occur should the AC plug be yanked out at the most inopportune moment. The journal results in a similar number of steps required to update data. With an SSD, the underlying hardware operates a similar CoW process no matter what filesystem you’re using. This is because SSD drives cannot actually overwrite data – they have to copy the data (with your changes) to a new location and then erase the old block entirely. An optimisation in this area is that an SSD might not even erase the old block but rather simply make a note to erase the block at a later time when things aren’t so busy. The end result is that SSD drives fit very well with a CoW filesystem and don’t perform as well with non-CoW filesystems.

To make matters interesting, CoW in the filesystem easily goes hand in hand with a feature called deduplication. This allows two (or more) identical blocks of data to be stored using only a single copy, saving space. With CoW, if a deduplicated file is modified, the separate twin won’t be affected as the modified file’s data will have been written to a different physical block.

CoW in turn makes snapshotting relatively easy to implement. When a snapshot is made the system merely records the new snapshot as being a duplication of all data and metadata within the volume. With CoW, when changes are made, the snapshot’s data stays intact, and a consistent view of the filesystem’s status at the time the snapshot was made can be maintained.

A new friend

With the above in mind, especially as Ubuntu has made btrfs available as an install-time option, I figured it would be a good time to dive into btrfs and explore a little. 🙂

Part 2 coming soon …

Share
Thursday, January 01st, 2009 | Author:

Apparently, what operating system you use can say a lot about you. If you’re using some form of *nix, which distro you’re using can say a lot as well. Redundancy aside, I believe that a Linux distribution depends absolutely on its package management and distribution system.

I liked apt-get (1, 2) but there was some technical problem at some point and it caused me to use aptitude instead. Using aptitude is slightly easier – it has more features automated into single, logical, commands where apt-get requires separate commands. Aptitude also has a curses-based GUI. If you’re not using the GUI then, other than brevity in terms of number of commands to learn, there is apparently no technical reason to prefer one over the other. Aptitude and apt-get serve K/X/Ubuntu and Debian well. From this point, I use the names Kubuntu and Ubuntu in a loosely interchangeable fashion.

In my use of CentOS (based on Red Hat), I’ve found I like yum. It seems to work in much the same as aptitude – one command to rule them all. It has some rather annoying default behaviour I’m not going to get into here as its most likely because I’m just not used to it. At least from a technical perspective, it is very good. I believe that Fedora also makes use of yum though my experience with Fedora is very limited.

the theory…

Fedora and Ubuntu are in a class of distributions that have a fairly rigorous release cycle. Ubuntu 8.10 (the version is named so for the year and month of its release) will not, except for major bugs and minor changes, have another major update until the next version, Jaunty Jackalope. Ubuntu users have the latest versions of most software on their desktops right now. In the months preceding the next release, however, they’re not going to be so lucky unless they like using “beta” releases. As I’m not very familiar with Fedora, I’m not going to bother going into its release cycle.

These 2 distributions are also within a class of distributions known as “binary” or “binary-based” distributions. This means that when you download an update, the files that are downloaded are precompiled and should run on any “supported” hardware. This isn’t specifically optimised for your desktop’s hardware, for example, your processor. Perhaps you have an AMD processor which has extra instruction support which Intel CPUs do not have. The reverse could also be true. For this reason, a binary-release distribution cannot optimise for one particular brand of hardware. Regardless of this “non-optimisation”, it should run at a decent pace.

the practice!

About 2 years ago I started using Kubuntu. After a few months of working with it, I started to learn more about its specifics. I’m not much of a fan of using GUI tools to update the system when, ultimately, its all happening on the command-line anyway. The GUI tools just hide the complexity I don’t mind seeing.

I ended up making a bash script, update, which would run all the steps required to get aptitude to just go ahead and upgrade already, kthx?©, perhaps stopping along the way to back up my configuration, remount the NFS network share where we keep an on-site repository, back up the local cache of aptitude’s installed packages, do some folder-link shuffling to use a local copy if the network share couldn’t remount, sync between the local copy and the network share if the previous update had a network share issue, and update lists of packages in the repository. In general, it wouldn’t go ahead if there were any errors though, as you can tell, this script became a messy beast that went above and beyond the original requirements. It worked well for me.

Until the day came to update between Kubuntu 6.10 to 7.04. I did this manually though, not with the script.

I ended up reinstalling from scratch as a result of the mess that ensued. At least, as a backup administrator should do well to demonstrate, it was easy to recover everything I really needed. 🙂

What else is out there?

Even before I had to reinstall Kubuntu, I was introduced to another distribution called Gentoo. There are 2 very distinct differences between Gentoo and Ubuntu’s update system. The first is that Gentoo is a source-based distribution. This means that when you update a package, the package manager downloads the source and compiles everything, hopefully optimising it for your system. This, I think, is very cool. The downside to this is that compiling everything takes a very long time.

Here are my (very unscientific) estimates for the length of time it takes to install a basic GUI OS to a desktop from installation media, excluding extraneous drivers (for example, the latest 3D graphics drivers):

OS: min – max (median)

Windows Vista: 15 – 30 (20) minutes

Ubuntu: 15 – 40 (20) minutes

Gentoo: 3 – 40 (6) hours

Gentoo also requires much tinkering with the config files in order to get things working – this is another reason for the extremely long delay between inserting the CD and booting your awesome* new desktop. Popular applications have binary packages available for download – though this isn’t a default option.

They see me rollin’

There is one more very important distinction Gentoo has from most other distributions. It is a “rolling-release” distribution. This means that there isn’t any rigorous version or “release” that the distribution adheres to. If you install Gentoo today… if you finish installing Gentoo today, you’re probably going to have the latest version of all the applications you installed. If some obscure application gets a major update tomorrow, within a few days, if you update your system, you’re going to have that latest version on your desktop.

The difference between this rolling release and the “other” distributions is rather staggering. For example: If KDE 4.2 were to be released tomorrow, you’d probably have to wait less than 2 weeks for it to be available on Gentoo. Ubuntu users might have to wait till 9.04 – that’s a 4-month wait.

Something more suitable?

Personally, I’m not willing to put in the 40 hours of effort to get my system working the way I want it to. My colleague had to reinstall recently for some obscure reason and it turns out he wasn’t willing to put in the 6 hours (he’s more experienced with Gentoo) of effort to get his system back to how it was running either. Instead, Arch Linux caught his eye. Arch Linux is a rolling-release (like Gentoo), binary-based (like Ubuntu) distribution. Its packages (well, the vast majority of them) don’t need much tinkering with their config files to get things working nicely either. Its the best of both worlds!

You still need to know what you’re doing* but if you’ve come to this juncture, it shouldn’t be such a giant leap of faith. Arch Linux’s package manager, called pacman, has built-in dependency and conflict handling. I use another package manager, yaourt (French for yoghurt), which has very quickly become popular with Arch users. Yaourt enhances the functionality of pacman by allowing you to download and install applications directly from the AUR, or Arch User Repository. This repository contains scripts that allow you to automatically download and install many applications that would otherwise be completely unsupported by Arch’s own core developers. It downloads and compiles the package into a chroot’d environment. It then packages the chroot’d environment into a pacman-compatible package tarball and uses pacman to deploy it into your system.

Also, the AUR supports a voting system whereby popular packages get placed into the more official [community] repository. Yaourt also supports an automated voting mechanism whereby, after installing a package via AUR, it asks if you want to vote for its inclusion in [community].

I estimate that the time taken for my Arch installation was about 90 minutes. I don’t recommend Archlinux for newbies though I do recommend it for any Linux user who’s gotten bored with other distros – and wants to get into the nitty gritty without having to install Linux From Scratch. Arch Linux has been getting pretty popular these days. Its currently at number 14 on Distrowatch.

* IF you know what you’re doing. AND YOU BETTER BLOODY KNOW WHAT YOU’RE DOING!
Share