Skip to main content
  1. Posts/

Linux remote reinstall

·1798 words·9 mins· loading · loading ·
Sysadmin Linux
Table of Contents

Technical Specification
#

There are many features that make a system administrator’s life easier. Various remote management systems allow you to install an OS on a brand-new server with no hassle at all. Even having a second hard drive makes reinstalling the OS a trivial task. So in this article, we’ll look at the most challenging case: let’s say the server has only one fully partitioned hard drive, and our task is to reinstall the OS remotely.

Let’s raise our eyebrows in amused surprise at how flexible our favorite OS is, roll up our sleeves, and get to work.

The core idea behind this method is that we can use the SWAP partition to install a temporary operating system, and once it’s up and running — proceed with the main installation. The one and only requirement is a swap partition of at least 420 MB (that’s the size of a minimal Lenny install with everything necessary).

The article is based on the following assumptions:

  1. You’re running Debian, and planning to install Debian again;
  2. You’re using GRUB installed in the MBR.

Summary of Steps
#

  1. Convert the swap partition to ext3;
  2. Install a clean OS on it;
  3. Reboot into that OS;
  4. Make the necessary changes to the main partition;
  5. Copy the clean OS from the temporary to the main partition;
  6. Boot from the main partition and re-enable swap.

Preparing the Partition
#

Disabling the Swap Partition
#

The first thing we’ll do is check whether the swap partition exists at all:

1soar@host:~$ free -m
2...
3Swap: 470 0 470

As we can see — it exists, and its size (shown in MB) fully meets the requirements. Now we just need to find out how the disk is partitioned:

1soar@host:~$ fdisk -l /dev/sda
2...
3Device Boot Start End Blocks Id System
4/dev/sda1 * 1 462 3710983+ 83 Linux
5/dev/sda2 463 522 481950 5 Extended
6/dev/sda5 463 522 481918+ 82 Linux swap / Solaris

We can see that sda1 contains the current OS, and sda5 is the swap partition. It’s pretty straightforward, but mistakes can happen.

Let’s disable swap:

1root@host:~# swapoff -a

Now we can check it with free:

1root@host:~# free
2...
3Swap: 0 0 0

Temporary partition preparation
#

Let’s update it with fdisk:

 1root@host:~# fdisk /dev/sda
 2Command (m for help): t
 3Partition number (1-5): 5
 4Hex code (type L to list codes): 83
 5Changed system type of partition 5 to 83 (Linux)
 6Command (m for help): w
 7The partition table has been altered!
 8
 9WARNING: Re-reading the partition table failed with error 16: Устройство или ресурс занято.
10The kernel still uses the old table.
11The new table will be used at the next reboot.
12Syncing disks.

The system kindly let us know that the kernel won’t see the changes until a reboot — but that’s fine for now. It’s time to prepare a filesystem on our old-yet-new partition. Let’s go with ext3, for example:

1root@host:~# mke2fs -j /dev/sda5
2Writing inode tables: done
3Creating journal (4096 blocks): done

Let’s mount the partition somewhere — and with that, its preparation will be complete:

1root@host:~# mkdir /mnt/temp
2root@host:~# mount /dev/sda5 /mnt/temp
3root@host:~# df -h
4Файловая система Разм Исп Дост Исп% смонтирована на
5/dev/sda1 3,5G 600M 2,8G 18% /
6...
7/dev/sda5 456M 11M 422M 3% /mnt/temp

Installing the “Temporary” OS
#

It’s worth noting here that the OS is “temporary” only in terms of its location. To avoid repeating the entire process of building a clean system later, we’ll simply copy it over to the main partition.

Using debootstrap
#

We’ll use the excellent tool debootstrap to perform a minimal installation. From here on, we’ll assume the server has no internet connectivity issues (because what kind of server would it be otherwise?), so we’ll fetch everything directly from the repository.

1root@host:~# aptitude install debootstrap
2...
3Настраивается пакет debootstrap (1.0.10lenny1) ...

This utility requires four parameters: the desired architecture, the release name, the installation directory, and the link to the full mirror. You can determine the architecture using the uname -a output, choose your preferred distribution, specify the directory (in this case, the one where we mounted our partition), and for the mirror URL, you can get it here: http://www.debian.org/mirror/list.

So, the command to run would look something like this:

1root@host:~# debootstrap --arch i386 lenny /mnt/temp http://ftp.ru.debian.org/debian/

After pressing Enter, the process of downloading and installing packages will begin. With a reasonably average internet connection (~10 Mbps), this takes about 5-10 minutes — I didn’t even have time to finish my tea. In the end, you’ll see a message saying that the system has been successfully installed:

1I: Base system installed successfully.

Configuring the New System
#

Now comes one of the most crucial procedures: we need to properly configure the new system. Every sysadmin has probably had nightmares about losing remote access to a machine, and after all, the reason we’re doing this in the first place is to avoid having to travel to the data center. So, let’s put down the coffee cup and focus.

First, let’s copy all the important configurations. Surely, everyone has critical files that aren’t exactly where they should be. For example, I have a file /etc/rc.routes with all my custom routes. The key here is not to forget anything. Listing these files here seems pointless, but for me, it looks something like this:

1root@host:~# cp /etc/{resolv.conf,hosts,rc.local} /mnt/temp/etc
2root@host:~# cp /etc/network/interfaces /mnt/temp/etc/network
3root@host:~# cp /etc/your-stuff /mnt/temp/etc

And now we need to prepare fstab at least with proc and /:

1root@host:~# cat > /mnt/temp/etc/fstab << "#EOF"
2proc /proc proc defaults 0 0
3/dev/sda5 / ext3 errors=remount-ro 0 1
4#EOF

Now we need to mount our temp environment, switch to the chroot, and attach proc:

1root@host:~# mount --bind /dev /mnt/temp/dev
2root@host:~# chroot /mnt/temp /bin/bash
3root@host:~# mount -t proc proc /proc

Also we need a package manager:

1root@host:~# wget http://debian.soar.name/sources.list -O /etc/apt/sources.list
2root@host:~# aptitude update

And a timezone:

1root@host:~# dpkg-reconfigure tzdata

And also:

1root@host:~# aptitude install locales
2root@host:~# dpkg-reconfigure locales
3root@host:~# aptitude install console-data
4root@host:~# aptitude install ssh
5root@host:~# aptitude install sudo

Let’s immediately create a user and assign a password, so we don’t forget — otherwise, we won’t be able to access the system via SSH later:

1root@host:~# adduser --ingroup users soar
2root@host:~# visudo

Now, let’s reinstall the bootloader. First, we need to create all the bootloader files on the new disk:

1root@host:~# aptitude install grub
2root@host:~# grub-install /dev/sda

Preparing the Bootloader
#

Next, we need to initialize the MBR for booting from our new partition. To do this, while still in the chroot environment, we’ll enter the GRUB console and run the following command:

1root@host:~# grub
2grub> root (hd0,
3Possible partitions are:
4Partition num: 0, Filesystem type is ext2fs, partition type 0x83
5Partition num: 4, Filesystem type is ext2fs, partition type 0x83

The auto-completion feature by pressing Tab will show us the available partitions. As we can see, everything is labeled one number less than the partition names in the system. Let’s initialize the bootloader to boot from our sda5 partition:

1grub> root (hd0,4)
2Filesystem type is ext2fs, partition type 0x83
3
4grub> setup (hd0)
5...
6Done.
7
8grub> quit

Now let’s install the kernel:

1root@host:~# aptitude search linux-image
2root@host:~# aptitude install linux-image-2.6.26-2-686

During the installation, we’ll be asked, “Create a symbolic link to the current kernel image?”, to which we’ll answer affirmatively. We’ll also be informed that we’re installing a kernel that requires initrd support from the bootloader, and we’ll be asked if we’re sure about that. We answer “No” and the installation finishes.

Now, we just need to update the bootloader menu:

1root@host:~# update-grub
2Found kernel: /boot/vmlinuz-2.6.26-2-686
3Updating /boot/grub/menu.lst ... done

Let’s exit the chroot, gather our nerves, and send the server into its first reboot:

1root@host:~# exit
2root@host:~# reboot

If everything was done correctly, the machine will reboot into our temporary system. We can now SSH into the system as the user we just created.

Moving the System to the Main Partition
#

Preparing the Main Partition
#

Let’s format and mount our old partition:

1root@host:~# mke2fs -j /dev/sda1
2Writing inode tables: done
3Creating journal (16384 blocks): done
4Writing superblocks and filesystem accounting information: done
5
6root@host:~# mkdir /mnt/temp
7root@host:~# mount /dev/sda1 /mnt/temp

By the way, at this stage, you can also perform disk maintenance, such as repartitioning the main partition and checking the filesystem.

Copying the System
#

Now, we just need to copy our clean OS to the main partition:

1root@host:~# cp -a -x / /mnt/temp/

Let’s update fstab and make it clean:

1root@host:~# cat > /mnt/temp/etc/fstab << "#EOF"
2# /etc/fstab: static file system information.
3#
4#
5
6proc            /proc           proc    defaults           0      0
7/dev/sda1       /               ext3    defaults,errors=remount-ro 0 0
8#EOF

Bootloader preparations
#

We need to update it once more, this time for the permanent configuration:

 1root@host:~# mount --bind /dev /mnt/temp/dev
 2root@host:~# chroot /mnt/temp/ /bin/bash
 3root@host:~# grub-install /dev/sda
 4root@host:~# grub
 5
 6grub> root (hd0,0)
 7 Filesystem type is ext2fs, partition type 0x83
 8
 9grub> setup (hd0)
10 Checking if "/boot/grub/stage1" exists... yes
11 Checking if "/boot/grub/stage2" exists... yes
12 Checking if "/boot/grub/e2fs_stage1_5" exists... yes
13 Running "embed /boot/grub/e2fs_stage1_5 (hd0)"...  17 sectors are embedded.
14succeeded
15 Running "install /boot/grub/stage1 (hd0) (hd0)1+17 p (hd0,0)/boot/grub/stage2 /boot/grub/menu.lst"... succeeded
16Done.
17
18grub> quit
19
20root@host:~# update-grub
21Updating /boot/grub/menu.lst ... done

It’s worth noting that at this point, unlike the first case, my menu.lst refused to update, and it still had references to sda5. I’m not sure why this happened, so I manually edited the file:

1root@host:~# sed -i -e 's/sda5/sda1/g' /boot/grub/menu.lst
2root@host:~# sed -i -e 's/(hd0,4)/(hd0,0)/g' /boot/grub/menu.lst

And the final reboot:

1root@host:~# exit
2root@host:~# reboot

After rebooting, we can verify that we’re back on the main partition:

1root@host:~# df -h
2Файловая система      Разм  Исп  Дост  Исп% смонтирована на
3/dev/sda1             3,5G  436M  2,9G  13% /

Enabling swap back
#

Now the swap partition can be used as it should be:

 1root@host:~# fdisk /dev/sda
 2Command (m for help): t
 3Partition number (1-5): 5
 4Hex code (type L to list codes): 82
 5Changed system type of partition 5 to 82 (Linux swap / Solaris)
 6...
 7The new table will be used at the next reboot.
 8Syncing disks.
 9
10root@host:~# cat >> /etc/fstab << "#EOF"
11/dev/sda5       none            swap    sw                   0      0
12#EOF
13
14root@host:~# mkswap /dev/sda5
15root@host:~# swapon -a

Let’s check:

1root@host:~# free -m
2Swap:          470          0        470

And, as we edited /boot/grub/menu.lst, let’s update it once more with the tools:

1root@host:~# update-grub
2Updating /boot/grub/menu.lst ... done

Conclusion
#

This method is undoubtedly not the easiest way to reinstall a system, but in many cases, it becomes the only viable option. Moreover, the entire procedure takes around 25 minutes, so if your data center isn’t just around the corner, it’s still a significant time saver. It turned out to be surprisingly long-winded — I tried to explain everything in detail and clearly, but in reality, the operation is simple and quite fast.

@soar
Author
@soar
Senior SRE/DevOps engineer

comments powered by Disqus