Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Motivation

History of DRBD at ITEG & Clazzes.org

For many years (since CentOS 5) we at ITEG (also the main force behind Clazzes.org) have used DRBD to mirror the partition of each virtual machine (then OpenVZ, now mostly LXC).

Kernel 4.6 hickups with DRBD 8.4.6 as initial suspect cause

After shredding some data last year (probably caused by a combination of aggressive settings and unfortunate module versions), it now (Debian jessie, Kernel 4.6 from jessie-backports, DRBD module 8.4.6) seemed to make troubles again.

For details see TBL.

Debianizing the most-recent upstream release of the kernel modul was our way to divide (as in divide and conquer).

We are currently as unsure as ever about wether DRBD is a cause or a trigger or just the first victim that manages to "cry out when hit".

Nevertheless, maintaining DKMS debs providing the most recent upstream kernel modul seems a good idea in general, so we'll keep it maintained for a while at least, #OpenSourceRules.

Up-to-date DRBD 8 packages in Clazzes.org' Debian repository

We are using the packages below on 5 nodes so far (as of 2016-09-09), with 3 more nodes eventually going to use them after the next reboot (although we are thinking about changing that).
So far it runs as good but also as bad as the module that comes with the Debian kernel 4.6.

Packages

Clazzes.org's Deb server deb.clazzes.org contains a repository "jessie-drbdpkg-8" providing 2 DRBD packages:

drbd8-dkms 8.4.8-1

This packages contains the most-recently available sources of the DRBD8 module, along with with DKMS integration for Debian jessie (probably usable for jessie-based derivates too).

On installation of drbd8-dkms 8.4.9-1 (or later installation of a new of linux-header-*-amd64 alongside the matching linux-image-*-amd64 package) the up-to-date DRBD module is automatically built and installed.

Remark: On some nodes dkms triggered the installation of a linux-image-3.2.0-4-rt-amd64 which can be removed afterwards.

drbd-utils 8.9.7-1

We 'cheated' a bit here: drbd-utils_8.9.7-1 is the package from Debian's unstable/experimental repositories, re-integrated in our DRBD8-repository.

This way any Debian jessie installation has access to up-to-date DRBD8 packages without the need to care about compiling manually or adding Debian unstable to sources.list.

We'll update this package once the mailing list settles down about the next version.

DKMS Promblems, Solutions, hints

Problem

1 of 5 hosts rebooted so far, going from 4.4 to 4.6 through the reboot, failed to load the DRBD module, claiming

modprobe: ERROR: could not insert 'drbd': Exec format error
Possible root cause

This could be due to ABI changes in the kernel, from 4.4 to 4.6. Our other reboots so far have been without changing the kernel version, 4.6 before and after.
However 2 more of our 8 nodes are running 4.5, and their /var/lib/dkms/drbd/8.4.8-1clazzes1/4.6.0-0.bpo.1-amd64/x86_64/module/drbd.ko has a different size from the those nodes with 4.6. Although, /lib/modules/4.6.0-0.bpo.1-amd64/kernel/drivers/block/drbd/drbd.ko has the same size and MD5 sum on machines running 4.5 or 4.6.

Solution used

I solved it with ...

apt-get remove drbd8-dkms
apt-get install drbd8-dkms
/etc/init.d/drbd start
Suspected faster solution

It would propably have been sufficient to just ...

dkms autoinstall
/etc/init.d/drbd start
Proposed check command

It might be a good idea to perform this every now and then:

dkms status

 

 

  • No labels