Why

Because you forget. Don’t waste time gathering information once again when you need to troubleshoot or change something, write it down for your own sake. If you have other users on the system, it will be beneficial for them as well. Also, it’s a great way to show off what kind of stuff you have functioning and what you managed to implement. Hopefully, it shortens the time it takes to fix a problem.

How

This is a Slashdot thread linked to me by a colleague, which contains an interesting discussion of methods and ideology. What I got out of it is that a wiki system is very useful, and a nice structure I adapted from mr. EEBaum:

==Purpose==
{What we want to do with this machine}

==Access==
 Hostname: 
 Alias:
 IP:
 MAC:
 Ethernet device: ethX

{How it is reachable, what ports are used}

==Services==
{What it does for us}

==Maintenance==
{What we have to do at regular intervals}

==Hacks==
{Unorthodox modifications that were required for
things to function as we want}

==Technical Specs==
{Hardware}

==Installation notes==
{The options and parameters given at installation}

==Incidents==
{Chronological log of complications and their fixes}

Separately, I have a sketch over the logical topology and the backup processes. The central authentication system deserves special attention too, as well as a physical topology.

I chose Mediawiki from the start for this documentation, which might have been a hasty decision. For my home network, I’ll go with DokuWiki, which contains some nice features and seems to be aimed at documentation.

Create a VirtualHost configuration in a file similar to /etc/apache2/sites-available/cloudspike.com

<VirtualHost *:80>
        ServerName cloudspike.com
        KeepAlive Off
        RewriteEngine On
        RewriteCond %{HTTP_host} ^[^./]+.[^./]+$
        RewriteRule ^/(.*)$ http://www.%{HTTP_host}/$1 [R=301,L]
</VirtualHost>

Enable mod_rewrite and our site (as root, of course):

a2enmod rewrite && a2ensite cloudspike.com && apache2ctl restart

Why

Having a secure chat server of your own is useful for companies, where internal discussion must never leave the corporate network, and very paranoid people, who think the authorities are sniffing their packets. Jabber is the most versatile protocol out there, having lots of features I probably won’t even use. It is possible to execute commands on the server, communicate with googletalk and other XMPP users

When I needed to decide what server software to use, I looked at popularity-contest lists for Debian. Packages “ejabberd” and “jabber” were at the top, of which ejabberd is significantly larger and written in Erlang. I haven’t previously ever used software written in Erlang or seen Erlang code. It seemed to have PAM configuration included, and since that was what I most of all wanted, to authenticate users with local unix account database (/etc/shadow), I chose ejabberd. The web chat called jwchat followed since it also was in apt sources tree and there were some pages about integrating it with ejabberd.

How

Ejabberd and a web client, jwclient, are included in apt:

sudo apt-get install ejabberd jwchat

Modifications to /etc/ejabberd/ejabberd.cfg:

%% Admin user
{acl, admin, {user, "me", "cloudspike.com"}}.
%% Hostname
{hosts, ["cloudspike.com"]}.

%% Change starttls to starttls_required in the line below
{5222, ejabberd_c2s, [ {access, c2s}, {shaper, c2s_shaper}, 
{max_stanza_size, 65536}, starttls_required, {certfile, "/etc/ejabberd/ejabberd.pem"} ]},
%% http_bind instead of http_poll
{5280, ejabberd_http, [ http_bind, web_admin ]}

%% Comment following line
%%{auth_method, internal}.
%% Uncomment this instead
{auth_method, pam}.
{pam_service, "ejabberd"}.

%% Add the following line in the list under "{modules"
{mod_http_bind, []},

Create /etc/pam.d/ejabberd:

#%PAM-1.0
auth        sufficient  pam_unix.so likeauth nullok nodelay
account     sufficient  pam_unix.so

Give “epam” sufficient permissions:

chown root:ejabberd /usr/lib/ejabberd/priv/bin/epam
chmod 4750 /usr/lib/ejabberd/priv/bin/epam

Switch the included certificate to use correct “Common Name”. If you in the e-mail field use the e-mail that the contact person for the domain name has, you get cool points, even though it’s just a self-signed certificate. Let’s do things properly:

sudo su
cd /etc/ejabberd/
mv ejabberd.pem ejabberd.pem.dist
openssl req -new -x509 -newkey rsa:1024 -days 3650 -keyout privkey.pem -out ejabberd.pem
    Country Name (2 letter code) [AU]:SE
    State or Province Name (full name) [Some-State]:Stockholms län
    Locality Name (eg, city) []:Stockholm
    Organization Name (eg, company) [Internet Widgits Pty Ltd]:Cloudspike
    Organizational Unit Name (eg, section) []:
    Common Name (eg, YOUR name) []:cloudspike.com
    Email Address []:admin@cloudspike.com
openssl rsa -in privkey.pem -out privkey.pem
cat privkey.pem >> ejabberd.pem
rm privkey.pem; chmod 640 ejabberd.pem; chown :ejabberd ejabberd.pem

Restart ejabberd by invoking stop and then start. The restart command in Ubuntu is too quick for the slow Erlang language and breaks the service. Processes “epmd” and “beam” need to be killed before ejabberd can be started again after such a botch.

sudo /etc/init.d/ejabberd stop
sudo /etc/init.d/ejabberd start

You can access the web administration interface at http://hostname:5280/admin

Edit /etc/jwchat/config.js:

var SITENAME = "cloudspike.com";

The configuration at /etc/apache2/sites-available/jwchat is quite broken. It doesn’t work out of the box, and it looks suspiciously similar to an unsecured proxy. This is how it should be, if mod_ssl is loaded in Apache:

<VirtualHost *:443>
  ServerName http://jabber.cloudspike.com
  DocumentRoot /usr/share/jwchat/www

  <Directory />
    Options FollowSymLinks
    AllowOverride None
  </Directory>

  <Directory /usr/share/jwchat/www>
    Options Indexes Multiviews FollowSymLinks
    AllowOverride None
    Order allow,deny
    allow from all
  </Directory>

  AddDefaultCharset UTF-8

  ProxyPass /http-bind/ http://cloudspike.com:5280/http-bind/
  ProxyPassReverse /http-bind/ http://cloudspike.com:5280/http-bind/
  <Proxy *>
    Allow from all
  </Proxy>

  CustomLog /var/log/apache2/jwchat_access.log combined
  LogLevel warn

  SSLEngine on
  SSLCertificateFile /etc/ejabberd/ejabberd.pem
  SSLOptions +StdEnvVars

  BrowserMatch ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0

</VirtualHost>

Don’t forget to reload apache with apache2ctl graceful.

Changing hostname

Erlang is quite a strange language, where software is running on nodes which can communicate with each other through built-in processes. That means that it’s easy to build software which can run on clusters, and ejabberd is no exception. What that also means is that it’s really anal about what host it runs on, and you can’t change your hostname willy-nilly. Furthermore, if you mess ejabberd up with a hostname change, the start script for it in init.d will not report any failure, and ejabberd will quit silently without writing any logs, which is a pain to debug.

If you haven’t messed your ejabberd up and want to change the hostname in its configuration:

ejabberdctl backup /tmp/ejabberd-oldhost.backup
# Change hostname here inbetween
ejabberdctl restore /tmp/ejabberd-oldhost.backup

If you have messed it up, you’ll have to completely wipe the /var/lib/ejabberd directory. This will remove all accounts if you use internal authentication (the Mnesia database for Erlang). If you use PAM authentication, your accounts are not affected, but you still lose any configuration made through the web interface.

If you don’t know what is wrong, and ejabberd doesn’t want to start after you’ve killed the “beam” and “epam” processes, try su ejabberd -c "ejabberdctl live" and google from there

Extra Security

It’s probably not so good to have http-bind and the administration interface on the default ports. If you change the http-bind port, don’t forget to change it as well in the proxy settings of jwchat’s apache configuration file.

If you don’t want to talk to other servers, you should comment out s2s functionality in the configuration file to avoid unnecessary traffic.

Inspired by

ejabberd 2.0.5 Installation and Operation Guide
XMPP Standards Foundation - Nice list of server and client software. For the record, I use the Gajim client.

So here is the success-story of Apache with it’s www-root on an samba share. Seems easy enough?
Wrong!

The background

One day I decided to put apache on a dedicated vmware-guest for um.. safety. (or just because i could now with a Xeon Quad Core)
Since the www-files already was on my host-OS (on the encrypted RAID5 array) i did not want to move them to the apache-VM but resolved to mounting them via samba instead. Some might argue that NFS is better but i already had samba-server on the host so there we go.
I never had problems using samba before so i did not tink about it much but just set it up.

The problem

A little while after moving apache to the new VM-guest i noticed that an image had stop loading completely. Only the top part loaded, the first 66kb actually (it did vary 1-2kb)
What also was strange was that if you hit refresh on the browser a second 66kb chunk of the image loaded. So after 4 refreshs the complete image was loaded. This was the same for all images.
If i moved the image to the local filesystem on the apache-VM everything was fine.

The troubleshooting

I started with removing
socket options = TCP_NODELAY SO_KEEPALIVE=1 SO_RCVBUF=8192 SO_SNDBUF=8192.
from the smb.conf
I also tried with a different webserver, lighttpd, but still the same problem.

The solution

After some googeling i found this:
http://httpd.apache.org/docs/2.0/mod/core.html#enablesendfile
Turned EnableSendfile off and suddenly i was very happy.

Why

In a multi-admin system, it is often important to know that all changes to configuration files are tracked, that changelogs can be viewed, that configurations files can be rolled back after something breaks. I suppose it also makes it easier to back /etc up with repository clone commands. It becomes especially interesting if you can create branches for all servers in the network.

Initially I was looking at svn, Subversion, for the repository. The horrible thing with that was that it creates .svn in every single directory, and I was afraid I would have to code something to catch changes done by apt. Thanks to my friend Jonta, who showed me etckeeper, it all became a lot simpler.

How

By default, etckeeper uses “git” in Ubuntu 8, and “bzr” in Ubuntu 9. I will only describe git here, but bzr has very similar commands.

Install etckeeper:

sudo su
apt-get install etckeeper
etckeeper init
etckeeper commit "initial import"
# Compress the repository:
git gc

To make sure git compresses the repository after apt has done its job, create file /etc/etckeeper/post-commit.d/99git-gc:

nano /etc/etckeeper/post-commit.d/99git-gc
#!/bin/sh
exec git gc

chmod +x /etc/etckeeper/post-commit.d/99git-gc
etckeeper commit "run git gc after apt"

To see a log of changes, use:

cd /etc
sudo git log
# Specific file:
sudo git log hosts

To check whether anything is to be commited:

cd /etc
sudo git status

Commit changes: (You can also use git’s commands, but this is easier, since it adds and removes files too)

sudo etckeeper commit "Changed something here"

Inspired by

Ubuntu 9.04 Server Guide
README included with etckeeper