Running NixOS on a Publicly-Facing Server - One Week In


A week ago I decided to take the plunge and finally host a "real", public-facing service on a server running NixOS. The experience so far has been very good and I thought it would be helpful to others to post some of my notes.

What Service?

So far, I'm only running my personal Nextcloud instance. It's a very simple, LAMP-style architecture that does not use any virtualization or containers of any kind. For someone who's been waist-deep in Containerland for a couple of years it feels like eating ice cream for breakfast.

Constraints

I've tried running NixOS in the past in a "hybrid" way where I try to install as much as I can using traditional, non-Nix methods. For example, I found the Emacs package config to be too confusing at first so I just compiled it from source and placed it under /usr/local/.

However, this time I want to do things the Nix way as much as possible, which means learning a heckuva lot more in the near future about the Nix language and derivations. For me this means that I'm going to try and avoid the following:

  1. Ansible - I really like Ansible and use it to manage every other device I use but it doesn't really seem to be Nix way. I'm therefore going to try and get the same results using Nix system alternatives.

  2. Installing packages that don't use Nix derivations - This one's hard for me because I still have a lot to learn about creating my own derivations/packages when one isn't available in the standard library.

Why Nix?

The best way I can summarize the goal of the Nix system is that it applies functional programming idioms to the practice of software compilation and deployment. By doing so, you create systems that are declarative, reliable and reproducible. You can think of it as an alternative to Ansible + Docker architectural pattern that is very popular in the DevOps world today.

I was first drawn to Nix because I like functional programming and I'm a former Systems Administrator, but I must admit that I didn't really understand what it was actually about at first. I was just playing around and picking up cool ideas that I could drop into my other projects (like using lorri and nix-shell to quickly build Python environments).

Now though, I see that it is an audacious, swing for the fences attempt at completely changing the way that software is built and distributed. And I ๐Ÿ’™ that. I love that a group of volunteers and academics created such a big idea and have nurtured it into the very robust and mature system that it is today. I also love that they have done so without falling in the trap of trying to create the next 7-minute abs of DevOps hotness.

High Points

Installation and Configuration

I can't get over how easy it was to setup Nextcloud on my Nix server using the standard package. I was able to copy most of this excellent tutorial by Jacob Neplokh and in under 30 minutes I had a new, fast and stable Nextlcoud instance with TLS courtesy of Let's Encrypt.

Backups

First I backup PostgreSQL with this postgres.nix file that I reference from my configuration.nix file:

{ config, lib, pkgs, ... }:

{
  services.postgresqlBackup = {
    enable = true;
    backupAll = true;
    startAt = "*-*-* 23:00:00";
  };
}

This backs up all of my databases under /var/backup/postgresql on a daily basis at 11:00 PM.

I then setup scheduled Borgbase backups using this excellent tutorial from Christine Dodrill. Easy-peasy.

Patching

I learned that there's a special small channel recommended for server users from this section of the NixOS manual. So naturally I set that up and also setup automatic, daily updates with reboots if required using information from the same section of the docs.

Scheduled Tasks

All of these scheduled tasks are managed using systemd timers, which is a new interface for me. I'm used to scheduling this type of work using cron (ugh) or Jenkins, which gives you significantly better results but requires much more setup. So I can add this to the list of fun new things I will learn with this experiment.

Challenges / Learning Opportunities ๐Ÿ˜บ

There's definitely a lot of new things to learn to make this system work and (hopefully) grow to meet my needs. The canonical beginner's learning resource in the Nix world is the Nix Pills series, which basically says, "Hey, you want to use this awesome system? Great! Now let's learn the basics of a functional DSL to build packages."

As strange as that may sound it's actually appropriate for Nix due the packaging system's declarative nature. And the tutorial / pills are well-written and the curriculum is well-designed. However, the learning curve is non-trivial, even for people who know a few things about Linux and functional programming. That can be frustrating, especially if you try to skip levels or just "hack something out" at 2 AM.

Thankfully, the fact that I have to learn a few new things to maintain a system that I no longer can live without is a feature, not a bug.