Browse Source

added services

DistroByte 5 months ago
No known key found for this signature in database GPG Key ID: 216AF164FD24BD37
8 changed files with 359 additions and 0 deletions
  1. +30
  2. +11
  3. +33
  4. +32
  5. +25
  6. +65
  7. +116
  8. +47

+ 30
- 0
docs/services/ View File

@@ -0,0 +1,30 @@
# Bind9 - `distro`, `ylmcc`

Bind9 is our DNS provider. Currently it runs on Paphos, but is being moved to Fred during the restructuring.

## Configuration

The config files for bind are located in `/etc/bind/master/`. The most important file in this directory is the
`` file.


You must never update this file without following the steps below first!


## Updating DNS

To update DNS:

1. Change directory to `/etc/bind/master`
2. Back up the `` file, usually to ``
3. Run `rndc freeze` - this stops changes to the file affecting dns while you edit it
4. Edit ``
5. Before changing any DNS entry in the file, you **must** edit the serial number on 4. You can increment it by one if
you want, or follow the format: `YYYYMMDDrev` where rev is revision
6. Once you are happy with your file, you can check it with `named-checkzone`
7. If this returns no errors, you are free to run `rndc thaw`
8. Check the status of bind9 by running `service bind9 status`

You can access more logs from bind9 by checking `/var/log/named/default.log`.

+ 11
- 0
docs/services/ View File

@@ -0,0 +1,11 @@
# CodiMD - `distro`

CodiMD lives on Zeus as a docker container. It is accessible through [](

CodiMD is built locally and is based on [codimd](, the docs for which are [here](

Hackmd auths against ldap and its configuration is controlled from docker-compose. Go to
`/etc/docker-compose/services/hackmd` on zeus to find the configuration.

See [CodiMD github]( for
more info on configuration. The important points are disabling anonymus users and the ldap settings.

+ 33
- 0
docs/services/ View File

@@ -0,0 +1,33 @@
# Gitea

Redbrick uses [Gitea]( as an open source git host.

- [Gitea docs](
- [Gogs docs](, not really important, but Gitea is built on [Gogs](
- [Link to Redbrick deployment](

## Deployment

Gitea and its database are deployed to Hardcase which runs NixOS

- The actual repositories are stored in `/zroot/git` and most other data is stored in `/var/lib/gitea`
- The `SECRET_KEY` and `INTERNAL_TOKEN_URI` are stored in `/var/secrets`. They are not automatically created and must be
copied when setting up new hosts. Permissions on the `gitea_token.secret` must be 740 and owned by `git:gitea`
- Make sure that the `gitea_token.secret` does NOT have a newline character in it.

## Other Notes

The Giteadmin credentials are in the passwordsafe.

## Operation

Gitea is very well documented in itself. Here's a couple of special commands when deploying/migrating Gitea to a
different host.

# Regenerate hooks which fixes push errors
/path/to/gitea admin regenerate hooks

# If you didn't copy the authorized_keys folder then regen that too
/path/to/gitea admin regenerate keys

+ 32
- 0
docs/services/ View File

@@ -0,0 +1,32 @@
# Icecast - `mcmahon`

Icecast is a streaming server that we currently host on Paphos.

We stream DCUFm's Broadcasts to their apps via a stream presented on ``.

They serve an audio stream (stream128.mp3) via butt on a desktop in their studio to `icecast2`.

Icecast requires root privilege to bind to Port 80; normally icecast2 runs as the `icecast2` user and binds to `8001`.

## Procedure

The configuration file for icecast is located at `/etc/icecast2/icecast.xml`.

<!-- Sources log in with username 'source' --> <-- This is the audio source.
<source-password>$password1</source-password> <-- This must be copied for the DCUFM buttrc.
<!-- Relays log in username 'relay' -->
<admin-user>admin</admin-user> <-- This is for the WebUI frontend


<bind-address></bind-address> <-- i.p. addr for A Record.

After that you must configure the default behaviour for the icecast server to allow icecast2 to bind to port 80.

Set `USERID` & `GROUPID` in `/etc/defaults/icecast2` to `root`.

+ 25
- 0
docs/services/ View File

@@ -0,0 +1,25 @@
# Preface

Here you will find a list of all the services Redbrick runs, along with some configs and some important information
surrounding them.

## Adding More Services

In order to add a new service, you will need to edit the [docs]( repository.

Adding a new service is as easy as creating a new file in `docs/services/` with an appropriate name, and adding the
reference to `mkdocs.yml` in the root of the repository.

The style guide for a service file should be as follows:

# ServiceName - `username`

Short description on how the service works and where it is running

## Configuration

Add some possible useful configs here, like a docker-compose file,
certain command you may have had to run, or something that is not very obvious.
Look at other services for hints on this.

+ 65
- 0
docs/services/ View File

@@ -0,0 +1,65 @@

## Redbrick InspIRCd

In 2016/2017 we began work to move to InspIRCd. This was due to the complications in ircd-hybrid and how old it was.
These complications stopped new netsocs joining us so we all agreed to move irc. $ 4 years later after multiple attempts
we had not migrated. Until TCD decided to shutdown their server breaking the network.

We run Inspircd v3 on Metharme. InspIRCd's docs can be found [here]( for configuration specifics.

IRC is available at `` on port `6697`. SSL is required for connection, we do not support non-SSL.

When connecting from a redbrick server a user will be automatically logged in. If connecting from an external server a
user must pass their password on login.

For the purpose of external peering of other servers the port `7001` is expose as well. Similarly to clients we only
support SSL on this port.

For docs on connecting and using an IRC client please refer to the [wiki](

## Installation

InspIRCd is installed with Nix. There is no Nix package for InspIRCd so we compile a specific git tag from source. See
[Nix package]( for details on how it is compiled.

Given we only support SSL and require LDAP, we need to enable both at compile time.

## Configuration

InspIRCd's configuration is in Nix [here](
This config will be converted to xml on disc.

### Important Configuration

_oper_ is a list of admin users on the irc server. Their `OPER` password will need to be manually hashed with
`hmac-sha256`, and placed in a secret on the server to be read in by inspircd.

_ldapwhitelist_ is a list of cidr addresses that do no require authentication. The list consists of Redbrick public and
private addresses as well as `oldsoc`.

_link_ is a list of all servers we peer with including the anope services server that runs on the same box.


`` is a server run by old TCD netsocers. All the users on it are the remaining TCD associates following the
shutdown of TCD IRCd. This server is maintained by its own users and has explicit permission to join IRC without LDAP auth.

## Anope

Redbrick runs Anope services for the entire network. As with
[inspircd we compile]( from source. Refer to anopes
[github docs]( for configuration specifics.

Our current Anope is configured with standard mods of chanserv, nickserv and operserv. All config is in [here](

Anope stores all info in a custom db file on disk.

## Discord Bridge - `butlerx`

We run a [bridge]( between the Redbrick Discord and irc. The configuration for
this is [here](

The bridge adds all users from discord with the suffix `_d2` and all irc users appear as them self but tagged as a bot
in discord. Not all discord channels are on IRC, the config above contains a mapping of irc channels to discord channels
id's. This needs to be manually updated to add more channels.

+ 116
- 0
docs/services/ View File

@@ -0,0 +1,116 @@
# NFS / Network File Storage

NFS is used to serve the notorious `/storage` directory on Icarus to all of Redbrick's machines, which in turn serves
`/home`, `/webtree` and some other critical folders.

## Deployment

- NFS is deployed with Nix on Icarus
- It is backed onto the PowerVault MD1200 with all its disk passed through single-drive RAID 0s toallow for setup of ZFS:
- 1 mirror of 2x 500GB drives
- 1 mirror of 2x 750GB drives
- 1 mirror of 2x 1TB drives
- Stripe across all the mirrors for 2TB of usable storage
- 1 hot spare 750GB drive
- ZFS is configured with compression onand dedup off
- The ZFS pool is called `zbackup`

## Redbrick Special Notes

On each machine where `/storage` is where NFS is mounted, but `/home` and `/webtree` are symlinks into there.

There are 2 scripts used to control quotas, detailed below.

NFS is backed up to Albus via [ZnapZend](/services/

## `zfsquota` and `zfsquotaquery`

These are two bash scripts that run as systemd services on Icarus to manage quotas. This is achieved through getting and
setting the `userquota` and `userused` properties of the ZFS dataset.

### zfsquota

ZFSQuota will read the `quota` field from LDAP and sync this with the userquota value on the dataset. It is not event
driven - it runs on a timer every 3 hours and syncs all LDAP quotas with ZFS. It can be kicked off manually, which is
described below. Users with no quota in LDAP will have no quota in `/storage`, and users who have their quota removed will
persist on ZFS.

Changing user names has no impact on this since it is synced with uidNumber.

### zfsquotaquery

ZFSQuotaQuery returns the quota and used space of a particular user. This is used to then inform `rbquota` which provides
the data for the MOTD used space report. Both of these scripts are defined and deployed in the Nix config repo. It runs on
port 1995/tcp.

## Operation

In general, there isn't too much to do with NFS. Below are some commands of interest for checking its status.

# On the NFS server, list the exported filesystems
showmount -e

# Get the real space usage + fragmentation percent from ZFS
zpool list zbackup

# Check a user's quota
zpool get userquota@m1cr0man zbackup
zpool get userused@m1cr0man zbackup

# Delete a quota from ZFS (useful if a user is deleted)
zpool set userquota@123456=none zbackup

# Get all user quota usage, and sort it by usage
zfs userspace -o used,name zbackup | sort -h | tee used_space.txt

# Resync quotas (this command will not return until it is finished)
systemctl start zfsquota

# Check the status of zfsquotaquery
systemctl status zfsquotaquery

## Troubleshooting

In the event where clients are unable to read from NFS, your priority should be restoring the NFS server, rather than
unmounting NFS from clients. This is because NFS is mounted in `hard` mode everywhere, meaning that it will block on IO
until a request can be fulfilled.

### Check The Server

# Check the ZFS volume is readable and writable
ls -l /zbackup/home
touch /zbackup/testfile

# Check that rpc.mountd, rpc.statd and rpcbind are running and lisening
ss -anlp | grep rpc

# Check the above services for errors (don't worry about blkmap)
systemctl status nfs-{server,idmapd,mountd}
journalctl -fu nfs-server -u nfs-idmapd -u nfs-mountd

### Check The Client

# Check for connection to NFS
ss -atp | grep nfs

# Check the fstab entry
grep storage /etc/fstab

# Check if the NFS server port can be reached
telnet 2049
# Entering gibberish should cause the connection to close

# Remount read-only
mount -o remount,ro /storage

# Not much left you can do but remount entirely or reboot

### Rolling Back or Restoring a Backup

See [znapzend](/services/znapzend/)

+ 47
- 0
docs/services/ View File

@@ -0,0 +1,47 @@
# ZnapZend

## Overview

[ZnapZend]( is used to back up the NFS ZFS dataset from our NFS server to Albus.
It can also be used to back up other ZFS datasets on other hosts, but at the time of writing NFS is the only
thing being backed up this way.

ZnapZend runs on the client and sends backups to Albus over SSH using `zfs send | zfs receive` piping.

The backup strategy can be viewed in the [NixOS configuration](

## Adding Another Backup

There is not much manual configuration to add a host to the ZnapZend backups.

1. Create an SSH key for the root user with no passphrase on the host you want to send the backups from. Use
`ssh-keygen -t ed25519`.
2. Add this new SSH public key to the rbbackup user's authorized keys on [Albus](
3. Try SSHing to `rbbackups@albus.internal` to load the host key and test the passwordless authentication.
4. Import the [znapzend service config](
on the sending host and configure `redbrick.znapzendSourceDataset` and `redbrick.znapzendDestDataset`. Then apply the config.
**NOTE** The `DestDataset` must be unique across all configured backups/servers.

## Debugging

Znapzend runs at the top of every hour to make backups. You can watch the progress with `journalctl -fu znapzend.service`.
Failures are usually caused by incorrect SSH configuration, so make sure that passwordless auth using the sending host's
root SSH key is working.

## Rolling Back NFS

If the NFS server is online and functional, you do not need to involve Albus to roll back changes, as all the snapshots
are kept on Icarus too.

1. Find the snapshot you want to restore with zfs list -t snapshot.
2. Run zfs rollback $snapshotname.

That's it! These instructions obviously work for backups other than NFS too, should any ever exist.

## Restoring NFS from a backup

If the NFS server has died or you are creating a copy of it, here's how to pull the dataset from Albus,

1. On Albus, find the snapshot you want to restore with `zfs list -t snapshot`.
2. Open a screen/tmux, and copy the snapshot to a dataset in your target ZFS pool with
`ssh albus zfs send -vRLec $snapshotname | zfs receive $newpool/$datasetname`.