permacomputing

eik.mdwn (13973B)

---

 Notes regarding `eik`
 
 ssh config
 ----------
 
     Host eik.permacomputing.net
     Hostname 193.170.194.218
     User <username>
     Port 22
     IdentityFile ~/.ssh/<ssh-private-key>
 
 disk volumes
 ------------
 
 `eik` uses LVM, and we recently added 5G to `/var` before realising that that was nearly all of the disk space we'd been allocated.  Growing the volume was as simple as:
 
     sudo lvextend --size +5G --resizefs /dev/mapper/permacomputing--vg-var
     sudo lvextend --extents +100%FREE --resizefs /dev/mapper/permacomputing--vg-root
 
 The first attempt to grow `/` had a typo (missing the `+`), so we reached out and got more disk added (possibly `/dev/vda2`: `vda1` is `/boot` and `vda5` is the LVM pool).  We will review our consumption needs after we complete more work, but we may want to return the space added on 6 May.
 
 
 upgrading debian
 ----------------
 
 We recently added space, which should make upgrades easier. Here are some tricks we developed when space on `/` was tight.
 
 
 * Make sure all old linux kernels are removed. You can upgrade/reboot and then check with `uname -a`/`uname -rms` to see what you're running. Then use `dpkg --list | grep -E 'linux-image|linux-headers'` to identify the old ones and directly `apt remove ...`.
 * Pay attention to what the `apt full-upgrade` disk space info is telling you. Check what you've got on the `/` partition with `df -h`. If you don't have enough, it won't work.
 * Delete a bunch of packages that you don't need if they're on the system.
 * You can usually get away with moving `/usr/include` / `/usr/share` into the `/home` partition and then symlinking them back. This is of course risky business and you should only do it as a last ditch effort.
 
 http certs
 ----------
 
 We leave the visitor the option to `http` or `https`. This is done in `nginx` with only the following instructions in each `site-enabled` config:
 
     listen 80;
     listen 443 ssl;
 
 However when authentication is required we force `https` like this:
 
     location /supercoolauth.cgi {
         return 301 https://$http_host$request_uri;
     }
 
 We use `acme.sh` for handling certs, using [the GANDI DNS API subsystem](https://github.com/acmesh-official/acme.sh/wiki/dnsapi#dns_gandi_livedns) to issue and renew our wildcard cert.  
 
 This relies on an environment variable in `/root/.acme.sh/acme.sh.env` called `GANDI_LIVEDNS_TOKEN`.  This token expires every year, and must be renewed in the first week of May.  
 
 
 cerca
 -----
 
 We maintain a single patch on `bbs-patches` to disable registration (See below for how we manage this with our own Git hosting).
 
 Our system configuration documentation has been upstreamed to [`docs/hosting.md`](https://github.com/cblgh/cerca/blob/main/docs/hosting.md) 🎉
 
 You can edit the CSS, about pages, registration instructions etc. all in `/var/www/bitrot.permacomputing.net/content/`. There are some other configurations lying around there which might be useful also. When making changes, you need to restart Cerca: `systemctl restart bbs-cerca`.
 
 If you run into issues, have ideas for improvements or otherwise wanna get in touch with the main developer, cblgh, they're very open to discussions on the issue tracker [here](https://github.com/cblgh/cerca/issues).
 
 ### Hacking cerca
 
 You can push/pull changes from our private temporary branch. You need to add the `eik` repository to your local checkout. You can do this with something like so: `git remote add pmc ssh://foo@eik:/var/www/git.permacomputing.net/repos/cerca.git` (see "git hosting" below for more details). Then you can `git fetch --all pmc` and you should have a copy of the `bbs-patches` branch. Push all your hacks there so we can get lost in hacks together 🤓
 
 ### Update cerca
 
 Due to resource constraints on `eik`, we are building `cerca` on our own machines and then uploading & replacing the binary to update a new version.
 
 You can build a static binary on your own machine with:
 
     go build -v \
       -ldflags="-s -w -linkmode 'external' -extldflags '-static'" \
       ./cmd/cerca
 
 ### warawara 🤖
 
 When there is activity on the forum we get XMPP notifications from [warawara](https://git.permacomputing.net/warawara). The binary is located at `/usr/local/bin/warawara` and if it needs to be restarted, you can `systemctl restart warawara`.
 
 ### compost 👩‍🌾
 
 Our own [image uploader](https://git.vvvvvvaria.org/pmc/image_upload) for bitrot. It's deployed on eik, `systemctl status compost` to learn more.
 
 motd message
 ------------
 
 See `/etc/update-motd.d/` for the scripts. When a user logs in, these scripts are all run. They must be executable to be run. Feel free to add some nice welcome screens!
 
 There are also some configuration knobs in `/etc/ssh/sshd_config` which related to showing the last log in and the default `/etc/motd`. These have been turned off for now.
 
 git hosting
 -----------
 
 Git repositories are located in `/var/www/git.permacomputing.net/repos`. Any user in the `gitusers` group has access to reading and writing these files.
 
 *  to add a new eik user to the group (replacing `USERNAME` as needed): 
 
    ```usermod --append --groups gitusers USERNAME```
 
 *  to create a new repository on eik (replacing `USERNAME` and `REPONAME`): 
 
    ```
    ssh USERNAME@eik
    cd /var/www/git.permacomputing.net
    ./new_repos.sh REPONAME "one line about the repos"
    ```
 
 *  to clone (locally) your new repo:
 
    ```
    git config --global init.defaultBranch main  # if not done already
    git clone USERNAME@eik:/var/www/git.permacomputing.net/repos/REPONAME.git
    ```
 
 *  to add a repository as a git remote (for `git push`, replacing `USERNAME` and `REPONAME`):
 
    ```
    git remote add pmc \
      ssh://USERNAME@eik:/var/www/git.permacomputing.net/repos/REPONAME.git
    ```
 
 stagit
 ------
 
 [stagit](https://codemadness.org/stagit.html) is used to produce the staticly generated web interface that is [git.permacomputing.net](https://git.permacomputing.net).
 
 If you follow the instructions above ("git hosting") to add a new git repository, then all the needed hooks and git repo modifications have been installed. Next time you push, you should see your changes reflected on [git.permacomputing.net](https://git.permacomputing.net) and [https://git.permacomputing.net/REPO](git.permacomputing.net/REPO).
 
 home pages
 ----------
 
 You can share web stuff via a local `public_html` directory.
 
     cd
     mkdir public_html
     echo "hello, world! > public_html/index.html
 
 Then visit `eik.permacomputing.net/~YOURUSERNAME`.
 
 The `nginx` incantation for this is configured in `/etc/nginx/sites-enabled/eik.permacomputing.net`.
 
 mosh
 ----
 
 [mosh](https://mosh.org) is installed on the server to make doing terminal ops more convenient on intermittent connections, long running commands and so on. See the website for more information. You simply replace `ssh` with `mosh` when you want to connect to the server. You need to install the `mosh` client on your machine also (e.g. `apt install mosh`).
 
 firewall
 --------
 
 [ufw](https://help.ubuntu.com/community/UFW) is installed as a "frontend" for `iptables`. Defaults are set to allow outgoing and block incoming traffic. Only specific ports are open and allow incoming.
 
 IDS
 ---
 
 We use `fail2ban` to dynamically firewall off IPs that appear to be doing suspicious things based on our logs.  If your IP gets banned by mistake, ask someone to find your IP in `sudo fail2ban-client banned` output, figure out which jail you're in, and then do
 
     fail2ban-client set $JAIL_NAME unbanip $IP_ADDRESS
 
 backups
 -------
 
 We are using [restic](https://restic.readthedocs.io/) for backups.
 
 On eik, we have created a specific user, following [these docs](https://restic.readthedocs.io/en/stable/080_examples.html#backing-up-your-system-without-running-restic-as-root).
 
 We're using a temporary homebrew server for remote backups because eik does not have enough storage capacity. You can find the connection details in `/home/restic/.ssh/config`. On the remote hbsc side, we've configured the `authorized_keys` to restrict access to only run the sftp command.
 
     # .ssh/authorized_keys
     restrict,command="/lib/sftp-server" ssh-ed25519 ...
 
 The restic user on eik has a nightly cron configured to run a full system backup. See `/home/restic/backup.sh` for more. Per-backup logs and other configuration are also provided in `/home/restic`.
 
 smtp relay
 ----------
 
 We are following the [LURK priort art](https://things.bleu255.com/runyourown/Postfix_Relay) with a postfix relay configured as a "satellite system". All services on eik can make use of this for "noreply" functionality. Please first check in the Toolshed group if that's OK. LURK want to keep a clean sheet with their mail server address and avoid problems with spam and block listing.
 
 Some handy commands for introspecting the mail system.
 
     mail                       # read mails on local account
     tail -f /var/log/mail.log  # postfix logs
 
 You need to `systemctl restart postfix` when you update the `main.cf`.
 
 You cand send a test mail with the following:
 
     echo "Subject: test from eik" | /usr/sbin/sendmail -v some@where.nice
 
 creating system users
 ---------------------
 
 We seem to be doing this.
 
     useradd --system --shell /sbin/nologin <username>
 
 And then using these in `systemd` files to run services with users with limited permissions. This is at least the case for `cerca` and `rauthy`. Documenting this in case we choose to do it differently in the future.
 
 rauthy
 ------
 
 ### build
 
 Install [rustup](https://rustup.rs), grab `musl-tools` and a copy of the Rauthy source code. You'll need to follow [the standard contrib docs](https://github.com/sebadob/rauthy/blob/main/CONTRIBUTING.md) to get a working development environment up and running.
 
 You can build a binary for eik with the following. Make sure to check out a tag and not build from main. Otherwise, we can potentially destroy our database schema if that binary makes it to eik and is run.
 
     sudo apt-get install musl-tools
     rustup target add x86_64-unknown-linux-musl
     cargo build --target=x86_64-unknown-linux-musl
 
 You can transfer to eik like so.
 
     scp target/x86_64-unknown-linux-musl/release/rauthy eik.permacomputing.net:
 
 Don't forget to `mv` it to `/usr/local/bin` and `chown rauthy:rauthy` on the binary. You should also make sure `rauthy` is stopped on eik before replacing the binary.
 
 ### configure
 
 The `rauthy` binary lives in `/usr/local/bin/rauthy` and it's configuration and data is to be found in `/home/rauthy/config.toml`. Here are the [configuration reference docs](https://sebadob.github.io/rauthy/config/config.html).
 
 You can `journalctl -fu rauthy` to follow along with the logs and the usual `systemctl restart rauthy` will restart Rauthy.
 
 Rauthy runs under the `rauthy` user, please prefix your commands with `sudo -su rauthy` if running commands directly on the configuration or data to avoid borking permissions.
 
 ### admin
 
 See the `/home/rauthy/config.toml` for the fallback admin email. Ask in Toolshed for the password. You can also create an account and be upgraded to administrator by applying the `rauthy_admin` role on user creation.
 
 ### arbitrary single sign-on
 
 It is possible to configure `nginx` to use the [`forward auth`](https://sebadob.github.io/rauthy/work/forward_auth.html#advanced-forward-auth) feature of `rauthy`. This means, you can put anything you serve on `nginx` behind a single sign-on.
 
 This avoids us having to hand out a HTTP basic auth username/password on top of the username/password that people configured for the specific application themselves. If this doesn't make any sense already then you know we had to change it. We have had several reports that this is super confusing and just stopping people logging in to our digital tools.
 
 The configuration is fairly hairy but once you get it, you get it. And yes, if `rauthy` is down, there is no access. It's as solid as HTTP basic auth. The `rauthy` [docs](https://sebadob.github.io/rauthy/work/forward_auth.html#advanced-forward-auth) cover it but the TLDR; if you're moving fast:
 
 * Create a confidential client (we don't use the password) on Rauthy with the correct allowed origin (the URL you want to protect) and redirect URI (the URL you want to protect + /callback)
 * Disable PKCE in the `rauthy` web client UI
 * Configure your Nginx configuration roughly like below. Refer to [the rauthy docs](https://sebadob.github.io/rauthy/work/forward_auth.html#advanced-forward-auth) for full context and tips. Please note, `<YOUR-CLIENT-ID>` must be replaced in the minimal configuration example below.
 
 Here's an example `nginx` configuration.
     
     upstream rauthy {
       server 127.0.0.1:8080 fail_timeout=5;
     }
 
     location /auth {
       internal;
 
       proxy_set_header X-Original-URI $request_uri;
       proxy_set_header X-Forwarded-Method $request_method;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Forwarded-Host $http_host;
       proxy_set_header X-Forwarded-URI $request_uri;
 
       proxy_set_header Content-Length "";
       proxy_set_header Connection "";
       proxy_pass_request_body off;
 
       proxy_pass http://rauthy/auth/v1/clients/<YOUR-CLIENT-ID>/forward_auth;
     }
 
     location = /callback {
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header X-Forwarded-Host $http_host;
       proxy_pass http://rauthy/auth/v1/clients/<YOUR-CLIENT-ID>/forward_auth/callback?$args;
     }
     
     location / {
         auth_request /auth;
         auth_request_set $redirection_url $upstream_http_location;
         error_page 401 =302 $redirection_url;
 
         # NOTE(d1): finally, serve your webshit
         try_files $uri $uri/ =404;
     }
 
 ### nss
 
 See `/etc/rauthy/rauthy-pam-nss.toml` for the configuration. You can `journalctl -fu rauthy-nss` to see what's going down. The simplest test is to run `getent hosts`. The hosts are configured under `Pam > Hosts` in the `rauthy` web UI.