Changes

This commit moves information from the Technical Working Group's (TWG) [git.coop wiki](https://git.coop/social.coop/tech/operations/-/wikis/home) into a subdirectory of the main wiki. The current TWG wiki on git.coop is currently public. These changes are predicated on the idea that we want to move forward with the [proposal](https://www.loomio.com/d/2Rwh5z3H/time-to-consolidate-wikis-/69) to consolidate wikis in preparation for the move to a more user friendly wiki platform. **Note:** This commit includes an update to the Tech Working Group section of the Bylaws to point to the relevant section in the general wiki rather than the old TWG specific git.coop wiki. Quite a bit of the information contained in these pages needs to be consolidated and/or updated. But for historical purposes the thought was to bring these pages over largely unchanged and then update them either here in Git, or once the content has been moved to a new wiki. If/when this merge request is pulled in the Tech Working Group pages should become available at wiki.social.coop. At that point the old https://git.coop/social.coop/tech/operations repository can be [archived](https://docs.gitlab.com/ee/user/project/settings/#archive-a-project).
Ed Summers · 849baa30
--- a/tech-working-group/Tech-Working-Group.md
+++ b/tech-working-group/Tech-Working-Group.md
+# Social.Coop Tech Working Group
+
+This is the home of the social.coop tech group!  If you are new to the group, please see the [new user guide](Getting-started-in-the-tech-group).
+
+- Newcomers: 
+  - [Getting started](Getting-started-in-the-tech-group)
+- Procedures:
+  - [How to add a social.coop email address](How-to-add-a-socialcoop-email-address) (including info on modifying other aliases like `tech.group@social.coop`)
+  - [How to get shell access to social.coop's servers](How-to-get-server-shell-access)
+- Entry points:
+  - [Infrastructure overview](Infrastructure-overview)
+  - [Recurring task table](Jobs) listing maintenance tasks to be allocated, currently out of date.
+- Other places to look for information:
+  - https://anagora.org/twg contains pointers.
+
+## Services
+
+The tech group is responsible for operating and maintaining the following services:
+
+| Primary services |  |
+|---|---|
+| Social.coop mastodon instance | [admin panel](https://social.coop/admin/dashboard) |
+| wiki.social.coop public wiki | https://wiki.social.coop |
+
+| Supporting services | |
+|---|---|
+| registrar | [gandi](https://www.gandi.net) |
+| registrant | organization: _Xarxa integral de professionals i usuaries_ |
+| DNS / DDOS protection| [cloudflare](https://www.cloudflare.com) | 
+| Mailgun for emails sent by mastodon | [mailgun](https://www.mailgun.com/) |
+| @social.coop email aliases | [webarch.mail](https://webarch.email/) |
+| Object store for backups and digital assets | [digital ocean spaces](https://cloud.digitalocean.com/login) |
+| Monitoring / metrics | [datadog](https://www.datadoghq.com/) |
+| Code repos | [git.coop/social.coop/tech](https://git.coop/social.coop/tech) |
+
+## Our git.coop repositories
+
+| Repo | purpose |
+|---|---|
+| [tech gitlab group](https://git.coop/social.coop/tech) | list of all repos |
+| [sauce](https://git.coop/social.coop/tech/sauce) | docker config and some systemd services (to be migrated)|
+| [ansible](https://git.coop/social.coop/tech/ansible) | server configuration |
+| [pass](https://git.coop/social.coop/tech/pass) | encrypted password store for shared passwords |
+| [wiki](https://git.coop/social.coop/tech/wiki.social.coop) | code for the public metalsmith wiki |
+
+## Administrative links
+- [issues](https://git.coop/social.coop/tech/operations/-/issues)
+- meetings
+    - every 2 weeks on Wednesday at 19:30 UTC on even numbered [ISO weeks](https://www.epochconverter.com/weeknumbers)
+    - [meeting pad](https://codi.kanthaus.online/social.coop)
+    - [tech meeting minutes](https://www.loomio.org/d/UwAeiBgE/tech-meeting-minutes)
+- communication
+    - [matrix chat #socialcoop-tech:matrix.org](https://riot.im/app/#/room/#socialcoop-tech:matrix.org)
+    - [tech loomio group](https://www.loomio.org/g/Mtb0FrvX/social-coop-tech-working-group)
+- tech governance
+    - [server access](https://www.loomio.org/d/jrbG5tue/server-access) (to vote for giving people access to server)
+
+
+# Mastodon runbook
+
+Our fediverse instance is the raison d`etre of the social.coop coop.  This is what the community signs up for and our primary responsibility.  The primary points of administration are:
+- the [admin panel](https://social.coop/admin/dashboard)
+- the [datadog dashboard](https://app.datadoghq.com/dash/host/640032656?from_ts=1667763442004&to_ts=1668368242004&live=true)
+- ssh cli access  Access via ssh on port 2022 e.g.  `ssh user@runko.social.coop -p 2022`. 
+
+If you need server access please see [server access](https://www.loomio.org/d/jrbG5tue/server-access) and [How to grant shell access](https://git.coop/social.coop/tech/operations/-/wikis/how-to-get-server-shell-access)
+
+Mastodon is running under [Docker-compose](https://docs.docker.com/compose/).
+
+Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.  We are using Systemd as the init system for docker-compose.
+
+## systemd services
+
+| service | purpose |
+|---|---|
+| social.coop-mastodon | a service to control the mastodon installation via docker-compose |
+| social.coop-remove-media | runs the media cleanup command to remove remote media >7 days old via a .timer |
+| certbot | runs the renewals via .timer |
+
+## logs
+
+| command | purpose |
+|---|---|
+| systemctl list-timers | lists timers! |
+| journalctl -f | tail ALL system logs |
+| docker-compose logs -f web | view and tail web logs (when in `/opt/social.coop/sauce/docker/`) |
+| docker-compose logs -f db | view and tail db logs (you get the pattern?) |
+| journalctl -f -u certbot | see when certbot was run |
+| journalctl -f -u social.coop-mastodon | see the output from the docker-compose commands ran with systemctl, but not the docker container logs themselves |
+| journalctl -f -u social.coop-remove-media | see what the remove media command is up to |
+
+## Service management
+
+All of these commands must be run on runko.social.coop in the `/opt/social.coop/sauce/docker/` directory.
+| command | purpose |
+|---|---|
+| `docker-compose ps` | List all Docker containers |
+| `docker-compose stop redis` | Stop a service |
+| `docker-compose start redis` | Start a service |
+| `sudo docker-compose scale sidekiq-default-q=5` | scale a service, in this case create 5 sidekiq default queue workers |
+| `docker-compose up -d` (or `systemd refresh social.coop-mastodon` - does same thing) | Redeploy (only changed things) |
+
+If you want to run commands make sure to use the `--rm` argument, or the containers will hang around.
+
+e.g. `docker-compose run --rm web rails console` (to get a rails console)
+
+## Backups
+
+Configure by the [pg-dump-to-s3](https://git.coop/social.coop/tech/ansible/-/tree/master/roles/pg-dump-to-s3) ansible role.
+
+Location of Postgres database files: `/opt/social.coop/var/lib/postgresql/data/`
+
+# Mastodon upgrade notes
+
+- it is recommended to run upgrades inside of `tmux` in case you are disconnected from the server.
+- get current version from social.coop
+- find next version from mastodon github releases
+- check upgrade notes
+- check whether there are DB migrations
+- make backup? `systemctl start pg-dump-to-s3.service`
+	- takes 15 mins or so?
+- separate command to see backup progress
+- make merge request on git.coop sauce repo to bump version in a couple of
+  places in docker-compose.yaml
+- `git diff v3.1.2..v3.1.3 -- docker-compose.yml` in mastodon repo after pulling
+  to check whether there were any changes we should consider mirroring to our
+  docker-compose file
+- could be cool to make these merge requests in advance
+- write a toot announcing upgrade and boost on admin account
+- touch file on server to activate maintenance mode
+- actually do the upgrade
+	- migration command creates a fresh web container and runs the migration
+	  command and then deletes that new container
+- turn maintenance mode off
+- we copy static assets outside of the container so they can be served by nginx
+	- there's a command for this which moves stuff into a temporary dir in nginx
+	  and pulls assets out of docker container into that folder in docker
+	  container
+- ssh forwarding is nice, then with `sudo -E -s` you have ssh access to stuff
+  you do from host machine(?)
+
+
+## Hardware
+
+Mastodon and Wiki run on runko.social.coop, a dedicated server at [Hetzner](https://www.hetzner.com/):
+
+- 32GB RAM (4x RAM 8192 MB DDR3)
+- i7-4770 CPU @ 3.40GHz).
+- 2x 250 GB disks (SSD)
+- Ubuntu 18.04
+
+### Storage
+
+We are using LVM.
+
+```
+NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
+sda             8:0    0 223.6G  0 disk 
+├─sda1          8:1    0     1G  0 part /boot
+└─sda2          8:2    0 222.6G  0 part 
+  ├─vg0-root1 253:0    0    25G  0 lvm  /
+  ├─vg0-root2 253:1    0    25G  0 lvm  
+  └─vg0-opt   253:2    0 396.1G  0 lvm  /opt
+sdb             8:16   0 223.6G  0 disk 
+└─sdb1          8:17   0 223.6G  0 part 
+  └─vg0-opt   253:2    0 396.1G  0 lvm  /opt
+```
+
+
+```
+  LV    VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
+  opt   vg0 -wi-ao---- 396.13g                                                    
+  root1 vg0 -wi-ao----  25.00g                                                    
+  root2 vg0 -wi-a-----  25.00g 
+```
+
+`opt` is mounted at `/opt`.
+
+
+# wiki.social.coop
+
+[wiki.social.coop](https://wiki.social.coop) has two main purposes:
+
+1. a public-facing site with information about social.coop
+2. the registration system for new users
+
+The code repo for the project is [tech/wiki.social.coop](https://git.coop/social.coop/tech/wiki.social.coop/) and the content comes from the [general wiki](https://git.coop/social.coop/general/-/wikis/home).
+
+It's configured/deployed via ansible using the [wiki.social.coop role](https://git.coop/social.coop/tech/ansible/-/tree/master/roles/wiki.social.coop) and the `wiki` tag, so `ansible-playbook server.playbook.yml --tags wiki` will set it up.
+
+The configuration secrets are stored in the [pass repo](https://git.coop/social.coop/tech/pass) at:
+```
+deployment/wiki/gitlab_token
+deployment/wiki/gitlab_username
+deployment/wiki/mailgun_password
+deployment/wiki/webhook_secret
+```
+
+On the server it lives at: `/opt/social.coop/wiki.social.coop/` and is running as the systemd service `wiki.social.coop.service`.