Changes

Page history
Consolidate Tech Working Group wiki authored by Ed Summers's avatar Ed Summers 
This commit moves information from the Technical Working Group's (TWG)
[git.coop
wiki](https://git.coop/social.coop/tech/operations/-/wikis/home) into a
subdirectory of the main wiki. The current TWG wiki on git.coop is
currently public. These changes are predicated on the idea that we want
to move forward with the
[proposal](https://www.loomio.com/d/2Rwh5z3H/time-to-consolidate-wikis-/69)
to consolidate wikis in preparation for the move to a more user friendly
wiki platform.

**Note:** This commit includes an update to the Tech Working Group section of the
Bylaws to point to the relevant section in the general wiki rather than
the old TWG specific git.coop wiki.

Quite a bit of the information contained in these pages needs to be
consolidated and/or updated. But for historical purposes the thought was
to bring these pages over largely unchanged and then update them either
here in Git, or once the content has been moved to a new wiki.

If/when this merge request is pulled in the Tech Working Group pages
should become available at wiki.social.coop. At that point the old
https://git.coop/social.coop/tech/operations repository can be [archived](https://docs.gitlab.com/ee/user/project/settings/#archive-a-project).
Show whitespace changes
Inline Side-by-side
tech-working-group/Infrastructure-overview.md 0 → 100644
View page @ 849baa30
# Infrastructure Overview
(The previous infrastructure overview is [here](Infrastructure-overview-(old)))
## A. Domain name
1. social.coop domain is registered under the https://gandi.net account of Enric of FairCoop, who should be notified and sent payment yearly before it expires
2. It would be good to get the domain transferred or at least administratively delegated to an account under social.coop control
3. DNS is managed via https://cloudflare.com where we have the option of turning on the DDOS protection and CDN/caching functionality if necessary
## B. Infrastructure
We have a dedicated server on Hetzner:
* 32GB RAM (4x RAM 8192 MB DDR3)
* i7-4770 CPU @ 3.40GHz).
* 2x 250 GB disks (SSD)
Disks are arranged as:
```
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 223.6G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 222.6G 0 part
├─vg0-root1 253:0 0 25G 0 lvm /
├─vg0-root2 253:1 0 25G 0 lvm
└─vg0-opt 253:2 0 396.1G 0 lvm /opt
sdb 8:16 0 223.6G 0 disk
└─sdb1 8:17 0 223.6G 0 part
└─vg0-opt 253:2 0 396.1G 0 lvm /opt
```
Using LVM such that we have:
```
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
opt vg0 -wi-ao---- 396.13g
root1 vg0 -wi-ao---- 25.00g
root2 vg0 -wi-a----- 25.00g
```
`opt` is mounted at `/opt`.
### Third party services
* DigitalOcean Spaces (S3 compatible, for storing Mastodon media and backups; also we use their transient VMs for development / testing)
* Email delivery by [mailgun](https://www.mailgun.com) (10,000 emails free every month)
* Email relaying for our social.coop domain is handled by a Mailcow server with Web Architects (no significant storage available). This is needed for git.coop registration.
* [datadog](https://www.datadoghq.com) and [pingometer](http://pingometer.com) for monitoring
* [cloudflare](https://cloudflare.com) for DNS management.
## C. Monitoring
## D. Regular Updates
1. Host systems (Ubuntu LTS package upgrades)
* security updates _will_ be handled via https://git.coop/social.coop/tech/ansible/merge_requests/1
2. datadog agent https://hub.docker.com/r/datadog/agent/tags/
4. Regular Mastodon upgrades
* Make sure to make backups first, then check for updates and setup instructions at https://github.com/tootsuite/mastodon/releases
## E. Security
1. HTTPS / SSL certificates
* Using https://certbot.eff.org
* Regenerates twice daily via systemd [service](https://git.coop/social.coop/tech/sauce/blob/rebuild/systemd/certbot.service) and [timer](https://git.coop/social.coop/tech/sauce/blob/rebuild/systemd/certbot.timer)
2. Backups
* work in progress, see https://git.coop/social.coop/tech/operations/issues/7
3. Firewalls
* _will_ be handled via https://git.coop/social.coop/tech/ansible/merge_requests/1
4. DDOS
* See about enabling Cloudflare
## F. Documentation & Communication
1. Document any new infrastructure / software / service / config
2. Keep all code in shared git version control (either in [sauce](https://git.coop/social.coop/tech/sauce) or [ansible](https://git.coop/social.coop/tech/ansible))
3. Keep configuration and private keys / passwords separate, and place all config files in shared git version control
4. Proactively communicate with Tech WG about reasons, approach and outcome of every change / update, and then add to documentation
5. Let fellow Ops Team members know before any prolonged unavailability (as much as possible)
6. Communicate with Ops Team during any emergency, or before doing anything that affects live services
7. Create/use individual accounts/passwords for each admin as much as possible
8. Use a [pass](https://git.coop/social.coop/tech/pass) for storing all shared secrets (like passwords)
## G. Fix unexpected issues
YMMV
# Handy commands, etc
## systemd services
| service | purpose |
|---|---|
| social.coop-mastodon | a service to control the mastodon installation via docker-compose |
| social.coop-remove-media | runs the media cleanup command to remove remote media >7 days old via a .timer |
| certbot | runs the renewals via .timer |
## logs
| command | purpose |
|---|---|
| systemctl list-timers | lists timers! |
| journalctl -f | tail ALL system logs |
| docker-compose logs -f web | view and tail web logs (when in `/opt/social.coop/sauce/docker/`) |
| docker-compose logs -f db | view and tail db logs (you get the pattern?) |
| journalctl -f -u certbot | see when certbot was run |
| journalctl -f -u social.coop-mastodon | see the output from the docker-compose commands ran with systemctl, but not the docker container logs themselves |
| journalctl -f -u social.coop-remove-media | see what the remove media command is up to |
## docker-compose
All of these commands must be run on runko.social.coop in the `/opt/social.coop/sauce/docker/` directory.
List all Docker containers: `docker-compose ps`
Stop a service, e.g. `docker-compose stop redis`
Start a service, e.g. `docker-compose start redis`
Redeploy (only changed things): `docker-compose up -d` (or `systemd refresh social.coop-mastodon` - does same thing)
If you want to run commands make sure to use the `--rm` argument, or the containers will hang around.
e.g. `docker-compose run --rm web rails console` (to get a rails console)
Location of Postgres database files: `/opt/social.coop/var/lib/postgresql/data/`
### Mastodon upgrades
Here are some **example** steps that you **might** take when upgrading Mastodon. Please note that the process may be different every time, and that issues may arise, so make sure to have a few hours ahead of you!
Depending on the significance of the upgrade, you may want to create a database backup. We don't currently have a ready made script for this.
This is a in-theory guide to doing the upgrade, please proceed carefully and understand what you are doing. If in doubt ask for someone to assist. If this guide is not up to date, please update it afterwards :)
* Update the mastodon docker image versions in https://git.coop/social.coop/tech/sauce/blob/rebuild/docker/docker-compose.yml
* `git pull` in `/opt/social.coop/sauce/`
* reload `systemctl refresh social.coop-mastodon` (will take a bit of time to download the new image)
* if you need to run database migration (check release notes):
* `docker-compose run --rm web rails db:migrate`
* if you need to update the assets (it'll say to regenerate them in release notes, but we just copy them out as they are prebuilt in the docker image), something like:
* `mkdir /tmp/updated-assets && chown 991:991 /tmp/updated-assets`
* `docker run --rm -v /tmp/updated-assets:/assets tootsuite/mastodon:v2.5.0 cp -r public/. /assets`
* then empty out `/opt/social.coop/var/www/mastodon` and fill with the stuff from `/tmp/updated-assets` and keep the permissions to 991:991 (I would probably keep the existing directory rather than mv, as it's mounted by docker and it might screw it up). rsync with `--delete` could be a good option. put the commands back in here once you worked out a good way!
\ No newline at end of file

Webarchitects | Forum | Status | SSH Fingerprints