Changes

Page history
Update Tech Working Group authored by Akshay Mankar's avatar Akshay Mankar
Show whitespace changes
Inline Side-by-side
tech-working-group/Tech-Working-Group.md
View page @ 7b4571d7
...@@ -18,12 +18,12 @@ This is the home of the social.coop tech group! If you are new to the group, pl ...@@ -18,12 +18,12 @@ This is the home of the social.coop tech group! If you are new to the group, pl
The tech group is responsible for operating and maintaining the following services: The tech group is responsible for operating and maintaining the following services:
| Primary services | | | Primary services | |
|---|---| |------------------|--|
| Social.coop mastodon instance | [admin panel](https://social.coop/admin/dashboard) | | Social.coop mastodon instance | [admin panel](https://social.coop/admin/dashboard) |
| wiki.social.coop public wiki | https://wiki.social.coop | | wiki.social.coop public wiki | https://wiki.social.coop |
| Supporting services | | | Supporting services | |
|---|---| |---------------------|--|
| registrar | [gandi](https://www.gandi.net) | | registrar | [gandi](https://www.gandi.net) |
| registrant | organization: _Xarxa integral de professionals i usuaries_ | | registrant | organization: _Xarxa integral de professionals i usuaries_ |
| DNS / DDOS protection | [cloudflare](https://www.cloudflare.com) | | DNS / DDOS protection | [cloudflare](https://www.cloudflare.com) |
...@@ -31,12 +31,12 @@ The tech group is responsible for operating and maintaining the following servic ...@@ -31,12 +31,12 @@ The tech group is responsible for operating and maintaining the following servic
| @social.coop email aliases | [webarch.mail](https://webarch.email/) | | @social.coop email aliases | [webarch.mail](https://webarch.email/) |
| Object store for backups and digital assets | [digital ocean spaces](https://cloud.digitalocean.com/login) | | Object store for backups and digital assets | [digital ocean spaces](https://cloud.digitalocean.com/login) |
| Monitoring / metrics | [datadog](https://www.datadoghq.com/) | | Monitoring / metrics | [datadog](https://www.datadoghq.com/) |
| Code repos | [git.coop/social.coop/tech](https://git.coop/social.coop/tech) | | Code repos | git.coop/social.coop/tech |
## Our git.coop repositories ## Our git.coop repositories
| Repo | purpose | | Repo | purpose |
|---|---| |------|---------|
| [tech gitlab group](https://git.coop/social.coop/tech) | list of all repos | | [tech gitlab group](https://git.coop/social.coop/tech) | list of all repos |
| [sauce](https://git.coop/social.coop/tech/sauce) | docker config and some systemd services (to be migrated) | | [sauce](https://git.coop/social.coop/tech/sauce) | docker config and some systemd services (to be migrated) |
| [ansible](https://git.coop/social.coop/tech/ansible) | server configuration | | [ansible](https://git.coop/social.coop/tech/ansible) | server configuration |
...@@ -44,9 +44,10 @@ The tech group is responsible for operating and maintaining the following servic ...@@ -44,9 +44,10 @@ The tech group is responsible for operating and maintaining the following servic
| [wiki](https://git.coop/social.coop/tech/wiki.social.coop) | code for the public metalsmith wiki | | [wiki](https://git.coop/social.coop/tech/wiki.social.coop) | code for the public metalsmith wiki |
## Administrative links ## Administrative links
- [issues](https://git.coop/social.coop/tech/operations/-/issues) - [issues](https://git.coop/social.coop/tech/operations/-/issues)
- meetings - meetings
- every 2 weeks on Wednesday at 19:30 UTC on even numbered [ISO weeks](https://www.epochconverter.com/weeknumbers) - every 2 weeks on Monday at 19:00 UTC on odd numbered [ISO weeks](https://www.epochconverter.com/weeknumbers)
- [meeting pad](https://codi.kanthaus.online/social.coop) - [meeting pad](https://codi.kanthaus.online/social.coop)
- [tech meeting minutes](https://www.loomio.org/d/UwAeiBgE/tech-meeting-minutes) - [tech meeting minutes](https://www.loomio.org/d/UwAeiBgE/tech-meeting-minutes)
- communication - communication
...@@ -55,10 +56,10 @@ The tech group is responsible for operating and maintaining the following servic ...@@ -55,10 +56,10 @@ The tech group is responsible for operating and maintaining the following servic
- tech governance - tech governance
- [server access](https://www.loomio.org/d/jrbG5tue/server-access) (to vote for giving people access to server) - [server access](https://www.loomio.org/d/jrbG5tue/server-access) (to vote for giving people access to server)
# Mastodon runbook # Mastodon runbook
Our fediverse instance is the raison d`etre of the social.coop coop. This is what the community signs up for and our primary responsibility. The primary points of administration are: Our fediverse instance is the raison d\`etre of the social.coop coop. This is what the community signs up for and our primary responsibility. The primary points of administration are:
- the [admin panel](https://social.coop/admin/dashboard) - the [admin panel](https://social.coop/admin/dashboard)
- the [datadog dashboard](https://app.datadoghq.com/dash/host/640032656?from_ts=1667763442004&to_ts=1668368242004&live=true) - the [datadog dashboard](https://app.datadoghq.com/dash/host/640032656?from_ts=1667763442004&to_ts=1668368242004&live=true)
- ssh cli access Access via ssh on port 2022 e.g. `ssh user@runko.social.coop -p 2022`. - ssh cli access Access via ssh on port 2022 e.g. `ssh user@runko.social.coop -p 2022`.
...@@ -72,7 +73,7 @@ Compose is a tool for defining and running multi-container Docker applications. ...@@ -72,7 +73,7 @@ Compose is a tool for defining and running multi-container Docker applications.
## systemd services ## systemd services
| service | purpose | | service | purpose |
|---|---| |---------|---------|
| social.coop-mastodon | a service to control the mastodon installation via docker-compose | | social.coop-mastodon | a service to control the mastodon installation via docker-compose |
| social.coop-remove-media | runs the media cleanup command to remove remote media >7 days old via a .timer | | social.coop-remove-media | runs the media cleanup command to remove remote media >7 days old via a .timer |
| certbot | runs the renewals via .timer | | certbot | runs the renewals via .timer |
...@@ -80,7 +81,7 @@ Compose is a tool for defining and running multi-container Docker applications. ...@@ -80,7 +81,7 @@ Compose is a tool for defining and running multi-container Docker applications.
## logs ## logs
| command | purpose | | command | purpose |
|---|---| |---------|---------|
| systemctl list-timers | lists timers! | | systemctl list-timers | lists timers! |
| journalctl -f | tail ALL system logs | | journalctl -f | tail ALL system logs |
| docker-compose logs -f web | view and tail web logs (when in `/opt/social.coop/sauce/docker/`) | | docker-compose logs -f web | view and tail web logs (when in `/opt/social.coop/sauce/docker/`) |
...@@ -92,8 +93,9 @@ Compose is a tool for defining and running multi-container Docker applications. ...@@ -92,8 +93,9 @@ Compose is a tool for defining and running multi-container Docker applications.
## Service management ## Service management
All of these commands must be run on runko.social.coop in the `/opt/social.coop/sauce/docker/` directory. All of these commands must be run on runko.social.coop in the `/opt/social.coop/sauce/docker/` directory.
| command | purpose | | command | purpose |
|---|---| |---------|---------|
| `docker-compose ps` | List all Docker containers | | `docker-compose ps` | List all Docker containers |
| `docker-compose stop redis` | Stop a service | | `docker-compose stop redis` | Stop a service |
| `docker-compose start redis` | Start a service | | `docker-compose start redis` | Start a service |
...@@ -120,25 +122,17 @@ Location of Postgres database files: `/opt/social.coop/var/lib/postgresql/data/` ...@@ -120,25 +122,17 @@ Location of Postgres database files: `/opt/social.coop/var/lib/postgresql/data/`
- make backup? `systemctl start pg-dump-to-s3.service` - make backup? `systemctl start pg-dump-to-s3.service`
- takes 15 mins or so? - takes 15 mins or so?
- separate command to see backup progress - separate command to see backup progress
- make merge request on git.coop sauce repo to bump version in a couple of - make merge request on git.coop sauce repo to bump version in a couple of places in docker-compose.yaml
places in docker-compose.yaml - `git diff v3.1.2..v3.1.3 -- docker-compose.yml` in mastodon repo after pulling to check whether there were any changes we should consider mirroring to our docker-compose file
- `git diff v3.1.2..v3.1.3 -- docker-compose.yml` in mastodon repo after pulling
to check whether there were any changes we should consider mirroring to our
docker-compose file
- could be cool to make these merge requests in advance - could be cool to make these merge requests in advance
- write a toot announcing upgrade and boost on admin account - write a toot announcing upgrade and boost on admin account
- touch file on server to activate maintenance mode - touch file on server to activate maintenance mode
- actually do the upgrade - actually do the upgrade
- migration command creates a fresh web container and runs the migration - migration command creates a fresh web container and runs the migration command and then deletes that new container
command and then deletes that new container
- turn maintenance mode off - turn maintenance mode off
- we copy static assets outside of the container so they can be served by nginx - we copy static assets outside of the container so they can be served by nginx
- there's a command for this which moves stuff into a temporary dir in nginx - there's a command for this which moves stuff into a temporary dir in nginx and pulls assets out of docker container into that folder in docker container
and pulls assets out of docker container into that folder in docker - ssh forwarding is nice, then with `sudo -E -s` you have ssh access to stuff you do from host machine(?)
container
- ssh forwarding is nice, then with `sudo -E -s` you have ssh access to stuff
you do from host machine(?)
## Hardware ## Hardware
...@@ -166,7 +160,6 @@ sdb 8:16 0 223.6G 0 disk ...@@ -166,7 +160,6 @@ sdb 8:16 0 223.6G 0 disk
└─vg0-opt 253:2 0 396.1G 0 lvm /opt └─vg0-opt 253:2 0 396.1G 0 lvm /opt
``` ```
``` ```
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
opt vg0 -wi-ao---- 396.13g opt vg0 -wi-ao---- 396.13g
...@@ -176,10 +169,9 @@ sdb 8:16 0 223.6G 0 disk ...@@ -176,10 +169,9 @@ sdb 8:16 0 223.6G 0 disk
`opt` is mounted at `/opt`. `opt` is mounted at `/opt`.
# wiki.social.coop # wiki.social.coop
[wiki.social.coop](https://wiki.social.coop) has two main purposes: wiki.social.coop has two main purposes:
1. a public-facing site with information about social.coop 1. a public-facing site with information about social.coop
2. the registration system for new users 2. the registration system for new users
...@@ -189,6 +181,7 @@ The code repo for the project is [tech/wiki.social.coop](https://git.coop/social ...@@ -189,6 +181,7 @@ The code repo for the project is [tech/wiki.social.coop](https://git.coop/social
It's configured/deployed via ansible using the [wiki.social.coop role](https://git.coop/social.coop/tech/ansible/-/tree/master/roles/wiki.social.coop) and the `wiki` tag, so `ansible-playbook server.playbook.yml --tags wiki` will set it up. It's configured/deployed via ansible using the [wiki.social.coop role](https://git.coop/social.coop/tech/ansible/-/tree/master/roles/wiki.social.coop) and the `wiki` tag, so `ansible-playbook server.playbook.yml --tags wiki` will set it up.
The configuration secrets are stored in the [pass repo](https://git.coop/social.coop/tech/pass) at: The configuration secrets are stored in the [pass repo](https://git.coop/social.coop/tech/pass) at:
``` ```
deployment/wiki/gitlab_token deployment/wiki/gitlab_token
deployment/wiki/gitlab_username deployment/wiki/gitlab_username
... ...
......

Webarchitects | Forum | Status | SSH Fingerprints