Social.Coop Tech Group: Operations repository

This is the home of the social.coop tech group!

Useful repos/places/links

• this repo tech/operations
• issues
• meetings
  • every 2 weeks on Wednesday at 19:30 UTC on even numbered ISO weeks
  • meeting pad
  • tech meeting minutes
• communication
  • matrix chat #socialcoop-tech:matrix.org
  • tech loomio group
• tech governance
  • server access (to vote for giving people access to server)
• server config
  • ansible
  • sauce repo (to be full replaced by ansible)

Domain name

Passwords

We use pass for password management, passwords are stored in our pass repo.

Email

Transactional email

We use mailgun for emails sent by mastodon.

@social.coop email addresses

@social.coop email is handled by webarch.mail. From there you can setup email aliases, forwarding, etc...

Aliases we use:

• tech.group@social.coop
• admin@social.coop

git

Our git home is git.coop/social.coop/tech

Media storage

We use DigitalOcean Spaces to store mastodon media images.

Backups

Database backups are stored in our DigitalOcean Space. Configure by the pg-dump-to-s3 ansible role.

There is nothing else stateful on the server (I think!).

Monitoring / metrics

We have some server metrics setup with datadog.

Server

We have a dedicated server at Hetzner:

• 32GB RAM (4x RAM 8192 MB DDR3)
• i7-4770 CPU @ 3.40GHz).
• 2x 250 GB disks (SSD)
• Ubuntu 18.04

Its ssh port is 2022.

The disks are arranged as:

NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda             8:0    0 223.6G  0 disk 
├─sda1          8:1    0     1G  0 part /boot
└─sda2          8:2    0 222.6G  0 part 
  ├─vg0-root1 253:0    0    25G  0 lvm  /
  ├─vg0-root2 253:1    0    25G  0 lvm  
  └─vg0-opt   253:2    0 396.1G  0 lvm  /opt
sdb             8:16   0 223.6G  0 disk 
└─sdb1          8:17   0 223.6G  0 part 
  └─vg0-opt   253:2    0 396.1G  0 lvm  /opt

Using LVM such that we have:

  LV    VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  opt   vg0 -wi-ao---- 396.13g                                                    
  root1 vg0 -wi-ao----  25.00g                                                    
  root2 vg0 -wi-a-----  25.00g 

opt is mounted at /opt.

wiki.social.coop

wiki.social.coop has two main purposes:

1. a public-facing site with information about social.coop
2. the registration system for new users

The code repo for the project is tech/wiki.social.coop and the content comes from the general wiki.

It's configured/deployed via ansible using the wiki.social.coop role and the wiki tag, so ansible-playbook server.playbook.yml --tags wiki will set it up.

The configuration secrets are stored in the pass repo at:

deployment/wiki/gitlab_token
deployment/wiki/gitlab_username
deployment/wiki/mailgun_password
deployment/wiki/webhook_secret

On the server it lives at: /opt/social.coop/wiki.social.coop/ and is running as the systemd service wiki.social.coop.service.

Mastodon upgrade notes

• get current version from social.coop
• find next version from mastodon github releases
• check upgrade notes
• we are on docker!
• check whether there are DB migrations
• make backup? systemctl start pg-dump-to-s3.service
  • takes 15 mins or so?
• separate command to see backup progress
• make merge request on git.coop sauce repo to bump version in a couple of places in docker-compose.yaml
• git diff v3.1.2..v3.1.3 -- docker-compose.yml in mastodon repo after pulling to check whether there were any changes we should consider mirroring to our docker-compose file
• could be cool to make these merge requests in advance
• write a toot announcing upgrade and boost on admin account
• touch file on server to activate maintenance mode
• actually do the upgrade
  • migration command creates a fresh web container and runs the migration command and then deletes that new container
• turn maintenance mode off
• we copy static assets outside of the container so they can be served by nginx
  • there's a command for this which moves stuff into a temporary dir in nginx and pulls assets out of docker container into that folder in docker container
• ssh forwarding is nice, then with sudo -E -s you have ssh access to stuff you do from host machine(?)