Deploy Rhizome

rhizome.social.coop now 1. exists and 2. should have everyone's keys, but otherwise is running stock Ubuntu 22 and is in need of further set up and deployment. Opening this bug to track:

1. Agreeing on a deployment strategy. To which extent do we want to replicate Runko 1:1 and to which extend do we intend to fix/upgrade aspects of the setup?
2. Actually executing the above.

About 1: one aspect that was brought up as potentially warranting discussion, for example, is whether we want to run nginx in a container instead of in the host 9as we do in Runko.)

For completeness sake, one aspect that I changed already w.r.t. Runko is the fact that Rhizome is running RAID 1 instead of RAID-0 logical volume striping. This should be more reliable.

At a high level I think the plan should be:

1. create a rhizome branch in the ansible repo
2. update it to build up rhizome with a functioning Mastodon instance at rhizome.social.coop
3. when we see it working, modify the playbook to bring up an instance for social.coop and import a backup

Ideally all services should be running in Docker containers. But if we need keep some things outside for now that's ok?

Sounds like a good plan.

Ideally all services should be running in Docker containers. But if we need keep some things outside for now that's ok?

I think it is OK as far as we don't install things manually. We can move things into docker later if we all agree.

It was agreed in TWG meeting on 2023-01-23 that we will try to migrate on 2023-01-30 at 19:00 UTC. We decided that we'll simply take down mastodon while doing the migration instead of trying a zero-downtime migration.

I think we should have a plan before we go into the meeting, so I propose this:

• [cloudflare] Reduce TTL for A record for social.coop to something like 300.
• [runko] Put social.coop in maintenance mode.
• [runko] Stop sidekiq and mastodon containers on runko.
• [runko] Backup postgresql.
• [runko] Backup elasticsearch.
• [localhost] Run rhizome playbook from https://git.coop/social.coop/tech/ansible/-/merge_requests/21. (Mastodon will most likely not come up due to DB not having been setup.
• [rhizome] explicitly stop mastodon and all sidekiq containers.
• [rhizome] Restore postgresql.
• [rhizome] Restore elasticsearch.
• [rhizome] Start all containers
• [cloudflare] Change A record for social.coop to point to rhizome, make sure TTL is small to help us revert if needed.
• [localhost] See if social.coop works.
• [localhost] See if wiki.social.coop works.
• [runko] Run certbot commands to revoke the previous certificates.
• [hetzner console] Decommission runko.

This is awesome @akshay! After the convo over in Matrix I copied this checklist over into our running notes doc so we can live edit it a bit easier: https://doc.anagora.org/social-coop-tech-group#2022-01-30

I checked as per our meeting notes and certbot was not working, https://git.coop/social.coop/tech/ansible/-/merge_requests/23/diffs should fix it (there's plenty of time, the certificate expires on April 17).

We are fully moved over to Rhizome now.

closed