Consolidate Tech Working Group wiki authored by Ed Summers's avatar Ed Summers
This commit moves information from the Technical Working Group's (TWG)
[git.coop
wiki](https://git.coop/social.coop/tech/operations/-/wikis/home) into a
subdirectory of the main wiki. The current TWG wiki on git.coop is
currently public. These changes are predicated on the idea that we want
to move forward with the
[proposal](https://www.loomio.com/d/2Rwh5z3H/time-to-consolidate-wikis-/69)
to consolidate wikis in preparation for the move to a more user friendly
wiki platform.

**Note:** This commit includes an update to the Tech Working Group section of the
Bylaws to point to the relevant section in the general wiki rather than
the old TWG specific git.coop wiki.

Quite a bit of the information contained in these pages needs to be
consolidated and/or updated. But for historical purposes the thought was
to bring these pages over largely unchanged and then update them either
here in Git, or once the content has been moved to a new wiki.

If/when this merge request is pulled in the Tech Working Group pages
should become available at wiki.social.coop. At that point the old
https://git.coop/social.coop/tech/operations repository can be [archived](https://docs.gitlab.com/ee/user/project/settings/#archive-a-project).
Hello and Welcome!
There are a few things you should sign up for:
- Our Tech group on [Loomio](https://en.wikipedia.org/wiki/Loomio). *(Join our main Loomio group [here](https://www.loomio.org/socialcoop/), then ask to join the tech group [here](https://www.loomio.org/g/Mtb0FrvX/social-coop-tech-working-group), it should also be visible in the side-bar when you're on our main group)*
- Our GitLab group on git.coop. *(If you're reading this you've probably done this already, but if not see [here](../Getting-a-social.coop-GitLab-project-account))*
- Our [matrix](https://en.wikipedia.org/wiki/Matrix_(communication_protocol)) chat channels on [riot.im](https://about.riot.im/what-is-riot/):
- The [public social.coop channel](https://riot.im/app/#/room/#SocialCoop:matrix.org) *(for general chat, but I suggest we use it as a rendezvous point when we notice an issue we need to coordinate on, this way everyone on the channel can check the situation)*
- The [private, encrypted tech group](https://riot.im/app/#/room/#tech.social.coop:matrix.org), for stuff we don't really want to discuss in the open *(note, messages posted before you join will not be readable, and there may be warnings about unverified devices, these are to be expected)*
## Passwords and other secrets!
We are currently devising a way to share log-ins and passwords, etc. for services we use which can't or won't allow a per-user log-in. The information is (or will) go in the [keyringer project](/social.coop/tech/keyringer) here in our GitLab group. Maintaining all these different log-ins is a bit of a headache, we'd like to have something unified but currently don't.
## Git.coop, GitLab and Git
Hopefully you will already know a bit about these, but it should suffice to say:
- [Git](https://en.wikipedia.org/wiki/Git) is a distributed version control system
- [GitLab](https://en.wikipedia.org/wiki/GitLab) is an open source web user interface built on top of it
- [Git.coop](https://www.webarchitects.coop/git) is the instance of GitLab which the [Web Architects Co-op](https://www.webarchitects.coop/about) host for their members.
## Documentation
The project whose [wiki](https://en.wikipedia.org/wiki/Wiki) you're currently viewing is where we keep our internal documentation. The wiki is backed by a Git repository, like the project's main repository but separate. The [GitLab wiki](https://docs.gitlab.com/ce/user/project/wiki/) has a slightly different user interface which makes it more wiki-like than the main repository's user interface.
The pages are all editable via the web user interface, or by directly editing files in the repository and pushing the changes back to the GitLab server. The documentation is mostly in [GitLab-flavoured Markdown](https://docs.gitlab.com/ce/user/markdown.html), but other formats are supported (for instance, [emacs org-mode mark-up](https://en.wikipedia.org/wiki/Org-mode)). Markdown is meant to about as hard as formatting a plaintext email message, but you should see a [link to a cheat-sheet](https://git.coop/help/user/markdown) about this at the bottom of the edit box when modifying a page in the web UI.
Supporting files and scripts can go in the main repository (see the [link](/social.coop/tech/operations/tree/master) on the left-hand side bar of the GitLab UI).
## Issues
Also in the left-hand sidebar is a [link](/social.coop/tech/operations/issues) to the GitLab issues database for this project. We use this to:
- keep track of problems we need to, or are in the process of resolving, as well as
- an ideas bucket and
- a wish-list.
In the tech group at least, there are a set of [labels](https://git.coop/groups/social.coop/tech/-/labels) which can be used to prioritise and categorise issues. There's a description [here](Jobs#suggested-classifications).
# Recurring tasks
We also have some recurring tasks which need to be done to maintain the servers, the organisation and documentation of which are a work-in-progress. The ambition is to have one or two people responsible for each one. Initially, we will need volunteers also to clarify the processes and draft the documentation, so that the skills are transferable among us.
See here for the [document outlining these tasks](Jobs).
## Outage
When there is a server outage, I recommend:
- Checking who is on the case already, if anyone, in the matrix channels (see links above).
- FIXME: We don't currently have a means to announce these externally, but perhaps we should
- Creating an issue for it labelled ~"Outage!", describing the initial problem and the time it occurred
- FIXME: We don't currently have a guide for fixing outages, but we do kinda need one.
I'd propose that everyone involved in resolving the issue keeps a record of what they did in the issue: i.e., steps taken, and where possible console commands and their output. Besides being a record, I hope this will create a common resource of prior-art and diagnostic tips and clues for the group, and especially newcomers. My aim is to be able to use this to condense down guides and how-tos which are easier to navigate. (They can later be found by searching for closed issues with the ~"Outage!" label. Some examples so far: #13, #14.)
# Datadog
We currently have a freebie [Datadog](https://en.wikipedia.org/wiki/Datadog) account, which is a very useful tool for monitoring our cluster of Docker services and visualising the result. (Free plan includes 1 day's logs retention and 150+ turn-key integrations, whatever that means.)
If you need to diagnose an outage, ask the group about this, we can allocate (probably a limited number) of additional log-ins there. The log-in page is:
https://app.datadoghq.com