Merge branch 'postfix' into 'master'

Postfix

See merge request !1

README.md

@@ -4,8 +4,24 @@ These Playbooks are designed to be used on Debian Stretch virtual servers.
 
 ## Discourse
 
-Login to the virtual server console, install `python`, enable root ssh access
-using keys by adding your keys to `/root/.ssh/authorized_keys`, edit
+Ansible Playbooks to install
+[Docker](https://store.docker.com/editions/community/docker-ce-server-debian)
+and [Discourse](https://github.com/discourse/discourse_docker) on a Debian
+Stretch virtual server and to configure the virtual server to use Postfix for
+incoming and outgoing emails (there is also a not-quite-working and, for now,
+abandoned [exim branch](https://git.coop/cotech/ansible/tree/exim)).
+
+The email setup is based on the [mail-reciever Docker
+container](https://github.com/discourse/mail-receiver) plus [this pull
+request](https://github.com/discourse/mail-receiver/pull/2) (which is now
+merged) and the [Postfix notes for using the host for outgoing
+email](https://meta.discourse.org/t/emails-with-local-smtp/23645/28), with an
+additional [Ruby
+script](https://git.coop/cotech/ansible/blob/master/roles/email/files/discourse-smtp-rcpt-acl).
+
+Before running these Playbooks, create a virtual server, runnng Debian Stretch
+then login to the virtual server's console, install `python`, enable root ssh
+access using keys by adding your keys to `/root/.ssh/authorized_keys`, edit
 `/etc/sshd/sshd_config` to set `PermitRootLogin prohibit-password`, run
 `service ssh restart` and then run the first Playbook:
 
@@ -40,24 +56,17 @@ then the `iptables` and `munin-node` roles will, as a minimum, need editing and
 might be best omitted. Also note that these Playbooks are based on using
 `mx.webarch.net` for incoming email -- this is an anti-spam gateway, if this
 wasn't used then SpamAssassin should probably be added to the mix.
-
-The email setup was originally based on the
-[mail-reciever](https://github.com/discourse/mail-receiver) Docker container
-plus the [outstanding pull
-request](https://github.com/discourse/mail-receiver/pull/2) and the [Postfix
-notes](https://meta.discourse.org/t/emails-with-local-smtp/23645/28) for using
-the host for outgoing email, but then we switched it over to use Exim.
  
 ### CoTech Community Discourse Settings
 
 Initial settings used for `community.coops.tech` when it was created:
 
 * title: Cooperative Technologists Community
-* site description: The intersection of co-operation and technology, the CoTech community forum.
-* contact email: community@coops.tech
-* contact url: https://www.coops.tech/
-* notification email: discourse@community.coops.tech
-* site contact username: system
+* site description: The intersection of co-operation and digtal technology, the CoTech community forum.
+* contact email: `community@coops.tech`
+* contact url: `https://www.coops.tech/`
+* notification email: `discourse@community.coops.tech`
+* site contact username: `system`
 * logo url: https://wiki.coops.tech/wiki/File:Cotech-blue.png
 * logo small url: https://wiki.coops.tech/wiki/File:Cotech-blue-text.png
 * company short name: CoTech
@@ -70,7 +79,7 @@ On the Email settings admin page:
 * reply by email enabled
 * reply by email address: `discourse+%{reply_key}@community.coops.tech`
 * manual polling enabled
-* email prefix: cotech-community
+* email prefix: `cotech-community`
 * email site title: CoTech Community
 
 On the Security page:
@@ -87,14 +96,6 @@ On the User Preferences page:
 
 The first post text:
 
-Welcome to the **Cooperative Technologists Community**, we are a network of technology focused cooperatives, [CoTech](https://www.coops.tech/), who are *"building a tech industry that's better for its workers and customers through co-operation, democracy and worker ownership."* This is our open community discussion forum, you don't have to be a member of a coop to join this community but you do need to support [the cooperative values and principles](http://ica.coop/en/whats-co-op/co-operative-identity-values-principles) and have an interest in technology, you can find out more [about us](https://www.coops.tech/about), read [our manifesto](https://www.coops.tech/manifesto), see who we are and who we have worked for and watch [a video made at our first gathering](https://vimeo.com/196080655) on [www.coops.tech](https://www.coops.tech/). We also have [a wiki](https://wiki.coops.tech/) and a decision making group on [Loomio](https://www.loomio.org/g/oVwtKDOn/digital-co-ops), [Slack channels](https://tech-coops.slack.com/) and (for now, we might close it and use Discourse) a public [email list](https://www.email-lists.org/mailman/listinfo/tech-coops). 
+Welcome to the **Cooperative Technologists Community**, we are a network of technology focused digital cooperatives, [CoTech](https://www.coops.tech/), who are *"building a tech industry that's better for its workers and customers through co-operation, democracy and worker ownership."* This is our open community discussion forum, you don't have to be a member of a coop to join this community but you do need to support [the cooperative values and principles](http://ica.coop/en/whats-co-op/co-operative-identity-values-principles) and have an interest in technology, you can find out more [about us](https://www.coops.tech/about), read [our manifesto](https://www.coops.tech/manifesto), see who we are and who we have worked for and watch [a video made at our first gathering](https://vimeo.com/196080655) on [www.coops.tech](https://www.coops.tech/). We also have [a wiki](https://wiki.coops.tech/) and a decision making group on [Loomio](https://www.loomio.org/g/oVwtKDOn/digital-co-ops), [Slack channels](https://tech-coops.slack.com/) and (for now, we might close it and use Discourse) a public [email list](https://www.email-lists.org/mailman/listinfo/tech-coops). 
 
 *Please read [our community guidelines](https://community.coops.tech/guidelines) before signing up for an account here.*
-
-
-
-
-
-
-
-

roles/api/tasks/main.yml

 ---
-- name: Stat "/etc/exim4/mail-receiver-environment.json"
+- name: Stat "/etc/postfix/mail-receiver-environment.json"
   stat:
-    path: "/etc/exim4/mail-receiver-environment.json"
+    path: "/etc/postfix/mail-receiver-environment.json"
   register: mail_receiver_environment
 
 - block:
@@ -9,9 +9,9 @@
   - name: Discourse scripts environmental variables file in place
     template:
       src: templates/mail-receiver-environment.json.j2
-      dest: /etc/exim4/mail-receiver-environment.json
-      mode: 0640
-      group: Debian-exim
+      dest: /etc/postfix/mail-receiver-environment.json
+      mode: 0644
+      group: root
       owner: root
 
   when: mail_receiver_environment.stat.exists == False

roles/discourse/tasks/main.yml

 ---
+- name: Group for Discourse present
+  group:
+    name: discourse
+    system: yes
+    state: present
+    gid: 1000
+
+- name: User for Discourse present
+  user:
+    name: discourse
+    system: yes
+    state: present
+    shell: /bin/bash
+    home: /home/discourse
+    createhome: true
+    groups: discourse,docker
+    uid: 1000
+
+- name: Stat /var/discourse/lost+found
+  stat:
+    path: "/var/discourse/lost+found"
+  register: var_discourse_partition
+
+- block:
+
+  - name: Delete lost+found directory if /var/discourse is a partition 
+    file:
+      dest: /var/discourse/lost+found
+      state: absent
+
+  when: var_discourse_partition.stat.exists == True
+
 - name: Directory for Discourse present
   file:
     dest: /var/discourse
     state: directory
+    owner: discourse
+    group: discourse
+
+- name: ssl-cert group present for UID mappings
+  group:
+    name: ssl-cert
+    system: yes
+    state: present
+    gid: 111
+
+- name: postgres group present for UID mappings
+  group:
+    name: postgres
+    system: yes
+    state: present
+    gid: 112
+
+- name: postgres user persent for GID mappings
+  user:
+    name: postgres
+    system: yes
+    group: postgres
+    createhome: false
+    shell: /bin/false
+    uid: 107
+
+- name: haproxy group present for UID mappings
+  group:
+    name: haproxy
+    system: yes
+    state: present
+    gid: 113
+
+- name: haproxy user persent for GID mappings
+  user:
+    name: haproxy
+    system: yes
+    group: haproxy
+    createhome: false
+    shell: /bin/false
+    uid: 108
+
+- name: redis group present for UID mappings
+  group:
+    name: redis
+    system: yes
+    state: present
+    gid: 114
+
+- name: redis user persent for GID mappings
+  user:
+    name: redis 
+    system: yes
+    group: redis 
+    createhome: false
+    shell: /bin/false
+    uid: 110
 
 - name: Discourse checked out
   git:
     repo: https://github.com/discourse/discourse_docker.git
     dest: /var/discourse
+  become: yes
+  become_user: 'discourse' 
+
+- block:
+
+  - name: Create lost+found directory 
+    command: mklost+found
+    args:
+      chdir: /var/discourse
+      creates: /var/discourse/lost+found
+
+  when: var_discourse_partition.stat.exists == True
 
 - name: Count how much swap is available
   shell: "free -g --si | awk '/^Swap:/{print $2}'"
@@ -79,6 +180,10 @@
   template:
     src: templates/standalone.yml.j2
     dest: /var/discourse/containers/app.yml
+  become: yes
+  become_user: discourse 
 
 - name: Rebuild Discourse app
   command: /var/discourse/launcher rebuild app
+  become: yes
+  become_user: discourse 

roles/docker/tasks/main.yml

@@ -6,7 +6,10 @@
     update_cache: yes
   with_items:
     - apt-transport-https
+    - ca-certificates
+    - curl
     - git
+    - software-properties-common
 
 - name: Docker GPG key present
   apt_key:
@@ -24,3 +27,8 @@
     name: docker-ce
     state: present
     update_cache: yes
+
+- name: Docker started
+  service:
+    name: docker
+    state: started

roles/email/files/00_local

-CHECK_RCPT_LOCAL_ACL_FILE = CONFDIR/check_rcpt_local_acl

roles/email/files/30_exim4-config_discourse

-discourse_transport:
-        driver = pipe
-        command = /usr/local/bin/receive-mail ${local_part}@${domain}
-

roles/email/files/450_exim4-config_discourse

-discourse_router:
-        driver = accept
-        transport = discourse_transport
-

roles/email/files/check_rcpt_local_acl

-# Local rcpt check
-  deny
-    message = No such discourse list
-    log_message = No such discourse list
-    !acl = acl_local_deny_exceptions
-    condition = ${run{/usr/local/bin/discourse-smtp-rcpt-acl $sender_address $local_part@$domain}{no}{${if eq {$runrc}{2}{yes}{no}}}}
-
-  defer
-    message = Temporary error checking discourse
-    !acl = acl_local_deny_exceptions
-    condition = ${if eq {$runrc}{1}{yes}{no}}
-
-
-

roles/email/files/discourse-smtp-fast-rejection

@@ -6,7 +6,7 @@ require 'uri'
 require 'cgi'
 require 'net/http'
 
-ENV_FILE = "/etc/exim4/mail-receiver-environment.json"
+ENV_FILE = "/etc/postfix/mail-receiver-environment.json"
 
 def logger
   @logger ||= Syslog.open("smtp-reject", Syslog::LOG_PID, Syslog::LOG_MAIL)
@@ -72,7 +72,11 @@ def maybe_reject_email(from, to, env)
   endpoint = "#{env['DISCOURSE_BASE_URL']}/admin/email/smtp_should_reject.json"
   key = env["DISCOURSE_API_KEY"]
   username = env["DISCOURSE_API_USERNAME"]
-
+  # just maker sure we have something in the from field
+  # so we can test for addresses remotely
+  if from == ''
+    from = 'test@example.org'
+  end
   uri = URI.parse(endpoint)
   fromarg = CGI::escape(from)
   toarg = CGI::escape(to)

roles/email/files/discourse-smtp-rcpt-acl

@@ -10,7 +10,7 @@ require 'net/http'
 # Returns 1 for defer
 # Returns 2 for reject
 
-ENV_FILE = "/etc/exim4/mail-receiver-environment.json"
+ENV_FILE = "/etc/postfix/mail-receiver-environment.json"
 
 def logger
   @logger ||= Syslog.open("smtp-reject", Syslog::LOG_PID, Syslog::LOG_MAIL)

roles/email/files/receive-mail

 #!/usr/bin/env ruby
 
-ENV_FILE    = "/etc/exim4/mail-receiver-environment.json"
+ENV_FILE    = "/etc/postfix/mail-receiver-environment.json"
 EX_TEMPFAIL = 75
 EX_SUCCESS  = 0
 

roles/email/tasks/main.yml

@@ -29,7 +29,26 @@
     dest: /usr/local/bin/discourse-smtp-rcpt-acl
     mode: 0755
 
-- name: Exim and related email packages installed
+- name: debconf-utils installed for Ansible
+  apt:
+    name: debconf-utils 
+    state: present
+
+- name: Debconf Postfix hostname set
+  debconf:
+    name: postifx
+    question: "postfix/mailname"
+    value: "{{ hostname }}"
+    vtype: string
+
+- name: Debconf Postfix set to be a internet server
+  debconf:
+    name: postfix
+    question: "postfix/main_mailer_type"
+    value: "Internet Site"
+    vtype: string
+
+- name: Postfix and related email packages installed
   apt:
     pkg: "{{ item }}"
     state: latest
@@ -38,31 +57,20 @@
     - curl
     - debian-archive-keyring
     - dnsutils
-    - exim4-daemon-light
     - mailutils
     - mutt
+    - postfix
     - pwgen
     - whois
 
-- name: Exim check_rcpt_local_acl in place
-  copy:
-    src: files/check_rcpt_local_acl
-    dest: /etc/exim4/check_rcpt_local_acl
+- name: Postfix smtpd_relay_restrictions set
+  command: postconf -e "smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated reject_unauth_destination"
 
-- name: Exim 00_local in place
-  copy:
-    src: files/00_local
-    dest: /etc/exim4/conf.d/main/00_local
+- name: Postfix set not to use /etc/aliases
+  command: postconf -e "alias_maps = "
 
-- name: Exim 450_exim4-config_discourse in place
-  copy:
-    src: files/450_exim4-config_discourse
-    dest: /etc/exim4/conf.d/router/450_exim4-config_discourse
-
-- name: Exim 30_exim4-config_discourse in place
-  copy:
-    src: files/30_exim4-config_discourse
-    dest: /etc/exim4/conf.d/transport/30_exim4-config_discourse
+- name: Postfix mydestination set to localhost 
+  command: postconf -e "mydestination = localhost"
 
 - name: Get the app container IP address
   command: "docker inspect --format '{''{ .NetworkSettings.IPAddress }''}' app"
@@ -71,16 +79,70 @@
 - debug:
     msg: "The Discourse app Docker container has the IP address {{ app_ip_address.stdout }}"
 
-- name: Exim config in place
+- name: Postfix my networks set to include {{ app_ip_address.stdout }}
+  command: postconf -e "mynetworks = 127.0.0.0/8, {{ app_ip_address.stdout }}" 
+
+- name: Postfix relay domains set to {{ hostname }}
+  command: postconf -e "relay_domains = {{ hostname }}"
+
+- name: Postfix smtpd_recipient_restrictions set
+  command: postconf -e "smtpd_recipient_restrictions = permit_mynetworks, check_policy_service unix:private/policy"
+
+- name: Postfix opportunistic TLS enabled
+  command: postconf -e "smtp_tls_security_level = may"
+
+- name: Postfix set to use sub-addresing
+  command: postconf -e "recipient_delimiter = +"
+
+- name: Postfix disable UTF-8 SMTP input 
+  command: postconf -e "smtputf8_enable=no"
+
+- name: Postfix Time Zone and Lang set 
+  command: postconf -e "export_environment='TZ LANG'"
+
+- name: Postfix set for ipv4 only
+  command: postconf -e "inet_protocols = ipv4"
+
+- name: Postfix set to use /usr/local/bin/receive-mail
+  command: postconf -M -e "discourse/unix=discourse unix - n n - - pipe user=nobody:nogroup argv=/usr/local/bin/receive-mail ${recipient}"
+
+- name: Postfix transport in place
   template:
-    src: templates/update-exim4.j2
-    dest: /etc/exim4/update-exim4.conf.conf
+    src: templates/transport.j2
+    dest: /etc/postfix/transport
+    mode: 0644
+
+- name: Postfix Transport Maps file set 
+  command: postconf -e "transport_maps=hash:/etc/postfix/transport"
+ 
+- name: Postmap run with Transport Maps file
+  command: postmap /etc/postfix/transport
 
-- name: Exim reconfigured
-  command: update-exim4.conf
+- name: Postfix set to reject incorrect email addresses 
+  command: postconf -M -e "policy/unix=policy unix - n n - - spawn user=nobody argv=/usr/local/bin/discourse-smtp-fast-rejection"
+
+- name: Stat "/var/discourse/shared/standalone/letsencrypt/{{ hostname }}/{{ hostname }}.cer" 
+  stat:
+    path: "/var/discourse/shared/standalone/letsencrypt/{{ hostname }}/{{ hostname }}.cer" 
+  register: le_cert
+
+- block:
+  
+  - name: Postfix configured to use Let's Encrypt RSA cert for incoming email
+    command: postconf -e "smtpd_tls_cert_file = /var/discourse/shared/standalone/letsencrypt/{{ hostname }}/{{ hostname }}.cer" 
+  
+  - name: Postfix configured to use Let's Encrypt RSA key for incoming email
+    command: postconf -e "smtpd_tls_key_file =  /var/discourse/shared/standalone/letsencrypt/{{ hostname }}/{{ hostname }}.key"
+
+  when: le_cert.stat.exists == True
+
+- name: Postfix stopped 
+  command: postfix stop
+
+- name: Postfix started
+  command: postfix start
 
 - name: Root .forward in place
   template:
     src: templates/forward.j2
     dest: /root/.forward
-

roles/email/templates/transport.j2

+{{ hostname }} discourse:

roles/email/templates/update-exim4.j2

-# /etc/exim4/update-exim4.conf.conf
-#
-# Edit this file and /etc/mailname by hand and execute update-exim4.conf
-# yourself or use 'dpkg-reconfigure exim4-config'
-#
-# Please note that this is _not_ a dpkg-conffile and that automatic changes
-# to this file might happen. The code handling this will honor your local
-# changes, so this is usually fine, but will break local schemes that mess
-# around with multiple versions of the file.
-#
-# update-exim4.conf uses this file to determine variable values to generate
-# exim configuration macros for the configuration file.
-#
-# Most settings found in here do have corresponding questions in the
-# Debconf configuration, but not all of them.
-#
-# This is a Debian specific file
-
-dc_eximconfig_configtype='internet'
-dc_other_hostnames='{{ hostname }}'
-dc_local_interfaces=''
-dc_readhost=''
-dc_relay_domains=''
-dc_minimaldns='false'
-dc_relay_nets='{{ app_ip_address.stdout }}/32'
-dc_smarthost=''
-CFILEMODE='644'
-dc_use_split_config='true'
-dc_hide_mailname=''
-dc_mailname_in_oh='true'
-dc_localdelivery='mail_spool'
-