Commit 50e2bd98 authored by Chris Croome's avatar Chris Croome

Rebuild with FD_SETSIZE 2048 plus text updates

parent 7bfdf29a
Pipeline #3591 passed with stage
in 109 minutes and 59 seconds
......@@ -2,7 +2,7 @@
This repo contains GitLab CI instructions to install Ansible and GPG and SSH secret keys to a Docker container and then run an Ansible playbook, `run.yml`, which runs three roles:
1. The build role download source debs for PHP, set the file descriptors limit to a higher value and then rebuilds and signs them and creates a apt repo layout.
1. The build role download source debs for Apache2, OpenSSL and PHP7.0, set the file descriptors limit to a higher value and then rebuilds and signs them and creates an apt repo layout.
2. The test role then installs the debs that have been built to test that they can be installed without errors and also checks the complied value of `enable-fd-setsize`.
3. The deploy role `rsync`'s the files to the apt repo.
......@@ -10,6 +10,33 @@ The apt repo that this code generates is available at [deb.webarch.net](https://
The variables in `run.yml` would need changing if these playbooks are used elsewhere or used for building other packages.
'''Note that before compiling a new version the number of Debian PHP7.0 patches
needs to be checked''' — the `php_patch_number` in `run.yml` might need
incrementing if there are more than ''49'', you can check the patches by going
to the [Debian PHP7.0 package page](https://packages.debian.org/stretch/php7.0)
and then following the link in the right hand column to the
''Debian Patch Tracker'', version `7.0.33-0+deb9u1` has
[49 patches](https://sources.debian.org/patches/php7.0/7.0.33-0+deb9u1/), so for
this version the `php_patch_number` in `run.yml` is set to `50`, the patch that
this code applies, using `quilt`, is generated from
[this template](https://git.coop/webarch/php/blob/master/roles/build/templates/fd-setsize.patch.j2).
A [blog post](https://www.blog.webarchitects.coop/2019/02/building-debian-php-packages-with-gitlab-ci-and-ansible) has been written about this.
# Environment variables
The following environmental variables are set in GitLab CI / CD Settings for this project:
* GPG_PRIVATE_KEY
* SSH_PRIVATE_KEY
* SSH_KNOWN_HOSTS
It would be more secure to use Ansible Vault or something…
The corresponding GPG public key is in `roles/build/files/pub.gpg` and the GPG key id is a variable in `run.yml`.
The corresponding SSH public key is installed on the host that the repo is rsync'ed to.
# PHP File Descriptors Limit
This is the issue we hit on shared web servers with 50+ clients when trying to send email with InvoicePlane:
......@@ -30,7 +57,7 @@ Filename: src/SMTP.php
Line Number: 1124
```
However it is not just a matter of recompiling with `--enable-fd-setsize=2048`, see [this bug thread](https://bugs.php.net/bug.php?id=69637), you need to add the following to `main/php.h` as well (however this suggestion isn't for php7.0 so perhaps it doesn't help):
However it is not just a matter of recompiling with `--enable-fd-setsize=2048`, see [this bug thread](https://bugs.php.net/bug.php?id=69637), so we also add the following to `main/php.h`:
```
#undef __FD_SETSIZE
......@@ -39,23 +66,25 @@ However it is not just a matter of recompiling with `--enable-fd-setsize=2048`,
#define FD_SETSIZE 2048
```
Or edit this line in `/usr/include/x86_64-linux-gnu/bits/typesizes.h`:
And edit this line in `/usr/include/x86_64-linux-gnu/bits/typesizes.h`:
```
#define __FD_SETSIZE 1024
```
Not that PHP 7.1.0 onwards has a variable, `PHP_FD_SETSIZE`, see [this bug](https://bugs.php.net/bug.php?id=43269).
Check your installed PHP for the value of `enable-fd-setsize`:
And this line in `/usr/include/linux/posix_types.h`:
```
apt install php7.0-dev
php-config --configure-options | sed 's/\s\+/\n/g' | grep enable-fd-setsize
--enable-fd-setsize=1024
#define __FD_SETSIZE 1024
```
Note that [the Plesk advice](https://docs.plesk.com/en-US/onyx/advanced-administration-guide-linux/enhancing-performance/increasing-the-number-of-domains-that-plesk-can-serve.68766/) is to also compile openssl and libc-client.
Note that PHP 7.1.0 onwards has a variable, `PHP_FD_SETSIZE`, see [this bug](https://bugs.php.net/bug.php?id=43269).
It was found that [the Plesk advice](https://docs.plesk.com/en-US/onyx/advanced-administration-guide-linux/enhancing-performance/increasing-the-number-of-domains-that-plesk-can-serve.68766/) to also compile openssl and apache2 was needed but the libc-client didn't compile without errors so has been omitted.
The [test script](https://git.coop/webarch/php/blob/master/roles/test/files/test.php) from [PHP bug 69637](https://bugs.php.net/bug.php?id=69637)
always fails when run in Docker via GitLab CI but works when run on regular virtual server, we are not sure of the reason for this, but this is why the test is
[commented out](https://git.coop/webarch/php/blob/master/roles/test/tasks/main.yml#L28).
Our shared hosting servers use a lot of file descriptors, for example the total number of open file handles of any sort:
......@@ -92,23 +121,9 @@ Add the following line to `/etc/sysctl.conf` to ensure that this persists across
fs.file-max = 1633852
```
# Environment variables
The following environmental variables are set in GitLab CI / CD Settings for this project:
* GPG_PRIVATE_KEY
* SSH_PRIVATE_KEY
* SSH_KNOWN_HOSTS
It would be more secure to use Ansible Vault or something…
The corresponding GPG public key is in `roles/build/files/pub.gpg` and the GPG key id is a variable in `run.yml`.
The corresponding SSH public key is installed on the host that the repo is rsync'ed to.
# GPG keypair
After several attempts to use a key pair with a passphrase a pair without one was generated:
After multiple attempts to use a key pair with a passphrase a pair without one was generated:
```bash
gpg --full-generate-key
......@@ -169,10 +184,9 @@ gpg --export --armor EB612F9FE81381F8F3E58874495739F6CAA6F12D > pub.gpg
gpg --export-secret-keys --armor EB612F9FE81381F8F3E58874495739F6CAA6F12D > sec.gpg
```
## References
Building `.deb` files generates a lot of output, and [this should fix it](https://git.coop/webarch/gitlab/commit/ea261dd42f08312e402f24441c134a8b9c08b232).
Building `.deb` files generates a lot of output, time outs needed to be increased, [see this](https://git.coop/webarch/gitlab/commit/ea261dd42f08312e402f24441c134a8b9c08b232).
* [Building debian packages with debuild](https://blog.packagecloud.io/debian/debuild/packaging/2015/06/08/buildling-deb-packages-with-debuild/)
* [Dch non-interactive mode](https://askubuntu.com/questions/579323/dch-non-interactive-mode)
......
......@@ -214,7 +214,8 @@
args:
chdir: "/build/src/{{ apache_dir.stdout }}"
# This code doesn't compile
# This libc-client doesn't compile, the Plesk page here suggested it is needed
# https://docs.plesk.com/en-US/onyx/advanced-administration-guide-linux/enhancing-performance/increasing-the-number-of-domains-that-plesk-can-serve.68766/
## Compile libc-client2007e
#- name: Download libc-client2007e source package
# command: apt-get -y source libc-client2007e
......@@ -289,18 +290,15 @@
- name: Generate a fd setsize patch file
template:
src: templates/fd-setsize.patch.j2
#dest: "/build/src/{{ php_dir.stdout }}/debian/patches/00{{ php_patch_number }}-fd-setsize.patch"
dest: "/build/src/fd-setsize.patch"
- name: Read the patch file contents
#command: "cat /build/src/{{ php_dir.stdout }}/debian/patches/00{{ php_patch_number }}-fd-setsize.patch"
command: "cat /build/src/fd-setsize.patch"
register: php_fd_setsize_patch
args:
warn: no
- debug:
#msg: "The contents of /build/src/{{ php_dir.stdout }}/debian/patches/00{{ php_patch_number }}-fd-setsize.patch : {{ php_fd_setsize_patch.stdout }}"
msg: "The contents of /build/src/fd-setsize.patch : {{ php_fd_setsize_patch.stdout }}"
- name: Apply all existing patches using quilt redirecting stderr to stdout
......
......@@ -4,26 +4,27 @@
--><!--#set var="PAGE_DESC"
value="The apt repo contains PHP{{ php_version }} packages for Debian {{ php_distro }} built with FD_SETSIZE / --enable-fd-setsize set to {{ php_sockets }}, rather than the default of 1024."
value="This APT repo contains Apache2, OpenSSL and PHP{{ php_version }} packages for Debian {{ php_distro }} built with FD_SETSIZE / --enable-fd-setsize set to {{ php_sockets }}, rather than the default of 1024."
--><!--#include virtual="/wsh/top.shtml" -->
<p>The PHP{{ php_version }} packages in this repo were built on <!--#config timefmt="%A, %e %B %Y" --><!--#echo var="LAST_MODIFIED" -->
and can be <a href="pool/main/p/php{{ php_version }}/">browsed here</a>, following are instructions for installing them on Debian {{ php_distro }}.
<p>The packages hosted here are automatically re-compiled versions of the Debian {{ php_distro }} source packages, to mitigate <a href="https://bugs.php.net/bug.php?id=69637">this PHP bug</a>,
they were were built on <!--#config timefmt="%A, %e %B %Y" --><!--#echo var="LAST_MODIFIED" --> and can be accessed via the directory listings for
<a href="/pool/main/a/apache2/">Apache2</a>, <a href="/pool/main/o/openssl/">OpenSSL</a> and <a href="pool/main/p/php{{ php_version }}/">PHP{{ php_version }}</a>,
and they can also be installed using <code>apt</code>, <code>apt-get</code> or <code>aptitude</code>, using the following are instructions.</p>
<h2 id="install">Install <a href="#install" class="a">#</a></h2>
<p>The following commands all assume you are <code>root</code>, run <code>sudo -i</code> first.
<p>To use the packages in this repo first ensure you have <code>apt-transport-https</code> installed:
<p>The following commands all assume you are <code>root</code>, run <code>sudo -i</code> first.
To use the packages in this repo first ensure you have <code>apt-transport-https</code> installed:</p>
<pre><code class="bash">apt install apt-transport-https</code></pre>
Then add <a href="pub.gpg">the public GPG key</a> (ID: <code>{{ php_gpg_id }}</code>) to your machine:</p>
Add <a href="pub.gpg">the public GPG key</a> (ID: <code>{{ php_gpg_id }}</code>) to your machine:</p>
<pre><code class="bash">wget -O - https://{{ php_repo_domain }}/pub.gpg | apt-key add -</code></pre>
<p>Then create a <code>/etc/apt/sources.list.d/php.list</code> file:</p>
<p>Create a <code>/etc/apt/sources.list.d/php.list</code> file:</p>
<pre><code class="bash">echo "deb https://{{ php_repo_domain }}/ {{ php_distro }} main" &gt; /etc/apt/sources.list.d/php.list</code></pre>
......@@ -51,6 +52,6 @@ apt dist-upgrade</code></pre>
<h2 id="code">Code <a href="#code" class="a">#</a></h2>
<p>The code that was used to rebuild the PHP{{ php_version }} packages for Debian {{ php_distro }} and build this apt repo is <a href="https://git.coop/webarch/php">available on our Git Lab servers at git.coop</a>.</p>
<p>The code that was used to build the Apache2, OpenSSL and PHP{{ php_version }} packages for Debian {{ php_distro }} and this apt repo is <a href="https://git.coop/webarch/php">available on our GitLab server at git.coop</a>.</p>
<!--#include virtual="/wsh/bot.shtml" -->
......@@ -4,7 +4,7 @@
vars:
php_version: 7.0
php_sockets: 20480
php_sockets: 2048
php_distro: stretch
php_repo_domain: deb.webarch.net
php_release_email: deb@webarch.net
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment