summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorVincent Sanders <vince@kyllikki.org>2018-12-21 20:10:12 +0000
committerVincent Sanders <vince@kyllikki.org>2018-12-21 20:10:12 +0000
commitda0510b2c6a1e9a1910a4ec7b8bbb7d5feb7dfde (patch)
treefea3b6fb5b86d898a040d02635c001809d8d6402
parent6f50fa1169888494aca906918be866a6de490943 (diff)
downloadnetsurf-wiki-master.tar.gz
netsurf-wiki-master.tar.bz2
add buster ci install pageHEADmaster
-rw-r--r--continuous_integration.mdwn2
-rw-r--r--continuous_integration/debian_buster_setup.mdwn349
-rw-r--r--continuous_integration/debian_stretch_setup.mdwn56
3 files changed, 379 insertions, 28 deletions
diff --git a/continuous_integration.mdwn b/continuous_integration.mdwn
index e494d59..fadaeeb 100644
--- a/continuous_integration.mdwn
+++ b/continuous_integration.mdwn
@@ -34,6 +34,8 @@ The majority of the build slaves are provided by our own
[[virtual host server|virtual_host_server]] excepting operating
systems which cannot be virtualised (principally MAC OS X).
+### [[Debian 10 (Buster) Setup|continuous_integration/debian_buster_setup]]
+
### [[Debian 9 (Stretch) Setup|continuous_integration/debian_stretch_setup]]
### [[Fedora 26 Setup|continuous_integration/fedora_26_setup]]
diff --git a/continuous_integration/debian_buster_setup.mdwn b/continuous_integration/debian_buster_setup.mdwn
new file mode 100644
index 0000000..b216b14
--- /dev/null
+++ b/continuous_integration/debian_buster_setup.mdwn
@@ -0,0 +1,349 @@
+[[!meta title="Continuous Integration Debian Buster Setup"]]
+[[!meta author="Kyllikki"]]
+[[!meta date="2018-12-21T12:46:14Z"]]
+
+
+[[!toc]]
+
+## Debian 9 (Buster) OS install
+
+### amd64 VDS install from media
+
+[[Virtual server setup|virtual_host_server]]
+
+Install minimal system from netinst CD (attached when VDS is created on
+phoenix) Config options:
+
+- In the "role" selection select "ssh server" and "system utilities" only.
+- The whole disc default partitioning is fine
+- The base user the install insists on creating should be the netsurf user.
+- Boot loader in MBR
+
+Once installed:
+
+- install sudo package and add netsurf user to sudo group
+- edit /etc/default/grub
+
+ GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8"
+
+as root update grub
+
+ update-grub2
+
+### arm64 VDS install
+
+[[ARM64 virtual server setup|virtual_host_server_arm64]]
+
+Once installed:
+
+- create netsurf user
+- install sudo package and add netsurf user to sudo group
+- edit /etc/inittab comment pty 3 through 6 and uncomment serial T0
+- enable backports
+
+`echo "deb `[`http://http.debian.net/debian`](http://http.debian.net/debian)` buster-backports main" > /etc/apt/sources.list.d/backports.list`
+`apt-get update`
+
+
+## Packaged CI worker install
+
+Do a base OS install
+
+The recommended hostname for CI workers is "nsciworker17" this allows us
+to clearly identify CI worker nodes. Note historically we have used
+"cislave1" which has been objected to by several users. See
+[[changing hostname|https://wiki.debian.org/howto/changehostname_changing_hostname]] on how to achive this.
+
+On master jenkins use "manage nodes" to create new node. Ensure "remote
+fs root" is set to /var/lib/jenkins add variable JENKINS\_HOME set to
+/var/lib/jenkins
+
+Note: replace arm64 with architecture name as required (armhf etc.)
+
+When configuring a scaleway instance the "Launch method" should be set
+to "start and stop this node on-demand" with the "Start script" set to
+`perl /usr/bin/pscw.pl start netsurf-ciworker17` and "Stop script" set
+to `perl /usr/bin/pscw.pl stop netsurf-ciworker17`. The Availability
+set to "Take this slave on-line when in demand and off-line when idle"
+and "in demand delay" set to 0 and "idle delay" set to 15. This will
+mean the node is turned off and not charged for when idle.
+
+As superuser:
+
+create jenkins user
+
+ adduser --system --group --home /var/lib/jenkins/ --disabled-login jenkins
+
+Install https transport
+
+ apt-get install apt-transport-https
+
+Add CI server repo to slave apt sources
+
+ echo "deb https://ci.netsurf-browser.org/debian/ buster/amd64/" >> /etc/apt/sources.list.d/netsurf-browser.list
+
+update repos
+
+ apt-get update
+
+install ci worker package. accept the large package list and the
+ unsigned package install for ns-ci-slave
+
+ apt-get install ns-ci-worker
+
+edit /etc/default/ns-ci-worker to set the correct url and secret parameters
+
+ensure /opt is setup correctly to allow toolchains to be built on the node
+
+ mkdir -p /opt/netsurf
+ chown jenkins:jenkins /opt/netsurf
+
+become jenkins user
+
+ su -s/bin/bash - jenkins
+
+create ssh keypair (accept defaults - no password)
+
+ ssh-keygen -t rsa -C "netsurf@nsciworker12.netsurf-browser.org"
+
+copy .ssh/id\_rsa.pub from worker to jenkins master node and append
+ to /home/netsurf/.ssh/authorized\_keys
+
+ scp /home/jenkins/.ssh/id_rsa.pub netsurf@ci.netsurf-browser.org:nsciworker12_id_rsa.pub
+
+exit jenkins user shell
+
+start CI worker daemon
+
+ /etc/init.d/ns-ci-worker start
+
+
+
+## Pbuilder setup
+
+This allows a worker to build Debian packages. The worker should be
+installed as a normal CI worker node and then:
+
+as superuser on node:
+
+ # apt-get install pbuilder
+ # addgroup pbuilder
+ # addgroup jenkins pbuilder
+
+create /etc/sudoers.d/pbuilder
+
+ jenkins ALL = NOPASSWD:/usr/sbin/pbuilder
+
+`visudo` and alter Defaults
+
+ Defaults env_reset,env_keep="DIST ARCH"
+
+replace `/etc/pbuilderrc`
+
+ # this is your configuration file for pbuilder.
+ # the file in /usr/share/pbuilder/pbuilderrc is the default template.
+ # /etc/pbuilderrc is the one meant for overwriting defaults in
+ # the default template
+ #
+ # read pbuilderrc.5 document for notes on specific options.
+
+ # List of Debian suites.
+ DEBIAN_SUITES=("sid", "buster", "jessie", "wheezy", "squeeze")
+
+ # List of Ubuntu suites.
+ UBUNTU_SUITES=("vivid" "utopic" "trusty" "saucy" "raring" "quantal" "precise" "oneiric" "natty" "lucid" "hardy")
+
+ # Mirrors to use. Update these to your preferred mirror.
+ DEBIAN_MIRROR="ftp.uk.debian.org"
+ UBUNTU_MIRROR="mirrors.kernel.org"
+
+ # set a default distribution if none is used.
+ : ${DIST:="$(lsb_release --short --codename)"}
+
+ # set the architecture to the host architecture if none set.
+ : ${ARCH:="$(dpkg --print-architecture)"}
+
+ NAME="$DIST"
+ if [ -n "${ARCH}" ]; then
+ NAME="$NAME-$ARCH"
+ DEBOOTSTRAPOPTS=("--arch" "$ARCH" "${DEBOOTSTRAPOPTS[@]}")
+ fi
+ BASETGZ="/var/cache/pbuilder/$NAME-base.tgz"
+ DISTRIBUTION="$DIST"
+ BUILDRESULT="/var/cache/pbuilder/$NAME/result/"
+ APTCACHE="/var/cache/pbuilder/$NAME/aptcache/"
+ BUILDPLACE="/var/cache/pbuilder/build/"
+
+ if $(echo ${DEBIAN_SUITES[@]} | grep -q $DIST); then
+ # Debian configuration
+ MIRRORSITE="http://$DEBIAN_MIRROR/debian/"
+ COMPONENTS="main contrib non-free"
+ if $(echo "$STABLE_CODENAME stable" | grep -q $DIST); then
+ OTHERMIRROR="$OTHERMIRROR | deb $MIRRORSITE $STABLE_BACKPORTS_SUITE $COMPONENTS"
+ fi
+ elif $(echo ${UBUNTU_SUITES[@]} | grep -q $DIST); then
+ # Ubuntu configuration
+ MIRRORSITE="http://$UBUNTU_MIRROR/ubuntu/"
+ COMPONENTS="main restricted universe multiverse"
+ else
+ echo "Unknown distribution: $DIST"
+ exit 1
+ fi
+
+The architecture is assumed to be the native one from
+
+ dpkg --print-architecture
+
+This can be set by passing ARCH to pbuilder (useful for i386 maybe?)
+
+for each distribution this node will build for:
+
+create pbuilder result directory and set ownership permissions
+
+ # mkdir -p /var/cache/pbuilder/buster-armhf/result
+ # chown root:pbuilder /var/cache/pbuilder/buster-armhf/result
+ # chmod g+w /var/cache/pbuilder/buster-armhf/result
+
+become jenkins user
+
+ su -s/bin/bash - jenkins
+
+create pbuilder base for distribution
+
+ sudo DIST=buster pbuilder create
+
+if desired additional packages and config can be made to the base with
+
+`sudo DIST=buster pbuilder login --save-after-login`
+
+## distcc worker node
+
+Do a basic OS install but \*not\* a CI worker setup.
+
+A recommended hostname for distcc worker is something like "cicpu0" this
+allows us to use systems as processing node for other purposes than just
+distcc in future. See debians
+[[changing hostname|https://wiki.debian.org/howto/changehostname_changing_hostname]] on how to achive this.
+
+The Netsurf repository has necessary updated packages in it and can be
+accessed by doing the following:
+
+Add CI server repo to worker apt sources
+
+ echo "deb https://ci.netsurf-browser.org/builds/debian/ buster/amd64/" >> /etc/apt/sources.list
+
+update repos
+
+ apt-get update
+
+use apt to install these packages:
+
+ build-essential
+ gcc
+ clang
+ distcc
+
+edit /etc/default/distcc
+
+ STARTDISTCC="true"
+ ALLOWEDNETS="192.168.211.0/24"
+ LISTEN="0.0.0.0"
+ JOBS="8"
+
+start the service
+
+ service distcc start
+
+ensure the client has hosts set to use the new worker
+
+
+
+## Manual CI worker install
+
+Caution these instructions may not be up to date.
+
+### required packages
+
+The Netsurf repository has necessary updated packages in it and can be
+accessed by doing the following:
+
+Add CI server repo to slave apt sources
+
+`echo "deb `[`http://ci.netsurf-browser.org/builds/debian/`](http://ci.netsurf-browser.org/builds/debian/)` buster/amd64/" >> /etc/apt/sources.list`
+
+update repos
+
+ apt-get update
+
+use apt to install these packages:
+
+`openjdk-7-jre-headless `
+`screen `
+`build-essential`
+`ccache`
+`clang`
+`git`
+`pkg-config`
+`check`
+`doxygen`
+`libjson0-dev (from our repo - needs bugfixes `[`http://ci.netsurf-browser.org/builds/debian/`](http://ci.netsurf-browser.org/builds/debian/)`)`
+`libexpat1-dev`
+`libxml-perl`
+`libxml-xpath-perl`
+`lcov`
+`gcovr (from our repo)`
+`gperf`
+`flex`
+`bison`
+`libpng-dev`
+`libjpeg-dev`
+`libmozjs185-dev`
+`libglib2.0-dev`
+`libcurl4-openssl-dev`
+`liblcms1-dev`
+`libxml2-dev`
+`librsvg2-dev`
+`libmng-dev`
+`libgtk2.0-dev`
+`libmozjs-dev`
+
+### config
+
+- on master jenkins use "manage nodes" to create new node. Ensure
+ "remote fs root" is set to /home/netsurf/jenkins
+- create netsurf user
+- as netsurf user:
+ - wget <http://ci.netsurf-browser.org/jenkins/jnlpJars/slave.jar>
+ - run screen
+ - create jenkins-slave.sh
+
+ #!/bin/bash
+
+ java -Djava.awt.headless=true -jar slave.jar -jnlpUrl https://ci.netsurf-browser.org/jenkins/computer/chimera/slave-agent.jnlp -secret 0123456789abcdef01234567890abcdef
+
+ - run jenkins-slave.sh
+ - create new screen tab
+ - create ssh keypair (accept defaults - no password)
+
+ ssh-keygen -t rsa -C "netsurf@cislave0.netsurf-browser.org"
+
+- - copy .ssh/id\_rsa.pub from slave to jenkins master node and
+ append to /home/netsurf/.ssh/authorized\_keys
+
+ scp ci.netsurf-browser.org:.ssh/id_rsa.pub .id_rsa.pub
+ cat id_rsa.pub >> .ssh/authorized_keys
+
+- - copy .ssh/id\_rsa.pub from master node to slave and append to
+ /home/netsurf/.ssh/authorized\_keys
+ - create reverse-ssh.sh (change tunnel port number!)
+
+ #!/bin/sh
+
+ ssh -R 22224:localhost:22 netsurf@ci.netsurf-browser.org 'bash -c "while true; do echo .; sleep 60; done"
+
+- - run reverse-ssh.sh
+ - on the master create a shell script to use the ssh tunnel
+ connection, thus firewalls etc are moot as long as the slave can
+ connect to the master
+
+ ssh netsurf@localhost -p 22223
diff --git a/continuous_integration/debian_stretch_setup.mdwn b/continuous_integration/debian_stretch_setup.mdwn
index 2333455..557472e 100644
--- a/continuous_integration/debian_stretch_setup.mdwn
+++ b/continuous_integration/debian_stretch_setup.mdwn
@@ -309,10 +309,10 @@ Add CI server repo to slave apt sources
update repos
- apt-getupdate
+ apt-get update
install ci worker package. accept the large package list and the
- unsigned package install for gcovr and ns-ci-slave
+ unsigned package install for ns-ci-slave
apt-get install ns-ci-worker
@@ -361,9 +361,9 @@ create /etc/sudoers.d/pbuilder
`visudo` and alter Defaults
-`Defaults        env_reset,env_keep="DIST ARCH"`
+ Defaults env_reset,env_keep="DIST ARCH"
-replace /etc/pbuilderrc
+replace `/etc/pbuilderrc`
# this is your configuration file for pbuilder.
# the file in /usr/share/pbuilder/pbuilderrc is the default template.
@@ -417,7 +417,7 @@ replace /etc/pbuilderrc
The architecture is assumed to be the native one from
-`dpkg --print-architecture`
+ dpkg --print-architecture
This can be set by passing ARCH to pbuilder (useful for i386 maybe?)
@@ -431,11 +431,11 @@ create pbuilder result directory and set ownership permissions
become jenkins user
-`su -s/bin/bash - jenkins`
+ su -s/bin/bash - jenkins
create pbuilder base for distribution
-`sudo DIST=stretch pbuilder create`
+ sudo DIST=stretch pbuilder create
if desired additional packages and config can be made to the base with
@@ -455,18 +455,18 @@ accessed by doing the following:
Add CI server repo to worker apt sources
-`echo "deb `[`http://ci.netsurf-browser.org/builds/debian/`](http://ci.netsurf-browser.org/builds/debian/)` stretch/amd64/" >> /etc/apt/sources.list`
+ echo "deb https://ci.netsurf-browser.org/builds/debian/ stretch/amd64/" >> /etc/apt/sources.list
update repos
-`apt-get update`
+ apt-get update
use apt to install these packages:
-`build-essential`
-`gcc`
-`clang`
-`distcc`
+ build-essential
+ gcc
+ clang
+ distcc
edit /etc/default/distcc
@@ -477,7 +477,7 @@ edit /etc/default/distcc
start the service
-`service distcc start`
+ service distcc start
ensure the client has hosts set to use the new worker
@@ -492,13 +492,13 @@ Caution these instructions may not be up to date.
The Netsurf repository has necessary updated packages in it and can be
accessed by doing the following:
-- Add CI server repo to slave apt sources
+Add CI server repo to slave apt sources
`echo "deb `[`http://ci.netsurf-browser.org/builds/debian/`](http://ci.netsurf-browser.org/builds/debian/)` stretch/amd64/" >> /etc/apt/sources.list`
-- update repos
+update repos
-`apt-get update`
+ apt-get update
use apt to install these packages:
@@ -542,33 +542,33 @@ use apt to install these packages:
- run screen
- create jenkins-slave.sh
-`#!/bin/bash`
-
-`java -Djava.awt.headless=true -jar slave.jar -jnlpUrl `[`http://ci.netsurf-browser.org/jenkins/computer/chimera/slave-agent.jnlp`](http://ci.netsurf-browser.org/jenkins/computer/chimera/slave-agent.jnlp)` -secret 0123456789abcdef01234567890abcdef`
+ #!/bin/bash
+
+ java -Djava.awt.headless=true -jar slave.jar -jnlpUrl https://ci.netsurf-browser.org/jenkins/computer/chimera/slave-agent.jnlp -secret 0123456789abcdef01234567890abcdef
-- - run jenkins-slave.sh
+ - run jenkins-slave.sh
- create new screen tab
- create ssh keypair (accept defaults - no password)
-`ssh-keygen -t rsa -C "netsurf@cislave0.netsurf-browser.org"`
+ ssh-keygen -t rsa -C "netsurf@cislave0.netsurf-browser.org"
- - copy .ssh/id\_rsa.pub from slave to jenkins master node and
append to /home/netsurf/.ssh/authorized\_keys
-`scp ci.netsurf-browser.org:.ssh/id_rsa.pub .id_rsa.pub`
-`cat id_rsa.pub >> .ssh/authorized_keys`
+ scp ci.netsurf-browser.org:.ssh/id_rsa.pub .id_rsa.pub
+ cat id_rsa.pub >> .ssh/authorized_keys
- - copy .ssh/id\_rsa.pub from master node to slave and append to
/home/netsurf/.ssh/authorized\_keys
- create reverse-ssh.sh (change tunnel port number!)
-`#!/bin/sh`
-
-`ssh -R 22224:localhost:22 netsurf@ci.netsurf-browser.org 'bash -c "while true; do echo .; sleep 60; done"'`
+ #!/bin/sh
+
+ ssh -R 22224:localhost:22 netsurf@ci.netsurf-browser.org 'bash -c "while true; do echo .; sleep 60; done"
- - run reverse-ssh.sh
- on the master create a shell script to use the ssh tunnel
connection, thus firewalls etc are moot as long as the slave can
connect to the master
-`ssh netsurf@localhost -p 22223`
+ ssh netsurf@localhost -p 22223