Skip to content

Latest commit

 

History

History
993 lines (673 loc) · 26.3 KB

Documentation SellYourSaas - Remote Backup server - EN.asciidoc

File metadata and controls

993 lines (673 loc) · 26.3 KB
Raw

Sell-Your-Saas installation and operation document

Table of contents

Introduction

This document present how to install a remote server for Backups in a SellYourSaas environement.

Choice of machine and OS and create it

  • Obtain a server with SSH access that can pass root (We will use Ubuntu LTS minimum 16.04 or 18.04 or 20.04 or 22.04)

  • Add DNS entries of the server(s) (Entry A for IP4 and entry AAAA for IP6)

Adding the harddisk of data (home of user instances and home of backups)

We will add on the RemoteBackupServer, an independent disk for user instances and backups.

With OVH Public Cloud or ScaleWay:

  • Create the disk of data from backoffice. You can imagine to reserve 250MB for each customer instance so choose a size in consideration.

  • Associate the disk with the server (each additional disk is added in /dev/vdb, /dev/vdc, /dev/vdd, …​). Note, the disk becomes visible with fdisk -l and lsblk

  • If it is a disk never partitioned, add the partition on the disk (Linux type) and format it by doing:

fdisk -l
fdisk /dev/vdx
option n then p (then choose the partition number, first and last sector) then w

fdisk -l

fsck -N /dev/vdxY
mkfs.btrfs /dev/vdxY

Whether the disk has just been formatted or whether it is an added disk already formatted, the rest of the procedure is identical:

  • Recover the value of the UUID at the end of the formatting which is displayed, otherwise, recover it with the command

blkid

  • Declare the assembly for an automatic assembly at each reboot by adding a line in /etc/fstab

UUID=94817f83-a2ad-46c4-81e0-06e6dd0e95f1 /mnt/diskX btrfs noatime,nofail 0 0 (does not block the server from starting)

  • Mount disk

mkdir /mnt/diskbackup; chown admin.admin /mnt/diskbackup
mount /dev/vdxZ /mnt/diskbackup

blkid

Note: A reboot may be required if disk or mount is not visible.

  • Optimize the filesystem by removing the update of the "atime" read access

To see options for optimizing filesystems:

tune2fs -l /dev/vdxY | grep features

return

Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize

To add -noatime to the filesystem in the /etc/fstab file:

UUID=94817f83-a2ad-46c4-81e0-06e6dd0e95f1 /mnt/diskX btrfs noatime,nofail 0 0

To take the change into account:

mount -oremount /dev/diskX/

To check:

cat /proc/mounts | grep diskX

Rem: If you need to recover data files from another disk, use:

rsync --info=progress2 -au serveursource:/mnt/diskSource/* /mnt/diskTarget

SSH and sudo

Unix admin account

Create the user account admin. It will be used to install and administer the system when root is not required.

groupadd admin; useradd -m -g admin admin; usermod -a -G adm admin;
mkdir /home/admin/logs; chown root.adm /home/admin/logs; chmod 770 /home/admin/logs;
chown admin.admin /mnt/diskbackup

Check that the id of this user admin is greater than or equal to 1000.

Create a user account for yourself (or other administrators), for example: myunixlogin. It will be used to log in.

adduser myunixlogin

ssh setup

Fix permission on /etc/ssh/sshd_config so only root has read and write access:

chmod go-rw /etc/ssh/sshd_config

Create a file /etc/ssh/sshd_config.d/sellyoursaas.conf to change login permissions with the following content:

# Privilege Separation is turned on for security
UsePrivilegeSeparation yes
# Permissions on files must be correct to allow login
StrictModes yes

# MaxSessions 10
MaxSessions 25

# Disallow login to root
PermitRootLogin no
# Disallow empty passwords
PermitEmptyPasswords no
# Do not support the "keyboard-interactive" authentication scheme defined in RFC-4256.
ChallengeResponseAuthentication no

# Define list of allowed method to authenticate
PasswordAuthentication yes
PubkeyAuthentication yes

DenyUsers guest

AuthorizedKeysFile     .ssh/authorized_keys .ssh/authorized_keys_support

AllowUsers admin
AllowUsers myunixlogin

Please note: replace myunixlogin with the correct value before taking changes into account with:

/etc/init.d/ssh reload

Add the following line in the /etc/sudoers file to reposition the HOME according to the user after a sudo -s:

Defaults set_home

Create a file /etc/sudoers.d/myunixlogin with the owner root.root and the permissions r-r----- and the content

myunixlogin ALL=(ALL) NOPASSWD:ALL

Test that you can connect using myunixlogin and make a sudo with

ssh -v [email protected]
sudo -s

Add your public key to your unix account.

ssh-copy-id [email protected]

Warning: Sometime ssh-copy-id copy the dss key instead of rsa key and ssh fails with dss.

Define or redefine the password for root, admin with a secure password.

passwd root
passwd admin

Launch ssh-keygen on each of thee 3 accounts root, admin and myunixlogin

Default shell

Modify the default shell to use bash (instead of dh sh or dash)

ln -fs /bin/bash /usr/bin/sh

Deletion of information files at login

In order not to give information to users doing SSH, on the deployment servers:

rm /etc/update-motd.d/10-help-text /etc/update-motd.d/20-runabove
rm /etc/update-motd.d/50-landscape-sysinfo /etc/update-motd.d/50-landscape-sysinfo
rm /etc/update-motd.d/9*-update*-available /etc/update-motd.d/92-unattended-upgrades

Add alias

Add at the end of /etc/bash.bashrc:

alias psld='ps -fax -eo user:12,pid,ppid,pcpu,pmem,vsz:12,size:12,tty,start_time:6,utime,time,cmd'

Hostname and IP configuration

Add an entry from the new server to the DNS provided by the domain provider.

Go to the OVH IP management interface, to add the reverse on the server IP.

Go to the management interface of OVH servers, to modify their short name. This will modify the /etc/hostname file automatically (if not manually modify) with the short name. The file will then have as sole content:

nameofserver

Connect and modify the file /etc/hosts with the entry of the new server

main.ip.of.server nameofserver.mysaasdomainname.com

Added support for IP v6 (optional, if ipv6 wanted but not yet enabled)

With ifupdown (Ubuntu 16.04)

  • To add a v6 IP dynamically for testing purposes at first:

ip addr add 2002:41d0:1234:1000::1234/128 dev eth0
ip -6 route add 2002:41d0:1234:1000::1 dev eth0
ip -6 route add default via 2002:41d0:1234:1000::1 dev eth0

  • For a persistent reboot definition, declare the interface in /etc/network/interfaces or in a file in /etc/network/interfaces.d (Ubuntu <17.10)

Example for an IPv6 2002:41d0:1234:1000::1234 with as gateway 2002:41d0:1234:1000::1

# To declare a persistent v6 IP (the mask is 128 at OVH in ipv6)
iface eth0 inet6 static
        address 2002:41d0:1234:1000::1234
        netmask 128
        post-up /sbin/ip -6 route add 2002:41d0:1234:1000::1 dev eth0
        post-up /sbin/ip -6 route add default via 2002:41d0:1234:1000::1 dev eth0
        pre-down /sbin/ip -6 route del default via 2002:41d0:1234:1000::1 dev eth0
        pre-down /sbin/ip -6 route del 2002:41d0:1234:1000::1 dev eth0

Rem: eth0 can be something else, for example ens3.

To take this into account, try this, otherwise, reboot.

/etc/init.d/networking restart

With netplan (Ubuntu 18.04 +)

Add a conf file /etc/netplan/51-ipv6-ovh.yaml. Note: OVH provides a /128 for ipv6 but netplan wants /64

Example for an IPv6 1234:41d0:1234:1000::1234 with as gateway 1234:41d0:1234:1000::1

network:
	version: 2
	ethernets:
		eth0:
			match:
				name: eth0
			addresses:
				- "1234:41d0:1234:1000::1234/64"
			gateway6: "1234:41d0:1234:1000::1"

Note: Use 4 spaces for tabulation.

netplan try
netplan apply

Rem: eth0 can be something else, for example ens3.

Add virtual IP (optional)

  • Add the virtual IP via the OVH manager.

  • Add and remove the virtual network interface on the server dynamically (for test).

Addition:

ifconfig eth0: 0 a.b.c.d

Deletion:

ifconfig eth0: 0 down

  • For a persistent reboot definition, declare the interface in /etc/network/interfaces or in a file in /etc/network/interfaces.d (Ubuntu <17.10)

Example for 2 virtual IPs:

auto eth0: 0
iface eth0: 0 inet static
            address a.b.c.d
            netmask 255.255.255.255
            broadcast a.b.c.d

# To declare a persistent virtual IP
auto eth0: 1
iface eth0: 1 inet static
            address e.f.g.h
            netmask 255.255.255.255
            broadcast e.f.g.h

Rem: eth0 can be something else, for example ens3.

To take this into account, try this, otherwise, reboot.

/etc/init.d/networking restart

  • Associate the virtual IP with the server from the OVH manager.

Creation of working directories

Only /mnt/diskbackup is required, so no creation of directory has to be done.

Installation of system and application components

Installation of packages

There are two scenario depending on your version of Ubuntu. Follow the instruction 18.04- OR the 20.04+ one

  • Installation of the 18.04- Ubuntu packages

sudo apt update
sudo apt install ntp git gzip zip zstd ncdu duc
sudo apt install rkhunter chkrootkit
sudo apt install spamc spamassassin clamav clamav-daemon
sudo apt install fail2ban

  • Installation of the 20.04+ Ubuntu packages

sudo apt update
sudo apt install systemd-timesyncd git gzip zip zstd ncdu
sudo apt install rkhunter chkrootkit
sudo apt install spamc spamassassin clamav clamav-daemon
sudo apt install fail2ban

Delete all snap packages (we don’t need snap for backup servers):

sudo apt remove --purge --assume-yes snapd
systemctl daemon-reload

Disabling automatic update

Uninstall the package unattended-upgrades if it was installed.

apt remove unattended-upgrades

Installation of the firewall

  • Create a firewall to accept input of SSH only and allow output for NTP and DNS and HTTPS

ufw allow 22

ufw allow out ntp
ufw allow out 53
ufw allow out 443

ufw status
ufw enable

Port 22 is for SSH, NTP is 123/upd, 53 if for DNS, 443 is to allow monitoring agents to push reporting data

Installation of fail2ban

  • Installation of fail2ban and activation of the following fail2ban rules: pam-generic, xinetd-fail

To do this, first create a /etc/fail2ban/jail.local file with this content:

Note
 The rules available may vary depending on the version of the OS installed.
Note
 Remember to also modify "mybusinessips" by your ip(s) separated by spaces as well as the parameter destemail by the supervision email of your company. 
# Fail2Ban configuration file.
#
# This file was composed for Debian systems from the original one
# provided now under /usr/share/doc/fail2ban/examples/jail.conf
# for additional examples.
#
# Comments: use '#' for comment lines and ';' for inline comments
#
# To avoid merges during upgrades DO NOT MODIFY THIS FILE
# and rather provide your changes in /etc/fail2ban/jail.local
#

# The DEFAULT allows a global definition of the options. They can be overridden
# in each jail afterwards.

[DEFAULT]
# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not
# ban a host which matches an address in this list. Several addresses can be
# defined using space separator.
ignoreip = 127.0.0.1/8 mybusinessips

# "bantime" is the number of seconds that a host is banned.
bantime  = 3600

# A host is banned if it has generated "maxretry" during the last "findtime"
# seconds.
findtime = 600
maxretry = 3

# "backend" specifies the backend used to get files modification.
# Available options are "pyinotify", "gamin", "polling" and "auto".
# This option can be overridden in each jail as well.
#
# pyinotify: requires pyinotify (a file alteration monitor) to be installed.
#            If pyinotify is not installed, Fail2ban will use auto.
# gamin:     requires Gamin (a file alteration monitor) to be installed.
#            If Gamin is not installed, Fail2ban will use auto.
# polling:   uses a polling algorithm which does not require external libraries.
# auto:      will try to use the following backends, in order:
#            pyinotify, gamin, polling.
backend = auto

# "usedns" specifies if jails should trust hostnames in logs,
#   warn when reverse DNS lookups are performed, or ignore all hostnames in logs
#
# yes:   if a hostname is encountered, a reverse DNS lookup will be performed.
# warn:  if a hostname is encountered, a reverse DNS lookup will be performed,
#        but it will be logged as a warning.
# no:    if a hostname is encountered, will not be used for banning,
#        but it will be logged as info.
usedns = warn

#
# Destination email address used solely for the interpolations in
# jail.{conf,local} configuration files.
destemail = [email protected]

#
# Name of the sender for mta actions
sendername = Fail2Ban


#
# ACTIONS
#

# Default banning action (e.g. iptables, iptables-new,
# iptables-multiport, shorewall, etc) It is used to define
# action_* variables. Can be overridden globally or per
# section within jail.local file
banaction = iptables-multiport

# email action. Since 0.8.1 upstream fail2ban uses sendmail
# MTA for the mailing. Change mta configuration parameter to mail
# if you want to revert to conventional 'mail'.
mta = sendmail



[pam-generic]

enabled = true


[sshd]

enabled = true



[xinetd-fail]

enabled = true

Then place the filter files supplied with the project in etc/fail2ban/filter.d in the directory of the same name /etc/fail2ban/filter.d

Test spamassassin

The process spamd must be running. Start it manually if it is not the case the first time.

To test that spamassassin client is working, create a file /tmp/testspam with content

Subject: Test spam mail (GTUBE)
Message-ID: <[email protected]>
Date: Wed, 23 Jul 2003 23:30:00 +0200
From: Sender <[email protected]>
To: Recipient <[email protected]>
Precedence: junk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
This is the GTUBE, the
 Generic
 Test for
 Unsolicited
 Bulk
 Email
If your spam filter supports it, the GTUBE provides a test by which you
can verify that the filter is installed correctly and is detecting incoming
spam. You can send yourself a test mail containing the following string of
characters (in upper case and with no white spaces and line breaks):
XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X
You should send this test mail from an account outside of your network.

Then test with:

spamc < /tmp/testspam
spamc -c < /tmp/testspam
echo $?

Installation of ClamAV

The process freshclam and clamd must be running. Start them manually the first time.

To test clamav tool, create a file /tmp/testvirus with content

X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*

And to test clamav command line and daemon:

clamscan /tmp/testvirus
clamdscan /tmp/testvirus --fdpass
service apparmor reload
service apparmor status

You should see into the status a line saying that Profile usr/sbin/clamd is disabled. It seems we must also restart apache to have this effective inside apache.

Setup of /etc/security/limits.conf (optionnel)

  • Editer le fichier /etc/security/limits.conf par exemple pour augmenter le nombre de fichiers max ouvert par processus

mysql           soft     nofile           4096
mysql           hard     nofile           32768

Pour voir les limites:

ulimit -a

Setup of logrotate

  • Add a line if not already present into file /etc/logrotate.conf

# use the syslog group by default, since this is the owning group of /var/log.
su root syslog

Setup of journalctl

Journals are stored into /var/log/journal/ (or into memory /run/log/journal/)

  • Edit the file /etc/systemd/journald.conf to define the max size for systemd journals

...
SystemMaxUse=1G
# Define max size of each file (there is 1 file per user). Default is 1/8 of SystemMaxUse.
SystemMaxFileSize=5M
...

Take into account the change with

systemctl stop systemd-journald
systemctl start systemd-journald

To force clear of journal:

journalctl --flush --rotate
journalctl --vacuum-size=1G
journalctl --vacuum-time=1d

To read journal:

journalctl --disk-usage
journalctl --header

Désactivation ou activation de apport (optionnel, "on" recommandé)

Pour activer:

sudo systemctl enable apport.service
sudo systemctl start apport.service
sudo systemctl status apport.service

Pour désactiver:

sudo systemctl disable apport.service
sudo systemctl stop apport.service
sudo systemctl status apport.service

Note: Reports are into /var/crash

Installation of cron tasks

Delete very old files every sunday by adding the cron task:

#0 18 * * 0 find /mnt/diskbackup/*/backupold_* -type f -mtime +50 -ls -delete > /var/log/find_delete_old_files.log 2>&1
#0 18 * * 1 find /mnt/diskbackup/*/backupold_* -type d -empty -ls -delete > /var/log/find_delete_empty_dir.log 2>&1
55 23 * * * btrfs subvolume delete /mnt/diskbackup/.snapshots/diskbackup-`date +\%j` >/var/log/sellyoursaas_btrfs.log; btrfs subvolume snapshot /mnt/diskbackup /mnt/diskbackup/.snapshots/diskbackup-`date +\%j` >>/var/log/sellyoursaas_btrfs.log; rm -fr /mnt/diskbackup/.snapshots/diskbackup-`date +\%j`/.snapshots >>/var/log/sellyoursaas_btrfs.log; btrfs property set -ts /mnt/diskbackup/.snapshots/diskbackup-`date +\%j` ro true >>/var/log/sellyoursaas_btrfs.log;
0 0 * * * find /mnt/diskbackup/.snapshots -maxdepth 1 -type d -mtime +40 -exec btrfs subvolume delete {} \; >>/var/log/sellyoursaas_btrfs.log;

Installation d’outils externes

Installation of DataDog (optionnel pour supervision)

  • Create an account on DataDog.

  • Install the agent on serveur with:

DD_AGENT_MAJOR_VERSION=7 DD_API_KEY=YOURDATADOGAPIKEY bash -c "$(curl -L https://raw.githubusercontent.com/DataDog/datadog-agent/master/cmd/agent/install_script.sh)"

Relancer datadog

sudo service datadog-agent stop
sudo service datadog-agent start

Exploitation - Supervision

Backup / Restauration

Backup system

La sauvegarde du serveur+bases peut se faire par un snapshot d’image de la VM. Il est aussi possible de ne faire un snapshot que des disques complémentaires.

Restauration system

Depuis l’espace "Snapshots" d’OVH, on peut demander à le restaurer sur un serveur (pour une image VM) ou sur un aute disque (pour une image disque complémentaire), à condition que la cible (serveur ou disque) soit supérieure ou égale en terme de capacité de stockage.

Increase size of disk

  • Faire le snapshot du disque à redimensionner pour sauvegarde. Créer un nouveau disque depuis ce snapshot et le rattacher à un autre serveur (voir chapitre [ajout_de_disque]) pour s’assurer qu’il est lisible et ainsi avoir les fichiers de la sauvegarde sous la main.

  • Unmount the filesystem:

umount /mnt/disk/

Rem: Pour voir les fichiers ouverts sur un disque si le démontage échoue:

lsof | grep "/mnt/disk"

  • Détacher le disque du serveur. S’assurer que son nom ne contient pas d’espaces ou caractères spéciaux. Changer la taille du disque depuis le manager du Public Cloud et le réattacher au serveur.

  • Agrandir la partition en lançant:

fdisk -l
parted /dev/vdX    (X=a, b, !!! SANS le chiffre, on veut le disque complet)
print all
resizepart
Y
999GB    (Ne pas saisir la valeur proposé par défaut mais la valeur max du disque qui a été affiché par le "print all")
q

  • Remonter le disque pour prise en compte et augmenter le formatage du filesystem sans effacement.

mount /mnt/disk/
resize2fs /dev/vdX9

Upgrade OS

Pour mettre à jour Ubuntu 16.04 vers 18.04 sur un serveur SellYourSaas:

apt dist-upgrade

Passage en mode rescue d’un serveur

Aller sur l’interface du service Cloud pour passer en mode rescue. Le serveur sera rebooté et un lien pour se logué sera fourni.

Trouver les disques attachées et montez le disque système.

lsblk
mount /dev/sdXY /mnt

Il est alors possible d’agir sur le disque en écriture accessible dans /mnt

Migrate a ext4 partition into btrfs

Run command to see partition types:

df -Th
lsblk -f

If the data disk is ext4, you can convert it into BTRFS with:

btrfs-convert -p /dev/vdX
lsblk -f

Then edit he /etc/fstab to the type ext4 to btrfs (change the UUID if it was changed)

Then you create a subvolume for each directory you want to be able to make snapshots.

BTRFS admin

To list existing subvolumes:

btrfs subvolume list /mnt/diskbackup

To create a subvolume for a server

btrfs subvolume create /mnt/diskbackup/xxx   # with xxx = home_serverX or backup_serverX
chown admin.admin /mnt/diskbackup/xxx
chmod -R o-rw /mnt/diskbackup/xxx
cp -ax --reflink=always src/. dest

To convert an existing directory into a subvolume or convert a subvolume into a common directory

export dirtoconvert=xxx		# with xxx = home_serverX or backup_serverX

# To convert directory into subvolume
mv /mnt/diskbackup/$dirtoconvert /mnt/diskbackup/$dirtoconvert-old; btrfs subvolume create /mnt/diskbackup/$dirtoconvert; cp -ax --reflink=always /mnt/diskbackup/$dirtoconvert-old/. /mnt/diskbackup/$dirtoconvert; rm -fr /mnt/diskbackup/$dirtoconvert-old;

# To convert subvolume into directory
mv /mnt/diskbackup/$dirtoconvert /mnt/diskbackup/$dirtoconvert-old; cp -ax --reflink=always /mnt/diskbackup/$dirtoconvert-old/. /mnt/diskbackup/$dirtoconvert; btrfs subvolume delete /mnt/diskbackup/$dirtoconvert-old;

 — Example of CLI to convert directoris not btrfs into btrfs subvolume — btrfs subvolume show /mnt/diskbackup/not 2>&1 | grep -c "Not a Btrfs subvolume" && mv /mnt/diskbackup/not /mnt/diskbackup/not-old && btrfs subvolume create /mnt/diskbackup/not && cp -ax --reflink=always /mnt/diskbackup/not-old/. /mnt/diskbackup/not && rm -fr /mnt/diskbackup/not-old

To create a readonly snapshot

btrfs subvolume snapshot /mnt/diskbackup /mnt/diskbackup/.snapshots/diskbackup-`date +%j`;
rm -fr /mnt/diskbackup/.snapshots/diskbackup-`date +%j`/.snapshots;
btrfs property set -ts /mnt/diskbackup/.snapshots/diskbackup-`date +%j` ro true;

To delete a readonly snapshot of a day. Note: space is not freed immediatly.

btrfs subvolume delete /mnt/diskbackup/.snapshots/diskbackup-`date +%j`;

To delete all old snapshot

find /mnt/diskbackup/.snapshots -maxdepth 1 -type d -mtime +60 -exec btrfs subvolume delete {} \; >>/var/log/sellyoursaas_btrfs.log;

TroubleShooting

See the chapter available into the Documentation SellYourSaas - Master and Deployment servers