OSE Staging Server
As of 2019, Open Source Ecology has exactly one staging server, and this article is specifically about this staging server.
For information about OSE's production server, see OSE Server.
For information about OSE's development server, see OSE Development Server.
Contents
Purpose
The OSE Staging Server (osestaging1) is an lxc container that lives on the OSE Development Server (osedev1). It is a severely under provisioned clone of our production server that is intentionally inaccessible from the Internet.
Access
To access the staging server, one must connect to the developer VPN.
If you are an OSE developer that requires access to the staging server, please see VPN#Developers:_How_to_request_access_to_the_dev_VPN for instructions on how to request VPN access from the OSE System Administrator.
Tips
This section will provide tips for interacting with the servers and clients that are connected to the VPN.
Accessing Staging Websites
After connecting to the developer VPN, your queries for OSE sites should now point to the staging server. This is done by using osedev1 as your DNS resolver. osedev1 returns the staging server IP address (mapped from the osedev1:/etc/hosts file) using dnsmasq.
To achieve this, the following setting is added to the server.conf openvpn config file on osedev1:
push "dhcp-option DNS 10.241.189.1"
And the following is added to the client.conf openvpn config file on a Developer's workstation/laptop:
# dns for staging script-security 2 up /etc/openvpn/update-resolv-conf down /etc/openvpn/update-resolv-conf
This can be validated by checking the contents of /etc/resolv.conf. If it does *not* say "nameserver 10.241.189.1", you can manually set it using the following command:
echo "nameserver 10.241.189.1" | sudo tee /etc/resolv.conf
If you find that you have to manually make the above change, please be very careful if you plan to make changes to the staging site in your web browser. Because if your DNS resets itself back again, you may find that you're making changes to the production site, not the staging site.
If you only need access to the staging server via a web browser, a potentially safer alternative to making potentially destructive and short-lived change to the system's DNS via /etc/resolv.conf is to just install firejail to force firejail to use osedev1 as the DNS server.
sudo apt-get install firejail firejail --dns=10.241.189.1 firefox
If the above commands result in a firefox window without any Internet access, you may be having an issue with the default firejail profile for firefox. You can fix this by clearing the profiles or using the '--noprofile' argument to `firejail`. These below commands should work; note that they also create an epeheraml firefox profile, which is safer in-case you have to, for example, create any tls certificate exceptions for the staging sites.
sudo firecfg --clean
tmpDir=`mktemp -d`
tmpProfileDir="${tmpDir}/firefoxProfile"
mkdir -p "${tmpProfileDir}"
firejail --noprofile --whitelist="${tmpProfileDir}" --dns=10.241.189.1 firefox -no-remote -new-instance -profile "${tmpProfileDir}" https://www.opensourceecology.org
Identifying Staging vs Production
The /root/bin/syncToStaging.sh script that lives on the Production server (and is responsible for syncing production state to the staging server) modifies the docroots of the apache vhosts by dropping a single file is_staging into the root of the docroot for all the vhosts in /var/www/html/ with the contents true.
Therefore, if you want to validate if the website you're viewing in your browser (ie: www.opensourceecology.org) is coming from the production or staging server, you can query '/is_staging' (ie: https://www.opensourceecology.org/is_staging). Production will return a 404, and staging will return a 200.
For example, this prints the first and last line of a curl to the '/is_staging' file on the www.opensourceecology.org website. It confirms that we're accessing the staging server:
user@ose:~$ curl -si https://www.opensourceecology.org/is_staging | awk 'NR==1; END{print}'
HTTP/1.1 200 OK
true
user@ose:~$
And here's the same command & different output on a different machine showing that we're not connected to the staging server (it's production):
user@disp9470:~$ curl -si https://www.opensourceecology.org/is_staging | awk 'NR==1; END{print}'
HTTP/1.1 404 Not found
</html>
user@disp9470:~$
Clearing DNS in firefox
After connecting to the VPN, your browser may still be caching the IP Address for the production server, so new requests for sites on the staging server (ie: www.opensourceecology.org) may still be communicating with production even after successfully connecting to the VPN and (automatically) updating your resolv.conf to resolve DNS records to the dev server.
You can clear your DNS cache in firefox by setting both network.dnsCacheExpirationGracePeriod and network.dnsCacheExpiration to 0 (default is 60) in about:config
Useful Commands
Sync Prod to Staging
Run this command on the OSE production server as root to trigger a sync of production to staging.
Note that this is a destructive operation that will clobber/delete most of the contents on the staging server to make it a nearly complete replica of the production server. Also note that this is a sloppy sync that may result in corrupt data being copied to staging since processes are not shutdown on production when the sync is initiated.
..But it *usually* works fine.
time nice /root/bin/syncToStaging.sh &> /var/log/syncToStaging.log
You can also sync specific directories by calling `rsync` directly. For example, the following will sync just the /var/www/ and mysql data
# sync web server files time nice rsync -e "ssh -p 32415 -i /root/.ssh/id_rsa.201910" --bwlimit=3000 --numeric-ids --delete --rsync-path="sudo rsync" -av --progress /var/www/ stagingsync@10.241.189.11:/var/www/ # sync mysql data dir time nice rsync -e "ssh -p 32415 -i /root/.ssh/id_rsa.201910" --bwlimit=3000 --numeric-ids --delete --rsync-path="sudo rsync" -av --progress /var/lib/mysql/ stagingsync@10.241.189.11:/var/lib/mysql/
Troubleshooting
Here are some useful commands to run if you have issues syncing prod to staging. For Example:
ssh: connect to host 10.241.189.11 port 32415: Connection timed out
# confirm that the server is connected to the vpn (this output shows it is not) [root@opensourceecology ~]# ip a show tun0 Device "tun0" does not exist. [root@opensourceecology ~]# # attempt to connect to the vpn [root@opensourceecology ~]# cd /etc/openvpn/client/ [root@opensourceecology client]# ls auth.txt ca.crt client.conf connect.sh hetzner2.crt hetzner2.key ta.key [root@opensourceecology client]# ./connect.sh ... Initialization Sequence Completed
