Main database is being filled constantly with data while the replicated database needs to be accessible by several services in read-only mode on different segment of the network. Because of that and need of local backup direct connection of those services to main database wasn’t an option. So, plan for postgresql replication was needed. Also, as these two servers aren’t connected by secure network or any kind of VPN somehow they needed to have communication channel between them for transfering of the replication data solved – in a secure and least obtrusive way possible. Data status on the slave side doesn’t affect main server and there is possiblity of temporary out-of-syncs (in case of some network failure) which is good enough for the requirements of the project so replication can be done asynchonously.

Unfortunately, postgresql (still) doesn’t have native replication mechanisms so some help from postgresql replication projects was needed. Although pgFoundry has quite a few replication projects listed only few were a real option (working with current versions of postgresql, no need to patch production databases, project in relatively stable state, simple master-slave asycnhronous replication). In the end SkyTools, an open sourced project by the skype developers (seems like they are using this also for replication of their postgresql cluster) was chosen for the job.

As for obtaining secure communication between the servers part – good ol’ ssh is used, one port needed, one-way connection (from slave replication server to master) to keep things secure as much as possible on the replicated server.

So, with tools chosen here is how it’s all put together, we’ll use some abbreviations to keep things simple:

• MasterIP  – IP of the master postgresql server
• MPort – port of the postgresql server on master server
• MDbase – master database name
• SPort – port of the postgresql server on slave
• MSPort – local tunneled port for postgresql master server on the slave
• SDbase – slave database name

Connection part

For the use, simple tunneling of the master postgresql port to slave with ssh -L MSPort:127.0.0.1:MPort MasterIP fulfills our needs. Although, as things go, once connection is broken it’s bye-bye replication so we need to make sure this connection stays up and keeps the tunnel open for the data.

So, meet autossh, a ssh connection monitoring program. After install (either thru apt/yum or manual compile) you just need to start it, this thing doesn’t have much features but it does what it’s intended for remarkably well. It doesn’t just monitor the ssh process, but also checks for traffic on the connection and checks for the ssh connectivity (so it doesn’t try to repeatedly login to server in case of wrong credentials or network failure). Traffic checking on the connection is made with few additional ports, their range is set by the -M parameter.

So to start the tunnel and keep it open (use of ssh authentication keys is recommended here):

autossh -M 20000 -f ssh -L MSPort:127.0.0.1:MPort MasterIP

Preparation of the postgresql server(s)

As already stated, very few modifications need to be done on the database servers itself, mainly it’s adding a few python functions and tables needed for skytools…

Install the SkyTools from pgFoundry page (simple ./configure && make && make install usually does the trick, although some dependencies might be needed; like postgresql-devel and python-psycopg2 packages). Additional note for the RedHat users: skytools modules by default get installed in “/usr/local/lib/python..”, while the system expects them at “/usr/lib/python…” – symlink or addition to path will solve this without much problems.

Slave server should also contain the same schema for the database that needs to be replicated as it’s existent on master server.

SkyTools configuration

SkyTools need to be installed on both master and slave (theoreticaly it could work from slave only but it would probably be too slow for this type of connection between the servers), and config files shall be created by hand; to keep things neat and clean we make /etc/skytools directory on both servers and make config files:

On master server

/etc/skytools/ticker.ini

[pgqadm]
job_name = pgqadm_ticker
db = dbname=MDbase port=MPort host=127.0.0.1
# how often to run maintenance [seconds]
maint_delay = 600
# how often to check for activity [seconds]
loop_delay = 0.1
logfile = /tmp/%(job_name)s.log
pidfile = /tmp/%(job_name)s.pid

On slave server

/etc/skytools/conf.ini

[londiste]
job_name = SPDLABreplicate_dbase
provider_db = dbname=MDbase port=MSPort host=127.0.0.1
subscriber_db = dbname=SDbase port=SPort host=127.0.0.1
# it will be used as sql ident so no dots/spaces
pgq_queue_name = SPDLABreplicator
logfile = /tmp/%(job_name)s.log
pidfile = /tmp/%(job_name)s.pid

Database preparation and replication start

After this is done, we can start creating the neccessary modifications to the databases and start the replication (commands should be executed by the postgresql superuser, each command states at which server):

ON MASTER - pgqadm.py /etc/skytools/ticker.ini install (installing the ticker sql on master database)

ON SLAVE – londiste.py /etc/skytools/conf.ini provider install (installing the londiste provider sql)

ON SLAVE – londiste.py /etc/skytools/conf.ini subscriber install (installing the londiste subscriber sql)

ON MASTER – pgqadm.py /etc/skytools/ticker.ini register SPDLABreplicator SPDLABreplicate_dbase (registering the consumer for ticker)

ON MASTER – pgqadm.py -d /etc/skytools/ticker.ini ticker (starting the ticker daemon; can be stopped with -s switch)

ON SLAVE – londiste.py /etc/skytools/conf.ini provider add –all (adding all tables on provider for replication)

ON SLAVE – londiste.py /etc/skytools/conf.ini subscriber add –all (adding all tables on subscriber for replication)

ON SLAVE – londiste.py -d /etc/skytools/conf.ini replay (starting the londiste daemon, can be stopped with -s switch)

Well, after this replication should have started and slave would be in sync and kept in that state. In case that’s not the so, check the log files defined in config files and use these commands for debugging:

ON MASTER – pgqadm.py /etc/skytools/ticker.ini status (checking the status of the ticker and updates)

ON SLAVE – londiste.py /etc/skytools/conf.ini subscriber tables (checking the state of the tables)

Additional notes

SkyTools seems to be quite undocumented regarding these procedures of installing and maintaing sync for replication and steps explained above seem to be working fine for me, while you might get problems somewhere along the way; so it’s good to keep an eye on the original documentation when some additional help is needed…

Additional notes 2

Sometimes things can go wrong and replication stops – in order to cope with that try to:

Restart the service ON SLAVE – londiste.py /etc/skytools/conf.ini -r

Still no progress?

Try to make a full new copy:

ON SLAVE – londiste.py /etc/skytools/conf.ini subscriber remove –all

ON SLAVE – londiste.py /etc/skytools/conf.ini subscriber add –all

ON SLAVE – disable constraints, triggers and such which could slow down the database

ON SLAVE – londiste.py /etc/skytools/conf.ini copy (if not possible try with –reset option to start as completely new copy with truncate, replay option can also be used but gives less info on the process)

ON SLAVE - In the end re-add the constraints on tables

When it’s really stuck, you can even try with dropping the whole database on subscriber end and reinstall and re-add subscriber stuff, along with initial copy (in the original docs it suggests you make initial copy before adding the tables for subscriber).

What will be explained here is mostly about open source implementation of ssh – OpenSSH, so some things might be different if you’re using something else.

Let’s first see about SSH security – is it really so secure? While it’s hard to be exact on this issue one thing is certain – if you’re using SSH protocol use only it’s version 2. SSH protocol version 1 is known to have exploits for some time now and it is definitely not advisable to use it. Sure, it’s not as bad as connecting with telnet, but any reasonably well configured SSH server nowdays will offer you a SSHv2 connection and even might be limited to this option only.

Some specific issues with SSH – if you’re new to it some things might confuse you. For example – when logging in a system that you’ve never logged in before it will prompt you for a authenticity check and display you a key fingerprint.

The authenticity of host ‘somerandomserver.com (0.0.0.0)’ can’t be established.
DSA key fingerprint is 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00.
Are you sure you want to continue connecting (yes/no)?

Huh? What happens here – to prevent man-in-the-middle attacks you must be sure that you are always communicating with the right system or you might be sending passwords and other confidential information to anyone who impersonates this system. IP addresses can be spoofed, DNS entries are easily changed ARP poisoning can redirect your traffic and so on… Once you’ve confirmed that this is exactly the system that you want to login to its key will be memorized for later use and you wont see it again …unless somebody tries to fool you – then you’ll get a nice big warning about not matching the hosts keys. Needless to say, if you are not absolutely sure that system you are logging to hasn’t changed it’s keys stay away from further login!

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the DSA host key has just been changed.
The fingerprint for the DSA key sent by the remote host is
00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:01.
Please contact your system administrator.

So, what exactly is it that is possible to achieve with this tool, or should I say toolbox?

Remote logins

Basic use of ssh client is
ssh hostname

Which will attempt login on host “hostname” on standard SSH port 22. After successful login you can work normally as with console or telnet.
Additional interesting basic options:
-p : port to which you are connecting to (default 22)
-l : login with different username then default (default is current users username; this can be also accomplished as with username@host syntax)
-q and -v : first is for no messages at all (quiet) and second is for verbose debugging messaging
-t and -T : disabling and enabling pseudo-tty generation at servers end (default enabled) – useful if you’re just making port forwarding
-C : request compression of the transmission data (useful for slow connections)
-o : define a option value so values from configuration file can be overridden
-1 or -2 : try only version 1 or 2 of the protocol

SFTP/SCP

Secure alternatives for commonly used (unsecure) protocols – FTP and rcp (remote copy)
scp and sftp accept most of the normal ssh client’s options (attention: port is here defined with -P unlike -p with ssh) but are used only for file transfers. scp for single file transfer and sftp for multiple file transfers.
scp use:
scp ~/somefile hostname:/tmp/somefile

This will copy “somefile” from your home directory to “/tmp/somefile” on remote ssh server. If you ommit path on remote server files will be copied to your home directory on server.
Retrieval of files from remote server isn’t much harder either:
scp hostname:/tmp/somefile ~/somefile_again

Passwordless logins

More appropriate for automated tasks but might help you if you tend to forget passwords. Basically have a private/public key with which you are authenticated on server. Of course, it’s more secure if you have password also setup on your private key – otherwise anyone who steels your private key can login into your servers. Yikes.
By default when connecting your private keys will be searched within ~/.ssh/ folder and are called id_dsa || id_rsa (depending on encryption used), but private key file location can easily be defined with -i option so you can use different keys for different servers:
ssh -i ~/secretfiles/mykey_at_hostname hostname

Of course the same applies for scp and sftp…

Remote command execution

Who says you even need a terminal?
ssh hostname df

If system command is added as parameter of the ssh connection command (in this example it’s df) after successful login it will be executed and logout happens as soon as it’s execution is finished. Useful for checking on some services; or in this case – file system capacity status.

Port forwarding (poor man’s VPN)

Forwarding a port thru remote system – weather you need it to secure traffic over not so secure link or connect to the service which is behind the firewall – this one is all time favorite ssh trick
So, presumably you need to check on some webpage which is available only inside LAN of your network from internet and you have only ssh access to server there. Imposible? Not with ssh:
ssh -L 8080:webserver:80 hostname
where 8080 is port on your client ssh computer, webserver is LAN address of the webserver in LAN and 80 is webservers http port

Now type in http://localhost:8080 in your browser and work like you’re there… As you can see -L is a very powerful option and it does forwards one port from some host (ports for localhost and remote hosts port can be set up as needed) to your local computer.
Additional use of this is anonymous surfing (or some other form of activity) – ports can be forwarded from whichever hosts your ssh server has connectivity to and port forwarding has a very interesting feature: all traffic made to that remote host seems like it originates only from ssh server and without any trace of your local computers address.

Reverse (remote) port forwarding

Forwarding a port on a local system to be available on remote system
Similar to port forwarding, but of reverse nature like it’s name says – expose your local network hosts to anybody who has access to remote servers forwarded port. Use with caution – you never know who’s listening on the other end…
ssh -R 8080:webserver:80 hostname
where 8080 is port on remote ssh server, webserver is LAN address of the webserver in LAN and 80 is webservers http port

Anyone who can access remote ssh servers 8080 port can now browse webserver in your LAN (do you really want this?). Also, with this your computer works as anonymous proxy for chosen webserver (or any other protocol depending on ports used).

SOCKS Proxy

Believe it or not, SSH can serve as SOCKS proxy
ssh -D 12345 hostname
where 12345 is port on your local computer

Huh? Is it this easy? Yep – dynamic port forwarding as it is usually called does just this – makes a SOCKS v4/5 out of your computer and all you need to do is point your SOCKS-aware applications to the chosen port…

Punching holes in HTTPS proxy

OK, now this isn’t exactly a ssh-only feature (proxytunnel does most of the work here) but might be the only way to make your way thru the corporate restrictive web policies

Port knocking as technique is used for a long time now and many implementations are already out there although for my specific use none of them seemed to be appropriate.
For complete listing of these implementations please visit portknocking.org website. There you can also find a complete explanation of the methods as well as recommendations of use.

Requirements (or why no other port knocking solution worked for me):
Knocking and service must share the same port (for example – let’s suppose you have a computer behind some router or firewall and have no access to this device; all you do have is one forwarded port to your computer – let’s say for ssh as most commonly used for system administration). I guess you can see now where the problem is with other implementations of port knocking: lack of ports that can be used for knocking.
Also, as I prefer a portable solution that is deployable anywhere (without any additional software clients) I want to make it possible so that it can be used with simple tools available on any system.

I’m trying to make this as simple as possible so all I’ll use is iptables. Why? Imagine a following scenario: something gets broken on a port knocking script/program/server and it’s not listening to knocks anymore. There comes a need for a visit to the computer to fix this directly as our ssh connection don’t let us do it remotely anymore. If iptables gets broken there is no firewall anyway so you’ll have a completely different set of problems, at least connecting to fix it wouldn’t be one of them.

One thing that is needed beside plain iptables is their recent module – either compiled in or as separate module. Few words about recent module parameters used here:
–name [name] defines a name of the list
–set puts the source IP matched in a list defined by –name (if name is not defined all matches go in some default list)
–rcheck does a rulecheck on a IPs from list
–seconds [seconds] is used to filter out the list for IPs seen within the indicated number of seconds
–hitcount [hits] filters out the list for minimum number of packets from some IP, can be combined with seconds to make the filtering more accurate

OK, so we’ll start with a standard firewall passing new ssh connections (I’ll use the iptables-save format as it’s easier to tweak the firewall and set comments though you can make this also with usual iptables commands) – port 22 is passed thru without any additional checks:

# pass established connections
-A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT
# pass new ssh connection
-A INPUT -m state –state NEW -m tcp -p tcp –dport 22 -j ACCEPT

For starters we’ll need to start listening on this port:

# pass established connections
-A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT
# listen to new connections on ssh port
-A INPUT -m state –state NEW -m tcp -p tcp –dport 22 -m recent –set –name sshknock -j LOG “knock knock: ”
# pass new ssh connection
-A INPUT -m state –state NEW -m tcp -p tcp –dport 22 -j ACCEPT

Now, after initiating a ssh connection we have a entry in /proc/net/ip_recent/sshknock list showing us the address and main parameters of the packet. As connection was initiated one rule later and all later packets were are picked up by ESTABLISHED state rule that’s all we’re going to get for now.

So let’s start filling in this list with new packets by dropping them instead of accepting and also enabling the port knocking mechanism:

# pass established connections
-A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT
# if at least 2 new packets inside 5 seconds open ssh port
-A INPUT -m state –state NEW -m tcp -p tcp –dport 22 -m –rcheck –name sshknock –hitcount 2 –seconds 5 -j ACCEPT
# listen to new connections on ssh port
-A INPUT -m state –state NEW -m tcp -p tcp –dport 22 -m recent –set –name sshknock -j LOG “knock knock: ”
# pass new ssh connection
-A INPUT -m state –state NEW -m tcp -p tcp –dport 22 -j DROP

We might stop at this as we have achieved our goal – 2 packet knock in a short amount of time is needed to open the port which will go under a radar for most port sniffers marking it as filtered port (of course, tweaking with more knocks and probably larger timespan is recommended to be more sure).
But, as we have a very well known port here which is targeted quite often let’s add an additional twist to this. By simply bombing the ssh port it will open without any defense at all.

# pass established connections
-A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT
# if more then 2 new packets inside 3 seconds arrive drop
-A INPUT -m state –state NEW -m tcp -p tcp –dport 22 -m –rcheck –name sshknock –hitcount 2 –seconds 3 -j ACCEPT
# if at least 2 new packets inside 5 seconds open ssh port
-A INPUT -m state –state NEW -m tcp -p tcp –dport 22 -m –rcheck –name sshknock –hitcount 2 –seconds 5 -j ACCEPT
# listen to new connections on ssh port
-A INPUT -m state –state NEW -m tcp -p tcp –dport 22 -m recent –set –name sshknock -j LOG “knock knock: ”
# pass new ssh connection
-A INPUT -m state –state NEW -m tcp -p tcp –dport 22 -j DROP

Another line to firewall is added – but what does it actually do?
If someone attempts to flood the port with packets it will refuse to open; and instead of only having number and time of knocks defined also rate at which these knocks must be employed is set – sending them at too high or too low frequency has no effect on opening the port – exactly 2 knocks is needed for 3rd packet to get to ssh server inside 3 – 5 seconds of the first knock.

So there it is, port knocking with only a few additional rules to iptables. Knocking packets can be generated with a simple telnet or ssh client. It most certainly isn’t the complete solution for the security and maybe isn’t as secure as some other port knocking solutions (additional encryption of the knocks, more complicated schemes of knocking) but it’s easily deployable and rises the security of the system significantly.