How to synchronize application development using Unison?

Skilled coder's tips on why and how use to use Unison
07 July

Every software developer at any time may need to work with the same files despite of where they originally were. For example, you have to move your precious backup the farther the better. Another case, you are developing a Rails application while running it on a remote machine: the remote runtime has to have all the file changes being applied at the correct time. Mybe you even want to create some dropbox-alike for you and your friends totally aside the popular cloud services which you for some reason don't trust anymore. So, you need a tool to sync your files with some remote box.

There are quite a multiple solutions for file synchronization; each of different complexity. Here I will cover the one both flexible and, say, allowing doing the simple things with the simple commands. It's Unison. It's written in OCaml(hello, practical FP-ers). It's compact (portable native binary with moderate memory footprint), fast (compresses files on-the-fly if needed while sending them over a slow link) and reliable (files metadata are stored on both ends; the transmission restarts if needed). Last but not least, Unison is well-documented: there's an online doc. Also, the comprehensive help is embedded within the binary (available via -help key). Stacking that up, you can have alldocumentation on any of multiple options Unison has. Btw, Unison is made for bidirectional file synchronization; all of its nuances are fine-tuneable. Also, you can sync two locations on the same machine: very useful if we need to sync a backup to a removable media.

Now let's install and run something useful. Our example is quite simple yet common: "here" and "there" we sync two source trees while editing the files on both sides of the link. "Here" is my Mac laptop; "There" is some remote Ubuntu instance (I use the one from my homelab).

Installation

Let's see how to install it on the different OS.

On the Mac

  1. Install Unison via Homebrew.

  2. Using pip and curl, install the helper for filesystem events monitoring.

For more information look at the docker-sync documentationdocker-sync is a tool used to sync local source tree with the remote Rails-dev docker container.

Let's see local synchronization:

So, Unison is run with unison [options] <here> <there>. "There" can be on either local or remote machine. We will experiment with the remote "there" in the next paragraph; now let's notice that being called without options Unison runs:

a) interactively;
b) in one run: it analyses the replicas once, syncs them and exits.

Different CLI options are there to adjust this basic behaviour if yow want something smarter, including the fully daemonized client-server solutions.

On the Ubuntu box

In order to use the working fs-monitoring python helper we need to install Unison from the source. It's not a problem because all we need is available as Opensource.

  1. Install OCaml

  2. Download Unison source; check out the correct version; compile'n'install the binaries:

Synchronization

Now, I get back to my Mac and we'll play with the actual files.

  1. Create the remote directory and do the first, interactive, run to establish the replicas metadata

    The output is eloquently desribing what's been done. Note the "ssh-url" where we specify the remote dir. Also, see that the local binary first starts the remote binary through SSH; then establishes ssh-tunnel between the too after which they negotiate and do the actual data transfer.

    Now, start the remote synchronization in "scan-syn-repeat" loop

    Unison processes together with their fs-monitors are started on both ends

    Now lets edit the file and see how the changes are propagated

    The Unison output shows how the magis is done

    Edit file "here", see the changes applied "there", and vice versa

    So it works: the changes are propagated in both directions. Our example was quite basic: one dir with one files. In my everyday development I sync the whole Rails project tree together with qute large number of upload files, etc. Also, I use Emacs which monitors the files itself and shows the changes on either end almost immediately.

Some caveats

  1. Unison isn't a networked file system so the locks aren't propagated and all the changes are being sent with some lag according to the network characteristics. You should take this into account and avoid the conflicts while editing a file on both ends simultaneously.
  2. Unison assumes the synced filesystems to be case-sensitive so beware and avoid syncing the files with case-only differing names if your filesystems are different in this sense. For instance, if you sync Mac fs vs. Linux ext4, etc.

Ocaml developer salary, August 2017

United States Ocaml developers labor market analysis, according to the results
29 August

We publish the analysis of the labor market of developers in the United States monthly. For Ocaml developers in August there were 58 vacancies. The vacancy rates were distributed as follows.

Ocaml Salary Estimate, August 2017
Ocaml Salary Estimate, August 2017

The most of the developers are required in Seatle, the least in San Francisco, CA, San Jose, CA, San Carlos, CA, Redwood City, CA, Menlo Park, CA, Illinois, Boston, MA, Eugene, OR.

Number of Ocaml vacancies in different cities, August 2017
Number of Ocaml vacancies in different cities, August 2017

Among the companies that hire Ocaml developers the leaders are: 

  • Oracle
  • IBM
  • Wade & Wendy 

Number of Ocaml vacancies in different companies, August 2017
Number of Ocaml vacancies in different companies, August 2017

According to the required experience, the vacancies are distributed as follows.  

Number of Ocaml vacancies by experience level, August 2017
Number of Ocaml vacancies by experience level, August 2017

The average salary and salary according to the experience level were distributed as follows.

Average Ocaml developer salary, August 2017
Average Ocaml developer salary, August 2017

The analysis was carried out by the Hype.codes portal method using the indeed.com data.