Bart Simons

Bart Simons


Thoughts, stories and ideas.

Bart Simons
Author

Share


Tags


.net .net core Apache C# CentOS LAMP NET Framework Pretty URLs Windows Server WireGuard WireGuard.io access log add analysis android api at the same time authentication authorization automate automation azure azurerm backup bash basics batch bootstrap build capture cheat sheet chromium chroot class cli click to close code snippet command line commands compile compiling compression containers control controller controlling convert cpu usage create credentials csv csvparser curl data dd deployment desktop detect devices disable diskpart dism distributed diy docker dom changes dotnet core drivers ease of access encryption example export file transfer files fix folders generalize getting started ghost ghost.org gui guide gunicorn gzip html html tables icewarp igd imagex import inotify install installation interactive ios iphone itunes java javascript jquery json kiosk kotlin linux live load data loading screen lock screen loopback audio lxc lxd lxml macos manage manually message messages minio mirrored mod_rewrite monitor monitoring mutationobserver mysql nexmo nginx no oobe node node.js nodejs not installing notification notifications object storage on desktop one command openssl owncloud parallels parallels tools parse perfect philips hue play port forwarding portainer.io powershell processing ps-spotify python quick raspberry pi record rip ripping rsync rtmp save save data sbapplication scraping script scripting scriptingbridge scripts security send server service sharedpreferences sms songs sonos spotify spotify api spotlight ssh stack streaming streamlink studio sudo swarm swift sync sysprep system audio systemd tables terminal tracking tutorial twilio ubiquiti ubuntu ubuntu 18.04 ui code unifi unlock unsplash source upnp uptime usb tethering wallpapers wasapi website websites webview windows windows 10 without itunes without oobe workaround xaml

Sync folders and files on Linux with rsync and inotify

So: you've got two or more clients and/or servers. They contain files that you want to have automatically synced when possible, because that would save a lot of time. Well, I got the solution for you: with a little bit of thinking in an innovative way I have found the solution that might bring you onto the right path as well. Rsync is a great solution, but having to run rsync manually would take a lot of unnecessary time away, right? And that is where inotify is for: real-time monitoring of your filesystem so that your files can be synced between multiple machines with the power of rsync!

Self-made sync daemon in a working state

Setting up a test scenario

I needed to get myself a nice development environment at first so I started off 3 virtual servers which all run Ubuntu 16.04, my personal favourite. All these 3 machines needed to be setup with the following software packages:

Also noteworthy is that these machines are absolutely not connected through a private network. All rsync traffic is supposed to be worked out over SFTP.

For who is this for?

I could see some potential for workflow improvement on these situations:

Working on it

First things first: we need to get all the dependencies installed on the 3 servers with this one-line command:

apt update && apt -y install openssh-server rsync inotify-tools

After that, let's create a specific folder that we want to sync. Let's call it SyncFiles:

mkdir /opt/syncfiles

And for secure file transfer, we want a public-private key link for the transfer link that rsync uses, this is how to configure it:

ssh-keygen -t rsa -f ~/rsync-key -N ''

# Paste the output in your destination servers' ~/.ssh/authorized_keys file:
cat ~/rsync-key.pub

# Removing public key for security purposes..
rm ~/rsync-key.pub

# Remember to execute this script on all servers separately! Then, copy the output of the script in all of your servers' authorized_keys files.

Now that you have got all the pre-configuration work done, it's about time to write a script that goes through an infinite loop with inotifywait in it:

#!/bin/bash

# Supposed to run on rsync-host01, change rsync-host02 to rsync-host01 to make a script that is meant to run on rsync-host02.

while true; do
  inotifywait -r -e modify,attrib,close_write,move,create,delete /opt/syncfiles
  rsync -avz -e "ssh -i /root/rsync-key -o StrictHostKeyChecking=no“  /opt/syncfiles/ root@rsync-host02:/opt/syncfiles/
done

I saved this script in the /opt directory as file-sync.sh.

To finish things off, lets create the systemd service file that can stop, start, and reset the script on demand or on specific events like a system bootup.

Create a file called sync.service in the directory /etc/systemd/system/ and put the following contents in it:

[Unit]
Description = SyncService
After = network.target

[Service]
PIDFile = /run/syncservice/syncservice.pid
User = root
Group = root
WorkingDirectory = /opt
ExecStartPre = /bin/mkdir /run/syncservice
ExecStartPre = /bin/chown -R root:root /run/syncservice
ExecStart = /bin/bash /opt/file-sync.sh
ExecReload = /bin/kill -s HUP $MAINPID
ExecStop = /bin/kill -s TERM $MAINPID
ExecStopPost = /bin/rm -rf /run/syncservice
PrivateTmp = true

[Install]
WantedBy = multi-user.target

Chmod this service file and reload the systemd daemon:

chmod 755 /etc/systemd/system/sync.service  
systemctl daemon-reload

You are all set! You can now use these commands to manage your self-made directory sync daemon:

# Start your service
systemctl start sync.service

# Obtain your services' status
systemctl status sync.service

# Stop your service
systemctl stop sync.service

# Restart your service
systemctl restart sync.service 
Bart Simons
Author

Bart Simons

View Comments