SynapseDirect

As you should know, all your scans are saved on the server in the cache-listener directory.  Some of you have more than 10,000 files there which need to be backedup.

Amazon offers secure storage on their servers, and this is called S3 - standing for Simple storage service.

I can create a client service that will upload all the files in the cache-listener directory to our S3 repository.  It will be client based so that the server is not blocked by uploading all the files, or, it could be server based with a special helper application to do the uploads.

If it's a client, you can just run it in the background ....

However, you will still need to backup your remr.fdb, and since this file can potentially be some Gbs in size, it's not realistic to back this up this way.

---

Graham Chiu
Beta Downloads and Documentation Wiki

I've had some more thought about this.

I can spawn a server process that does the backup to S3, and you can get a report back in the client instead.

Once the files are on S3, there is no longer a need for them to be stored locally - because they can be served from S3 storage instead.  It might even be a little faster if you're using Synapse as an ASP.

One scenario is that I often see patients out of town.  I take a copy of the remr.fdb with me in case I can't access the main server ... but there are just too many files in the cache-listener for me to take as well.  So, in that case I rely on the metadata.  But if I had S3 storage, I could choose to download from there instead of my localhost server, or, my remote server.

---

Graham Chiu
Beta Downloads and Documentation Wiki

I see that the cost of S3 is relatively minimal, the most expensive being GB of data requested (17 cents per month per GB for first 10 TB).  Is it possible to store remr.fdb itself on S3, and have the odbc point to it?  Then the cache-listener files would be generated on S3 and stay there maybe?  Would that arrangement be too slow in practice?

---

The idea would be to host the whole shebang on EC2 instead but that would cost $70/month for computing time.

As for just the database on S3, not possible as Firebird needs a real hard drive to access the database.  That's why it can't even be on a mapped drive.

---

Graham Chiu
Beta Downloads and Documentation Wiki

 

Current working configuration screen for backup.

---

Graham Chiu
Beta Downloads and Documentation Wiki

My client is now able to tell the server to start a gbak backup.  The log from that is now stored in Synapse and is viewable by the client in a report screen.

I'm going to split the backup file ( remr.fbk ) into 2Mb splits like RAR, and upload each one ... 

eg remr.001 ... remr.100

so that I can easily store this on S3.

The question is how often do you want this to happen?  We could be talking about transferring 100Mb each day if you're doing a backup each day.

---

Graham Chiu
Beta Downloads and Documentation Wiki

I think even 100 megs a day would be fine.  My own remr.fdb file is pretty small, as I'm still building and things.  Even if I had a several GB remr.fdb, it's only 15 cents per gig per month for storage, and 10 cents per gig transfer so we're only talking say $3-4 / month, and probably less.  That's pretty cheap for reasonable peace of mind.  I'm going to assume transfers up would (hopefully) be rare.

---

Over 2 years of data. Hourly backups.

Not one REMR.FDB problem.  Firebird and Synapse work great together.

18 months of that I ran the Server off a Client PC with Windows XP. (not that I suggest that).

 

---

I buy insurance .. not because I expect my house to burn down!

Having automated offsite backup is pretty reassuring for me.

I didn't have time to do the S3 stuff this last release.  I need to add some encryption and compression as well prior to uploading to S3.

---

Graham Chiu
Beta Downloads and Documentation Wiki

I think the S3 backup thing is a great idea.

 

 

---

Preliminary testing .. S3 backup is now working for me.

 

---

Graham Chiu
Beta Downloads and Documentation Wiki

First glitch ... if your server is out by 2 mins from real time, Amazon will reject the request .. so need to build in some time verification routines.

But encrypted backup is now working .... once I fixed the time on my server by 1 min!

 

---

Graham Chiu
Beta Downloads and Documentation Wiki

I've got just over 4000 scans to backup.  In the past 3 hours, I have backed up 500 files.  So, I estimate I will finish in 24 hours from start to finish.

Why so slow?  Because the agent pauses between uploads, and does them in batches of 10, to stop grabbing all the internet bandwidth and CPU time.

 

---

Graham Chiu
Beta Downloads and Documentation Wiki

1000 up, and 3000 to go ... taking a little longer than i had estimated.

 

---

Graham Chiu
Beta Downloads and Documentation Wiki

Backup completed a few hours ago .. so about 36 hours for 4100 files.  Should be quicker for those in the USA or Europe ... we are as far from Amazon's servers as you can be without being at the pole!

The backup was also throttled deliberately so that it would not max out the CPU or internet connection.

From now on, only the new files will be backed .. so only a few mins each hour at the most.

---

Graham Chiu
Beta Downloads and Documentation Wiki