How to sync files lightning fast, and yet use so little memory?!


Well, those of you who are really serious about comparing different file sync tools and software, may have already notice that Bamboo File Sync and Backup's file processing engine is extremely fast and robust.  And most amazing of all, it has a very very small memory foot print.

Let me explain a little bit.  For anyone designing file processing tool, huge volume jobs are their nightmare.  Logically, the tool have to analyze the folders to be synchronized entirely and thoroughly, before they can draw the entire plan of action.  If the folders to be synchronized contains hundreds of thousands of files and sub-folders, the processing engine will eat up a lot of the computer's memory, and in extreme cases, eventually bring the whole system down.

There are two challenges inside this problem.  The first one is that when the program enumerates huge folders, its memory consumption gradually builds up until it takes up all the physical memory (RAM) from all other running programs, and even keep the whole system swapping virtual memory from your hard drive.  You definitely do not want your computer being slowed down to an un-usable state when you start running a file sync program.  In some extreme cases, there are tools in the market that simply cannot complete huge sync jobs forever because the memory usage has grown into a racing condition with virtual memory swapping.

The second challenge is to display the analyze results or history of the synchronization.  Usual user interface elements like list views or tree views would have a very hard time displaying hundreds of thousands of records.  Without special and highly technical design, they would either become extremely slow or eat up very huge amount of memory, or both.  So, there are tools in the market simple avoid the problem by do not (cannot?) display the entirely list of analysis result to the user for examination.

The following is an example of memory usage when Bamboo is syncing a 100GB+ folder with 400,000+ files and 20,000+ sub-folders with a low-end laptop.



From start to finish, Bamboo had never consume more the 12MB of physical memory!  And when you want to view the history of the sync job in details later, it just opens up instantly!  Yet still using no more than a few MB of RAM.  In fact, Bamboo's memory consumption is virtually independent from the size of the job you give it to run!



What exactly have we done to overcome these challenges?  Of course, I will not name it here. J

But as an end-user who just want to enjoy a well designed and implemented quality software, I think you couldn't care less.  Feel free to just leave it to us!


Print | posted on Sunday, June 13, 2010 12:08 AM

Copyright © Haxar Limited

Design by Bartosz Brzezinski

Design by Phil Haack Based On A Design By Bartosz Brzezinski