The Dropbox API used by beorg has to fetch metadata about all the remote files in batches. I can't remember how many files are included in each batch, but possibly about 100. If that is the case then for that number of files there are 20 individual calls to Dropbox just to get the file listing. Possibly it is this which is taking the time. If you look at the beorg debug log there should be timestamps which may show how long each part of the sync is taking. If you are able to post your findings here then that will help me look at the right part of the sync process and work out what is slowing it down.
Also, how many files on average per sync are changed on each sync?
(It would be great if you could ask Dropbox for a list of changed files since a particular date, in a particular directory hierarchy - but as far as I'm aware that isn't possible.)
Hey Matthew,
With zero changes, the sync takes about 40 seconds for me right now. Looking at the log it does seem like polling dropbox is what's taking the most time.
I dont see a way to export the log. What is the best way to do it?
Check out https://www.beorg.app/support/debug/ for how to view just the text of the log, so you can copy/paste out of beorg.
Hi Matthew,
Here's the log: https://gist.github.com/dmitrym0/1cc06c661d704a85b9e9555c542f416d
Here's my init.org: https://gist.github.com/dmitrym0/e1ec73b1d76a5b5bb22cc3a5974aa316
I sanitized my log to avoid any PII, hopefully that's ok.
Sorry for taking so long to look at your log. It is the case that the interaction with Dropbox is taking the most time. The first sync in the file times out, waiting for nearly a minute after the third Dropbox call. The second sync takes nearly 20 seconds to, this time successfully, get all of the Dropbox file metadata - there are ~30 individual calls to Dropbox made.
I personally make sure that "Auto sync on modify" is turned off on Dropbox so that a sync happens only when the app moves into the background.
It's a pity that the Dropbox API doesn't have (as far as I'm aware) a call to say - give me the metadata for all files which have been modified after this date. That would speed up sync massively!
Levi Crews
I have 3000+ org files total split across subfolders (org-roam!), so syncing subfolders takes forever. The app has been syncing since yesterday morning and it's only gotten through <500 files.
In Emacs, you can specify a list of org-agenda-files, including files in subfolders. I only use beorg to read my agenda, so I only need my org-agenda-files synced. With the new subfolders feature (+ maybe the ability to exclude particular files), is it possible to selectively sync only my list of agenda files, even if some of them are in subfolders?
In response to my related ticket, Matthew suggested one potential solution for more advanced users: write a Scheme function that could be defined to return true if a particular file is to be included in the sync. Unfortunately I don't know much Scheme, so any help or alternative ideas would be appreciated!