Here is an example of how this will work in the next release:
(define (sync-ignore-file path) (check-prefixes path '("ignore1/" "ignore2/" "private.org")))In the above example beorg will ignore any files in the ignore1 and ignore2 folders, and also the file private.org. A user could define sync-ignore-file to do more complex matching if required.
I hope to get this update out a little later this month.
sync-ignore-file has been released. If you would like to use this just redefine that function in your init.org file.
I've just uploaded the latest library.org file to beorg.app. You can find this at https://beorg.app/manual/library-org/. It is linked to from the Learning > Scripting page. You can also find this in beorg itself (which is always up-to-date from the REPL, button top right on the Settings tab.
I've added links so you can download library.org on the website and view in Emacs or another text editor. I have also included the BiwaScheme source file that beorg uses and some additional JavaScript which extends the Scheme functions. This should hopefully give a complete picture of what is available for use in the Scheme scripting in beorg.
Here's what's sort of working for me:
```
(define blacklist '("file.org"))
(log-debug (format "-- Examining file ~s" path))
(member path blacklist))
I have 600 entries in the blacklist. I suspect that this is causing performance degradation instead. In the log, I see "Examining file" entries and they last for about 20 seconds. Mean while I can't make any changes to the agenda. So this is not the way to go.
Any other thoughts on how to optimize synchronization?
Sorry for taking so long to look at your log. It is the case that the interaction with Dropbox is taking the most time. The first sync in the file times out, waiting for nearly a minute after the third Dropbox call. The second sync takes nearly 20 seconds to, this time successfully, get all of the Dropbox file metadata - there are ~30 individual calls to Dropbox made.
I personally make sure that "Auto sync on modify" is turned off on Dropbox so that a sync happens only when the app moves into the background.
It's a pity that the Dropbox API doesn't have (as far as I'm aware) a call to say - give me the metadata for all files which have been modified after this date. That would speed up sync massively!
Hi Matthew,
Here's the log: https://gist.github.com/dmitrym0/1cc06c661d704a85b9e9555c542f416d
Here's my init.org: https://gist.github.com/dmitrym0/e1ec73b1d76a5b5bb22cc3a5974aa316
I sanitized my log to avoid any PII, hopefully that's ok.
Check out https://www.beorg.app/support/debug/ for how to view just the text of the log, so you can copy/paste out of beorg.
Hey Matthew,
With zero changes, the sync takes about 40 seconds for me right now. Looking at the log it does seem like polling dropbox is what's taking the most time.
I dont see a way to export the log. What is the best way to do it?
The Dropbox API used by beorg has to fetch metadata about all the remote files in batches. I can't remember how many files are included in each batch, but possibly about 100. If that is the case then for that number of files there are 20 individual calls to Dropbox just to get the file listing. Possibly it is this which is taking the time. If you look at the beorg debug log there should be timestamps which may show how long each part of the sync is taking. If you are able to post your findings here then that will help me look at the right part of the sync process and work out what is slowing it down.
Also, how many files on average per sync are changed on each sync?
(It would be great if you could ask Dropbox for a list of changed files since a particular date, in a particular directory hierarchy - but as far as I'm aware that isn't possible.)
Hey Matthew, revisiting this to see if you have any thoughts. My org dir has grown to 2000+ files and Dropbox sync has slowed down significantly.
I spent a bit of time cleaning things up and crafting an init.org that has taken the sync from 2m+ to just under 1 minute. Here's the relevant portion: https://gist.github.com/dmitrym0/9e81b746136a538ed5e65fb047433b05
Do you see any ways to speed the sync up further or is it just a consequence of having to iterate over so many files?
I'm submitting a beorg update to Apple which I'm hoping will help. It looks as though the issue wasn't in the sync itself, but when beorg was trying to work out what name to use for each file - and making too many requests to it's database of cached .org files. I was experiencing a similar issue and with the fix beorg feels much snappier. Look out for 3.27.0 in the coming days.
Hi Matthew,
I have roughly 1700 .org files and I use dropbox.
I tried using iCloud which seems more performant but it stopped syncing my org directory at some point, I suspect because of the .git folder -- I use git-auto-commit module.
Hi Dmitry. What sync method are you using (and approximately how many files is beorg checking to see what needs syncing)?
I personally have "Auto sync on modify" turned off on the Settings tab. beorg will still sync when moved into the foreground/background but not after each change.
Oh, just took a look at Beorg > Learning > Scripting and found the link to library.org. One problem:
Library.org has some long lines and the box it is displayed in on the web page cuts off the lines. There is a horizontal scrollbar way at the bottom of the box, but, if you scroll all the way down to find it, you lose context and it is impossible to see how far to scroll right.
Can that be fixed?
Levi Crews
I have 3000+ org files total split across subfolders (org-roam!), so syncing subfolders takes forever. The app has been syncing since yesterday morning and it's only gotten through <500 files.
In Emacs, you can specify a list of org-agenda-files, including files in subfolders. I only use beorg to read my agenda, so I only need my org-agenda-files synced. With the new subfolders feature (+ maybe the ability to exclude particular files), is it possible to selectively sync only my list of agenda files, even if some of them are in subfolders?
In response to my related ticket, Matthew suggested one potential solution for more advanced users: write a Scheme function that could be defined to return true if a particular file is to be included in the sync. Unfortunately I don't know much Scheme, so any help or alternative ideas would be appreciated!