FreeCAD Sync Proposal

FreeCAD Sync

It should be possible to translate individual user edits on a particular file/assembly into git commits, transparently and without user-intervention. Microtransactions. This is different than the traditional use of git, where the end-user manually versions a worktree, writes a commit message, and pushes/pulls/merges.

Microtransactions

These microtransactions will be sent to a configured repository, such as on GitHub/GitLab, an OSE server, a peer or peers, or just the local filesystem. Others concurrently working on the same assembly would see the assembly being updated as the microtransactions are pulled in. As opposed to git polling the remote repo and subrepos at some interval (other than on launch), it would be ideal to have a subscription service set up to tell subscribers to update, over UPnP would be less secure than a cloud or OSE server.

The use of a custom XML merge driver, with underlying knowledge of the FCStd file format, should mitigate the event of a merge conflict from occurring. Were a merge conflict to occur, be it online or offline, I hope to present a suitable interface for the end-user to make a decision, such as to branch their conflicting microtransaction or be presented with a diff of the XML element? Paired with FreeCAD being patched to not store BREP data in the persistent layer and only store it in memory, regenerating it on demand. realthunder may have already done this in his branch.

Sub-assemblies

Each linked sub-assembly of a parent assembly will have their own document, in their own git branch, with their own history, allowing other assemblies to make use of and update that sub-assembly without changing the original parent assembly.

Transparent Git features

The use of git as a protocol for synchronization also allows for re-use of existing features. Such as a version history and other git features.

At intervals the end-user may elect to version/tag their progress, entering in a commit message into a FreeCAD GUI explaining their changes.

Some of the below plumbing in support of synchronization of FreeCAD data requires custom user configuration, such as the re-zip tools. I anticipate the FreeCAD Module would hide everything git behind the scenes, where the user does not interact with git at all. The Module would allow for visually browsing the version history, automatically generating thumbnails (maybe the caching should only be done here).

Contrast to present OSE Merge Workflow

This proposal is my intuition of what is possible. My impression of the present Merge_Workflow is that it is optimized for the existing constraints/limitations of FreeCAD in the spirit of mass collaboration. But it does impose additional steps on end-users Talk:Merge_Workflow#False Dichotomy.

The memory FreeCAD requires for sub-assemblies in large assemblies is a problem that the OSE Merge Workflow mitigates in the procedure of deleting the part history. But that should be do-able from within FreeCAD, leaving that data on the persistent layer and dynamically loading it into memory on demand. Less data-management overhead were this to be implemented in FreeCAD. Easiest way to do implement this feature would be to script the procedure within FreeCAD as sub-assemblies are loaded: load full model, then prune. I need to look at internals.

FreeCAD Plumbing in support of Sync

FreeCAD v0.17 and newer

As of FreeCAD version 0.17 the WebTools Workbench allows Git integration through manual steps.

However *.fcstd files are binary files, specifically zip archives, and binary files aren't really suitable for version control tools like Git.

To prevent the explosion in space usage this mal-use would cause, a git filter should be set up for these *.fcstd files, such as:

1. hoijui/ReZipDoc Open Source Ecology Germany, requires JRE 8 or newer writeup
2. callegar/Rezip, written in bash, won't work on Windows (unless cygwin is used). Might have unwanted permissions changes on files due to the rezip?
3. Zippey, written in python. Zippey uses a different strategy than the other two so isn't compatible with them.

The rezip tools essentially re-zip the zip archive with a compression of 0, to allow git to compress better given changes through time. While Zippy is a bit different than the rezip tools... I'm not sure at a glance the implications of using it.

If you have Java then hoijui/ReZipDoc would be more robust. callegar/Rezip in bash seems fine if you don't have Java installed and the git-config filter name matches what is in gitattributes. I expect any user can use either they want, as long as they have the global git config filter set up for the tool they have, and that filter name matches what is in the repo's .gitattributes

If a user hadn't properly set up a rezip tool and pushes their changes, then there won't be any problems other than a size increase. Which I suppose can be manually fixed later, but that is a headache.

To see the results of the better compression, run `git gc`.

However, FreeCAD in a future version will support better integration with version control without such re-zip workarounds and additional workflow steps (e.g. FreeCAD automatically controlling git after a save instead of the user controlling git after saving the file).

FreeCAD v0.18 and newer (or v0.19 and newer?)

Brief view of Assembly3 assembly, document, and link organization. Sub-assemblies which are linked to an external document, will result in that external document being loaded and displayed as shown. While frozen they won't consume much memory.

Has realthunder's Assembly3 workbench. Supports links, which help eliminate prior memory limitations with larger assemblies, and need for such workarounds like File_Simplification. So long as the linked document is opened and the link is frozen, the assembly is displayed within the parent document as a single unmodifiable geometry. When the linked document is closed it won't be displayed anymore in the parent document.

I think it would be best if when the Assembly3 workbench isn't active,

That a simplified view of the model is presented in the Combo view, without the Constraints, Elements, Parts and additional hierarchy that Assembly3 at present introduces to FreeCAD:

The clutter of external documents in the tree view can be hidden by the setting 'View -> TreeView actions -> Single document'. But I'm still looking for a way to hide the Constraints, Elements, Parts containers- which would entail changing the TreeView to use a proper model-view separation. This change would also allow the ability to reorder items easily in the TreeView, something others would like to see implemented anyway.

FreeCAD v0.20 and newer

realthunder has made changes in FreeCAD to better support Version Control, however he writes "It is still in my fork. I think this feature will most likely land in the next 0.20 release".

Save as directory for version control integration in FreeCAD v0.20 and newer and realthunder's fork

realthunder has put in the plumbing to allow an FreeCAD module to automatically manage the version control of the CAD files. I'm unsure how what the performance or implementation constraints are at present.

ickby's Colaborative Addon

ickby created an collaborative addon in 2021. Supports v0.18 and v0.19. Doesn't use git for sync plumbing. Doesn't support true concurrency at present. Certainly an interesting approach to the problem though.

Looks like it doesn't use UPnP at a glance, and doesn't handle NAT? Listens on port 7000, has a bootstrap server. Unsure what function the bootstrap server has, maybe it is a data proxy mitigating the need for forwarding firewall port 7000 from 0.0.0.0...? That's going to be attacked.

I suppose that deserves some thought, what the most secure architecture would be. I suppose I'd only use key-based ssh on a centralized server, or perhaps there is a secure peer-to-peer library that meets my expectations out there somewhere... not worth the effort, the subscription notifications I envision in my proposal will be low bandwidth and will be fine centralized, just JSON sent to subscribers, triggered via a git hook, that URL would be to the centralized server, which would send that JSON to on-the-fly subscribers. Certainly a bazillion IoT libraries out there that do this.

Discussion

While the solution with the FreeCAD WebTools Workbench and rezip should work... better possibilities exist, which will:

1. Better handle multiple documents, allowing re-use of work done (why invent the wheel twice!)
   1. Each linked sub-assembly of a parent assembly will have their own document, in their own git branch, with their own history, allowing other assemblies to make use of and update that subassembly without changing the original parent assembly.
2. Transparent git integration
   1. End-user does not have to interact with git or do any additional steps, other than save and enter a commit message

Use FreeCAD v0.20 once available, or realthunder's present fork.

Set up a Module to respond to the signals. The signal data will include what files were modified/deleted/added. The module will:

1. Interact with git. e.g. Commit the changes to a new revision.
2. Push the new commit to a remote repository, if a remote repository is designated in the document properties.
3. Put sub-assemblies into their own branch, and use git sub-repo to include those sub-assemblies in a subdirectory in the main branch (likely master). Assign a GUID to that sub-assembly on creation and have that be the branch name?
4. If a remote repository was designated for any sub-assembly, the module will push any change for that sub-assembly to that sub-assembly's repo.

In addition to the module:

1. Add a means of adding properties to the document, so that the FreeCAD user can use the GUI to designate a remote version control repository to push/pull the changes to.
2. Add a means for the user to enter a commit message when they save the document.

Annotated Visual History

An optimal way to implement the plumbing behind Annotated_Visual_History.

1. An ".annotated_visual_history_cache" folder, which is listed in .gitignore (not versioned)
2. FreeCAD on opening a version controlled directory without said cache, will interact with the module as the module iterates through git revisions, to cache each revision. In the background async, or scheduled when idle, so as to not interfere with the user's work. In-memory would be quick. Might be best to do this only when the user attempts to view the history so as to not slow down their other work.
3. On each revision, take a set of isometric snapshots rotating around the assembly, resized to say 256x256 px.
   1. The module can possibly iterate through each revision without opening up the FreeCAD GUI.
4. An image processing utility will assign a complexity score to each snapshot, and the snapshot with the maximal complexity score will be selected as the thumbnail. A complexity score being a proxy measure of what image best illustrates the part!
   1. Filesize might suffice as an indirect measurement of complexity, the largest image being the most complex, otherwise,
   2. ImageMagick's identify might suffice, with a grep of a -verbose output, or a -features measurement, or
   3. FreeCAD 'Preferences->Document->SaveThumbnail' boolean indicates that a suitable feature may be in the FreeCAD API somewhere.
5. Have a FreeCAD workbench, such as the WebTools workbench, able to interpret the git log and present these thumbnails in a meaningful way for the user

Or simpler would be to not make a cache, but just use FreeCAD's thumbnail generation as part of each versioned document. But this is binary and shouldn't be versioned if at all possible... caching is the optimal solution as illustrated earlier.

Related docs: Collaboration_Architecture, FreeCAD_101 Merge_Workflow Microfactory_Boot_Camp_-_Every_Hardware_Build_is_a_Fork