ok stsp@

Now that the handling of including sys/queue.h is better, there's no need to remove those lines from the source. Copy the location of those original sys/queue.h lines from upstream at the same line number, so as to avoid any conflicts in the future.

ok op, tracey

Thanks to the design of Got, the libexec helpers don't need any resource (in fact they run under pledge "stdio recvfd" on OpenBSD) and so using cap_enter(2) on FreeBSD is dead-easy. While the main process can't be sandboxed on FreeBSD (needs to exec the helpers), all the tough work is done by these small libexec helpers which is also the biggest attack surface. tested by naddy, ok thomas

The old code did not work correctly if only a single object Id was to be sent to got-read-pack. Make got-read-pack error out if the list of commits for object enumeration is empty to catch this problem if it occurs again. Found by the send_basic test, which was failing with GOT_TEST_PACK=1 ok tracey

This allows the main process to avoid looping over all object IDs again in case the pack file used for enumeration is complete. ok op@

The problem that was found in the earlier version has been fixed. ok op@

This implementation marked commits and trees as enumerated before all trees which they depend on were enumerated. This behaviour leads to incomplete pack files when a tree is only partially packed and got-read-pack hits a missing tree entry as a result. The algorithm must be reworked such that packed leave nodes are marked enumerated first, then bubble-up. Found by op@

Fixes an error where got-read-pack errors out with "bad object data" during 'got send' because we ended up handing a tag object to the commit object parser.

ok op@

This approach uses more memory but is much faster. To offset the additional memory usage somewhat the cache now stores very small deltas only. However, overall memory usage goes up. Hopefully we will find a way to reduce this later. ok op@

Avoids some extra malloc/free in a performance-critical path. ok op@

This significantly speeds up the deltification step of packing by avoiding imsg traffic. gotadmin no longer requests individual raw deltas from got-read-pack to check whether it can reuse them. Instead, got-read-pack obtains a list of objects we want to pack, and hands back the list of all deltas in its pack file which can be reused. Messages are now batched such that imsg buffers are filled as much as possible. Another advantage is that deltas we are not going to reuse will no longer be written to the delta cache file, saving disk space. Before this patch, any raw delta candidate was written to the delta cache file by got-read-pack, and the decision whether to reuse the delta happened afterwards in the gotadmin process. Code for reading individual raw deltas is now unused and could be removed at some point. ok op@

This reduces memory and disk space consumption during packing. with tweaks + memleak on error fix from op@ ok op@

Saves us from doing a malloc/free call for every item on the list. ok op@

tested by thomas, naddy, and myself

landlock is a new set of linux APIs that is conceptually similar to unveil(2): the idea is to restrict what a process can do on a specified part of the filesystem. There are some differences in the behaviour: the major one being that the landlock ruleset is inherited across execve(2). This just restricts the libexec helpers by completely revoking ANY filesystem access; after all they are the biggest attack surface. got send/fetch/clone *may* end up spawning ssh(1), so at the moment is not possible to landlock the main process. From Omar Polo.

This speeds up the creation of pack files by about 30%.

This adds the capability to compile got-portable on FreeBSD.