Server-side of fetch/pull

[Byron]

What would be needed to allow a server to send a pack?

Tasks

Server fetch/pull (server to client)

• git-odb

The below is a very early draft - it would be better to study existing implementations first to get a better overview on what (not) to do.
This one starts with the fun part to allow writing tests early and experiment with different diff algorithms and potentially their performance.

• generate a pack from objects received by an iterator producing (see issue)
  • base objects only
  • re-use existing delta objects
  • A mechanism to declare some bases to be 'out of pack' for thin pack support
• Iterator to feed pack generation efficiently
• pack creation

• git-transport

Certainly needs more research, but roughly…

• Server side accept()

  • http(s)
  • ssh
  • daemon probaby only used in testing, and we might implement it if it's useful for us as well

• git-protocol

  • Server side chatter to negotiate a pack for
    • protocol V2
    • protocol V1 (probably not worth it, let's see)

• gix-serve

Probably more like a toy at first merely for testing operation against various git clients.

• A server able to answer via
  • http(s)
  • file protocol (or remote invocation via SSH)
  • git://

Notes

• Could something like gittorrent be build using the plumbing of the server? Is it desirable even? Can there be some differentiation to allow custom transport layers easily?

[masklinn]

Server side accept()

http(s)
ssh

Just to be clear, this doesn't mean reimplementing things from accept() upwards, but supporting these as inputs using existing libraries, similar to what git-transport currently does on the client side?

[Byron]

If I understand the question correctly, the answer is yes. The client of git-transport could be handled with what's provided by accept(), while the actual interaction patterns would be abstracted in git-protocol.

This is very desirable to be able to quickly implement your own self-hosted git server (existing ones in other langs require too much memory and therefore are a poor fit for cheap vms).

[vlad-ivanov-name]

I think it might be enough to accept AsyncWrite and AsyncRead for transport without worrying too much where those come from. Or, at the very least, the whole HTTP and SSH plumbing, providing alternatives to tools like git-http-backend and git-upload/receive-pack, should be separate.

[willstott101]

I've been experimenting with writing a server (in a private repo so far). My interest here is in using parts of gix for the protocol, but leaving the storage up to pluggable backends. I'm very curious about git-on-db, and git-on-object-storage, and git-on-kv-store, and mixtures of the three. Step 1 of this is to have a clean working HTTP git server written with gitoxide, using it's filesystem access as the only storage backend.

I think I have ls-refs working and I'm starting to investigate fetch. The protocol is very command-based, and in HTTP AsyncRead & AsyncWrite can't really exist at the same time. Regardless we'd want to leave it up to server implementations to authenticate during the connection, find the relevant repo, authorize the parsed command, then hand back to gitoxide to respond.

So for me a sketch might look like this:

• Server parses HTTP headers & path to verify protocol v2, authenticate
• async fn read_command(source: AsyncRead) -> Result<Command>
• Server authorizes the command and writes response headers
• async fn execute_command(cmd: Command, repo: ..., dest: AsyncWrite) -> Result<()>

It's possible that a sufficiently advanced AsyncWrite could have a buffer and be appended to the HTTP response after writing the headers (in-case the command parsing wanted to reply with errors in packet-line format?). I have also spent no time so far learning about the SSH transport. But those are my thoughts so far.

I also have some questions about packfile construction. I haven't spent a great deal of time investigating yet but I currently haven't found much in the existing gitoxide codebase. Especially relating to resolving packfiles between two peers, but there must be some logic in gix for this somewhere. I'll keep looking, and if anyone is particularly interested in collaborating let me know and I can un-private the repo, I'm just quite enjoying the messy private sandpit atm.

[Byron]

That's great news! Please be sure to let us know here once the repo goes public!

Regarding the sketch, the server would probably also reject V1 requests. But then, read_command() would read the command itself and arguments to it, but I wonder if that's not a liability as it might read more than it has to given that the server might reject the command itself. That probably also depends on what information the server wants to use to reject the command, but I can imagine that a step-wise process would be better. Read the command-name, then read its arguments, but maybe that is implied.

In general, it's probably OK to just cobble it together and then refactor.

pack creation

Regarding packs, you can try gix free pack create and see from there how the API works. In general, packs can start streaming quickly, but they won't be the most efficient as they don't delta-compress on the fly. But that might even be a beneficial trade-off at first.

Transports

You can check the client-side (use gix --trace clone ssh://… to see how ssh is typically invoked. From there you will see that it definitely requires its own binary, but that should then be the easiest implementation as it's the same as gix --trace clone file://local/path. However, getting the server-side SSH server going is probably it's own set of problems to solve.

[AleDema]

@willstott101 Have you managed to write a working git server using gix crates? I'm also looking into doing something similar and I'd like to know whether you found any major roadblocks.

[willstott101]

I'm afraid I really didn't put a lot of time into it in the end. No major roadblocks, but I only had a super basic fetch functioning. Decided I needed to refactor to a different server lib and I lost steam when I realized it was time to make integration tests :)

I hope you're planning your work to be open source. I would be interested to pick this up again.

[AleDema]

Yes, I do plan on open sourcing it.

@Byron do gix packages already provide the necessary primitives to implement a basic http git server?

[Byron]

The packetline format at the lowest level in gix-packetline is available, and the client-side of the fetch operation is in gix-protocol. But there is no client-side push, nor any of the server side implementation.
One would be able to create packs though, so implementing the server-side protocol of say fetch would be possible only by implementing the protocol part.

So the answer would be: yes, the necessary primitives are available.

If things have to go fast (as in 'fast' to implement) though, I'd definitely recommend shelling out to git.

[willstott101]

I've invited both of you to my "mizzle" repo where I was giving it a go. Current license file is MIT but I can make it whatever I don't mind. No guarantee there's anything useful in there 🤷