Openrsync: An implementation of rsync, by the OpenBSD team (github.com)

277 points by sph 11 hours ago

Panino 5 hours ago

I've been using openrsync here and there since it was announced and it's definitely improved over time. I'm looking forward to when I can use it exclusively.

The one place in my usage where it doesn't match Samba rsync is with the following:

openrsync --rsync-path=openrsync -av -e ssh /etc/services example.com:/tmp/services

I would expect openrsync to create a remote file /tmp/services, but instead it creates /tmp/services/services.

Normal directory mirroring as in -av -e ssh /path/to/src/ example.com:/path/to/dst/ works as it does with Samba rsync.

wtetzner 2 hours ago

> The one place in my usage where it doesn't match Samba rsync is with the following:

> openrsync --rsync-path=openrsync -av -e ssh /etc/services example.com:/tmp/services

This appears to match "normal" `rsync` behavior as well. I think you need a trailing slash after `services` to sync only the contents.

EDIT: actually my "normal" rsync is openrsync on macOS...

genxy 4 hours ago

Was there already a /tmp/services directory on the dest?

One of the biggest points of confusion with rsync is how directories and trailing slashes are handled.

anyfoo 3 hours ago

I hear that a lot, but I familiarized myself with it once and ever since it makes a lot of sense to me.

Source ending in “/“: You want what’s inside. Source not ending in “/“: You want the thing (i.e. directory itself). For the destination, it does not matter whether it ends in “/“ or not, but for consistency I like adding a “/“ anyway (I want to put thing inside the directory).

eichin 2 hours ago

It's a big source of confusion with cp. One of the UI reasons to use rsync (for mundane non-remote copying) is that it doesn't do different things based on what's present on the target.

Panino 4 hours ago

> Was there already a /tmp/services directory on the dest?

No. And just to make sure, I ran a quick 'rm -rf /tmp/services' on the remote host, then re-ran openrsync on the client. Same result. This is OpenBSD 7.9 on both sides.

And I 100% agree about trailing slashes.

hnarn 4 hours ago

> I would expect openrsync to create a remote file /tmp/services, but instead it creates /tmp/services/services.

As someone who has also suffered uncountable years of abuse from rsync, I understand the impulse, but I think it makes a lot more sense (and is a safer default) to create a second ”services”.

If we have a chance to change rsync defaults to something less insane and save future generations from this mess I think we should.

kbenson 4 hours ago

We don't, since we're not implementing a UI from scratch, we're matching something else.

Of the two possible worlds where in one this reimplementation matches what some see as annoyances in the interface or in another they mostly match the interface except for a few cases where the purposefully diverge (for no good technical reason), IMO the latter is far worse and causes more enexpected behavior.

At most, add a special flag to opt into different default behavior so nobody is surprised by running the same command on different systems and getting different behavior.

hilsdev 3 hours ago

If you use a trailing slash on the source it copies from the directory, if you omit the trailing slash it copies the directory itself. AFAIK this is pretty standard across POSIX tools

SoftTalker 28 minutes ago

It's not, for example cp -R doesn't change behavior on the basis of a trailing slash on directory names.

denysvitali 6 hours ago

There's also a Go implementation by Michael Stapelberg / the Gokrazy team: https://github.com/gokrazy/rsync

salvesefu 4 hours ago

For those needing context for the development of this package; this project is presently being developed as part of a RPKI validator.

https://medium.com/@jobsnijders/a-proposal-for-a-new-rpki-va...

thefilmore 6 hours ago

This is the version used in macOS since 15.0.

mrdomino- 5 hours ago

Was it 15.0? I seem to recall it coming in one of the minor point releases in the 15.x line - and I remember it breaking some scripts mysteriously.

EDIT: ah, fun: they did include it in 15.0, but they decided to save the breaking change that removed backwards compatibility for 15.4. https://apple.stackexchange.com/a/479297

onedognight 3 hours ago

They don’t support any recent rsync protocol, so there’s no 64bit timestamp support, so you can never actually sync metadata across newer filesystems.

Bender 8 hours ago

The actual work of porting is matching the security features provided by OpenBSD's pledge(2) and unveil(2). These are critical elements to the functionality of the system. Without them, your system accepts arbitrary data from the public network.

https://justine.lol/pledge/

I am not seeing pledge on Alpine Linux in edge. Have people been testing Pledge on Linux? Did I perhaps misunderstand the risk of using Openrsync without pledge? Or is this article just for OpenBSD users?

saidnooneever 5 hours ago

Linux has no such features as pledge or unveil, nor capsicum. it has cgroups, namespaces and a mess ofnother things u need to combine to try and do similar things. (it was built iteratively as many systems interacting and being combined to form 'sandboxing' or isolation/limiting of capabilities rather than specific isolation as an entire concept with specific system calls and kernel paths to enable it).

there might be newer stuff in linux land now i see comments about landlock but i assume those will build on the linux primitives rather than whole new ones. - total assumption there but it would seem logical to reuse rather than make new.

part of likely what they mean by 'mess' is that its all over the place. many different ways to try and lock things down. hard to pick what is best etc. without thoroughly diving into the different subsystems entirely. (as opposed to just have 1 or 2 relatively simple system calls)

thomashabets2 5 hours ago

No, landlock is a separate thing. It's the first of its kind on Linux that doesn't completely suck, like seccomp does (https://blog.habets.se/2022/03/seccomp-unsafe-at-any-speed.h...).

e12e 7 hours ago

From above your quote:

> The only officially-supported operating system is OpenBSD, as this has considerable security features.

And below your quote:

> This is possible (I think?) with FreeBSD's Capsicum, but Linux's security facilities are a mess, and will take an expert hand to properly secure.

It is portable in the sense that it compiles and runs, not in the sense that it has the same security features.

I'd love to see pledge/unveil on (upstream) Linux - but I'm not holding my breath.

papercrane 6 hours ago

> I'd love to see pledge/unveil on (upstream) Linux - but I'm not holding my breath

There is Landlock now, I believe it would be possible to implement unveil and pledge on top of that.

e12e 44 minutes ago

isityettime 2 hours ago

Bender 7 hours ago

Ok that makes more sense, thankyou.

justinsaccount 5 hours ago

that quote seems to be a bit of an oversimplification to the point of being completely wrong.

> Without them, your system accepts arbitrary data from the public network.

Neither of these features change if you are accepting arbitrary data from the public network. They limit what an exploited process can do. It's explained properly in the 'Security' section, so I'm not sure where this came from.

Bender 5 hours ago

that quote seems to be a bit of an oversimplification to the point of being completely wrong.

Under Portability [1] I don't have access to update that repo. I deleted my accounts when Microsoft took over.

[1] - https://github.com/kristapsdz/openrsync

skeledrew 8 hours ago

This attempt to avoid things that use AI is increasingly looking like some weird kind of reverse whack-a-mole where each targeted hole becomes radioactive after. Just grabbing some popcorn to watch.

ranger_danger 8 hours ago

I feel bad for people with the real name Claude.

xp84 6 hours ago

Yeah, and we thought the most unlucky people were the ones named Alexa.

SoftTalker 5 hours ago

grishka an hour ago

queuebert an hour ago

I don't know, Claude Shannon did okay.

cozzyd 4 hours ago

I think it would be funny to have a grad student named Claude for the hilarious ambiguity it would create.

hideout_berlin 3 hours ago

formerly_proven 8 hours ago

It took me quite some time to realize what an utterly presumptuous product name Claude Code actually is, but only because Shannon is rarely mentioned with his first name. It's golden calf levels of hubris, even more so if you consider how incapable it was on release. It's like renaming calc.exe Einstein. Incredibly poor taste, but entirely in line with AI tech bro mentality.

kstrauser 5 hours ago

1over137 5 hours ago

Yeah, especially since most Americans don't know how to properly pronounce Claude.

txru 5 hours ago

reaghs 3 hours ago

Thanks for the heads-up! I wasn't aware that Tridge is using Claude. I shall use Openrsync from now on.

einpoklum 3 hours ago

How about the attempt to avoid things that use AI promiscuously and start exhibiting bugs? :-(

skeledrew 2 hours ago

Push for better guardrails and QA structures. Avoidance helps nobody in the long run, and isn't possible anyway without going completely cold turkey. Like literally in a few months every project worth using will directly or indirectly involve AI.

groundzeros2015 an hour ago

echelon an hour ago

Wasting their precious limited time on this planet for performative hand wringing.

AI is only going to get better and better. Eventually manually writing software by hand with programming languages will be thought of as the punch-card phase of software development.

Do these people think we'll be writing software in 200 years time? That anybody will be maintaining rsync, let alone this "moral human hands only" version of it?

The anti-AI lot are trying to make all AI content wear a Scarlett letter. I wish they would wear one themselves so that we could filter them from our timeline.

This "effort" is entirely wasted.

23hartr an hour ago

Ask your AI girlfriend how "performative" is usually used. AI boosters here are really deranged and dumb.

What "effort" do you make? I'm sure you only produce SEO slop garbage.

tptacek 7 hours ago

rsync has specific running modes for the super-user. It also pumps arbitrary data from the network onto your file-system. openrsync is about 10 000 lines of C code: do you trust me not to make mistakes?

No, but that's why almost nobody runs it outside of strict trust boundaries. This security section would make more sense if rsync was like curl, which routinely deals with hostile counterparties. If the other side of your rsync is hostile, you probably have bigger problems!

(I'm not an rpki person so I don't know if there's some part of that problem domain that changes this equation. I'm not dunking on the project, just saying this snagged me in the README).

cperciva 7 hours ago

No, but that's why almost nobody runs it outside of strict trust boundaries. This security section would make more sense if rsync was like curl, which routinely deals with hostile counterparties. If the other side of your rsync is hostile, you probably have bigger problems!

I disagree. While rsync is most often used to transfer data between "friendly" systems, it's inherently crossing a security boundary. It's important to make sure that an attacker can't leverage it to transform the breach of one system into the breach of multiple systems.

eikenberry 3 hours ago

It is almost universally hooked up using ssh tunneling so ssh takes care of the security boundary and ssh is well trusted.

cperciva an hour ago

delusional 6 hours ago

> almost nobody runs it outside of strict trust boundaries.

I guess you can define "strict" however you want, but from what I saw ~10 years ago, most linux distros handled mirroring with rsync. That's a lot of usage in a pretty core part of the foundational open source ecosystem.

tptacek 6 hours ago

OK, I agree, that's bad.

akerl_ 6 hours ago

Many distros use rsync for that but also support unencrypted HTTP.

They’re layering on checksums and signing such that they mostly don’t think about the trustworthiness of mirrors or the networks between them.

triggis 9 hours ago

No-slop version for the sane of us

Context: https://mastodon.gamedev.place/@JeremiahFieldhaven/116654345...

ranger_danger 8 hours ago

akerl_ 8 hours ago

+1 to this. Other than people's reflexive anger or fear about AI coming for their code, I don't see anything to suggest that these are bugs that are due to the inclusion of AI vs bugs in a program with a bunch of complex interop with the filesystem and network.

48terry 3 hours ago

triggis 8 hours ago

throwaway27448 3 hours ago

I'm confused. Isn't rsync already free software? What are we doing here. Why are we trying to cuck ourselves for capital.

I like open bsd but this just seems like burning cash

somat 2 hours ago

My understanding is that much of the point of openrsync is to create a second implementation of a protocol so the standards bodies don't balk at including it in their standards.

Or to put it more concretely, people working on the rpki standard(who happened to also be openbsd devs) wanted to use rsync to transfer bulk data. The standards body was hesitant, while rsync is ostensibly a documented protocol, there was only one implementation. So in true openbsd fashion they rolled up their sleeves and wrote that second implementation.

On use, there is nothing wrong with openrsync, however it may never hit feature parity with rsync, that is not a goal of the project, they want a specific subset of rsync features to support their rpki needs. If anyone else finds this useful that is great. So I suspect users will be those who want a bsd licensed rsync(apple) or them who are willing to give up features for openbsd quality code(myself).

SubiculumCode 3 hours ago

I'm going to ask a question. I could ask chatgpt. I could Google it. I am asking a question because it is human to do so.

Ubuntu's packaged rsync, is it Samba rsync? Why reimplement it?

hideout_berlin 3 hours ago

samba is different topic

eichin 2 hours ago

it's tridge rsync; samba is another project by the same guy. (rsync was originally a PhD thesis...)

jmclnx 8 hours ago

I have not checked with OpenBSD 7.9, but as of 7.8 it did not support --exclude or -z. But outside of that openrsync works great.

(EDIT: --exclude is now supported on 7.9. Not sure when that was added, nice!)

But seems avoiding "slop" is getting very hard. I saw postfix now has a bit of AI code in it.

https://mastodon.sdf.org/@[email protected]/1...

nineteen999 8 hours ago

Somewhat ironic Postfix has a record of no root/RCE in the default install, where opensmptd hasn't (CVE-2020-7247). Time will tell if it stays that way.

agwa 7 hours ago

Where do you see that about Postfix? I followed the links and the only thing I see is that AI is being used to find bugs, not write code.

jmclnx 7 hours ago

>Claude assisted code found in external/ibm-public/postfix/dist...

That is from the original post in the thread. Is that really due to LLM ? I do not know since I avoid AI as much as I can.

But the person also posted this link too:

https://github.com/NetBSD/src/commit/f764ddf4062e855f73fe2e3...

agwa 7 hours ago

Bender 8 hours ago

Exclude is very commonly used in automation jobs to avoid duplicating big git repos and other big files. I think that would be a show stopper for a number of people.

jmclnx 7 hours ago

I just tried openrsync(1) on OpenBSD 7.9, --exclude now works.

I have not tried using exclude in openrsync in a while, but I can see it now works on OpenBSD 7.9!

WD-42 8 hours ago

What's the deal with the name? Openrsync implies to me that it's an open source alternative to a closed source program. But the original Rsync is GPL? Is this just the pushover license making it "more open"?

jtickle 8 hours ago

OpenBSD folks would consider the GPL to be less open due to the requirement to apply the GPL to any derivative works.

ranger_danger 8 hours ago

And GNU folks would say the GPL is actually the more open choice because it forces the project to stay open.

Two different ways of thinking about it I guess... it's nice to have choices and I don't think one is more or less "correct", more a matter of opinion/taste I guess.

gblargg 5 hours ago

thayne 3 hours ago

isityettime 2 hours ago

gilrain 8 hours ago

ranger_danger 8 hours ago

Many projects closely associated with OpenBSD start with "open"... openssh, openbgpd, openntpd, opensmtpd etc.

SoftTalker 5 hours ago

Notable exception, OpenSSL already had the Open prefix so the OpenBSD project is called LibreSSL.

hamdingers 8 hours ago

Not many are reimplementations of existing, much more popular, already open source projects.

throw0101a 8 hours ago

gpvos 7 hours ago

chasil 5 hours ago

There is also a (stub) web page:

https://www.openrsync.org/

The problem with this fragmentation of rsync is that Apple and Android will prefer it, but the Linux and greater GPL world will adhere to the original implantation due to inertia. Power users will just have to know the quirks of each version.

The only way to stop this is for the original author(s) to release this under a BSD license.

Edit: For those assuming equivalent/identical behavior, study these words carefully: "accepts only a subset of rsync's command-line arguments."

yjftsjthsd-h 4 hours ago

> The only way to stop this is for the original author(s) to release this under a BSD license.

Would that stop it? My understanding was that at least OpenBSD tended do redo things for technical reasons, not just licensing

eikenberry 3 hours ago

The only option should not to be to take away user freedoms. BSD licenses are popular with proprietary software writers because they can use it without any of the restrictions that seek to preserve the rights of the end user. Instead you get proprietary software stacks like Apple and Android that seek to lock the users out of anything not granted by the company.

The correct way to stop this is to file bugs against the software for not matching the de-facto standard of the copied software.

spauldo 4 hours ago

It's really no different than every other BSD utility (and SysV utility, if you're running one of those) being different than the GNU ones. We've coped with it for fifty years at this point.

qalmakka 5 hours ago

Basically like GNU Tar/CPIO and BSD Tar/CPIO. I've largely standardised towards using the bsd variant everywhere (especially since now even Windows ships it and it handles lots of other archive formats using the `tar` command) but it's always a pain to install it everywhere

yjftsjthsd-h 4 hours ago

Yeah, I'm leaning towards strongly preferring bsdtar since it's happy to work on e.g. zip files:) Does it have any real downsides?

SoftTalker 27 minutes ago

asveikau 4 hours ago

> Apple and Android will prefer it,

My thought upon reading this is why would Apple or Android bother including rsync? I've noticed that I've needed to install it manually on fresh installs of Debian, FreeBSD...

But then, I just checked a recent Mac that I don't use much and haven't put much on, and it's installed.