Hacker News

Something else that git isn't good at: permissions. In gamedev, you might have proprietary work that you want to restrict to certain users. In P4, you can add restrictions to certain directories for only those who have signed the required NDAs. That's not something that you can do in git: it's all or nothing. Maybe you can set something up with submodules, but that's going to upend your repository if you hadn't planned for it.

Its important to understand that in Game Dev a ’git clone’, aka ’p4 sync’, can be a terabyte of stuff.

Git is bad at such volumes of binary assets, textures, models, sounds, etc.

Git LFS is a major PITA, and if you use GitHub is even worse since there are quotas and rate limits that are charged separately.

I am building a small asset heavy game. Ran into a similar problem. Built a storage cost efficient tool for exactly this [1].

[1] https://github.com/debarshibasak/assets

I wonder how useful this could be as a generalized version control for regular user systems, as a way to rollback, or scrub through history. Presumably if this is designed to work at Epic and Big Game studio scale, it should work at home computer scale

P4 is more "industry standard" than "state of the art"... But it does handle large files and partial checkouts without feeling bolted on.

There's also a new player called diversion (diversion.dev) which I think may be a YC startup? Anyway it takes a different approach of being more like Google drive but bringing in VCS behavior making it more indie and designer friendly.

> Git is fine for text based files like code, but it's really bad at stuff like textures, 3D models, audio files, and other non-text files

Git-annex ?

P4 is also really well integrated into IDEs and UE Editor so that I don't need to think about it as much as I need, compared to Git. Locking assets, releasing them, merging into streams etc., is overall pretty streamlined. When it works, it's great, but when it doesn't work though, it's pretty hard to diagnose issues.

A significant part of my job, unfortunately, is helping people fix their workspaces when Perforce (p4) goes bad, or creating guardrails and wrappers to stop Perforce doing bad things.

In fairness, p4 predates most of the VCSes we consider "modern", so I empathize with a lot of the underlying architecture decisions. However, it has and continues to utterly fail at improving at a reasonable pace.

For example:

  - p4 tracks file metadata of client workspaces on the server (sync'ed locally, opened for edit, file revision, etc) and uses this as the basis to avoid doing unneeded work.  If this becomes desync'ed, a reconcile or force sync must be used.  A reconcile can take hours, potentially days; it tries do detect file moves by default, so likely at least O(c^n) for some c>1. I have never personally seen a default reconcile operation _complete_ over any modestly large game code base, and in practice, people accumulate a litany of workarounds and scripts to fix this for themselves.

  - Scripting p4 is a nightmare. Documentation is poor, schemas do not exist, and all the language-specific libraries are just thin wrappers over its C++ API.

  - By default, p4 "helps" you with text files by "correcting" line endings on sync or even converting between encodings. This works until you have a mixed-OS environment, and discover a part of the pipechain that _must_ have a certain style.  There are various levers to pull to make this better, but I've yet to find something fool proof.

  - By default, p4 keeps flies read-only, only unlocking them when explicitly marked as being edited.  This means, to avoid having to do this manually, every tool you use needs to be p4-aware.  Or, you can turn this off, and choose to contend reconcile instead. (See above)

  - Branching a modest game project, with, say, Unreal source code, can take hours.  And this is the quick version where you ask the server to simply create new metadata, with no file transfer to a client.

  - p4 is licensed by the user-account. Every user entity in p4 not intended exclusively for performing backups and maintenance operations counts toward this, including users required to integrate with other services.  Plus, often times, these integration users must have admin access to be useful. The security posture is horrific.

Git is certainly not great with binary assets, but calling perforce SOTA ... ouch.

If perforce is the best there is out there for large binary asset management, then there is a blue ocean worth of potential improvement for git.

Perforce is a piece of crap, a relic of the 20th century that must die in a fiery inferno.

That sucks, git is so absolutely horrible. It's crazy to me that nobody has made anything better yet. Although I could start that myself and yet have not.

Jonathan Blow found it convenient to represent all assets in a large number of text files, to enable merging. For instance he'd have one text file per entity on a map. The game and editor could read either this or the compiled binary version.

Git LFS has file locking, and no VCS can provide you with the tools for diffing binary assets. I don't see any meaningful difference between Perforce, Diversion or Lore and git + LFS + file locking. Unless there's a meaningful performance impact for large projects (I only work on small / medium projects), the capabilities are the same. However, I get excellent git support for code in any editor, as opposed to Diversion or Lore which have none.

Every Beagle command:

    gritzko@spot ~/beagle $ be get
     19:07  get ?#0ac49e6a
     16:58  post ?0ac49e6a#POST-018 put:/post: banner on stdout
     19:07  new beagle/test/be-post-put-banner.sh
     19:07  upd dog/INDEX.md
     ...more stuff...
     19:07  del test/post/01-bare-msg/01.put.err.txt
     19:07  del test/post/01-bare-msg/02.post.err.txt
     19:07  get abc?4222dfab

Objects are your files. Underlying git is a content-addressable filesystem.

The objects are referenced by trees. A tree is just a directory.

The trees are then referenced by commits and/or tags into a DAG with named pointers into various parts of it (which are your branch and tag references):

https://git-scm.com/book/en/v2/Git-Internals-Git-Objects

Because it would be terribly in-efficient to have a bunch of loose objects, git periodically groups them together into packs. To save space, the objects are compressed against one another (delta compression) within the packs.

https://git-scm.com/docs/git-pack-objects

https://github.com/git/git/blob/master/Documentation/technic...

When pushing or pulling, the git transfer protocol basically enumerates what objects each side has so that it only needs to transfer the difference. On top of that, it delta compresses the objects on each side that aren't already grouped into packs against each other to save space.

https://github.com/git/git/blob/master/Documentation/technic...

Because git is an open-source project written by nerds, it shows you all of this information. Feel free to ignore it!

But if you really want to know, it's all documented both in the git book and git documentation directory, both linked above.

(Caveat: I'm working from memory and surely got some detail at least slightly wrong.)

I think we can all agree that information should be behind a -v CLA. It's probably just something no one has thought of doing. I've learned over the decades to just ignore it.

Every place I worked at has a git introduction where all new employees learn about how git works internally. Takes 1h, and all junior devs stops memories random commands and actually start to understand. I highly recommend to you to poking around in the .git directory.

The git support for new employees drops basically to zero.

The lights are blinking, so everything must be working!

I actually like this underlying logs. Could have a concise / project level summary though.

Those are just the sounds that animal makes. Live with the animal long enough, you learn how the sounds correspond to its internal states, even if you don’t really know what they mean.

I’d be a bit worried if git didn’t heave that particular contented sigh when I ask it to push

I'd rather see some gobbledegook than extended pauses or idealized (read: fake) information. Those are specific tasks it is doing when you run that command, there's a simplicity to it.

Not saying Lore's approach is bad, but sometimes "worse is better".

Git as a data structure is clever, but Git as a CLI is atrocious.

This is what happens when a kernel developer creates tools that need some kind of UX (I say this both as a shitty UX developer and Linus fan)

I use GitHub desktop app that pushes to my local Gitlab. It’s a nice and simple GUI, it might be what you’re looking for.

I suspect the use of Rust rather than C++ might have something to do with the fact that Simon Peyton Jones and Lennart Augustsson (both of Haskell fame) both work at Epic and there would have been a strong internal push to do this in a language with some functional programming features. Rust rather than Verse because that would probably not be the right tool for the job (even if Simon works on it). Rust rather than Haskell probably because of performance -- DARCS never caught on partly for performance reasons.

I don't think the needs are exactly the same. I believe in AI the big binary files are normally written once, while in gamedev, they are constantly updated.

That already warrants different storage architectures.

Yeah, I'm still concerned about crystalline entities suddenly showing up. Have they ever fixed it? I don't see anything in the issue tracker, probably because no one was left last time to report it...

If it were Data instead of Lore, it would reject any commit messages that used contractions such as "I'm" and "can't" ;-)

Dunno, this seems fully functional?

Oh brother.

Don't be too hard on Lore.

All Data is Lore. I mean lore is a superset of data. I mean data is lore with a special attribute.

I'm not just picking nits here. And this is not cynicism.

so there you go.

"Lore" is appropriate. Epic games is a very unethical company that steals from people. Myself in paticular. I bought Rocket League the game for linux from Psyonix. Epic bought Psyonix and immediately removed the game clients for linux and mac os. I can no longer play. They stole from me and many others. It'd be one thing if they just shut down the game entirely, but stealing it from only some people while keeping it going for others is worse than just killing games.

https://epicgames.github.io/lore/roadmap/#desktop-client

> Open-source the desktop client so the community can build on its full graphical experience, not just download it. An early desktop client already exists as a binary download, but it isn't open source yet — it depends on some proprietary components, including Epic's internal design system. We're working to make all of it available in the open so that the client can ship as source alongside the rest of Lore. Lore is an open project, so it is important that the desktop client — which will be one of the main ways many people will interact with Lore — is also fully open so that the community is free to review, extend, and shape it.

to be fair, that's just the desktop client. You can use or build on top of the CLI

they do say they will open source it, but who knows:

"It isn’t open source yet—it currently depends on some proprietary components, including Epic’s internal design system—but we’re committed to open-sourcing it in the future"

There’s a “Repositories” link at the top of the page that links to https://lore.org/#repositories, WhyHow links to various GitHub repos, including https://github.com/EpicGames/lore, which claims to have code for the CLI. I see no reason to suspect that claim is incorrect. The code likely lives in https://github.com/EpicGames/lore/tree/main/lore-client)

This is a very long document that says nothing about chunking at first skim. If chunking is actually wrong, then just explain why, here. Wasting space is not actually a problem if it’s optimized for other purposes instead.

Git-ish CLI is great. The GUI is more important though. Non-programmers don't want to dabble with CLI. One reason why Perforce is the defacto standard IMO. The GUI covers 99% of daily used operations and is easy to use.

The five command part isn't really possible but you can use custom diffs for merges, git diff, etc. pretty easily. There are projects like diffsitter ( https://github.com/afnanenayet/diffsitter ) for doing more intelligent diffs like this for supported languages.

EDIT: and then an example for the merge stuff I couldn't find while typing before: https://mergiraf.org/ and HN discussion a few years ago: https://news.ycombinator.com/item?id=42093756

I had wanted the same thing for a long time and jj + difftastic has satisfied me.

About the recent track record, is there some technical problem or just drama with Unreal Engine that I'm not aware of? (I already have the same opinion about using AI-coded VCS.)

They are so anti linux that they ship a linux version of Lore.

Yeah, they should have posted it to LoreHub! I just checked for the availability of lorehub.com, and you can buy it for only ~$13,000 and start a competing business to GitHub.

If you've used git+LFS for any extended period of times, you'd know how often it breaks, especially when used with forges like GitHub. Both GitHub and Git treat LFS as an after-thought and second class citizen.

The fact that it's even referred to as git+LFS instead of just git... If I needed to work with large files frequently, I wouldn't want such a basic feature bolted on. Not a criticism of git, just can see why Epic doesn't use git.

I've used git+LFS for unity projects without much issue. If I was working with more than 2 people, I would definitely reach for something centralized.

What sort of scale are we talking about when mentioning tech company's monorepo though?

Lore itself is not an example of a program that meaningfully benefits from any of the key features of Lore.

Lore is meant for situations where your repository is going to contain gigabytes of binary files, such as art assets for games. Git is technically great at everything but that, and even the external solutions for that situation still kind of suck.

This is primarily going to be targeted at Unreal Engine users, for whom the source control tooling they use is the smallest of concerns when it comes to being tied to Epic Games.

I reluctantly agree. I was interested to read the system design doc[1] but it's so many pages long and so full of redundant statements and needless details[2] that I gave up a couple sections in. The numerous "it's not X, it's Y" constructions give the game away. If they can't be bothered to read these docs themselves, why should I?

Anyway it's probably fine software and I am genuinely going to give it a shot for a usecase I have involving large image files. But the LLM-generated docs don't inspire confidence.

[1] https://epicgames.github.io/lore/explanation/system-design/

[2] They literally have a section header "10.1 Revision state as a 320-byte fragment". The byte size isn't even relevant in the code as an implementation detail, much less belongs in a design doc. No one read this doc before publishing it.

Yeah, I agree about the docs. I started on the system design page and my head started to swim after about 5 minutes. So exhausting to read!

On the flip side, I expect the project itself will be workable - well, assuming they're actually using it themselves! UE is a big pile of Stuff on its own, and Fortnite must have god knows how much additional crap in there, so if this is (or will be) their replacement for Perforce internally, then it'll be getting a good deal of testing. (If they're just chucking it over the wall, though... well, sheesh, you first...)

(Perforce is the standard thing for games, and pretty well it works too, and hopefully this will deliver it a well-needed kick. It was sold to private equity about 10 years ago, and it feels like they've been coasting ever since. (Perhaps users in other sectors are happier though?))

Yeah, I hope I'm wrong, but it feels a little bit like "planned project no longer has internal support, let's see if we can make it open-source to garner goodwill and recoup some of our investment." Which isn't the worst thing that can happen.

Git does great with text files, but game development contains a lot of binary assets (textures, videos, 3d models) and correspondingly huge repos. git-lfs tries to patch around that, but that makes a complex tool that creatives struggle to understand even more complex. Perforce is a pretty popular solution, and was used by Epic in the past

> It’s optimized for projects—including games and entertainment—that combine code with large binary assets, and caters for the needs of developers and artists alike.

When you have a game that weighs in at 100GB, only a tiny fraction of that is built from code. The rest of it is binary assets that most VCSs struggle with. What makes this worth considering is the fact that they designed it to handle binary assets.

Apparently, how it handles binaries when developing games.

Why not?

Well it seems to be intended for repos with large blobs like video games so it kind of makes sense that their own pure source code is still managed with git.

Epic's Unreal Engine development is made in Perforce, and then it was mirrored to git/Github. Or that's how it was a couple years ago. That's kind of expected, VCS is one thing, a forge is another thing.

It is not a DVCS.

> 3.2 Explicit non-goals¶

> Peer-to-peer decentralization. Lore is centralized by design. Two clients communicate through the remote, not directly.

My first thought exactly :(

I can’t remember the last time anyone actually played the game they got for free on epic’s store

It has an MIT License