Skip to content
This repository was archived by the owner on Apr 16, 2020. It is now read-only.
This repository was archived by the owner on Apr 16, 2020. It is now read-only.

[EPIC] Towards better file system APIs #71

@djdv

Description

@djdv

If we wish to support file system based package managers, we'll need to accommodate a range of existing expectations, and hopefully exceed them.
In general, there should be an easy (and ideally transparent/drop-in) way to both add package repositories into IPFS, and get packages out. From the perspective of both maintainers and users.

Getting to that point will require work in multiple areas to cover multiple use cases.
Sections of the work done here: #67
will likely become relevant, as we find out what specific issues are present today, and how we may alleviate them.

For example, using data on IPFS with existing tools will certainly involve mounting IPFS and interfacing with APIs like FUSE.
Creating new tools, such as a repo syncing tool, IPFS package manager PoC, etc. on top of existing APIs, may prove challenging for certain workloads, or at the very least, contain a lot of overlap. So we should find ways to improve, or supersede these APIs.
In the cases where existing things are fit for certain tasks, it may still be hard to make them work together. So we should find better ways to interoperate.

Short term, we can collect and discuss known problem points that are relevant to package management. What exists today, and why it is/is not viable?
e.g.

  • MFS; for importing and working with new or existing datasets
    • {conceptually} deals with an isolated filesystem, using a unix-like (subset) specification
      • core implementations should try to unify namespace handling so things like ipfs files ls go get obviated by ipfs ls; (at an API level as well)
    • {go-mfs} suboptimal performance
    • {go-mfs} implementation is not well understood
  • ipfs mount; for interacting with IPFS using existing utilities
    • {go-ipfs} suboptimal performance
    • {go-ipfs} lacks write support
    • {go-ipfs} Currently only deals with 2 namespaces/APIs (IPFS, IPNS, in a rigid way)

...

Long term, we can build towards a virtual file system interface (VFS), that supersedes MFS and better integrates with other APIs that may be relevant for filesystem based operations (such as UnixfsV2).

In between, I plan to work towards creating and maintaining an experimental VFS API that provides a means of interacting with IPFS (using new methods, and our newer APIs).
As well as using this API to provide an experimental version of ipfs mount that should aid us in testing and development. Particularly, this should help nail down a common set of file system expectations that IPFS implementations provide to developers and users.

This draft is a good starting reference: ipfs/kubo#6036
recapping work that's already been done on this effort.
Recent talk around this was done at IPFS camp, notes are here: https://github.com/ipfs/camp/blob/master/DEEP_DIVES/31-mounting-an-ipfs-filesystem.md

As we proceed, these goals will have to be better divided up and defined. Expect smaller issues to pop up around this, in this repo, and linked back here as work is done on them.
(in a simillar fashion to: ipfs/kubo#5003)

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions