Skip to content
This repository has been archived by the owner on Feb 12, 2024. It is now read-only.

⚡️ v0.41.0 RELEASE 🚀 #2656

Closed
53 of 57 tasks
alanshaw opened this issue Dec 9, 2019 · 6 comments
Closed
53 of 57 tasks

⚡️ v0.41.0 RELEASE 🚀 #2656

alanshaw opened this issue Dec 9, 2019 · 6 comments
Labels
status/ready Ready to be worked topic/release Topic release

Comments

@alanshaw
Copy link
Member

alanshaw commented Dec 9, 2019

One giant leap forward

🗺 What's left for release

🚢 Estimated shipping date

Feb 14

🔦 Highlights

🦁 Async Await and Async Iterables

🎶 In the jungle, the mighty jungle the lion sleeps tonight!
🎶 async await, async await, async await, async await...

We've completed a HUGE refactor to js-ipfs internals 🥳, switching to using Promises and async/await over Callbacks and using async iterables instead of Node.js and Pull Streams. Ok, tell a lie, it's actually not just js-ipfs internals, it's the whole stack, including libp2p, IPLD and multiformats, you might call it a ground up re-write, but, you know, we don't like to brag.

🚨 Oh, wait, serious note - this release brings big breaking changes to the core API so please consult the "API Changes" section below for all the information.

It's been a long and emotional voyage but this refactor brings a plethora of incredible immediate and future benefits to consumers, contributors and core developers that make all the work and effort worthwhile. To summarise, we've:

  • Switched to streaming APIs by default to reduce memory pressure
  • Reduced API surface area by removing buffering, Node.js and Pull Stream APIs
  • Reduced the amount of code in the code base and number of dependencies we depend on (for smaller browser bundles and faster install times)
  • Switched to using async iterables to stream data to help make streaming more approachable
  • Switched to using async/await so we’ll get better error stack traces and improved readability and maintainability

Using async/await in JavaScript is gaining a lot of traction in the ecosystem and is rapidly becoming the de facto way of writing idiomatic JS. We want js-ipfs to move with the times and continue to be attractive to contributors by using modern JS features, techniques and practices. The big idea with these changes is for the code to be easier to contribute to, easier to understand, easier to maintain, and be faster and smaller than ever.

This change is so big, and so significant, we wrote a whole blog post about it to explain the motivations behind the changes...in excruciating delightful depth 🤣!

We've also compiled some stats on this refactor for your viewing pleasure:

...and a lot of those stats are just for js-ipfs and js-ipfs-http-client - the tip of the iceberg! We saw changes similar to this for between 60-70 dependencies across IPFS, libp2p, IPLD and multiformats.

🌗 UnixFS v1.5

Turns out, it's really important for package managers to retain file metadata, particularly last modified time (mtime). File mtime allows them to selectively sync only data that has changed. Up until now if you wanted to host a large data set on IPFS, like a package manager's repository, it would be difficult to update.

"What about the permenant web?" I hear you cry. Well, this absolutely doesn't prevent a particular snapshot of a package manager's repository from being permenantly available. Metadata just enables diffs to be imported, instead of the whole thing. So, when I say "difficult to update", like I did up there, I mean slow and/or impossible. When you have Terabytes (or more) of package data and someone publishes a new package, it's kinda inconvenient to import everything again, when only a little part changed. File mtime is a really good indicator of which things have changed, so you can use it in IPFS now! 🥳

For example, there's two new options to jsipfs add that allow mode and mtime to be preserved as the file is added to IPFS:

$ jsipfs add -r --preserve-mtime --preserve-mode ~/Desktop/gif
added QmT6WX9McZyx5ZoisRgpsjYKDBWnYpMnBLpfAgjW5kavBA gif/yesthisisdog.jpg
added QmXMrFfZ9zHLZKN7xP2dX76YFFhvBJsQkd4fLnTDkyR31Q gif

Ok, no big changes there aside from the new options, buuut, now when you list directory contents you get Mode and Mtime info:

$ jsipfs ls QmXMrFfZ9zHLZKN7xP2dX76YFFhvBJsQkd4fLnTDkyR31Q -v
Mode       Mtime                           Hash                                           Size  Name
-rw-r--r-- Apr 16, 2018, 12:20:33 PM GMT+1 QmT6WX9McZyx5ZoisRgpsjYKDBWnYpMnBLpfAgjW5kavBA 87779 yesthisisdog.jpg

The coolest thing is that it's completely backwards compatible. The CID for a given file/directory only changes if you opt in to metadata, otherwise the CIDs remain the same. Hooray!

There's a bunch of changes that add metadata capability to the CLI, HTTP and core API both for inputs and outputs. There's also a couple of new MFS commands touch and chmod which allow you to change the metadata whenever you like! Magic 🧙‍♂️.

See the API Changes sections below for details of all the new UnixFS v1.5 stuffs.

🏗 API Changes

Core API

There are significant and breaking core API changes in this release. Please see the migration guide.

  • IPFS is not a class that can be instantiated - use IPFS.create. An IPFS node instance is not an event emitter.

  • The init option passed to IPFS.create will now not take any initialization steps if it is set to false. Previously, the repo would be initialized if it already existed. This is no longer the case. If you wish to initialize a node but only if the repo exists, pass init: { allowNew: false } to the constructor.

  • Instance .ready property has been removed. Please use IPFS.create instead.

  • IPFS.createNode has been removed, please use IPFS.create instead.

  • Callbacks are no longer supported on any API methods. Please use a utility such as callbackify on API methods that return Promises to emulate previous behaviour. See the migration guide for more info.

  • Delegated peer and content routing modules are no longer included as part of core (but are still available if starting a js-ipfs daemon from the command line). If you wish to use delegated routing and are creating your node programmatically in Node.js or the browser you must npm install libp2p-delegated-content-routing and/or npm install libp2p-delegated-peer-routing and provide configured instances of them in options.libp2p. See the module repos for further instructions:

  • add now returns an async iterable.

  • add now accepts mode and mtime options on inputs to allow setting mode and mtime metadata for added files. See the core interface docs for more info.

  • add results now contain a cid property (a CID instance) instead of a string hash property.

  • 🆕 add results now include mode and mtime properties if they were set.

  • addReadableStream, addPullStream have been removed. Please see the migration guide for more info.

  • addFromStream has been removed. Use add instead.

  • addFromFs has been removed. Please use the exported globSource utility and pass the result to add. See the glob source documentation for more details and an example.

  • addFromURL has been removed. Please use the exported urlSource utility and pass the result to add. See the URL source documentation for more details and an example.

  • bitswap.stat result has changed - wantlist and values are now an array of CID instances and peers is now a string[] of peer IDs.

  • bitswap.wantlist now returns an array of CID instances.

  • block.rm now returns an async iterable.

  • block.rm now yields objects of { cid: CID, error: Error }.

  • block.stat result now contains a cid property (whose value is a CID instance) instead of a key property.

  • dht.findProvs, dht.provide, dht.put and dht.query now all return an async iterable.

  • dht.findPeer, dht.findProvs, dht.provide, dht.put and dht.query now yield/return an object { id: string, addrs: Multiaddr[] } instead of a PeerInfo instance(s).

  • 🆕 files.chmod has been added. See the core interface docs for info.

  • 🆕 files.flush now returns the root CID for the path that was flushed (/ by default)

  • files.lsPullStream and files.lsReadableStream have been removed. Please see the migration guide for more info.

  • files.ls now returns an async iterable.

  • files.ls results now contain a cid property (whose value is a CID instance) instead of a hash property.

  • 🆕 files.ls results now include mode and mtime properties if they were set. See the core interface docs for more info.

  • files.ls no longer takes a long option (in core) - you will receive all data by default.

  • 🆕 files.mkdir now accepts mode and mtime options to allow setting mode and mtime metadata. See the core interface docs for more info.

  • files.readPullStream and files.readReadableStream have been removed. Please see the migration guide for more info.

  • files.read now returns an async iterable.

  • files.stat result now contains a cid property (whose value is a CID instance) instead of a hash property.

  • 🆕 files.stat result now includes mode and mtime properties if they were set. See the core interface docs for more info.

  • 🆕 files.touch has been added. See the core interface docs for info.

  • 🆕 files.write now accepts mode and mtime options to allow setting mode and mtime metadata. See the core interface docs for more info.

  • get now returns an async iterable. The content property value for objects yielded from the iterator is now an async iterable that yields BufferList objects.

  • id result has changed, the addresses property is now a Multiaddr[]

  • name.resolve now returns an async iterable. It yields increasingly more accurate resolved values as they are discovered until the best value is selected from the quorum of 16. The "best" resolved value is the last item yielded from the iterator. If you are interested only in this best value you could use it-last to extract it like so:

    const last = require('it-last')
    await last(ipfs.name.resolve('/ipns/QmHash'))
  • 🆕 object.get now accepts a timeout option. It will cause the method to throw with a TimeoutError if no data is received within the timeout window. It can be passed as a number or a string. If a number is passed it is interpreted as milliseconds, if a string is passed it is interpreted as a human readable duration.

  • ls now returns an async iterable.

  • ls results now contain a cid property (whose value is a CID instance) instead of a hash property.

  • 🆕 ls results now include mode and mtime properties if they were set. See the core interface docs for more info.

  • pin.add results now contain a cid property (a CID instance) instead of a string hash property.

  • 🆕 pin.add now accepts a timeout option. It will cause the method to throw with a TimeoutError if no data is received within the timeout window. It can be passed as a number or a string. If a number is passed it is interpreted as milliseconds, if a string is passed it is interpreted as a human readable duration.

  • pin.ls now returns an async iterable.

  • pin.ls results now contain a cid property (a CID instance) instead of a string hash property.

  • pin.rm results now contain a cid property (a CID instance) instead of a string hash property.

  • ping now returns an async iterable.

  • refs and refs.local now return an async iterable.

  • 🆕 refs now accepts a timeout option. It will cause the method to throw with a TimeoutError if no data is received within the timeout window. It can be passed as a number or a string. If a number is passed it is interpreted as milliseconds, if a string is passed it is interpreted as a human readable duration.

  • repo.gc now returns an async iterable.

  • stats.bw now returns an async iterable.

  • swarm.peers now returns an array of objects with a peer property that is a string, instead of a PeerId instance.

  • swarm.addrs now returns an array of objects { id: string, addrs: Multiaddr[] } instead of PeerInfo instances.

HTTP API

  • 🆕 /api/v0/add now supports the following additional multipart headers to allow mode and mtime metadata to be set on individual files:
    • mode file mode to apply to created UnixFS entries [string]
    • mtime modification time in seconds before or since the Unix Epoch to apply to created UnixFS entries [number]
    • mtime-nsecs modification time fraction in nanoseconds [number]
  • 🆕 /api/v0/add now returns file Mode, Mtime and MtimeNsecs if set.
  • /api/v0/file/ls has been removed, please use /api/v0/ls instead.
  • 🆕 /api/v0/files/chmod has been added and supports the following query string args:
    • arg path of file to apply mode to [string]
    • mode file mode to apply [string]
  • 🆕 /api/v0/files/ls now returns file Mode, Mtime and MtimeNsecs if set.
  • 🆕 /api/v0/files/mkdir now supports supports the following additional query string args:
    • mode file mode to apply [string]
    • mtime modification time in seconds before or since the Unix Epoch to apply [number]
  • 🆕 /api/v0/files/stat now returns file Mode, Mtime and MtimeNsecs if set.
  • 🆕 /api/v0/files/touch has been added and supports the following query string args:
    • arg path of file to apply mode to [string]
    • mtime modification time in seconds before or since the Unix Epoch to apply [number]
  • 🆕 /api/v0/files/write now supports the following additional multipart headers:
    • mode file mode to apply to created UnixFS entries [string]
    • mtime modification time in seconds before or since the Unix Epoch to apply to created UnixFS entries [number]
  • 🆕 /api/v0/ls now returns file Mode, Mtime and MtimeNsecs if set.

CLI

  • 🆕 jsipfs add [file...] now supports the following flags to respect and apply mode and mtime metadata of files added from the file system or explicitly set them:
    • --preserve-mode automatically apply permissions to created UnixFS entries from the file system [boolean] [default: false]
    • --preserve-mtime automatically apply modification time to created UnixFS entries from the file system [boolean] [default: false]
    • --mode file mode to apply to created UnixFS entries [string]
    • --mtime modification time in seconds before or since the Unix Epoch to apply to created UnixFS entries [number]
    • --mtime-nsecs modification time fraction in nanoseconds [number]
  • jsipfs file ls has been removed, please use jsipfs ls instead.
  • 🆕 jsipfs files chmod [mode] [path] has been added.
  • 🆕 jsipfs files ls now prints file mode and mtime.
  • 🆕 jsipfs files mkdir now supports the following flags:
    • --mode file mode to apply to created UnixFS entries [string]
    • --mtime modification time in seconds before or since the Unix Epoch to apply to created UnixFS entries [number]
  • 🆕 jsipfs files stat now prints file mode and mtime.
  • 🆕 jsipfs files touch [path] has been added and supports the following flags:
    • --mtime modification time in seconds before or since the Unix Epoch to apply to created UnixFS entries [number]
  • 🆕 jsipfs files write now supports the following flags:
    • --mode file mode to apply to created UnixFS entries [string]
    • --mtime modification time in seconds before or since the Unix Epoch to apply to created UnixFS entries [number]
  • 🆕 jsipfs ls now prints file mode and mtime.

Other changes

  • libp2p has been upgraded to 0.27, which also includes breaking changes to it's core API. Please see the release announcement post for more info:
  • The protocol name for peer IDs in multiaddrs has changed from 'ipfs' to 'p2p'. There's no changes to data on the wire but this change is seen when multiaddrs are converted to strings.

✅ Release Checklist

❤️ Huge thank you to everyone that made this release possible

🙌🏽 Want to contribute?

Would you like to contribute to the IPFS project and don't know how? Well, there are a few places you can get started:

⁉️ Do you have questions?

The best place to ask your questions about IPFS, how it works and what you can do with it is at discuss.ipfs.io. We are also available at the #ipfs channel on Freenode.

@alanshaw alanshaw added status/ready Ready to be worked topic/release Topic release labels Dec 9, 2019
@alanshaw alanshaw pinned this issue Dec 10, 2019
@wehriam
Copy link

wehriam commented Dec 10, 2019

What steps are required to define the issue and include a resolution for #2161 in this release? Transmitting large files between browsers seems like a primary use case.

@alanshaw
Copy link
Member Author

With any luck this will be magically be fixed in this version with the new libp2p release which includes libp2p/js-libp2p-mplex#100

I agree this is an important bug to get squashed. If you’re willing it would be good to get your testing when the prerelease comes out.

@alanshaw
Copy link
Member Author

Screenshot 2020-01-24 at 17 53 51

Screenshot 2020-01-27 at 14 21 27

@alanshaw
Copy link
Member Author

alanshaw commented Feb 3, 2020

🥚 0.41.0-rc.0 is out now!

@alanshaw
Copy link
Member Author

🐣 0.41.0-rc.1 is out now!

@alanshaw
Copy link
Member Author

alanshaw commented Feb 11, 2020

🐥 0.41.0-rc.2 is out now with a bitswap fix.

@alanshaw alanshaw unpinned this issue Feb 13, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
status/ready Ready to be worked topic/release Topic release
Projects
None yet
Development

No branches or pull requests

2 participants