Skip to main content

Idiyanale P1 - Milestone 2

· 14 min read
Daniel Maricic
Adriano Dalpane
info

This article is part of the Web3 Grant Program PR 719 deliverable.

Overview

In the Milestone 1 article we explained the core idea of Anagolay Network and its approach to the rights management, as well as defined the basics of Workflows and Operations. If you need a recap about Anagolay basics, we recommend going back to the Milestone 1 blogpost.

Milestone 2 brings plenty of new stuff: WebSocket microservice, brand new UI for workflow building, two operations, a new publisher job and rust code generation for workflows, deterministic build of WASM artifacts and two demo applications.

New operations

In Milestone 1, we built the op_file, an Operation that reads a file from an URL or a local path and returns its bytes. In this release we introduce two new Operations:

  • op_multihash: accepts bytes as input and produces a hash (bytes) using one of the two hashing algorithms: Blake3 256 or Sha256
  • op_cid: accepts a hash (bytes) as input and produces a string representing the v1 Content Identifier (CID) base32 encoded

The combination of these three Operations is producing a unique identifier (CID) for any file reachable over HTTP (where CORS would allow this).

Workflow -- auditable, transparent, and trusted process

We've learned that each operation is a self-containing WASM file and users can execute them manually and in any order, given that they manually serialize and deserialize the inputs and outputs. This approach is good for prototyping but not when we require code reusability and above all if we care about sharing the process with others.

This is where the Workflow comes in. We can use the web app to create the Workflow manifest. The Workflow manifest is just a JSON file containing the information about the Workflow and how it is executed and it's useless without the generated source code (rust) and built binaries (WASM).

Workflow manifest​

The Workflow and Operation implement the [AnagolayStructure](https://ipfs.anagolay.network/ipfs/bafybeicjwe3dc7bg3fqwcwb535qqrjlldnxg5tk5iceyxg75fqvufcqpyu/anagolay_support/struct.AnagolayStructure.html) struct, which means that both have the IDs that are content-sensitive and for the same manifest we will always ge the same ID. This is one of the assurances that there can be only ONE thing workflow doing exactly ONE thing in ONE specific way.

The manifest looks like this:

  • id: workflow_cidv1(data)
  • data: the structure payload, namely:
    • name: name of the Workflow transformed to lowercaseWithUnderscore(). doc
    • creators: a list of identity identifiers, currently they are limited to substrate-based addresses. doc
    • description: a short description of the Workflow. doc
    • groups: Tells which groups the Workflow belongs to. doc
    • segments: a list of Segment definitions

Most of the fields are self-explanatory and easy to understand. The segments need a little bit more explanation; they are the execution order. Every time when we start the Workflow, the first segment with -1 in the input field gets called with the data we pass in. It will keep executing the same segment until it reaches the end, then it will either wait for the user input or exit because there are no more segments to run. What gets executed in the segment is the Operation source code (rust) which is compiled together with other Operations in the same segment.

All Operations which don't require any additional input but the output of the previously executed Operation can be executed sequentially in a Segment without any user intervention and without the need to cross the WASM boundary to propagate the previous output toward the next input. This approach maximizes the efficiency and increases the performance which is the most visible in the Browsers and smartphones due to the limited resources.

Once a Segment finishes the execution, its result is used as an input to next Segment or Segments and so on until we reach the last segment and print out the result. Here is an example of a complex workflow which is used to prove certain data via the QrCode:

PoCLO workflow with Segments

The data flow from top to bottom: at the top, the user inputs data, and at the bottom the final result of execution. In the example, Segments are numbered in one potential order which is stored and cannot be changed. The execution always starts from the least dependent Operation, for example, it cannot start from Segment 5 since its dependent on the result of Segment 4 and Segment 3. The segments are in an executable dependency tree:

PoCLO Workflow can also be represented by the Segment dependency tree.

It appears clear now that one possible order of execution would be: 1,2,0,4,3,5 - even though it's not the only possibility. This sequence is described in the Workflow manifest and is always exactly reproduced.

WASM bindings​

Workflows have a WASM binding, just like Operations do. While possible, executing manually Operation after Operation through their WASM interface, in the order that satisfies their dependencies, serializing every output, and deserializing every input is not only cumbersome and inefficient but also repetitive and error-prone. This is why Workflows exist, reducing boilerplate, improving the developer experience, and better performance.

ℹ️ There is no WASM boundary crossing in executing operations of the same segment, and the Segment result is deserialized for the caller only at the end of the Workflow.

The native Workflow interface and its WASM binding expose the following methods:

  • new(): creates a new instance of the Workflow, initializing its state
  • next(): accepts the external inputs and invokes the execution of the next segment. Only user (or external) inputs are needed as parameters; the input coming from previously executed Segments is known in the Workflow state and is handled automatically

Execution​

It may help to think about a Workflow as an application of the generator pattern. According to this definition, every call to next() returns an object with the following properties:

  • done: boolean that indicates if the Workflow execution has been completed
  • output: only available in the last Segment execution, when done is true since producing a result every call to next() implies a performance penalty
  • segmentTime: performance measurement of the time taken to execute the segment
  • totalTime: performance measurement of the time taken to execute the Workflow up to the current Segment

ℹ️ In Rust, these fields are exposed as getters from the interface SegmentResult. In order to deal with the type variance in input and outputs, Rust makes use of type Any, whose reference can be downcast to the expected type.

Code to execute the Workflow can be written in JavaScript (left) and in Rust (right). These examples are taken from the [Deliverable Support repo](https://github.com/anagolay/w3f-grant-support-repo/tree/project-idiyanale-phase1_milestone-2)

Code to execute the Workflow can be written in JavaScript (left) and in Rust (right). These examples are taken from the Deliverable Support repo

Anagolay blockchain

In the previous article, we've introduced a couple of pallets:

  • anagolay-support: provides common functionalities like types definition, content identifiers computation, and artifact registry
  • operations: provides the extrinsic API to publish an Operation in its initial Version

We now have a new pallet that, in the future, will more broadly cover the scope of managing and maintaining Workflows:

  • workflows: provides the extrinsic API to create a Workflow in its initial Version

Versioning​

Operation code, as all source code, evolves; to fix bugs or to update dependencies. For this reason, it's not an Operation that is part of a Workflow, but one of its Versions (typically the latest). This concept of incremental versioning cascades to Workflows when it comes to updating the versions of the Operations that compose it, so both entities are now versioned on the blockchain and they can share the same artifact registry.

The required steps for Workflow creation are:

  • run the Workflow Builder UI, get the workflow manifest, and the build information
  • from a Workflow template, generate the source code in rust containing the Operation Version
  • store the source code and the Workflow WASM artifacts on IPFS
  • with the manifest and the version, use the Anagolay workflow.create an extrinsic to store the new Workflow

The Workflow Version manifest, and as said above the Operation Version manifest too, look like this:

  • id = workflow_cidv1(data)
  • data
    • entityId — in this case, it's a Workflow ID; can be queried in the workflow pallet
    • parentId — present only in case of a non-initial (improved) implementation
    • artifacts — a collection of artifacts
      • artifactType — check the WorkflowArtifactType for a full list of the artifact types
      • fileExtension — indicates the extension of the file that will be downloaded
      • ipfsCid— and IPFS CID artifact address
  • extra — a key-value pair for adding extra fields that are not part of the data. In our case, it is the createdAt because recalculation of the CID is not possible in the verification state, since the time never stops, as far as we know it always goes forward.

The manifest data contains the entityId, this makes sure that the association between an entity and its version is immutable and uniquely identifiable.

Publish Service

This micro-service is our custom build system which prepares, builds, and publishes the Operations and Workflow. It does not do any extrinsic calls to the Anagolay network, this is done in the CLI as a part of either operation publish or workflow create. It exposes a couple of API endpoints protected by auth mechanism and it's not publicly available.

The implementation has been improved to deal with the build of a Workflow and these are the steps of publishing it:

  • clone the git repository containing the template of a Workflow crate
  • a template engine uses the Workflow manifest and the build information to produce the source code that executes the chosen Operations in Segments
  • store the generated Workflow git repository on IPFS
  • build the Workflow code and produce WASM artifacts
  • store the artifacts on IPFS
  • cleanup the working directory
  • return the IPFS CID of every hosted content

Anagolay CLI

The starting point to create a Workflow in the Anagolay Network is the CLI. The new command is anagolay workflow create which is the entry point for the workflow creation. The purpose of the create command is to give a user a unique link for workflow creation. This way we have solved the man-in-the-middle attack where 2 consecutive runs of the create command will produce different workflow URLs and sandbox the code build and extrinsic call.

The UI prevents the user to generate invalid workflows, avoiding the following situations:

  • Workflows that have more than one entry point
  • Operations connected in a cyclic path
  • Mismatching the output type of the previous Operation and input type of the next one
  • Fields of the manifest like name, description, or groups are not filled correctly

Playground

This second milestone intends to release an environment that "just works" so that anybody can get to run the prototype and experiment with it. Therefore we provide a VSCode devcontainer and gitpod setup that will take care of launching the backends, along with the simple example of execution of a workflow directly from Rust. To sum up, two scenarios are covered for the workflow:

  • execution
  • creation

There is more information in the Playground README.md file than what's provided here, be sure to check it out. We recommend going and checking it out and playing around → https://github.com/anagolay/w3f-grant-support-repo/tree/project-idiyanale-phase1_milestone-2

Building the Operations and Workflow

To run the demos, you need the generate the source code and artifacts. If this would be a real-world scenario, you would get them via a package manager, since this is a self-containing support repo, you need to build them yourself.

I used the gitpod but you can use the devcontainer, the workspace paths will be different, the devcontainer will have /workspace instead of /workspace/w3f-grant-support-repo.

For the simplicity of the executions, I will add only outputs in the correct order for you to see and execute the first lines.

re-build op_file

gitpod /workspace/w3f-grant-support-repo/operations/op_file (main) $ anagolay operation publish
✔  success   Sanity checks, done!
  ◝ Checking if the remote job is done. This can take a while.
ℹ  info      Connected to Anagolay Node v0.3.0-0bd52ee-x86_64-linux-gnu
? Which account do you want to use to sign the transaction? Use Alice
> TX is at blockHash 0x50f4616f27f3a83f877d80bcb9e7b6c4e4a33c4af915ac18f743ff27c7126e3b
> Manifest ID is bafkr4igxckwhvpd47nrhdjbdun3wrw24cnhrodnvydxxo27bdifep5dr7q
Artifacts and their types.
┌─────────┬──────────────────┬───────────────────────────────────────────────────────────────┐
│ (index) │       type       │                              cid                              │
├─────────┼──────────────────┼───────────────────────────────────────────────────────────────┤
│    0    │  { Git: null }   │ 'bafybeic56c277zlm543hkptt55rmdtsg6nfjlnvezj62fokhxmggqobxq4' │
│    1    │ { Wasm: 'Esm' }  │ 'bafybeifio5pvhwrzwosro6lenjll7thnikmxmqvjnzstz2vwlpv2k2qsne' │
│    2    │ { Wasm: 'Web' }  │ 'bafybeieusvzktswmwzag3ulu7ulbsf6mls6zopbqcjdsvity4dbeumcexe' │
│    3    │ { Wasm: 'Cjs' }  │ 'bafybeigi6hhz6gxv3elmxdcfqjhcaqvjvfzgwvsj6hy77ulef2udky6z2q' │
│    4    │ { Wasm: 'Wasm' } │ 'bafybeibrod5nsubitx5v4uolnts4qu5fgrcuhrbfkum3f664vtim53anjy' │
│    5    │  { Docs: null }  │ 'bafybeibtseepqurn3l4hizsibenovmcvdqdambziggqh2ods7ty2iaa32y' │
└─────────┴──────────────────┴───────────────────────────────────────────────────────────────┘
Total execution elapsed time: 4:47.043 (m:ss.mmm)
✔  success   DONE 🎉🎉!

build op_multihash

gitpod /workspace/w3f-grant-support-repo/operations/op_multihash (main) $ anagolay operation publish
✔  success   Sanity checks, done!
ℹ  info      Connected to Anagolay Node v0.3.0-0bd52ee-x86_64-linux-gnu
? Which account do you want to use to sign the transaction? Use Alice
> TX is at blockHash 0x65b7de2a0a0bf7b049b5c2b306e3df6ccf64baaf71f3c02c78d284c35e083cc6
> Manifest ID is bafkr4id2aod4g3vg3b5exzi2rorvu44my63o6fcjfqihydkvcdrsd33hlq
Artifacts and their types.
┌─────────┬──────────────────┬───────────────────────────────────────────────────────────────┐
│ (index) │       type       │                              cid                              │
├─────────┼──────────────────┼───────────────────────────────────────────────────────────────┤
│    0    │  { Git: null }   │ 'bafybeifdnvwcj6lxnyfjytief3ftvcqqavwel3bfgf7ee3o4norawp5rne' │
│    1    │ { Wasm: 'Esm' }  │ 'bafybeiatou2kkkzrzxzidbz7y2mho2y7md7csbg7ufm5tu32icy45qwuda' │
│    2    │ { Wasm: 'Web' }  │ 'bafybeigus5n53n7ebv22jzlkamfbnsrew3nx5smxdwhdnu63abu5op4yru' │
│    3    │ { Wasm: 'Cjs' }  │ 'bafybeiggb2q4vt2oj3vyqrl4x2ax6bumrumkxos2vfl2lwwiawf6dt26ma' │
│    4    │ { Wasm: 'Wasm' } │ 'bafybeibn7lffaeqhln3i7eumkepwmkxgb5cisfpfnva3uo4m2nxmr66ln4' │
│    5    │  { Docs: null }  │ 'bafybeigzkhk44xchamjsgtpmtigorkwh3q2efr4ud6vbelooweoff42uae' │
└─────────┴──────────────────┴───────────────────────────────────────────────────────────────┘
Total execution elapsed time: 2:51.183 (m:ss.mmm)
✔  success   DONE 🎉🎉!

build op_cid

gitpod /workspace/w3f-grant-support-repo/operations/op_cid (main) $ anagolay operation publish
✔  success   Sanity checks, done!
ℹ  info      Connected to Anagolay Node v0.3.0-0bd52ee-x86_64-linux-gnu
? Which account do you want to use to sign the transaction? Use Alice
> TX is at blockHash 0xce44d350cf94ed84d55428c0f9e62ac9afc509b7a462707a2908641c89fb34f6
> Manifest ID is bafkr4if466wjv6qrwp7gppvobhaa5hwahvowu2ox2cppydykx6h3ygqz5i
Artifacts and their types.
┌─────────┬──────────────────┬───────────────────────────────────────────────────────────────┐
│ (index) │       type       │                              cid                              │
├─────────┼──────────────────┼───────────────────────────────────────────────────────────────┤
│    0    │  { Git: null }   │ 'bafybeib4em2zvtqjsrrd2hg2n2ms4g7efmrmod3nilwalroj3dzl47cnea' │
│    1    │ { Wasm: 'Esm' }  │ 'bafybeiaaeanc56ebhxj55rgvasmrkufv4pb2ih6zm224s4plexcnvgugdq' │
│    2    │ { Wasm: 'Web' }  │ 'bafybeihtencrpcdgif4dp3z2uwtdwa6nf3bnrozuhm3njk3shx3a3jxi3y' │
│    3    │ { Wasm: 'Cjs' }  │ 'bafybeib2kzhoaori4vkogjiwigzp4ilwxpjm4yzxm4c4moj5gej2wzgj34' │
│    4    │ { Wasm: 'Wasm' } │ 'bafybeicrv7zh5guoztp2be4hl45xu77ijbabet6frzlesgbwywxoz4pitu' │
│    5    │  { Docs: null }  │ 'bafybeidwtifwwtr344kywifknwurbh5lieh27cz6rshawd5kmuilir6kxy' │
└─────────┴──────────────────┴───────────────────────────────────────────────────────────────┘
Total execution elapsed time: 2:40.845 (m:ss.mmm)
✔  success   DONE 🎉🎉!

Now when you have operations, it's time to build the Workflow.

In any directory run anagolay workflow create, then click on the link to get to an UI builder:

Anagolay Workflow UI

Use this for new workflow:

name        = Workflow Compute CIDv1
description = Generic CIDv1 computation of any data. Use base32 encoding with Blake3-256 hasher.
groups      = Generic, SYS

Connect the operations like on the image then click save. Check the terminal you will see the saving the workflow to the chain. The output should be similar to this:

Workflow Compute CIDv1

It's very important to note that all WASM artifacts are deterministic for a given manifest. The only one that it's not is the git artifact because the git commit messages contain the created time which breaks the deterministic build. This means that if you try to build the workflow with the above data on the same machine as we did ( this support repo ) you must get the same CIDs for WASM artifacts. Code assurance at its best!

ℹ️ Note that the ALL artifacts will be the same in your build. This is the 100% source code and execution code assurance we are bringing to the proof verification and creation.

Useful Links

All Anagolay code is open source and can be found in the following repositories:

Get in touch with us

Join us on Discord, Twitter, or Matrix to learn more and get our updates.

Want to join the team? See our Careers page.