Introduction

Author: Matt Henderson @dafacto

This document details migrating from a public/private Spacemesh node setup to a single node managing four postdata sets, using the new “1:n” feature of go-spacemesh version 1.4, on a 2023 Mac mini.

Preparation

Spacemesh wizard @Earl in Discord recommends backing up the following before migrating from version 1.3 to 1.4:

• from post-data:
  • key.bin
  • nipost_builder_state.bin
  • postdata_metadata.json
• from data-dir (node):
  • node_state.sql
  • p2p.key
• config.json

Overview of the process

Here’s an outline of the process, followed by the contents of my specific configuration files. (The Spacemesh documentation for all the below can be found here, though I found it a bit difficult to follow, as it’s more generic in nature.)

1. Quit any running go-spacemesh nodes.
2. Download and install the latest 1.4+ release binaries. Note that the release includes both the go-spacemesh binary that runs your node, and the post service binary that manages your postdata.
3. Add grpc-post-listener on port 9094 in your startup scripts, modifying the port of any other listeners that conflict with that port. This allows go-spacemesh to communicate with post service processes, each of which does the proving for a particular postdata set.
4. Modify your poet-server configurations to include “pubkeys”. (See examples below.)
5. Remove any direct private peers from your public node configuration, if you’re migrating from a private/public node setup like me.
6. Configure your public node configuration to not smesh! (See examples below.)
7. Configure each private node to use your public node configuration. (This in preparation for the next step.)
8. Start every node you have, running 1.4, and let it sync. This will:
   1. Migrate the node to 1.4, creating some new files, new databases and run some migrations on others. It’s important that each node has been migrated to 1.4 before running the merge tool discussed in a later step.
   2. It will also backup your key.bin file in your postdata folder to key.bin.bak, move and rename the original to local.key in a new identities folder in your node’s node-data folder.
9. Shutdown each node, find all those local.key files in those identities folders, and rename them to something corresponding to your particular nodes. In my case, I ended up with node1.key, node2.key, node3.key and node4.key. Important!—if you leave your main node key named local.key the migration will not work!
10. Use the new merge-nodes utility delivered with go-spacemesh to merge all your nodes into one, e.g. Here I’m merging node2 into node1: ./merge-node --from ~/Spacemesh/node2/node-data --to ~/Spacemesh/node1/node-data. This will move/consolidate all those node.key files into the identities directory of your future single node. (I used node1 as my eventual master, single node.)
11. Modify your node startup script to remove all smeshing parameters, since the node itself will no longer be smeshing, but rather the multiple service processes you’ll run. (See below.)
12. Create multiple startup scripts to start a separate services process for each postdata set you have. (See example below.)
13. After confirming you’ve made all the right configuration changes, you can start go-spacemesh on a single node, and then start your four service processes.

At this point you’re finished! You have one node running, communicating with multiple service processes, smeshing multiple postdata sets!

Let’s now look at my specific configurations, so you can see some real-world examples.

Configuration files

start-node.sh

This is the script that I run to start my node. You’ll see that instead of leaving it named node1, I simply renamed it node.

• Note the removal of all smeshing related parameters, and the important addition of the --grpc-post-listener parameter.

#!/bin/bash

/Users/mhenders/Spacemesh/app/go-spacemesh \
--config /Users/mhenders/Spacemesh/node/config/config.json \
--listen /ip4/0.0.0.0/tcp/7515 \
-d /Users/mhenders/Spacemesh/node/node-data \
--filelock /Users/mhenders/Spacemesh/node/lock/spacemesh.lock \
--grpc-public-listener 0.0.0.0:9092 \
--grpc-private-listener 0.0.0.0:9093 \
--grpc-post-listener 127.0.0.1:9094 \
--grpc-json-listener 0.0.0.0:9095

go-spacemesh config.json

• Note that I use Team24’s poet servers.
• Note the important addition of the "smeshing-start": false parameter, so that the node itself isn’t smeshing.
• The "post-k3": 1 configuration is to enable distributed ATX verification.

{
  "main": {
    "layer-duration": "5m",
    "layers-per-epoch": 4032,
    "poet-servers": [
      {
        "address": "https://poet-1.team24.co",
        "pubkey": "AHlrgUZq5dFqK2oa2C9qVq4qIE2beJ/mG7sryynfqx0="
      },
      {
        "address": "https://poet-2.team24.co",
        "pubkey": "8SatjpHxbrfeIKTFSAFcZklxPnhUElloRvAUAy1xOxo="
      },
      {
        "address": "https://poet-3.team24.co",
        "pubkey": "givyUYkXx4kVe7gOngNEeIA4YrNBPojB7lAH1RLA62c="
      },
      {
        "address": "https://poet-4.team24.co",
        "pubkey": "C7/aEeZ716W17Z/Gz1GHtCVEcK8IN2JXrhVyIF43aEA="
      }
    ]
  },
    "logging": {
        "p2p": "error"
    },
    "poet": {
        "phase-shift": "288h",
        "cycle-gap": "24h",
        "grace-period": "2h"
    },
    "p2p": {
        "min-peers": 30,
        "low-peers": 60,
        "high-peers": 100,
        "bootnodes": [
            "/dns4/mainnet-bootnode-14.spacemesh.network/tcp/5000/p2p/12D3KooWRkZMjGNrQfRyeKQC9U58cUwAfyQMtjNsupixkBFag8AY",
            "/dns4/mainnet-bootnode-16.spacemesh.network/tcp/5000/p2p/12D3KooWDAFRuFrMNgVQMDy8cgD71GLtPyYyfQzFxMZr2yUBgjHK",
            "/dns4/mainnet-bootnode-18.spacemesh.network/tcp/5000/p2p/12D3KooWMJmdfwxDctuGGoTYJD8Wj9jubQBbPfrgrzzXaQ1RTKE6"
        ],
        "direct": []
    },
    "smeshing": {
        "smeshing-start": false,
        "smeshing-coinbase":"{address}"
    },
    "post": {
        "post-k3": 1
    }
}

start-service1.sh

Here’s one of the four start-service.sh scripts, that start four post services, one for each postdata set I have.

• Note the operator-address parameter, that will let you monitor the post service on the local web URL http://127.0.0.1/50051/status. Of course, you need to change the port to something unique for each service. I used 50051 through 50054.
• Note that threads have been set to 0, which let’s the operating system’s scheduler figure out how many threads to assign to each process.
• Note that I kept 288 nonces from my previous setup. (There is a complicated profiler process if you want to try to optimize threads and nonces.)

#!/bin/bash

/Users/mhenders/Spacemesh/app/service \
--threads 0 \
--address=http://localhost:9094 \
--dir /Volumes/smesh1/postdata \
--nonces 288 \
--operator-address=127.0.0.1:50051

LaunchAgents

On the Mac, you need to create a LaunchAgent, which goes in ~/Library/LaunchAgents/, to control your post services, just like you should have one to control the launch of go-spacemesh. This will launch them all on machine startup, and relaunch them if they ever crash. This is similar to systemd on Linux machines.

You pretty much need to use the LaunchControl app for Mac, to load and run your launch agents, making the process much easier than manually managing them.

I have four service launch agents, one for each start-service.sh script.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>KeepAlive</key>
    <dict>
        <key>Crashed</key>
        <true/>
    </dict>
    <key>Label</key>
    <string>com.spacemesh.service1</string>
    <key>Program</key>
    <string>/Users/mhenders/Spacemesh/node/start/start-service1.sh</string>
    <key>RunAtLoad</key>
    <true/>
    <key>StandardErrorPath</key>
    <string>/Users/mhenders/Spacemesh/node/logs/error-service1.log</string>
    <key>StandardOutPath</key>
    <string>/Users/mhenders/Spacemesh/node/logs/service1.log</string>
</dict>
</plist>

Monitoring

I use the Mac app Keyboard Maestro to monitor my setup once per hour. Here’s what that looks like, collapsing the macro contents into the highest level groups:

• Version check /Users/mhenders/Spacemesh/app/go-spacemesh version
• Sync status /opt/homebrew/bin/grpcurl --plaintext -d "{}" localhost:9092 spacemesh.v1.NodeService.Status
• Presence of Smesher IDs /opt/homebrew/bin/grpcurl --plaintext localhost:9093 spacemesh.v1.SmesherService.SmesherIDs
• Check that the post services are alive http://127.0.0.1:%PostPort%/status where %PostPort% is the http port on which the particular process was configured. (Here’s the official documentation that contains all the status responses you might receive.
• Note activity of each post service /opt/homebrew/bin/grpcurl --plaintext localhost:9094 spacemesh.v1.PostInfoService.PostStates
• Confirm postdata volume is mounted. (I use a Keyboard Maestro specific macro function to confirm this.)

This Keyboard Maestro macro writes all monitoring data to a file that I can inspect at any time, and is configured to email me if anything is wrong.

Conclusion

Here’s what my directory system looks like, after finishing this migration.

Thanks to @nj, @fastmat, @flare, @hakehardware, @earl and others for helping me get all this figured out!

If you have any questions, I’m @dafacto in Discord, or ping me on Twitter.