Cloud Sync

In the examples we provided below, we use rclone to synchronize the data with Google Drive and Dropbox, and we use CLI to setup cloud sync with AWS.

Google Drive

Prerequisite: Active Google Drive account

NetMind operates at the account level. Users are advised to have a dedicated Google Drive account for their NetMind-related activities, as opposed to using personal accounts. This separation aids in maintaining a clear distinction and enhanced security for NetMind operations.

Install and setup rclone on your rental GPU

Follow the instruction here to install rclone on the SSH instance you rent.

Here we show an example of how to install rclone v1.65.2 to your SSH instance.

// Download rclone
wget https://github.com/rclone/rclone/releases/download/v1.65.2/rclone-v1.65.2-linux-amd64.deb

// Install rclone
apt install ./rclone-v1.65.2-linux-amd64.deb

// Config rclone in "Rent GPU" instance
root@702760ea4881:~/workspace/test# rclone config
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

Enter name for new remote.
name> remote

Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
...
18 / Google Drive
   \ (drive)
...
Storage> drive

Option client_id.
Google Application Client Id
Setting your own is recommended.
See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
If you leave this blank, it will use an internal key which is low performance.
Enter a value. Press Enter to leave empty.
client_id> 

Option client_secret.
OAuth Client Secret.
Leave blank normally.
Enter a value. Press Enter to leave empty.
client_secret> 

Option scope.
Comma separated list of scopes that rclone should use when requesting access from drive.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / Full access all files, excluding Application Data Folder.
   \ (drive)
 2 / Read-only access to file metadata and file contents.
   \ (drive.readonly)
   / Access to files created by rclone only.
 3 | These are visible in the drive website.
   | File authorization is revoked when the user deauthorizes the app.
   \ (drive.file)
   / Allows read and write access to the Application Data folder.
 4 | This is not visible in the drive website.
   \ (drive.appfolder)
   / Allows read-only access to file metadata but
 5 | does not allow any access to read or download file content.
   \ (drive.metadata.readonly)
scope> 

Option service_account_file.
Service Account Credentials JSON file path.
Leave blank normally.
Needed only if you want use SA instead of interactive login.
Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
Enter a value. Press Enter to leave empty.
service_account_file> 

Edit advanced config?
y) Yes
n) No (default)
y/n> 

Use web browser to automatically authenticate rclone with remote?
 * Say Y if the machine running rclone has a web browser you can use
 * Say N if running rclone on a (remote) machine without web browser access
If not sure try Y. If Y failed, try N.

y) Yes (default)
n) No
y/n> n

Option config_token.
For this to work, you will need rclone available on a machine that has
a web browser available.
For more help and alternate methods see: https://rclone.org/remote_setup/
Execute the following on the machine with the web browser (same rclone
version recommended):
        rclone authorize "drive"
Then paste the result.
Enter a value.
config_token> 

Authorization on Google Drive

Install rclone to your own machine that has a browser and do Google Drive authentication.

We are only using Google Drive as an example. You can use a cloud storage service that is more convenient for you to synchronize data in the SSH instance. For more tools, please refer here.

1. Execute "authorize" command in terminal

(base) xxxxxx@xxxxxxxxx rclone-v1.65.2-osx-arm64 % ./rclone authorize "drive"

1. Allow rclone in "System Setting" (if your are using Mac)

1. Sign in your account in browser

Paste the authorize code into SSH instance

config_token> eyJ0...SJ9

Sync files to Google Drive

// Sync folder from Google Drive
root@702760ea4881:~/workspace/test# rclone sync remote:test /root/workspace/test

// Sync folder to Google Drive
root@702760ea4881:~/workspace/test# rclone sync remote:test /root/workspace/test

---

AWS

Prerequisitie: Active AWS account

For NetMind, we advise against using an existing IAM user. As NetMind connects at a user level, it is preferable to create a new IAM user with specific authorizations tailored for the data you plan to store on NetMind servers. This approach ensures more precise and secure data management.

Install Cloud Sync with CLI

root@xxxxxxxxxx:~/workspace/test# pip3 uninstall awscli
...
root@xxxxxxxxxx:~/workspace/test# aws configure
AWS Access Key ID [None]: xxxxxxxxxxxx
AWS Secret Access Key [None]: xxxxxxxxxxxxxxxxxxxxxxx
Default region name [None]: 
Default output format [None]:

Sync Files to AWS

// Sync files from S3
aws s3 sync s3://test0131/ /root/workspace/test

// Sync files to S3
aws s3 sync s3://test0131/ /root/workspace/test