Storj
Storj is a decentralized peer-to-peer file storage system, and new protocol being added to FileZilla.
Source code is available at https://svn.filezilla-project.org/filezilla/FileZilla3/trunk and depends on https://docs.storj.io/dcs/api-reference/storj-client-libraries
To check out the code with subversion:
svn co https://svn.filezilla-project.org/svn/FileZilla3/trunk filezilla
And enable using:
./configure --enable-storj
Quick setup
- Add a new site in the Site Manager and select Storj as the protocol
- Enter gateway.us1.storjshare.io as the host and 443 as the port
- Note: us1, eu1, and ap1 are the current gateways. Use the closest gateway.
- Enter the bridge user name and password
- If you have not created an Encryption Key before, generate a new one, otherwise add your existing key
- Connect to the server and begin uploading and downloading files
Introduction
FileZilla is a fast and reliable cross-platform FTP, FTPS, SFTP and Storj client with many useful features and an intuitive graphical user interface. A server-client is also available.
Below are some of the FileZilla client features:
- Easy to use
- Supports Storj, FTP, FTP over SSL/TLS (FTPS) and SSH File Transfer Protocol (SFTP)
- Cross-platform. Runs on Windows, Linux, *BSD, Mac OS X and more
- IPv6 support
- Available in many languages
- Supports resume and transfer of large files >4GB
- Tabbed user interface
- Powerful Site Manager and transfer queue
- Bookmarks
- Drag & drop support
- Configurable transfer speed limits
- Filename filters
- Directory comparison
- Network configuration wizard
- Remote file editing
- Keep-alive
- HTTP/1.1, SOCKS5 and FTP-Proxy support
- Logging to file
- Synchronized directory browsing
- Remote file search
FileZilla incorporates the Storj asynchronous multi-platform C library for encrypted file transfer on the Storj network into its back end to ensure the highest level of performance and stability. This library is also the back end to the native libstorj CLI.
In this guide, we will run through the process of setting up FileZilla to be able to up and download painlessly to the Storj Cloud.
Setting up a Storj account
Before we can start uploading and downloading to the Storj network, it is necessary to open an account on a Storj Satellite. The Satellite acts as an interface to the Storj network and manages all the complicated stuff in the background, making it completely painless for a user to start using the Storj network.
Registering an account via Satellite (STORJ Signup)
If you already have a Storj account, you can proceed directly to section 3.2.
To register a new account, click on the Sign Up link (Figure 3.1). Once on the registration page, enter your email address and desired password, then click the check box to agree to the terms of service and finally click on "Sign Up" (Figure 3.2).
Now check your email inbox and click on the account activation link sent to you by Storj. A web page should open with the account activation confirmation. Note that sometimes the email automatically gets moved to the spam/trash folder by some email providers, so if the email is not in the inbox, make sure to check the spam folder. Now that the Satellite account is created, you can login (Figures 3.1) and you will be presented with your Satellite dashboard page as show in Figure 3.3.
Configure billing to upload/download beyond the free tier limit
Currently the free tier for Storj renters is structured as follows:
- Uploading limit: 50GB/month.
- Downloading limit: 50GB/month.
(Data obtained from: Storj Billing. If you want to upload or download beyond the free tier, a payment option can be added to the Billing tab of your account on storj.io.
Payments options currently include:
- STORJ token (this currently needs to be requested by email to hello@storj.io, automatic deposit option will be added later to the GUI)
- Bitcoin
- Credit Card
Once your billing information is added, the free tier limit will be lifted and you will now be able to upload and download as much as you like/can afford!
These are the current storage and bandwidth prices (they apply only after you exhausted your monthly free tier limits):
- Storage -> $4 per TB per month
- Download Bandwidth -> $7 per TB downloaded
- Upload bandwidth -> free
This information was extracted from pricing.
Important things to note regarding the billing structure:
- You only pay for the time span that your data was stored on the network, if you delete some of your files today you will not pay for them anymore tomorrow, you will stop getting charged for storage right after you delete the files.
- Your storage usage is added daily to your "Current Usage" in the billing section. However, if your billing method is by credit card, you will only be billed once per month (normally on the 22nd of every month). For pre-paid billing methods like STORJ token or BTC, your pre-paid balance will be reduced daily according to your usage but again actual amount deducted from your balance will be reconciled at the end of the billing cycle taking into account your free tier allotment.
- The free tier allotment is subtracted from your paid tier every month, thus if you upload 100GB in a given month, you will only pay for 75GB (100GB – 25GB free tier) at the end of the billing cycle.
Configure FileZilla so it can be used with Storj
After the main FileZilla client window opens do the following:
- Click on
File->Site Manager...
- In the resulting Site Manager Popup Window, first click on the
New Sitebutton at the bottom left-hand side. You canRenamethe site name to a name of your liking, for exampleStorj. Then in theGeneraltab on the right side, forHost:enterapi.storj.io, forPort:enter443and forProtocol:selectStorj - Decentralized Cloud Storage.
- In the
Userfield enter your api.storj.io email address. In thePasswordfield enter the matching password to the previous email address.- If you are new to Storj click on
Generate...to the right of theEncryption keyfield. This will generate an encryption key which will be used to encrypt all the data you upload to Storj, including all the folders. This key will also be used to decrypt the data you download from your Storj account. Be sure to write down the encryption key in a very safe place as you will lose access to your uploaded data if you don´t have the encryption key. - If you were previously running libstorj see the next chapter to export the encryption key from libstorj. Once the encryption key is exported, you can paste it into the
Encryption keyfield in FileZilla. This way you will be able to access any files you previously uploaded to Storj with libstorj. The same applies if you were using core-CLI or any other toolkit to upload to Storj.
- If you are new to Storj click on
- On the bottom right, there is a space to add
Commentsto the account which is particularly handy if you want to add more accounts to FileZilla. - Finally, click on
Connect. FileZilla Should now connect to your account. If all goes well and the login details are correct, no errors will be returned and the Site Manager will close (Figure 4.3).
Note: If already added any buckets (folders in FileZilla) to you account with another application, you will get the following error:
Error: Wrong encryption key for at least one bucket
You do not have to worry about this error, it just means you cannot access one of the folders because you do not have a valid decryption key for it. In certain use cases this is actually desirable because sometimes you want to keep the file encryption key private for a certain machine/environment.
Note: It is absolutely vital that you backup the encryption key to a safe location and preferentially offline. Printing the key on paper is a very secure common practice.
Export encryption key from libstorj
If you were previously running libstorj, you can export the encryption key by executing the following command (Figure 4.4):
storj export-keys
Now copy the Encryption key string exported from libstorj and paste it into the Encryption Key field in the FileZilla Site Manager menu.
Secure the FileZilla keys with a master password
To protect the encryption keys and bridge password, it is advisable to encrypt these passwords with a top-layer master password (and/or use full disk encryption and home directory encryption). This can be done as follows:
- . In the FileZilla top menu, click on
Edit->Settings... - . Once in the
Settingsmenu, click onInterface. - . Under
PasswordsselectSave passwords protected by a master password. - . Now enter a password of your choice (preferentially a long one), repeat the password for confirmation and click "OK".
Done!. Now in case your computer gets compromised, it will be impossible for the hacker to retrieve your bridge keys and encryption key without using a key logger while you're unlocking the password.
Adjust simultaneous transfers and timeout details
We can adjust the number of simultaneous transfers to a value from 1 to 10. Setting this value to 10 allows maximum bandwidth utilization during transfers of a large number of files, both for uploading and downloading.
This can be done as follows:
- In the FileZilla top menu, click on
Edit->Settings.... - Once in the
Settingsmenu, click onTransfers. - In the
Transferssection set theMaximum simultaneous transfersto a number of your choice between 0 and 10.
Note: You can also set transfer speed limits in KiB/s for both downloads and uploads in this settings menu. However, this isn't currently available with Storj, please watch https://github.com/Storj/libstorj/issues/9 for when it's added.
Timeouts can occur at the default timeout value. We can adjust this value upwards to where timeout errors no longer occur. In the example below the timeout was set to 60 seconds. This will affect all commands except for uploads and downloads.
Note: The Maximum number of retries is also a useful setting to adjust in case downloads or uploads fail after the first 1-2 retries.
Creating a folder, uploading and downloading files
This step is very easy. Once connected to our account, there are two main sections of the GUI, namely:
- Local site (your computer) - Left pane
- Remote site (the Storj cloud) - Right pane
If you already have some folders (buckets) in your Storj account, they will be displayed in the Remote site pane. You can easily create a folder by right clicking somewhere in the remote site pane and selecting Create directory (Figure 5.1).
Note: You can add as many folders as you want and also add sub-folders!.
You can now browse to the files/folders you want to upload in the Local site pane and upload them by dragging-and-dropping them to the Storj folder you created in the Remote site pane.
Tip: If you can't add a directory, you can reconnect to Storj by doing the following:
- Go to
Server->Reconnect(Figure 5.3).
Downloading files is just as easy, only this time drag-and-drop the files from your folder in your Remote Site pane to a folder of your choice in the Local Site pane. That's it!
Frequently Asked Questions
What is an Encryption Key?
The Encryption Key is a series of 12 to 24 words that represents 128 to 256 bits of entropy that is used to encrypt files before they are uploaded into the network.
Here is an example Encryption Key:
legal winner thank year wave sausage worth useful legal winner thank year wave sausage worth useful legal will
If you're familiar with BTC, this will be recognizable as it's based on the BIP39 standard.
How do I export settings from libstorj CLI?
First run this command to get the bridge user, password and the encryption key:
storj export-keys
And then enter those settings into FileZilla in the Getting Started section.
What does it mean if I see "Wrong encryption key for at least one bucket"?
This means that a bucket was added using a different Encryption Key and you won't be able to download the files within that bucket as it won't be able to decrypt or verify their integrity.
Why does the progress bar go above 100% for uploads?
Files are encoded with Reed Solomon error correction before they are uploaded to the network, so there is more data being uploaded compared to the original size of the file. This provides the ability to repair parts of the file that are missing or inaccessible. The number of parity shards is currently calculated as 2/3 of the original file size.
How are files encoded before being uploaded to the Storj network?
Files are first encrypted using the Encryption Key and then encoded using Reed Solomon into various size shards. For specific details please read SIP5. These shards are then uploaded to different nodes in the Storj network directly after communicating with the Storj bridge to negotiate which nodes to communicate with. After the file is uploaded, it is then mirrored to other nodes in the network, so that in the event that a shard goes missing after this initial upload, a shard can be repaired. The total number of shards that can be repaired is around 1/2 of the original file.
Note: For larger files the time it takes to do this preparation step can be more noticeable.
Can I rename or move files?
Not currently, this feature can be supported in the future, when file ids are unique, see deprecation messages at https://github.com/Storj/service-storage-models/blob/master/lib/models/bucketentry.js#L115-L119
What do I do if I see "Transfer rate limit" error?
This means that a rate limit on the number of bytes uploaded or downloaded has been reached. These are calculated time spans by hour, day and month. Accounts that have billing information entered currently do not have these rate limits, and instead have a rate limit on the total number of requests, which can be seen as a "Too many requests" error.
How many files can I upload?
There currently isn't a hard limit an the total number of bucket entries that can be added to a bucket. However there is limited support for more than 5000 buckets and more than 2000 bucket entries.
Why can't I upload a file at the root?
FileZilla currently maps buckets as directories at the root level and each file within those buckets is a bucket entry. Sub-directories are created by having a forward slash in the bucket entry name, which are parsed and shown as a directory within FileZilla. For this reason, it's not possible to add a file to the root because there isn't a bucket to add a bucket entry.
Conclusion
In this simple guide, we first created a Storj account and added a billing option. We then imported our account details into FileZilla and adjusted a few parameters in the client settings. Finally, we learned how to add a folder to out Storj account and how to up- and download files to and from this folder. FileZilla has many more advanced features and settings, so if you want to learn more about FileZilla, please have a look at the official FileZilla wiki page.
We are constantly improving Storj and all of its tools to make sure that both renters and farmers can interact harmoniously in the Storj ecosystem. We would love to hear from you, no matter if you have suggestions, comments, questions or criticisms you would like to share.
- Community chat: Storj Community.
- Storj feedback: Storj product feedback.
Community & Support
Please sign up for the Storj community chat at: https://community.storj.io/ and ask any questions you may have in the #filezilla channel.
Debugging
It's possible to view full detailed logs for file transfers. To enable it (you'll need to do this each time you start up):
- Open up the Preferences/Settings
- Go to Logging and optionally enable timestamps and select a file to log
- Go to Debug and select "4 - Debug"
- View logs in the command console
