From FIL to Shelby: The Evolution of Decentralization Storage

Storage has been one of the hottest sectors in the blockchain industry. Filecoin, as the leading project of the last bull market, once had a market cap exceeding 10 billion USD. Arweave, with its selling point of permanent storage, reached a maximum market cap of 3.5 billion USD. As the availability of cold data storage is being questioned, the necessity of decentralization in storage has also sparked controversy. The emergence of Walrus brings new vitality to the long-dormant storage narrative, while the Shelby project launched by Aptos and Jump Crypto aims to elevate decentralized storage to new heights in the hot data sector. So, can decentralized storage rise again and provide a wide range of application scenarios? Or is it merely another round of speculation? This article will explore the evolution of the decentralized storage narrative by analyzing the development trajectories of four projects: Filecoin, Arweave, Walrus, and Shelby, and attempt to answer this question: How far is the path to the popularization of decentralized storage?

FIL: A Name for Storage, A Reality for Mining

Filecoin is one of the early rising cryptocurrency projects, with its development direction centered around Decentralization. This is a common feature of early alternative coins - seeking the meaning of decentralization in various traditional fields. Filecoin links storage with decentralization and emphasizes the trust risks associated with centralized data storage service providers. However, certain aspects sacrificed for the sake of decentralization have become pain points that later projects like Arweave or Walrus attempted to address. To understand that Filecoin is essentially just a mining coin, one must recognize the objective limitations of its underlying technology, IPFS, which is not suitable for handling hot data.

IPFS: Decentralization architecture transmission bottleneck

IPFS (InterPlanetary File System) was introduced around 2015, aiming to disrupt the traditional HTTP protocol through content addressing. The biggest drawback of IPFS is its extremely slow retrieval speed. In an era where traditional data service providers can achieve millisecond-level responses, retrieving a file from IPFS still takes several seconds, making it difficult to promote in practical applications and explaining why it is rarely adopted by traditional industries, except for a few blockchain projects.

The underlying P2P protocol of IPFS is mainly suitable for "cold data", which refers to static content that does not change frequently, such as videos, images, and documents. However, when it comes to handling hot data, such as dynamic web pages, online games, or artificial intelligence applications, the P2P protocol does not have significant advantages over traditional CDNs.

Although IPFS itself is not a blockchain, its directed acyclic graph (DAG) design concept is highly compatible with many public chains and Web3 protocols, making it inherently suitable as a foundational building framework for blockchains. Therefore, even if it lacks practical value, it is sufficient as a foundational framework to carry the blockchain narrative. Early alternative coin projects only needed a runnable framework to initiate grand visions, but as Filecoin developed to a certain stage, the limitations brought by IPFS began to hinder its progress.

The logic of mining coins under the storage cloak

The original intention of IPFS's design is to allow users to not only store data but also be part of the storage network. However, in the absence of economic incentives, users find it difficult to voluntarily use this system, let alone become active storage nodes. This means that most users will only store files on IPFS without contributing their storage space or storing others' files. It is against this backdrop that FIL was born.

In the token economic model of Filecoin, there are mainly three roles: users are responsible for paying fees to store data; storage miners receive token incentives for storing user data; retrieval miners provide data when users need it and receive incentives.

This model has potential malicious space. Storage miners may fill garbage data after providing storage space to obtain rewards. Since this garbage data will not be retrieved, even if it is lost, it will not trigger the penalty mechanism for storage miners. This allows storage miners to delete garbage data and repeat this process. The Filecoin replication proof consensus can only ensure that user data has not been privately deleted, but cannot prevent miners from filling garbage data.

The operation of Filecoin largely depends on miners' continuous investment in the token economy, rather than on the real demand from end users for distributed storage. Although the project is still iterating, at this stage, the ecological construction of Filecoin aligns more with the "mining coin logic" rather than the definition of an "application-driven" storage project.

Arweave: The Double-Edged Sword of Long-Termism

If Filecoin's design goal is to build an incentivized, provable decentralized "data cloud" shell, then Arweave takes a different extreme direction in storage: providing the capability for permanent storage of data. Arweave does not attempt to build a distributed computing platform; its entire system revolves around a core assumption - important data should be stored once and preserved forever on the network. This extreme long-termism makes Arweave fundamentally different from Filecoin in terms of mechanisms, incentive models, hardware requirements, and narrative perspectives.

Arweave takes Bitcoin as a learning object and attempts to continuously optimize its permanent storage network over a long cycle measured in years. Arweave does not care about marketing, nor about competitors and market development trends. It is simply moving forward on the path of iterating its network architecture, not minding if no one pays attention, because this is the essence of the Arweave development team: long-termism. Thanks to long-termism, Arweave was highly sought after during the last bull market; and also because of long-termism, even if it falls to the bottom, Arweave may still survive through several rounds of bull and bear markets. The only question is whether there is a place for Arweave in the future of decentralized storage. The existence value of permanent storage can only be proven through time.

The Arweave mainnet has evolved from version 1.5 to the recent version 2.9. Although it has lost market attention, it has been dedicated to enabling a broader range of miners to participate in the network at minimal cost, while incentivizing them to maximize data storage, thereby continuously enhancing the robustness of the entire network. Arweave is well aware that it does not align with market preferences and has adopted a conservative approach, not embracing the mining community, leading to a complete stagnation of the ecosystem. It upgrades the mainnet at minimal cost while continuously lowering hardware thresholds without compromising network security.

Review of the upgrade path from 1.5 to 2.9

The Arweave version 1.5 exposed a vulnerability where miners could rely on GPU stacking rather than real storage to optimize block production chances. To curb this trend, version 1.7 introduced the RandomX algorithm, limiting the use of specialized computing power and instead requiring general-purpose CPUs to participate in mining, thereby weakening the centralization of computing power.

In version 2.0, Arweave adopts SPoA, transforming data proofs into a concise path of Merkle tree structure, and introduces format 2 transactions to reduce synchronization burdens. This architecture alleviates network bandwidth pressure, significantly enhancing node collaboration capabilities. However, some miners can still evade the responsibility of true data ownership through centralized high-speed storage pool strategies.

To correct this bias, 2.4 introduced the SPoRA mechanism, which incorporates global indexing and slow hash random access, requiring miners to genuinely hold data blocks to participate in effective block production, thereby weakening the effect of hash power stacking from a structural perspective. As a result, miners began to focus on storage access speed, promoting the application of SSDs and high-speed read-write devices. 2.6 introduced hash chain control of block production rhythm, balancing the marginal benefits of high-performance devices and providing fair participation space for small and medium-sized miners.

Subsequent versions further enhance network collaboration capabilities and storage diversity: 2.7 adds collaborative mining and pool mechanisms to improve the competitiveness of small miners; 2.8 launches a composite packaging mechanism that allows large-capacity low-speed devices to participate flexibly; 2.9 introduces a new packaging process in replica_2_9 format, significantly improving efficiency and reducing computational dependencies, completing the closed loop of the data-oriented mining model.

Overall, Arweave's upgrade path clearly presents its storage-oriented long-term strategy: while continuously resisting the trend of computational power centralization, it aims to lower the participation threshold and ensure the long-term viability of the protocol.

Walrus: Embracing Innovative Attempts with Hot Data

The design philosophy of Walrus is completely different from that of Filecoin and Arweave. The starting point of Filecoin is to create a decentralized and verifiable storage system, at the cost of cold data storage; Arweave's starting point is to build an on-chain library of Alexandria that can permanently store data, at the cost of too few scenarios; Walrus's starting point is to optimize the storage costs of a hot data storage protocol.

Magic Modified Error Correction Code: Cost Innovation or Old Wine in New Bottles?

In terms of storage cost design, Walrus believes that the storage overhead of FIL and Arweave is unreasonable. The latter two adopt a fully replicated architecture, whose main advantage is that each node holds a complete copy, providing strong fault tolerance and independence among nodes. This type of architecture ensures that even if some nodes go offline, the network still maintains data availability. However, this also means that the system requires multiple copies for redundancy to maintain robustness, which in turn increases storage costs. Especially in Arweave's design, the consensus mechanism itself encourages node redundancy storage to enhance data security. In contrast, FIL is more flexible in cost control, but the trade-off is that some low-cost storage may carry a higher risk of data loss. Walrus attempts to find a balance between the two, aiming to enhance availability through structured redundancy while controlling replication costs, thus establishing a new compromise path between data availability and cost efficiency.

The Redstuff created by Walrus is a key technology for reducing node redundancy, originating from Reed-Solomon (RS) coding. RS coding is a very traditional erasure code algorithm, which is a technique that allows for the doubling of datasets by adding redundant fragments, enabling the reconstruction of original data. From CD-ROMs to satellite communications and QR codes, it is frequently used in everyday life.

Erasure codes allow users to take a block, for example, 1MB in size, and "expand" it to 2MB, where the additional 1MB is special data known as erasure codes. If any byte in the block is lost, the user can easily recover those bytes using the codes. Even if up to 1MB of the block is lost, the entire block can still be recovered. The same technology enables computers to read all data on a CD-ROM, even if it has been damaged.

The most commonly used is RS coding. The implementation method is to start with k information blocks, construct the relevant polynomial, and evaluate it at different x-coordinates to obtain the encoded blocks. Using RS erasure codes, the possibility of randomly sampling large missing data blocks is very small.

For example: divide a file into 6 data blocks and 4 parity blocks, totaling 10 parts. As long as any 6 of them are retained, the original data can be completely restored.

Advantages: Strong fault tolerance, widely used in CD/DVD, fault-tolerant hard disk arrays (RAID), and cloud storage systems (such as Azure Storage, Facebook F4).

Disadvantages: Decoding calculations are complex and have high overhead; not suitable for data scenarios with frequent changes. Therefore, it is usually used for data recovery and scheduling in centralized off-chain environments.

Under a decentralized architecture, Storj and Sia have adjusted traditional RS coding to meet the practical needs of distributed networks. Based on this, Walrus has proposed its own variant - the RedStuff coding algorithm - to achieve lower costs and a more flexible redundancy storage mechanism.

What is the biggest feature of Redstuff? By improving the erasure coding algorithm, Walrus can quickly and robustly encode unstructured data blocks into smaller shards, which are distributed across a network of storage nodes. Even if up to two-thirds of the shards are lost, the original data block can be quickly reconstructed using partial shards. This is made possible while maintaining a replication factor of only 4 to 5 times.

Therefore, it is reasonable to define Walrus as a lightweight redundancy and recovery protocol redesigned around a decentralized scenario. Compared to traditional erasure codes (such as Reed-Solomon), RedStuff no longer pursues strict mathematical consistency, but instead makes realistic trade-offs regarding data distribution, storage verification, and computational costs. This model abandons the immediate decoding mechanisms required for centralized scheduling, opting instead for on-chain.