The Evolution of Decentralized Storage: From FIL to Shelby

Storage has been one of the hottest tracks in the blockchain industry. Filecoin, as the leading project of the last bull market, once had a market cap exceeding ten billion dollars. Arweave focuses on permanent storage, reaching a peak market cap of 3.5 billion dollars. However, as the limitations of cold data storage have been revealed, the necessity of permanent storage has been questioned, sparking widespread discussions on whether decentralized storage can truly take off. The emergence of Walrus brings new hope to the long-dormant storage track, while the Shelby project launched by Aptos and Jump Crypto aims to elevate decentralized storage in the hot data sector to new heights. This article will analyze the narrative changes of decentralized storage through the development paths of four representative projects: Filecoin, Arweave, Walrus, and Shelby, and explore the future development prospects of decentralized storage.

Filecoin: Surface Storage, Essentially Mining

Filecoin is one of the representative projects that emerged early on, with its development direction centered around Decentralization, a common feature of early blockchain projects. Filecoin combines storage with Decentralization, attempting to address the trust issues associated with centralized data storage service providers. However, certain aspects sacrificed for the sake of Decentralization became pain points that later projects like Arweave or Walrus focused on solving. To understand that Filecoin is actually a mining coin project, one needs to recognize the objective limitations of its underlying technology IPFS, which is not suitable for handling hot data.

IPFS: Transmission bottleneck of Decentralization architecture

IPFS(, the InterPlanetary File System, was introduced around 2015, aiming to disrupt the traditional HTTP protocol through content addressing. The biggest drawback of IPFS is its extremely slow retrieval speed. In an era where traditional data services can achieve millisecond-level responses, retrieving a file from IPFS still takes tens of seconds, making it difficult to promote in practical applications, which also explains why it is rarely adopted by traditional industries, except for a few blockchain projects.

The underlying P2P protocol of IPFS is mainly suitable for "cold data", which refers to static content that does not change frequently, such as videos, images, and documents. However, when it comes to handling hot data, such as dynamic webpages, online games, or artificial intelligence applications, the P2P protocol does not have a significant advantage over traditional CDNs.

Although IPFS itself is not a blockchain, its design concept of directed acyclic graph )DAG( is highly compatible with many public chains and Web3 protocols, making it inherently suitable as a foundational framework for blockchain. Therefore, even if it has no practical value, as a foundational framework carrying the blockchain narrative, it is already sufficient. Early projects only need a functioning framework to open new imaginative spaces. However, as Filecoin develops to a certain stage, the limitations brought by IPFS begin to hinder its further development.

) Mining Coin Logic Under the Storage Cloak

The original intention of IPFS's design is to allow users to not only store data but also be part of the storage network. However, in the absence of economic incentives, users find it difficult to voluntarily use this system, let alone become active storage nodes. This means that most users will only store files on IPFS but will not contribute their own storage space or store others' files. It is against this backdrop that FIL emerged.

In the token economic model of Filecoin, there are mainly three roles: users are responsible for paying fees to store data; storage miners receive token incentives for storing user data; and retrieval miners provide data when users need it and receive incentives.

This model has potential malicious space. Storage miners may fill garbage data after providing storage space to reap rewards. Since this garbage data will not be retrieved, even if they are lost, it will not trigger the penalty mechanism for storage miners. This allows storage miners to delete garbage data and repeat this process. The Filecoin proof-of-replication consensus can only ensure that user data has not been privately deleted, but it cannot prevent miners from filling garbage data.

The operation of Filecoin largely relies on miners' continuous investment in the token economy, rather than on the actual demand for distributed storage from end users. Although the project is still iterating, at this stage, the ecological construction of Filecoin aligns more with the "mining coin logic" rather than the "application-driven" definition of storage projects.

Arweave: A Double-Edged Sword of Long-Termism

If the design goal of Filecoin is to build an incentivized and verifiable decentralized "data cloud" shell, then Arweave takes a different extreme in the direction of storage: providing the capability for permanent storage of data. Arweave does not attempt to construct a distributed computing platform; its entire system unfolds around a core assumption—that important data should be stored once and remain forever on the network. This extreme long-termism makes Arweave vastly different from Filecoin in terms of mechanisms, incentive models, hardware requirements, and narrative perspectives.

Arweave uses Bitcoin as a learning object, attempting to continuously optimize its permanent storage network over long periods measured in years. Arweave does not care about marketing, nor does it care about competitors and market trends. It is simply moving forward on the path of iterating its network architecture, indifferent to whether anyone pays attention, because this is the essence of the Arweave development team: long-termism. Thanks to long-termism, Arweave received enthusiastic support during the last bull market; and because of long-termism, even if it falls into the depths, Arweave may still survive through several cycles of bull and bear markets. The only question is whether there will be a place for Arweave in the future of decentralized storage. The existence value of permanent storage can only be proven over time.

Since the Arweave mainnet version 1.5 to the recent version 2.9, although it has lost market attention, it has been committed to enabling a broader range of miners to participate in the network at minimal cost, and to incentivize miners to maximize data storage, thereby continuously enhancing the robustness of the entire network. Arweave is well aware that it does not meet market preferences, hence it adopts a conservative approach, not embracing the miners' community, with the ecosystem completely stagnant, upgrading the mainnet at minimal cost, while continuously lowering the hardware threshold without compromising network security.

A Review of the Upgrade Journey from 1.5 to 2.9

The Arweave 1.5 version exposed a vulnerability that allowed miners to rely on GPU stacking instead of actual storage to optimize block generation chances. To curb this trend, version 1.7 introduced the RandomX algorithm, limiting the use of specialized computing power and instead requiring general-purpose CPUs to participate in mining, thereby weakening power centralization.

In version 2.0, Arweave adopts SPoA, transforming data proofs into a concise path of the Merkle tree structure, and introduces format 2 transactions to reduce synchronization burdens. This architecture alleviates network bandwidth pressure, significantly enhancing the collaborative capabilities of nodes. However, some miners can still evade the responsibility of holding real data through centralized high-speed storage pool strategies.

To correct this bias, 2.4 launched the SPoRA mechanism, introducing global indexing and slow hash random access, requiring miners to genuinely hold data blocks to participate in effective block production, thereby weakening the stacking effect of computing power from a mechanistic perspective. As a result, miners began to focus on storage access speed, driving the application of SSDs and high-speed read-write devices. 2.6 introduced hash chain control for block production rhythm, balancing the marginal benefits of high-performance devices and providing a fair participation space for small and medium miners.

Subsequent versions further enhance network collaboration capabilities and storage diversity: 2.7 introduces collaborative mining and pool mechanisms to improve the competitiveness of small miners; 2.8 launches a composite packaging mechanism, allowing large-capacity low-speed devices to participate flexibly; 2.9 introduces a new packaging process in replica_2_9 format, significantly increasing efficiency and reducing computational dependencies, completing the closed loop of data-driven mining models.

Overall, Arweave's upgrade path clearly presents its storage-oriented long-term strategy: while continuously resisting the trend of computing power centralization, it aims to lower participation thresholds and ensure the long-term viability of the protocol.

Walrus: A New Attempt at Hot Data Storage

The design philosophy of Walrus is completely different from that of Filecoin and Arweave. Filecoin's starting point is to create a decentralized verifiable storage system, at the cost of cold data storage; Arweave's starting point is to create an on-chain library of Alexandria that can permanently store data, at the cost of too few scenarios; Walrus's starting point is to optimize the storage costs of hot data storage protocols.

RedStuff: Improved version of erasure code

In terms of storage cost design, Walrus believes that the storage overhead of Filecoin and Arweave is unreasonable, as both adopt a fully replicated architecture. The main advantage of this model is that each node holds a complete copy, providing strong fault tolerance and independence among nodes. This type of architecture ensures that even if some nodes go offline, the network still maintains data availability. However, this also means that the system requires multiple copies for redundancy to maintain robustness, thereby increasing storage costs. Especially in the design of Arweave, the consensus mechanism itself encourages node redundancy storage to enhance data security. In contrast, Filecoin is more flexible in cost control, but this comes at the cost of potentially higher data loss risks for some low-cost storage options. Walrus attempts to find a balance between the two, controlling replication costs while enhancing availability through structured redundancy, thus establishing a new compromise between data availability and cost efficiency.

The Redstuff created by Walrus is a key technology to reduce node redundancy, derived from Reed-Solomon ### RS ( encoding. RS coding is a very traditional erasure code algorithm, and erasure codes are a technique that allows for doubling a dataset by adding redundant fragments ) erasure code ( to reconstruct the original data. From CD-ROMs to satellite communications to QR codes, it is frequently used in daily life.

Erasure codes allow users to obtain a block, for example, 1MB in size, and then "expand" it to 2MB, where the additional 1MB is called special data known as erasure codes. If any byte in the block is lost, users can easily recover those bytes using the codes. Even if up to 1MB of the block is lost, you can still recover the entire block. The same technology allows computers to read all the data on a CD-ROM, even if it has been damaged.

The most commonly used is RS coding. The implementation method starts with k information blocks, constructs the relevant polynomial, and evaluates it at different x coordinates to obtain the code blocks. Using RS erasure codes, the probability of randomly sampling large chunks of data loss is very small.

For example: Divide a file into 6 data blocks and 4 parity blocks, totaling 10 pieces. As long as any 6 pieces are retained, the original data can be completely restored.

Advantages: Strong fault tolerance, widely used in CD/DVD, fault-tolerant hard disk arrays ) RAID (, and cloud storage systems ) such as Azure Storage, Facebook F4(.

Disadvantages: Decoding calculations are complex and expensive; not suitable for scenarios with frequent data changes. Therefore, it is usually used for data recovery and scheduling in off-chain centralized environments.

Under a decentralized architecture, Storj and Sia have adjusted traditional RS coding to meet the actual needs of distributed networks. Walrus has also proposed its own variant based on this - the RedStuff coding algorithm, to achieve lower costs and a more flexible redundancy storage mechanism.

What is the biggest feature of Redstuff? By improving the erasure coding algorithm, Walrus can quickly and robustly encode unstructured data blocks into smaller shards, which are distributed across a storage node network. Even if up to two-thirds of the shards are lost, the original data block can be quickly reconstructed using partial shards. This is made possible while maintaining a replication factor of only 4 to 5 times.

Therefore, it is reasonable to define Walrus as a lightweight redundancy and recovery protocol redesigned around a Decentralization scenario. Compared to traditional erasure codes ) like Reed-Solomon (, RedStuff no longer pursues strict mathematical consistency, but instead makes realistic trade-offs for data distribution, storage verification, and computational costs. This model abandons the immediate decoding mechanism required for centralized scheduling, opting instead to verify through on-chain proofs whether nodes hold specific data copies, thus adapting to a more dynamic and marginalized network structure.

RedSt