Michael G. Solomon - Blockchain Data Analytics For Dummies

Although transaction data is stored in blocks on the blockchain, Ethereum stores state data not in blockchain blocks but in an external (off-chain) database. Each block stores a hash value that points to the root of that block's state trie in the off-chain database, which stores the block’s state data. Storing data using a trie structure makes it possible to query the trie for a value, and validate the integrity of that value, without having to read the entire trie.

Refer to Figure 3-5, which shows the contents of a block header, including the state root hash that points to the root of the state values for that block (which is stored in the off-chain database).

Unlike blockchain data, state data can change. Each time a function in a smart contract runs, it may change the value of one or more state data items. The transaction that caused the change is stored in a block on the blockchain, and log entries may be created by events, but state data changes are stored in the off-chain database.

Each Ethereum client can select its own database for storing state data. For example, the Geth client uses LevelDB, and the Parity client uses RocksDB. Each database uses different methods for access, so the blockchain client you use for analysis should support a database that is familiar.

Now that you know about the basic categories of data a blockchain stores, you can start to dig into what each type of data you’ll find might mean. Few hard-and-fast rules for storing data exist. Each smart contract sets its own rules for defining and maintaining the data it needs to do its work.

Every transaction contains basic information about crypto-asset ownership and transaction cost. The From, To, and Value fields record, respectively, the account that owns the value at the start of the transaction, the account to which the value is transferred, and the amount or cost of the asset that is transferred by the transaction.

The Input Data field may contain additional information about the transaction. This data field is often very different from one transaction to another. When the To field of a transaction refers to a regular Ethereum account, the Input Data field may contain supporting or additional transaction details. In these cases, the transaction serves primarily to record a transfer of cryptocurrency from one account to another. If the To field contains the address of a smart contract, the Input Data field will contain information about the function that the transaction invokes and the data sent to the function.

Part of the challenge in extracting blockchain data for analysis is classifying and making sense of the input data. The blockchain analytics process is more than just reading data and building models.

Although transaction data can reveal what a client requested and what value was transferred, it doesn’t always provide many details of how the transaction played out. In other words, transaction data doesn’t provide more than summary information. If you want to explore details of what happened during a transaction, you’ll have to look elsewhere.

Because smart contract code can include complex calculations and data, it's often beneficial, and sometimes necessary, to store messages and data at points within the transaction. Most complex business transactions involve multiple steps, and mirroring real-world processes in code makes sense. For example, if you want to import wood from a foreign country to manufacture furniture, you’d follow a general sequence of steps:

1 The importer requests the product from the exporter.

2 The importer applies for a letter of credit from a local bank on behalf of the exporter.

3 The importer’s bank issues a letter of credit and sends it to the exporter’s bank.

4 The exporter ships the product.

5 Based on the terms of the letter of credit, the exporter may receive partial payment while the product is in transit. If partial payment is executed, the exporter’s bank claims the payment due and the importer’s bank transfers the specified payment.

6 When the importer receives the product, the importer notifies the bank and the importer’s bank transfers the remaining funds to the exporter’s bank.

Believe it or not, this process is simplified! I didn’t even touch on export licenses or bills of lading. Even with this simple scenario, you can see that the process has many steps. In a real application, some of these steps might occur at different times and some might occur at the same time (such as in Step 5). It would be helpful if the blockchain stored status information about how the transaction was carried out, as opposed to storing just the amount transferred from one account to another.

Event logs provide that functionality. No events occur by default; smart contract programmers must request each one. Most smart contract code includes at least minimal event invocations. A best practice when developing a smart contract is to invoke an event any time a package of work of interest to the application’s user is completed. That description of when to use events is a loose one and open to interpretation.

Knowing how the smart contract code that will supply data for your analytics projects works is important. In Chapters 5and 6, you learn how to get smart contract source code and how to use it to build your data acquisition plan. But until then, remember that analyzing data in a blockchain environment requires familiarity with far more than just blockchain data.

Although understanding the data available through transactions, events, and contract state is important, you must understand what that data represents before you can make much sense out of it. An important part of any data analytics project (blockchain or traditional data) is to align data with the real world. In a blockchain environment, that understanding starts with smart contracts.

You can think of smart contracts as programs that contain data and the functions to manipulate that data. One way to help understand smart contracts is to think of state data as nouns and functions as verbs. Associating smart contract elements with parts of speech helps to understand each element’s purpose. You store data that represents something in the real world, such as an order, a product, or a letter of credit.

Functions provide the actions that applications take on data, such as creating an order, createOrder(), shipping a product, shipProduct(), or requesting a letter of credit, requestLoC(). Data analytics is focused on extracting meaningful and actionable information from data. It is important to understand the data available to you, along with how that data was created and what real-world things and processes it represents. Smart contract functions provide the roadmap to how data gets added to the blockchain and what that data means.

One process early in any data analytics project is assessing your available data. In a blockchain environment, that step should include assessing any events related to the smarts contracts you’ll examine. One way to view events is as documentation of internal operations. These microtransaction artifacts often provide a level of granular data that you can’t get anywhere else. Don’t ignore the event logs — they may provide your best description of blockchain data and what it really represents.