Classification of Tools Available for Ethereum Smart Contracts
There are many different tools available for Ethereum Smart Contracts. These tools have been gathered from research publications and through Internet searches. In this section, we have classified the various tools available into different categories, so that the end users can easily find which tool to use for their particular application.
Even though our work is primarily concerned with Security Tools, the other tools are included for the reader’s convenience.
Security Tools
These are tools which take as input either the source code or the bytecode of a contract and give outputs on the security issues present. These are the tools that we are primarily concerned in with our work.
Visualization Tools
Visualization tools help give graphical outputs like control flow graphs, dependency graphs, etc. of the given contract to help in analysis. Tools like solgraph and rattle fall under this category.
Disassemblers and Decompilers
A dis-assembler converts the binary code back into the high level language code while a decompiler converts the binary code to a low level language for better understanding. evm-dis is a popular dis-assembler for smart contracts.
Linters
Linters are static analysis tools primarily focused on detecting poor coding practices, programming errors, etc. Ethlint is a common linting tool of ethereum smart contracts.
Miscellaneous Tools
This includes tools like SolMet which help give common code metrics like number of lines of code, number of functions per contract, etc. for solidity source files.
Methods employed by the Security Tools
Static Analysis
Static Analysis essentially means evaluating the program code without actually running it. It looks at the code structure, the decompiled outputs, and control flow graphs to identify common security issues. SmartCheck, Slither and RemixIDE are static analysis security tools for Ethereum smart contracts.
Symbolic Execution
Symbolic execution is considered to be in the middle of static and dynamic analysis. It explores possible execution paths for a program without any concrete input values. Instead of values, it uses symbols and keeps track of the symbolic state. It leverages constraint solvers to make sure that all the properties are satisfied. Mythril and Oyente are the popular Symbolic Execution tools for smart contract security.
Formal Verification
Formal Verification incorporates mathematical models to make sure that the code is free of errors. Bhargavan et. al conducted a study of smart contracts using F*. However, the work is not available as open source to the best of our knowledge.
Creation of the Vulnerability Benchmark
Need for a Vulnerability Benchmark
It is observed that many security tools have come up for Ethereum smart contracts over the years. However, it is also observed that these tools are usually tested on different test-instances and in some cases even the ground truth is unknown. Therefore, as a smart contract developer or a user, it becomes difficult to actually compare the performance of different tools without a proper benchmark.
Dika tried to solve this issue. However, he tested only three tools on just 23 vulnerable and 21 audited-safe contracts. A contract was called vulnerable if it had any vulnerability. However, it was not checked that a tool properly detected the vulnerability claimed and the results were presented as is.
Benchmark Creation Methodology & Statistics
To create the benchmark, we collected contracts known to be vulnerable from various sources. This included –
• Smart Contract Weakness Classification (SWC) Registry
• (Not So) Smart Contracts
• EVM Analyzer Benchmark Suite
• Research papers, theses and books
• Various blog posts, articles, etc.
After collecting all the instances, we manually removed the duplicate contracts – this was important as we found that there was notable overlap between contracts gathered from different sources. After this, we manually checked the contracts and classified them as per the new taxonomy. Finally we compiled the smart contracts into run-time bytecode. However, as each contract required a different version of solidity, and solc-select did not support such a large range of compiler versions, we leveraged Remix IDE to manually generate the run-time byte-codes for each contract and stored it separately. This was not done for the on-chain contracts and the run-time byte-codes for these were directly taken from the blockchain. A summary of the benchmark creation methodology is depicted in Figure 4.1.
It has been observed that Ethereum smart contracts have been used for creating ponzi-schemes to scam innocent people into loosing money by promising extraordinarily high returns. Even though a ponzi contract might not be a direct security vulnerability, we have included them in our study because of the high monetary impact of such contracts. Apart from ponzi schemes like GovernMental, FirePonzi and Rubixi which have already been included, we added ponzi schemes that exhibited one or more of the following properties – contracts that do not refund and contracts that allow the owner to withdraw funds. These properties are typical in ponzi schemes, however they cannot be classified as security vulnerabilities directly without knowing the context. Table 4.2 lists the additional ponzi schemes which have been included in the benchmark because they exhibit the above properties and had not been included before in the benchmark.
The final benchmark consists of 180 contracts spread over all the categories. Out of these we have 162 unique contracts. This includes 40 on-chain contracts (including six additional ponzi schemes). This is very high in comparison to the 23 vulnerable smart contracts identified by. Figure 4.2 shows the distribution of the benchmark instances across different categories.
4.3 Experimental Setup
For the purpose of the study, we select the security tools that are actively maintained, opensourced, ready for use and cover a fairly large section of the vulnerabilities. Keeping the above constraints in mind, the following tools were selected –
Remix IDE
Remix IDE is primarily an integrated development environment (IDE) for developing Solidity smart contracts. It can connect to the Ethereum network using Metamask and developers can directly deploy smart contracts from Remix. It is developed and maintained by the Ethereum Foundation.
The IDE has a security module to help developers with common security issues like re-entrancy, etc. It requires the solidity file of the contract to work. As a web interface was available, the testing using the benchmark instances was carried out manually.
SmartCheck
SmartCheck is a static analysis tool for Solidity and Vyper smart contracts. It is developed by SmartDec and the University of Luxembourg. Like other static analysis tools, it does not work on byte-codes and requires the source codes to be present for analysis.
It works by transforming the source codes into an intermediate representation which is XML-based. This representation is then checked against XPath patterns to highlight potential vulnerabilities in the code.
Slither
Slither is a static analysis tool for solidity source files written in Python 3. It is open sourced and is developed by by Trail of Bits. It works on contracts written in solidity >= 0.4 and requires the solidity files for analysis.
It leverages an intermediate representation call SlithIR for code analysis. However, it requires the correct solidity version to be installed in the system. For this, we utilize another tool by Trail of Bits called solc-select to switch to the right compiler version which is predetermined manually.
Oyente
Oyente is one of the earliest security tools for Solidity smart contracts. It was developed by security researchers at the National University of Singapore and is now being maintained by Melonport. Oyente leverages symbolic execution to find potential vulnerabilites in the smart contracts. It works with both byte-codes and solidity files.
Being one of the first tools in this area, Oyente has been extended by many researchers over the years. For example, the control flow graphs generated by Oyente are also used by EthIR, which is a high level analysis tool for Solidity. A web interface for the tool is also available.
Securify
Securify has been created by researchers at ETH Zurich in collaboration with ChainSecurity for security testing of Ethereum smart contracts. It works on both solidity source files and bytecodes. It has also received funding from the Ethereum Foundation to help mitigate the security issues in smart contracts. It analyzes the contract symbolically to get semantic information and then checks against patterns to see if a particular property holds or not. A web interface is also available at.
Mythril
Mythril is a security tool developed by ConsenSys. It uses as a combination of symbolic execution and taint analysis to identify common security issues. Recently, a new initiative called MythX was launched with a similar core as Mythril for smart contract developers to provide security as a service. However, it is still in beta testing and is not available as open source. Therefore, we use Mythril Classic for our testing purposes.
All the experiments were carried out on a machine running Ubuntu 18.04.2 LTS on an Intel Core i7-4770 CPU with 16GB DDR3 RAM. Also, the tools that worked on both solidity and
bytecode les were tested on bytecode les only. The results output by each tool were then
converted to the new taxonomy as shown in Table 4.4 to allow us to compare the tools uniformly.
Results and Analysis
Results on the Vulnerability Benchmark
Remix IDE
The performance of Remix IDE is surprisingly good. As seen in figure 4.4, it detects vulnerabilities like tx.origin authorization, use of assembly, unchecked call and timestamp dependence with 100% accuracy. However, we find that these vulnerabilities are caught by mere presence of certain constructs without checking whether they actually result in a vulnerability or not. For example, Timestamp Dependence flag is raised if timestamp is used anywhere in the code. Similarly, tx.origin flag is raised if tx.origin is used anywhere within the code without checking if any it causes any security issue or not. The selfdestruct module works similarly. However it could not detect the Parity Bug because it uses the older suicide construct. It was also observed that for solidity versions 0.3.1 and prior, the check-effects and the selfdestruct modules gave an error. This resulted in the famous DAO contract not being analysed by the tool.
SmartCheck
The performance of SmartCheck is given in figure 4.5. It has a good performance in only a few of the many categories that it can detect. The include security issues like use of deprecated functions, unchecked call, use of assembly, etc. However, the performance on other instances is not very good.
Slither
Slither has a very good performance across most of the categories that it detects. There was no category that it could not detect even one instance from. The biggest drawback of slither is that does not work with older solidity versions (prior to 0.4) and requires the correct version of solidity to be present on the system. Because of this, a lot of contracts in the benchmark gave errors with slither. However, it is a very good tool for smart contract developers who are developing in newer versions of solidity.
Oyente
Being one of the earliest tools, Oyente is now showing it’s age. It covers a very low number of vulnerabilities. The results are shown in figure 4.7. Average EVM code coverage for the entire benchmark set was found to be 75.98%. Also, there was not a single report of integer overflow or underflow across the complete benchmark. We believe this is some bug in the tool causing this
behaviour.
Securify
Securify is the only tool that reports a contract as ‘safe’ from a particular vulnerability. If the contract contains a vulnerability, and Securify reports it as ‘safe’, we call it false negative. From figure 4.8 we can see that Securify reports a lot of false negatives. However, it has a decent performance on re-entrancy bug detection.
Mythril
The performance of Mythril on the benchmark is shown in figure 4.9. It is able to detect the attacks with a fair accuracy, however it encounters a lot of errors. This makes its performance inferior to some static analysis tools like Slither.
4.4.2 Analysis
Figure 4.10 depicts the time taken by each tool on the benchmark. Static Analysis tools are faster than symbolic execution tools as expected. Mythril is the slowest, while Slither is the fastest tool. Remix could not be included in this comparison as it was used manually.
Slither and Mythril gives the most errors as showin in figure 4.11. Remix and Oyente are the only tools that do not give any errors on any contract in the benchmark.
Table 4.5 shows the tool vulnerability matrix as claimed by the tools. Here, Y implies that the tool claims to detect a particular vulnerability. Also PS stands for Ponzi Schemes. It is clear from this table that most of the vulnerabilities have been covered by one tool or the other. Slither and Oyente cover the most vulnerabilities closely followed by Mythril. Also, the most popular vulnerabilities with the tools are re-entrancy, unchecked call and timestamp dependence.
4.5 Summary
In this chapter, we have looked at the various tools available for smart contracts. For proper comparison of the tools, we have created a new vulnerability benchmark by collecting vulnerable
smart contracts from different sources. These are then run against the different tools. To ensure uniformity, and to enable comparison of different tools we create a mapping between the outputs given by the tools and our taxonomy as given in Chapter 3. The effectiveness of the tools in detecting the vulnerabilities in the benchmark is shown in Table 4.6. The cells highlighted in green indicate that all the instances of that vulnerability present in the benchmark are successfully detected, while grey highlights the maximum vulnerabilities (though not all) accurately detected across all the tools. We observe that many vulnerabilities are not being detected by the tools. We also observe that even though the tools cover a wide spectrum of vulnerabilities, they are not very accurate in detecting them. The best tool from our study is Slither. It covers a wide range of vulnerabilities and is the only tool that detected at-least one from each category it could successfully evaluate. The only drawback is that it works on solidity versions greater than 0.4.0. Nevertheless, it is still a good tool for new smart contract developers.
This article has been published from the source link without modifications to the text. Only the headline has been changed.
[ad_2]
Source link
