In this edition of ICON Weekly, South Korea looks to invest $48.6 billion into “Industry 4.0”, Balanced integrates Band for decentralized price feeds, ICON Foundation responds to last week’s 100x transaction fee incident, and a look into MyID and VisitMe adoption in South Korea.

South Korea to Invest in Blockchain

Earlier this month, ZDNet Korea reported that South Korea would be investing ₩58.2 trillion - yes, trillion - into various “Industry 4.0” technologies by 2025. For those of us who are more familiar with USD, that’s $48.6 billion.

According to ZDNet Korea, the South Korean government is targeting areas like 5G, AI, and blockchain for investment. The overarching strategy and theme here is hard to ignore. Over the next decade, South Korea is looking to move to a society that relies less on “face-to-face interactions”. ZDNet’s article gives a few examples of how this is expected to play out.

  • Fast WiFi will be installed in 380,000 schools around the country, faculty laptops will be replaced with newer models, and an official online education platform will be developed.
  • 18 “smart hospitals” with real-time patient monitoring powered by 5G and IoT (Internet of Things) will be built.
  • Smart logistics and shipping centers powered by blockchain will be established in 11 locations.

To me, this is clearly a response to the current world we live in. COVID-19 won’t be the last pandemic we face as a species, so perhaps this is South Korea’s way of preparing itself for future pandemics. Utilizing new technologies to reduce face-to-face interactions makes complete sense for a forward-thinking and technologically-capable country like South Korea.

This post is made possible by RHIZOME. If you enjoy reading in-depth and thoughtful content about the ICON ecosystem, consider subscribing to our newsletter, checking out our podcast, and voting for RHIZOME.

The last two items are especially interesting from an ICON perspective. In the past, ICON, or maybe ICONLOOP (it’s hard to tell sometimes), has conducted blockchain integration concepts with various hospitals and insurance firms in South Korea. Similarly, ICON has also seem some action in the logistics sector with its loopchain technology being used by Korea Customs Service.

Every month, it looks like South Korea is comitting more and more to the idea of digitizing, automating, and interconnecting society. I wouldn’t be surprised if ICONLOOP ends up playing a huge role in designing the blockchains and smart contracts that will power everyday life in Korea a decade from now. Connecting society will require some form of interoperability between different services and blockchains, so I really think there is a non-zero chance that the ICON public chain will see adoption on a massive scale.

However, to improve the probability of something like that happening, ICON Foundation needs to be more methodical about how it approaches technical development of the public chain. Specifically, it needs to start viewing the public chain as a flagship enterprise solution, instead of an afterthought.

Balanced Integrates Band Protocol

Earlier this week, Balanced announced an upcoming integration with Band Protocol for decentralized oracle services.

Balanced is thrilled to announce that we’ve partnered with Band Protocol, and will integrate them as our oracle solution upon launch.

We chose Band Protocol not only because of their secure and decentralized protocol, but also because of their willingness to work closely with us, almost acting as an extension of the Balanced team.

To start, Band will provide Balanced with an ICX/USD price feed. Balanced centers around “giving ICX another use case” – collateral. In order for ICX to be used as collateral, it’s value needs to be known. It’s important for DeFi apps like Balanced to receive asset prices in a decentralized manner. If Balanced only queried a single exchange for the price of ICX, it would be fairly trivial for someone to manipulate Balanced – that defeats the purpose of DeFi.

Balanced decentralizes price feeds with multiple validators and sources.
Source: Balanced

Band has multiple layers of decentralization built in. First of all, price feeds are maintained by multiple validators, each of which are economically incentivized to provide honest and accurate data. Secondly, each validator sources the ICX/USD price from a variety of exchanges and sources, which effectively decentralizes risk even further. Specifically, in order to compromise the price feed, a malicious actor would have to compromise Binance, OKex, UpBit, Huobi, and CoinGecko at the same time – not impossible, but not likely.

A Post Mortem from ICON Foundation

In last week’s update, I shared my thoughts on the recent 100x transaction fee spike caused by an untested smart contract on ICON. In that post, I highlighted the importance of proper testing (especially for a project vying for enterprise usage) and the presence of a “technical lead” on the ICON team to handle post mortems and “next steps”.

This past week, Bong from the ICON team shared a post mortem, which was then followed up by a series of tweets from Min. I’ve spent the better part of the week reflecting on how ICON handled this situation, and I’m now extremely convinced that ICON needs to hire a technical individual to co-lead the project ASAP. The quality of the post mortem and Min’s subsequent commentary displayed a high degree of ignorance towards the gravity of the situation. It was egotistical, and not humble – that’s a problem.

Before I share my thoughts, here’s the post mortem.

Hello Everybody,

We would like to share a post-mortem on what happened with the transaction fees on the ICON Network recently.

At block height 21,789,395, the ICON Foundation submitted a Step Price Proposal to adjust the step price to 12,500,000,000 loop (0.00000000125 ICX). However, at block height 21,809,081, the proposal was passed and the step price changed to 1,258,425,417,728 loop (0.000001258425417728 ICX). We further analyze why this happened below.

Problem

The original design of the Step Price Proposal was to convert the value entered in the “Value” parameter to decimal. The Value parameter on the Network Proposal is generally designed to support the String format to process various types of proposals. Therefore, the Step Price Proposal was supposed to convert the input “String” to decimal. However, the Step Price Proposal converted the value to hexadecimal as opposed to decimal, resulting in a 1,258,425,417,728 loop proposal instead of the intended 12,500,000,000 loop.

Solution

ICON Foundation modified the Governance SCORE to add logic to identify both decimal and hexadecimal numbers to prevent this issue from happening again. ICX Station submitted a followup proposal after the Governance SCORE was updated. Main P-Reps approved this proposal and we are now at the intended level of transaction fees.

Here’s Min’s follow-up commentary.

My thoughts on recent Step Price Proposal bug/error. You can read the post mortem in the link here. First of all, there is no excuse for mistakes. Period. We take full responsibility for this and cover any damage.

It’s more important to learn from this incident. Bugs are part of dev process. We’ve recently seen that even big co’s like Twitter w more engineers + resources make mistakes. The truth is mistakes are going be made. It’s inevitable. The Q is how do you prep to minimize risks.

At ICON, we try to balance speed (shipping codes) based on our risk assessment. In other words, we prefer to move fast and break things if we can afford it. Sometimes, learning by breaking is cheaper than spending hours on testing.

We’d eventually get to a point where ICON network is large enough so it’s absolutely necessary and more cost efficient to push proposals and codes more slowly and carefully. You’ll know when we get there.

Every mistake looks amateurish and preventable in hindsight. We humans tend to simply overlook answer to easiest question b/c we assume it’s correct. And usually 99 out 100, we are correct.

I don’t lose sleep over mistakes like this. ICONLOOP are pros. They have an extensive experience w enterprises. ICONLOOP sets up enterprises for system failures/errors. This is exactly why we’ve been designing a hybrid blockchain.

The Purpose of a Post Mortem

First off, I really appreciated Bong’s post mortem on the ICON community forum. I think it could’ve contained some more insight into how the team plans to restructure internal QA protocols following this incident, but overall it was a decent explanation of what happened. On the other hand, Min’s commentary was 100% unnecessary – we’ll get into that later.

A post mortem is a detailed document that covers three things.

  1. How you f***ed up.
  2. How you solved it.
  3. What you’re going to do to prevent it from happening again.

ICON’s post mortem covered the first two things, but it didn’t reveal what the team is going to do to ensure this doesn’t happen again six months or six years from now. In the previous update, I mentioned it would be good to use this opportunity to launch a full-fledged testnet to reproduce the effects of network proposals before repeating the voting process on mainnet. Alternatively, announcing a third-party smart contract audit focused on human and smart contract interaction would’ve helped ease developer and investor concern.

To get a sense of what a proper post mortem looks like, check out this post from Cloudflare, which details a network outage caused by a bad software deploy. The level of detail in this post mortem is commendable, but I think the section below is by far the most important part.

What’s happened since last Tuesday

Firstly, we stopped all release work on the WAF completely and are doing the following:

  • Re-introduce the excessive CPU usage protection that got removed. (Done)
  • Manually inspecting all 3,868 rules in the WAF Managed Rules to find and correct any other instances of possible excessive backtracking. (Inspection complete)
  • Introduce performance profiling for all rules to the test suite. (ETA: July 19)
  • Switching to either the re2 or Rust regex engine which both have run-time guarantees. (ETA: July 31)
  • Changing the SOP to do staged rollouts of rules in the same manner used for other software at Cloudflare while retaining the ability to do emergency global deployment for active attacks.
  • Putting in place an emergency ability to take the Cloudflare Dashboard and API off Cloudflare’s edge.
  • Automating update of the Cloudflare Status page.

In the longer term we are moving away from the Lua WAF that I wrote years ago. We are porting the WAF to use the new firewall engine. This will make the WAF both faster and add yet another layer of protection.

Why is this section important? It shows a certain degree of humility, and it gives users and investors insight into what’s changing internally to reduce the probability of a similar incident occuring in the future. In fact, I’d argue this is the only truly important part of a post mortem.

By the time a post mortem is published, people already know what happened, and they may even know how the issue was solved (real-time updates on Twitter). What “outsiders” don’t have access to is insight into how a particular incident affected internal protocols, and that’s the kind of stuff you should be sharing in a post mortem if you truly care about being transparent.

Regarding ICON’s post mortem, I hope to see a more transparent document detailing how the 100x transaction fee incident will shape and improve the project’s internal workflows, but I’m not holding my breath.

Min’s Commentary

Min is headstrong, there’s no doubt about that. This personality trait isn’t necessarily a bad one to have as a leader, but there is a time and place for humility as well. Min’s follow-up commentary to 100xgate was the perfect opportunity to display a sense of humility. Unfortunately, he did the exact opposite. Ironically, this move undermines trust in ICON instead of reinforcing it.

Let me explain.

Disclaimer: I harbor no ill feelings towards Min Kim. I, along with RHIZOME, am HEAVILY invested in ICX. This is a realistic outlook on the project from someone with a tech background, with advice on how ICON can improve in a way that benefits everyone from the top to the bottom.

In his opening statement, Min refers to the recent incident as a “bug/error”. I have to admit that when I first saw this series of tweets, I had to stop reading after I saw “bug/error”. This incident was not the result of a bug or error. The fact that Min referred to it as a “bug/error” tells me, once again, ICON needs a technical lead to handle these kinds of situations. There was nothing buggy or error-y in what happened last week because the smart contract code performed exactly as it should have. Instead, the incident was caused by two things.

  1. Sloppy internal coding practices that don’t account for how humans interact with computers.
  2. Sloppy internal QA and testing protocols.

Even my wife tests WordPress plugin updates before deploying to her live site. Why can’t ICON test potentially network-breaking changes before deploying to mainnet? The answer is simple – because the processes don’t exist. Blaming this incident on a “bug/error” shows a complete misunderstanding of what happened last week. That’s very concerning.

First of all, there is no excuse for mistakes. Period. We take full responsibility for this and cover any damage.

I guess there’s a dash of humility here. After Min’s statement, I fully expect ICON Foundation to cover the fee difference for all transactions that occured during the incident period.

At ICON, we try to balance speed (shipping codes) based on our risk assessment. In other words, we prefer to move fast and break things if we can afford it. Sometimes, learning by breaking is cheaper than spending hours on testing.

This section doesn’t make sense to me. What is ICON’s “risk assessment” protocol? In terms of risk, a network-wide transaction fee change seems worthy of testing, especially when it’s being done for the first time. If such a change that affects literally every network participant does not qualify for testing under ICON’s risk assessment, then what is worth testing?

I don’t understand how “learning by breaking is cheaper than spending hours on testing”. In some cases, that statment is totally true, but not in this case. ICON could’ve simply deployed the fix to testnet first and observed the effects before deploying to mainnet. This is not “spending hours on testing”. It’s a normal and standard workflow for software development. While this incident may not have affected ICON’s bottom line in terms of cash holdings, it has undoutedy damaged community trust in ICON’s technical capabilities. I hope ICON never forgets the community is everything.

We’d eventually get to a point where ICON network is large enough so it’s absolutely necessary and more cost efficient to push proposals and codes more slowly and carefully.

As someone who works in tech, I get to observe the processes of our development team every single day. In software development, testing is important no matter how small or big your app or network is. To suggest that testing code makes software development less cost efficient is absolutely insane. Testing should be an embedded cost that represents a mandatory part of a software development workflow.

If a feature is going to be pushed to production, it should be tested in a staging environment or testnet. This is especially true for a product like ICON that transfers value across parties. Again, it seems like ICON is only concerned about the bottom line. I just can’t understand how deploying a transaction fee change straight to mainnet with zero testing on testnet is “okay”.

I don’t lose sleep over mistakes like this. ICONLOOP are pros. They have an extensive experience w enterprises. ICONLOOP sets up enterprises for system failures/errors. This is exactly why we’ve been designing a hybrid blockchain.

I say this in the nicest way possible – Min, this is not about you losing sleep! In fact, it has nothing to do with how well you sleep. This “mistake” caused people to pay 100x higher fees. This is a situation where you don’t want to highlight how “pro” ICONLOOP is because “not accepting human input in hex-encoded format” is something you learn in Human Interface Design 101. When you take a situation like this, and counter it with how “pro” ICONLOOP is because of their “extensive experience w enterprises”, it only tells us one of two things.

  1. ICONLOOP developers are not actually “pro”.
  2. ICONLOOP developers cuts corners when developing non-enterprise software like the ICON public chain.

I’m leaning towards #2 because it’s clear that ICONLOOP is capable of developing enterprise-ready products like MyID and other blockchain products for various well-known companies and insitutions in South Korea. With that in mind, why does ICON Foundation, after receiving $45 million in ICO funding, not demand the same level of rigorous development and testing for a public chain that transfers MONETARY VALUE everyday?

I hope you understand my point here. To summarize, I am happy that ICON Foundation provided a post mortem. That’s certainly a step in the right direction compared to how technical failures were handled in the past.

What I’m not happy about is Min’s nonchalant commentary that clearly exposes his lack of technical understanding for the world to see. It’s not a good look for ICON, especially when nearly every single one of its competitors is led by someone with deep technical knowledge. Zoom out – it’s no coincidence that the world’s top technology software companies (Google, Facebook, GitHub, Basecamp, Netlify, the list goes on and on) are lead by people with technical backgrounds. I’ll say it again. ICON needs to onboard a technical lead ASAP.

MyID App

Many of us are eagerly awaiting the release of MyID in South Korea. The communication from ICONLOOP isn’t super clear, but I think it’s supposed to go live in July with a chance of a delay into August. In either case, a few ICONists pointed out a new MyID-related app developed by ICONLOOP, Inc. in the Apple App Store and Google Play Store. This app has been around for a while. I think I first saw it back in June sometime. The interesting thing is that it was recently updated on July 23, 2020 after a two month period with no activity.

Is this the MyID app?
Is this the MyID app?

I’m not sure if the recent update means anything, but it does indicate that the app was updated for some reason. Hopefully we’ll see some official news about the MyID release sometime in August.