The Long Road to SegWit: How Bitcoin’s Biggest Protocol Upgrade Became Reality

The Critique

But despite initial excitement, Segregated Witness had its critics, too.

Concerns about the proposed protocol upgrade varied. Jeff Garzik, the former Bitcoin Core contributor — who would soon after found his own development company Bloq — did not consider SegWit a sufficient short-term scaling solution. Bitcoin XT lead developer Mike Hearn, meanwhile, was not convinced by the proposal at all: He dismissed the solution as an “accounting trick” and completely quit Bitcoin development shortly after. 

Jonathan Toomim, developer for alternative software client Bitcoin Classic, argued that the proposal was “ugly and awkward,” suggesting it would be better implemented as a hard fork. Even Bitcoin Core contributor Peter Todd had his concerns, in particular related to mining.

Most of these issues were deemed either solvable, unconvincing or worth the trade-off by the Bitcoin Core development team at large, however. Development of the soft-fork upgrade commenced.

The Development

Even though a version of Segregated Witness was already implemented on Elements, the code for the Bitcoin main chain version mostly had yet to be written, not only because it needed to be implemented as a soft fork, but also because SegWit for Bitcoin would enjoy a range of new features not present in Elements: for example, the “witness discount” necessary to increase the block size, new compatibility for the peer-to-peer network and more.

The concrete Bitcoin Improvement Proposal for SegWit, BIP141, was authored by Pieter Wuille, Ciphrex CEO Eric Lombrozo and independent Bitcoin Core contributor Johnson Lau. By early January 2016, in the midst of a heated scaling debate, these and other Bitcoin Core contributors launched an initial dedicated test network for the protocol upgrade, which was dubbed SegNet. Another two weeks later, this testnet was taken public for the wider Bitcoin development community to experiment with. And by March, SegNet was upgraded to support test versions of the lightning network.

Development continued for the months to come, taking in feedback from Bitcoin’s development community, fixing bugs, improving the codebase accordingly and launching several more iterations of SegNet(s).

The SegWit GitHub page, where development and other issues are publicly visible for anyone to keep track of and contribute to.

Meanwhile, Bitcoin Core contributors also reached out to the broader Bitcoin industry, which over time led to a consistently growing list of companies and projects committing to supporting Segregated Witness.

By June, the Segregated Witness code counted 4,743 lines of code (including test code) and proposed removing or modifying 554 existing lines of Bitcoin Core code. After more review from several contributors, Bitcoin Core lead maintainer, Wladimir van der Laan, merged it into Bitcoin Core’s “master branch” by the end of that month.

The Meetings

At the same time that SegWit was being developed, block size tensions in the Bitcoin community were once again heating up. This time spearheaded by Bitcoin Classic, a number of Bitcoin companies and miners appeared determined to hard fork in order to increase the block size limit to 2 megabytes.

In what is perhaps best described as an emergency meeting, once again in Hong Kong, several Bitcoin Core contributors, mining pool operators and other Bitcoin industry members met to discuss the scaling issue.

The meeting led to an agreement that came to be known as the “Bitcoin Roundtable Consensus” (or the “Hong Kong Agreement”). The Bitcoin Core contributors present at the meeting agreed to work on a block size limit increase hard fork to be proposed to the Bitcoin Core development team and the wider Bitcoin community. The miners, in turn, agreed to run a SegWit release in production by the time such a hard fork would be released in a version of Bitcoin Core. The crisis seemed to have been averted — even though it quickly became clear that not everyone was happy about the agreement.

Several months later, an even bigger group of Bitcoin Core contributors and mining pool operators met in California. The Bitcoin Core contributors present at this meeting left convinced that Segregated Witness would be activated by the miners.

The Release

About six months behind on the initial schedule — the release was originally set for April — Segregated Witness was officially introduced October of 2016, in Bitcoin Core version 0.13.1. The protocol upgrade was also implemented in several other Bitcoin implementations, like Bitcoin Knots and Bcoin.

Using an activation method called “VersionBits” (BIP9), designed to minimize network disruption, 95 percent of miners (by hash power) had to signal support for SegWit to activate on the Bitcoin network. This miner signaling was to start on November 15th. Meanwhile, users were encouraged to upgrade their clients, which over time, it seemed, many did.

As of August 2017, the vast majority of the Bitcoin network consists of SegWit-ready nodes.

Based on the meetings with mining pool operators, as well as a general conviction that SegWit would be a boon for Bitcoin, many expected that the soft fork would be activated rather quickly.

The Politics

But that’s not what happened. As it turned out, several attendees of the Hong Kong Roundtable Consensus disagreed over what they had actually signed onto.

Bitmain co-CEO Jihan Wu, in particular, indicated he would only be willing to activate SegWit if the Bitcoin Core development team also implemented a hard fork to increase the block size limit in their codebase. Other mining pools, including F2Pool, HaoBTC and bitcoin.com didn’t signal support for the soft fork either.

Bitmain (and subsidiary AntPool) demand a hard fork block size limit increase in return for SegWit activation.

Moreover, a new Chinese mining pool emerged: ViaBTC. With close ties to Bitmain, ViaBTC alone garnered enough hash power to single-handedly block SegWit activation. And its operator, Haipo Yang, positioned himself as a staunch critic of the proposed protocol upgrade.

SegWit activation seemed far away.

The UASF

The avatar of pseudonymous Bitcoin and Litecoin developer Shaolinfry

In February 2017, a little over three months after the official release of SegWit, a new opportunity presented itself.

The pseudonymous developer “Shaolinfry,” who had previously contributed to Litecoin, dropped a new proposal in the Bitcoin development mailing list and the popular bitcointalk.org forum: a “user activated soft fork” or “UASF.”

Shaolinfry argued in his email that the hash power activation mechanism that had become the standard for soft forks was never intended to be a “vote.” “[T]he signaling methodology is widely misinterpreted to mean the hash power is voting on a proposal and it seems difficult to correct this misunderstanding in the wider community,” he wrote.

Shaolinfry proposed an alternative: a user activated soft fork (UASF). Instead of hash power activation, a user activated soft fork would have a “‘flag day activation’ where nodes begin enforcement at a predetermined time in the future.” As long as such a UASF is enforced by an economic majority, this should compel a majority of miners to follow (or activate) the soft fork.

The idea immediately generated buzz throughout Bitcoin forums and social media. And when former BTCC COO and outspoken SegWit proponent Samson Mow set up a bounty fund for the development of a UASF software implementation, it seemed like the proposal could become a reality.

The Patented Technology

In the first week of April 2017, Gregory Maxwell dropped what was widely considered a bombshell revelation on the Bitcoin development mailing list.

Maxwell claimed to have reverse-engineered a specialized ASIC-mining chip and found that it included patented AsicBoost technology. What’s more, Maxwell revealed that covert use of the technology would be incompatible with a soft-forked version of SegWit. “An incompatibility would go a long way to explain some of the more inexplicable behavior from some parties in the mining ecosystem,” he noted.

While no specific ASIC-manufacturer was mentioned in Maxwell’s email, Bitmain acknowledged that it had implemented the patented technology in their mining chips — though it denied having used it on Bitcoin’s mainnet.

Either way, for some users the revelation added to the desire to have the Segregated Witness soft fork activated on the Bitcoin network. And, as hash power activation seemed even less likely now, a user activated soft fork was increasingly looking like the solution to accomplish that.

The BIP148 Proposal

Shortly after proposing the general idea of a UASF, Shaolinfry and about a dozen other members of the Bitcoin community opened a UASF channel on the Bitcoin Core Community Slack.

The channel became a central point of discussion and organization for the initiative. A flag date was picked, initially for October 1, then later moved to August 1 to better account for potentially low hash power support. Shaolinfry authored a concrete Bitcoin Improvement Proposal: BIP148. Open Dime founder Rodolfo Novak also established an informational website to promote the idea.

The initial plan was to get exchanges and other businesses on board with the UASF. If these companies would support the proposal and enforce the soft fork, it would go a long way in realizing a desired economic majority.

But the UASF did not gain the level of traction some of its proponents hoped for. While a number of companies and some developers seemed onboard with BIP148, no major exchanges or other businesses declared support and some even spoke out against the initiative.

And, by mid-April, Gregory Maxwell on the Bitcoin development mailing list stated that he believed BIP148 to be a bad idea as well. Coming from one of the most respected and influential Bitcoin Core contributors, his rejection of the initiative had an impact: This version of a UASF appeared to be losing all momentum.

Instead, some began to work on an alternative UASF: BIP149.

The Altcoins

Many altcoins are based on Bitcoin’s codebase. This means that the SegWit code, while developed for Bitcoin, is largely compatible with these alternative cryptocurrencies. Unsurprisingly, therefore, several altcoins decided to implement SegWit. The first to activate Segregated Witness was Groestlcoin as early as January 2017.

But other coins were struggling. Litecoin, Vertcoin and Viacoin all seemed to have been caught in Bitcoin’s political game. These coins relied on the same miners as Bitcoin, to a large extent, and most were not signaling support for the upgrade.

This was allegedly due to technical issues or other stated reasons, but, as Viacoin lead developer Romano noted, “It seems more likely they want to refrain from activating Segregated Witness on altcoins because that would give them even less reason to hold off activation on Bitcoin.”

By April of 2017, this attitude led Litecoin creator Charlie Lee to advocate for a user activated soft fork on “his” coin. His initiative was eagerly picked up among Litecoin users; it didn’t take long for Litecoin miners, Lee, and other members of the Litecoin ecosystem to arrange an online meeting, the result of which was the Litecoin Global Roundtable Resolution. In exchange for some commitments by Lee, miners agreed to activate SegWit. ShaolinFry and others considered the UASF effort a success.

Within a week after SegWit activation on Litecoin, an unknown person made a bold move. He (or she) sent $1 million worth of the cryptocurrency to a SegWit-protected address, challenging anyone to steal the funds if they could. To this date, the bounty remains untouched, further strengthening confidence in the technology.

The New York Agreement

Meanwhile, the block size debate raged on. Another software client to increase Bitcoin’s block size limit per hard fork, Bitcoin Unlimited gained traction among Bitcoin’s mining community. Endorsed by Bitmain’s Wu in particular, the project appeared to be heading toward a potential (and controversial) hard fork.

This looming threat, and the possibility of a “split” in Bitcoin’s blockchain, was reason for DCG founder and CEO Barry Silbert to organize a meeting ahead of the Consensus 2017 conference in New York. Initially announced on a private email list for Bitcoin entrepreneurs and other prominent industry members, the meeting would bring together a significant chunk of the Bitcoin industry, including miners — though, notably, no Bitcoin Core contributors.

The outcome of that meeting is typically referred to as the “New York Agreement.” Participants agreed on what they deemed to be a compromise between those who wanted to increase Bitcoin’s block size with a hard fork and those who preferred SegWit. Based on an idea originally proposed by RSK founder Sergio Demian Lerner, SegWit would be activated under specific conditions, while there would also be a hard fork to double Bitcoin’s “base block size limit.”

The New York Agreement and its two concrete points of action

But while it sufficed to say not everyone in the Bitcoin ecosystem supported the agreement, one specific problem stood out in particular. The conditions for SegWit activation were largely incompatible with those proposed by the Bitcoin Core development team, for which the code was already widely adopted by Bitcoin users.

The Intolerant Minority

Imagery by Samson Mow in support of the BIP148 UASF

While the BIP148 UASF seemed to have lost a lot of steam in favor of BIP149, not everyone had given up on this first UASF proposal completely.

Shaolinfry had proposed the concept under the assumption that it would be backed by an economic majority and thought it should be aborted before the flag day otherwise. But a group of users on the UASF Slack channel had a different idea. Some of them — including Bitcoin Core and Bitcoin Knots developer Luke Dashjr — were contemplating activating the soft fork regardless of what the rest of the Bitcoin ecosystem would do. Even if they were a minority, and even if they’d effectively spin themselves off into a new altcoin, they would move forward with the upgrade.

Around mid-May, Alphonse Pace linked this determination to a game-theoretical concept described by statistician and author Nassim Nicholas Taleb: the “intolerant minority.” In short, this idea presupposes that even an economic minority should be able to compel miners to activate the Segregated Witness soft fork. They would unnecessarily lose a chunk of their “customer base” (Bitcoin users) otherwise.

Seemingly fuelled by the AsicBoost scandal, the SegWit activation on Litecoin and discontent regarding the New York Agreement — and this time backed by game theory — BIP148 support started to snowball into somewhat of a viral phenomenon on social media and message boards once again.

Several more articles discussed the growing potential of the UASF and much debate on social media, YouTube channels other discussion platforms followed. Meanwhile, Eric Lombrozo also threw his weight behind the effort, and UASF hats distributed by Samson Mow became the rage. Inspired by the codename for an upcoming Electrum Wallet release, August 1 was dubbed “Bitcoin Independence Day.”

The only problem: activation methods for BIP148 and the New York Agreement were as incompatible as the New York Agreement was with the activation methods proposed by the Bitcoin Core development team.

The Kludge

It was Bitmain Warranty engineer James Hilliard who came to the rescue. Hilliard proposed a slightly complex but clever solution that would make everything compatible: Segregated Witness activation as proposed by the Bitcoin Core development team, the BIP148 UASF and the New York Agreement activation mechanism. His BIP91 could keep Bitcoin whole — at least throughout SegWit activation.

As long as a majority of miners would activate BIP91 before August 1, all Bitcoin nodes should remain part of the same network. It was a relatively small time window, since the solution was only proposed by late May, but Jeff Garzik, the main developer attached to the New York Agreement, adopted the proposal and planned to release the software client resulting from that agreement weeks before August 1. It was doable.

The Activation

Informational website XBT.eu at the time of BIP91 lock-in

By mid-July, Bitcoin miners had missed their window to activate SegWit through the method proposed by the Bitcoin Core development team in time to be compatible with BIP148. As a result, markets seemed to get nervous about a potential “split” between a BIP148 chain and a non-BIP148 chain. In the span of only a week, bitcoin’s exchange rate tumbled from around $2500 to $1900: the lowest it had been in well over a month.

Possibly startled by these market movements, Bitcoin’s mining community started to rapidly signal support for BIP91, even ahead of the schedule set forth by the New York Agreement. And on July 20, ten days before BIP148’s August 1 flag day for activation, BIP91 locked in. It activated a little over two days later.

With BIP91 locked in, it was only a matter of time before Segregated Witness itself would lock in. This ultimately happened on August 9 — the point of no return having been reached on August 8.

Bitcoin would “officially” get SegWit after another two-week grace period.

The Adoption

Segregated Witness logo designed by Albert Dros

The final step for Segregated Witness is, of course, actual user adoption. Since SegWit has only just activated at the time of publication of this article, it’s impossible to know how quickly and how much the upgrade will actually be used. Some critics, perhaps most notably Garzik, predict that widespread adoption could take up to a year or even longer. Others, including a number of wallet and library developers, think they can utilize the feature within weeks, or they are prepared already. And other technologies that depend on the upgrade, such as the Lightning Network, but also Merkelized Abstract Syntax Trees (MAST), atomic swaps, faster transaction signing for hardware wallets, and TumbleBit in payment processor mode, are in various stages of development as well.

It’s been a long road, but anyone who wants to use Segregated Witness should now be able to do so, starting today.