Coronavirus could cause “high risk of widespread outages”, ICANN says

There’s a “high risk of widespread outages” in the DNS if ICANN can’t get enough people in the same room for its next root DNSSEC ceremony because of the coronavirus pandemic.

That’s according to ICANN’s own board of directors, which yesterday published a contingency plan that — in the worst case scenario — could see parts of the internet come to a screeching halt in July.

The problem is with the elaborate “ceremonies” that ICANN and its IANA/PTI unit uses to make sure the internet can support DNSSEC — the secure version of the DNS protocol — all the way from the root servers down.

Every quarter, ICANN, Verisign and a select few “Trusted Community Representatives” from all over the world meet in person at one of two secure US-based facilities to generate the public Zone Signing Keys for the root.

In addition to the complex cryptographic stuff happening in the computers, there’s a shedload of physical security, such as retinal scans, PIN-based locks, and reinforced walls.

And the “secret key-holders”, memorably fictionalized in a US spy drama a few years ago, actually have physical keys that they must bring to these ceremonies.

The events are broadcast live and archived on YouTube, where they typically get anything from a few hundred to a few thousand views.

Obviously, with the key-holders dotted all over the globe and most under some form of coronavirus-related lockdown, getting a quorum into the same facility at the same time — originally, Culpeper, Virginia on April 23 — isn’t going to be possible.

So IANA has made the decision to instead move the ceremony to the facility in El Segundo, California, within easy driving distance of ICANN’s headquarters, and have it carried out almost entirely by ICANN staff, wrapped in personal protective equipment and keeping their distance from each other.

The TCRs for El Segundo live in Mauritius, Spain, Russia, Tanzania, Uruguay and on the east-coast of the US, according to ICANN.

Four of these key-holders have mailed their keys to different IANA staff “wrapped in opaque material” and sealed in “tamper-evident bags”. These IANA employees will stand in for the TCRs, who will be watching remotely to verify that nothing fishy is going on.

Verisign and the independent auditors will also be watching remotely.

That’s the current plan, anyway, and I’ve no reason to believe it won’t go ahead, but ICANN’s new contingency plans do provide four alternatives.

It’s already discarded the first two options, so if the current, third, plan for the ceremony can’t go ahead before June 19 for some reason, all that would be left is the nuclear option.

Option D: Suspend signing of the DNS root zone

This is the final option if there is no conceivable way to activate the KSK and perform signing operations. There would need to be a massive education campaign at short notice to have resolver operators disable DNSSEC validation. There is a high risk of widespread outages as it is not possible to ensure global implementation, and high risk this will fatally compromise trust in DNSSEC in general as a technology.

This is considered highly unlikely, but nonetheless the final option. Without exercising the option, in the absence of a successful key signing ceremony, DNSSEC validation would be unsuccessful starting in July 2020.

The reason for this scenario is that DNSSEC keys have a finite time-to-live and after that period expires they stop functioning, which means anyone validating DNSSEC on their network may well stop resolving the signed zones.

ICANN typically generates the keys one quarter in advance, so the current key expires at the start of July.

However, the planned April 23 ceremony will generate three quarters worth of keys in advance, so the root should be good until the end of March 2021, assuming everything goes according to plan.

Clearly, the idea that half the planet might be on the verge of lockdown wasn’t taken into consideration on February 12, the last ceremony, when ICANN’s biggest problem was that it couldn’t get into one of its safes.

If you’re interested in more about the ceremony and the coronavirus-related changes, info can be found here.

The internet is still working after KSK roll

The first-ever change to the security keys at the top of the DNS tree appears to have been a non-event.
While ICANN received reports of some disruptions after last Thursday’s KSK rollover, the impact appears to have fallen short of the millions of users that had been speculated.
ICANN said yesterday:

After evaluation of the available data, there does not appear to be a significant number of Internet end-users who have been persistently and negatively impacted by the changing of the key.
The few issues that have arisen appear to have been quickly mitigated and none suggested a systemic failure that would approach the threshold (as defined by the ICANN community) to initiate a reversal of the roll. In that context, it appears the rollover to the new Key Signing Key, known as KSK 2017, has been a success.

The KSK, also sometimes called the “trust anchor”, is the ultimate cryptographic key in the chain that secures all DNSSEC queries on the internet.
October 11 was the first time it had been changed since the first version came online in 2010.
While changing the key was broadly considered sound security practice, the roll was delayed by a year after it was discovered that potentially millions of endpoints were using DNS resolvers not properly configured to use the 2017 key.
After much research, outreach and gnashing of teeth, it was decided that the risk posed by rolling the KSK now fell within acceptable parameters of collateral damage.
Experts from the likes of Google and Verisign, and one ICANN director, had urged caution and said perhaps the roll should be delayed further while more data was gathered.
But they were in the minority, ICANN went ahead anyway, and it seems their fears have not come to pass.
The KSK is now likely to be rolled regularly — it could be as little as once every five years, or more frequently.
It also gives ICANN the opportunity to eventually update the system to swap out its current RSA keys for keys based on elliptical curve cryptography, which could reduce the traffic load on the DNS as a whole.

KSK vote was NOT unanimous

ICANN’s board of directors on Sunday voted to approve the forthcoming security key change at the DNS root, but there was some dissent.
Director Avri Doria, a Nominating Committee appointee, said today that she provided the lone vote against the DNSSEC KSK rollover, which is expected to cause temporary internet access problems for potentially a couple million people next month.
I understand there was also a single abstention to Sunday’s vote.
Doria has released a dissenting statement, in which she said the absence of an external, peer-reviewed study of the risks could prove a problem.

The greatest risk is that out of the millions that will fail after the roll over, some that are serious and may even be critical, may occur; if this happens the lack of peer reviewed studies may be a liability for ICANN, perhaps not legal, but in terms of our reputation as protectors of the stability & security of internet system of names.

She added that she was concerned about the extent that the public has been notified of the rollover plan, and questioned whether the current risk mitigation plan is sufficient.
Doria said she found comments filed by Verisign (pdf) particularly informative to her eventual vote, as well as comments from the At-Large Advisory Committee (pdf), Business Constituency (pdf) and Registries Stakeholder Group (pdf).
These groups had called for more study and data, better outreach, more clearly defined success/failure benchmarks, and more delay.
Doria noted in her dissenting statement that the ICANN board did not have a chance to quiz any of the minority of the members of the Security and Stability Advisory Committee who had called for further delay.
The board’s resolution, apparently arrived at after two hours of formal in-person discussions in Brussels at the weekend, is expected to be published shortly.
The rollover, which has already been delayed a year, is now scheduled to go ahead October 11.
Any impact is expected to be felt within a couple of days, as the change ripples out across the DNS.
ICANN says that any network operator impacted by the change has a simple fix: turn off DNSSEC. Then, if they want, they can update their keys and turn it back on again.

Set buttocks to clench! ICANN approves risky KSK rollover

ICANN has approved the first rollover of the domain name system’s master security key, setting the clock ticking on a change that could cause internet access issues for millions.
The so-called KSK rollover, when ICANN deletes the key-signing key that has been used as the trust anchor for the DNSSEC ecosystem since 2011 and replaces it with the new one — will now go ahead as planned on October 11.
The decision was made yesterday at the ICANN board of directors’ retreat in Brussels.
ICANN chief technology officer David Conrad posted this to an ICANN mailing list this morning:

The Board voted to approve the resolution for ICANN org to move forward with the revised KSK rollover plan. So barring unforeseen circumstances, the KSK-2017-signed ZSK will be used to sign the root zone on 11 October 2018.

The rollover was due to happen October 11 last year, but ICANN delayed it when it emerged that many DNS resolvers weren’t yet configured to use the new key.
That’s still a problem, and nobody knows for sure how many endpoints will stop functioning properly when the new KSK goes solo.
While most experts weighing in on the rollover, including Conrad, agreed that the risk of more delay outweighed the risk of rolling now, that feeling was not unanimous.
Five members of the 22-member Security and Stability Advisory Committee — including top guys from Google and Verisign — last month dissented from the majority view and said ICANN should delay again.
The question now is not whether internet users will see a disruption in the days following October 11, but how many users will be affected and how serious their disruptions will be.
Based on current information, as many as two million internet users could be affected.
ICANN is likely to take flak for even relatively minor disruptions, but the alternative was to continue with the delays and risk an even bigger impact, and even more flak, in future.
The text of ICANN’s resolution and the rationale behind it will be published in the next day or so.

ICANN faces critical choice as security experts warn against key rollover

Members of ICANN’s top security body have advised the organization to further delay plans to change the domain name system’s top cryptographic key.
Five dissenting members of the influential, 22-member Security and Stability Advisory Committee said they believe “the risks of rolling in accordance with the current schedule are larger than the risks of postponing”.
Their comments relate to the so-called KSK rollover, which would see ICANN for the first time ever change the key-signing key that acts as the trust anchor for all DNSSEC queries on the internet.
ICANN is fairly certain rolling the key will cause DNS resolution problems for some — possibly as much as 0.05% of the internet or a couple million people — but it currently lacks the data to be absolutely certain of the scale of the impact.
What it does know — explained fairly succinctly in this newly published guide (pdf) — is that within 48 hours of the roll, a certain small percentage of internet users will start to see DNS resolution fail.
But there’s a prevailing school of thought that believes the longer the rollover is postponed, the bigger that number of affected users will become.
The rollover is currently penciled in for October 11, but the ultimate decision on whether to go ahead rests with the ICANN board of directors.
David Conrad, the organization’s CTO, told us last week that his office has already decided to recommend that the roll should proceed as planned. At the time, he noted that SSAC was a few days late in delivering its own verdict.
Now, after some apparently divisive discussions, that verdict is in (pdf).
SSAC’s majority consensus is that it “has not identified any reason within the SSAC’s scope why the rollover should not proceed as currently planned.”
That’s in line with what Conrad, and the Root Server System Advisory Committee have said. But SSAC noted:

The assessment of risk in this particular area has some uncertainty and therefore includes a component of subjective judgement. Individuals (including some members of the SSAC) have different assessments of the overall balance of risk of the resumption of this plan.

It added that it’s up to the ICANN board (comprised largely of non-security people) to make the final call on what the acceptable level of risk is.
The minority, dissenting opinion gets into slightly more detail:

The decision to proceed with the keyroll is a complex tradeoff of technical and non-technical risks. While there is risk in proceeding with the currently planned roll, we understand that there is also risk in further delay, including loss of confidence in DNSSEC operational planning, potential for more at-risk users as more DNSSEC validation is deployed, etc.
While evaluating these risks, the consensus within the SSAC is that proceeding is preferable to delay. We personally evaluate the tradeoffs differently, and we believe that the risks of rolling in accordance with the current schedule are larger than the risks of postponing and focusing heavily on additional research and outreach, and in particular leveraging newly developed techniques that provide better signal and fidelity into potentially impacted parties.
We would like to reiterate that we understand our colleagues’ position, but evaluate the risks and associated mitigation prospects differently. We believe that the ultimate decision lies with the ICANN Board, and do not envy them with this decision.

SSAC members are no slouches when it comes to security expertise, and the dissenting members are no exception. They are:

• Lyman Chapin, co-owner of Interisle Consulting, a regular ICANN contractor perhaps best-known to DI readers for carrying out a study into new gTLD name collisions five years ago.
• Kimberly “kc claffy” Claffy, head of the Center for Applied Internet Data Analysis at the University of California in San Diego. CAIDA does nothing but map and measure the internet.
• Jay Daley, a registry executive with a technical background whose career includes senior stints at .uk and .nz. He’s currently keeping the CEO’s chair warm at .org manager Public Interest Registry.
• Warren Kumari, a senior network security engineer at Google, which is probably the largest early adopter of DNSSEC on the resolution side.
• Danny McPherson, Verisign’s chief security officer. As well as .com, Verisign runs the two of the 13 root servers, including the master A-root. It’s running the boxes that sit at the top of the DNSSEC hierarchy.

It may be the first time SSAC has failed to reach a full-consensus opinion on a security matter. If it has ever published a dissenting opinion before, I certainly cannot recall it.
The big decision about whether to proceed or delay is expected to be made by the ICANN board during its retreat in Brussels, a three-day meeting that starts September 14.
Given that ICANN’s primary mission is “to ensure the stable and secure operation of the Internet’s unique identifier systems”, it could turn out to be one of ICANN’s biggest decisions to date.

Root crypto rollover now slated for October

ICANN has penciled in October 11 as the new date for rolling the DNS root’s cryptographic keys, a delay of a year from its original plan.
The so-called KSK rollover will see ICANN remove the deprecated 2010 Key Signing Key, leaving only the 2017 KSK active.
The KSK acts as the “trust anchor” for DNSSEC across the whole internet.
After the rollover, any network not configured to use the latest KSK would see a service interruption.
This could mean many millions of internet users being affected, but ICANN doesn’t know the extent of the possible impact for sure.
ICANN told us in November that it knows of 176 organizations in 41 countries, fairly evenly spread across the globe, that are currently not prepared to handle the new KSK.
But its data is patchy because only a tiny number of DNS resolvers are actually configured to automatically report which KSKs they’re set up to use.
Key rollovers are recommended by DNSSEC experts to reduce the risk of brute force attacks against old keys. At the root, the original plan was to roll the keys every five years.
ICANN had named October 11 2017 as the date for the first such rollover, but this was pushed back to some time in the first quarter after ICANN became aware of the lack of support for the 2017 KSK.
This was pushed back again in December to Q3 at the earliest, after ICANN admitted it still didn’t have good enough data to measure the impact of a premature roll.
Since then, ICANN has been engaged in (not always successful) outreach to networks it knows are affected and has kicked off discussions among network operators (there’s a fairly lively mailing list on the topic) to try to gauge how cautious it needs to be.
It’s now published an updated plan that’s the same as the original plan but with a date exactly one year late — October 11, 2018.
Between now and then, it will continue to try to get hold of network operators not ready to use the new keys, but it’s not expecting to completely eliminate damage. The plan reads:

Implicit in the outreach plan is the same assumption that the community had for the earlier (postponed) plan: there will likely be some systems that will fail to resolve names starting on the day of the rollover. The outreach will attempt to minimize the number of affected users while acknowledging that the operators of some resolvers will be unreachable.

The plan is open for public comment and will require the assent of the ICANN board of directors before being implemented. You have until April 2 to respond.

Second delay for domain security key rollover

ICANN has decided to delay changing the security keys to the DNS for the second time.
The “KSK Rollover” had been rescheduled from October 11 to some time in the first quarter 2018, but that will no longer happen. We’re now looking at Q3 at the earliest.
“We have decided that we do not yet have enough information to set a specific date for the rollover,” VP of research Matt Larson said in a blog post. “We want to make clear, however, that the ICANN org is committed to rolling the root zone KSK”.
The root KSK, or Key Signing Key, is the cryptographic key pair at the very top of the security hierarchy specified by DNSSEC, the security extension for DNS.
The current, first-ever, root KSK has been in operation since 2010, but ICANN’s policy is to roll it every five years or so.
The October date was delayed after newly available data showed that hundreds of DNS resolvers were still only configured to use the 2010 keys and not the 2017 keys that have already been deployed in tandem.
This would mean a rollover would cut off access to DNSSEC-signed zones to potentially millions of internet users.
ICANN found that 4% of the 12,000 DNSSEC-validating resolvers — roughly 500 IP addresses — it surveyed in September were not ready for KSK-2017.
Larson told us last month that at least 176 organizations in 41 countries were affected.
Since the first delay, ICANN has been trying to contact the owners of the 500 incompatible IP addresses but has run into some serious problems, Larson blogged.
First, a significant number of these addresses are dynamically allocated (such as to home broadband hubs) meaning tracking down the owners of the misconfigured devices would be next to impossible. Others were forwarding DNS queries on behalf of other devices, creating a similar problem.
Additionally, it seems ICANN has still not received responses from owners of 80% of the affected IP addresses.
Due to the lack of reliable data, it’s difficult for ICANN to figure out how many users’ internet access will be affected by a rollover.
The threshold called for by current policy is about 20 million people.
So ICANN has delayed the event to some point after Q1. Larson wrote that the organization will publish a plan on January 18 which will be open for public comment and discussed at the ICANN 61 meeting in Puerto Rico next March.
A final plan is not expected until ICANN 62, which happens in late June, so Q3 would be the earliest the rollover could actually occur.
Larson encouraged anyone interested in discussing the plan to join this mailing list.

Up to 20 million people could get broken internet in domain security rollover

Twenty million people losing access to parts of the internet is considered an acceptable level of collateral damage for ICANN’s forthcoming DNS root security update.
That’s one of a number of facts and figures to emerge from recent updates from the organization, explaining its decision to delay the so-called “KSK rollover” from October 11 to some time in the first quarter next year.
The rollover will see a new Key Signing Key, used as the trust anchor for all DNSSEC-signed domains, replace the seven-year-old original.
DNSSEC protects internet users and registrants from domain-based man-in-the-middle attacks. It’s considered good practice to roll keys at each level of the DNS hierarchy periodically, to reduce the risk of successful brute-force attacks.
The root KSK update will affect hundreds of millions of people who currently use DNSSEC-compatible resolvers, such as Google DNS.
ICANN delayed the rollover after it, rather fortuitously, spotted that not all of these resolvers are configured to correctly handle the change.
The number of known incompatible servers is quite small — only about 500 of the 11,982 DNSSEC-using recursive servers initially surveyed (pdf). That represents only a very small minority of the world’s internet users, as most are not currently using DNSSEC.
Subsequent ICANN research, presented by principal researcher Roy Arends at ICANN 60 last week, showed that:

• There are currently about 4.2 million DNS resolvers in the world.
• Of those, 27,084 are configured to tell the root servers which KSKs they support (currently either the KSK-2011 or KSK-2017).
• Of those, 1,631 or 6.02% do not support KSK-2017

It was only possible to survey servers that have turned on a recent update to DNS software such as BIND and Unbound, so the true number of misconfigured servers could be much higher.
Matt Larson, ICANN’s VP of research, told DI that ICANN has identified 176 organizations in 41 countries that are currently not prepared to handle the new KSK. These organizations are fairly evenly spread geographically, he said.
Since making the decision to delay the rollover, ICANN has hired a contractor to reach out to these network operators to alert them to potential problems.
ICANN’s CEO Goran Marby has also been writing to telecommunications regulators in all countries to ask for assistance.
After the rollover, people using an incompatible resolver would be unable to access DNSSEC-signed domains. Again, that’s still quite a small minority of domains — there are only about 750,000 in .com by some accounts and apparently none of the top 25 site support it.
ICANN could roll back the change if it detects that a sufficiently large number of people are negatively affected, but that number turns out to be around 20 million.
According to its published rollover plan:

Rollback of any step in the key roll process should be initiated if the measurement program indicated that a minimum of 0.5% of the estimated Internet end-user population has been negatively impacted by the change 72 hours after each change has been deployed into the root zone.

According to InternetWorldStats, there were around 3,885,567,619 internet users in the world this June. It’s very likely more people now.
So a 0.5% threshold works out to about 19 million to 20 million people worldwide.
Larson agreed that in absolute terms, it’s a big number.
“The overall message to take away from that number, I suggest, is that a problem would have to be pretty serious for us to consider rolling back,” Larson, who was not on the team that came up with the threshold, said.
“I think that’s a reasonable position considering that, in the immediate aftermath of the rollover, there are two near-immediate fixes available to any operator experiencing problems: update their systems’ trust anchors with the new key or (less desirable from my perspective but still effective) simply disable DNSSEC validation,” he said.
He added that the 0.5% level is not a hard and fast rule, and that ICANN could be flexible in the moment.
“For example, if when we roll the key, we find out there’s some critical system with a literal life or death impact that is negatively affected by the KSK roll, I think I can pretty confidently state that we wouldn’t require the 0.5% of Internet user threshold to be met before rolling back if it looked like there would be a significant health and safety risk not easily mitigated,” he said.
The chances of such an impact are very slim, but not impossible, he suggested.
It’s not ICANN’s intention to put anyone’s internet access at risk, of course, which is why there’s a delay.
ICANN’s plan calls for any rollover to happen on the eleventh day of a given calendar quarter, so the soonest it could happen would be January 11.
Given the complexity of the outreach task in hand, the relative lack of data, and the holiday periods approaching in many countries, and ICANN’s generally cautious nature, I’d hazard a guess we might be looking at April 11 at the earliest instead.

ICANN just came thiiis close to breaking the internet

ICANN has decided to postpone an unprecedented change at the DNS root after discovering it could break internet for potentially millions of users.
The so-called KSK Rollover was due to go ahead on October 11, but it’s now been pushed back to — tentatively — some time in the first quarter 2018.
The delay was decided after ICANN realized that there were still plenty of ISPs and network operators that weren’t ready for the change.
Had ICANN gone ahead anyway with the change anyway, it could have seen subscribers of affected ISPs lose access to millions of DNSSEC-supporting domain names.
So the postponement is a good thing.
A KSK or Key Signing Key is a public-private cryptographic key pair used to sign other keys called Zone Signing Keys. The root KSK signs the root ZSK and is in effect the apex of the DNSSEC hierarchy.
The same KSK has been in operation at the root since 2010, when the root was first signed, but it’s considered good practice to change it every so often to mitigate the risk of brute-force attacks against the public key.
While it’s important enough to get dramatized in US spy shows, in practice it only affects ISPs and domain names that voluntarily support DNSSEC.
ICANN estimates that 750 million people use DNSSEC, which is designed to prevent problems such as man-in-the-middle attacks against domain names.
That’s a hell of a lot of people, but it’s still a minority of the world’s internet-using population. It’s not been revealed how many of those would have been affected by a premature rollover.
When DNSSEC fails, people whose DNS resolvers have DNSSEC turned on (Comcast and Google are two of the largest such providers) can’t access domain names that have DNSSEC turned on (such as domainincite.com).
Preventing the internet breaking is pretty much ICANN’s only job, so it first flagged up its intention to roll the root KSK back in July last year.
In July this year, the new public KSK was uploaded as part of a transition phase that is seeing the 2010 keys and 2017 keys online simultaneously.
Last year, CTO David Conrad told us the long lead time and cautious approach was necessary to get the word out that ISPs needed to test their resolvers to make sure they would work with the new keys.
In June, ICANN CEO Goran Marby spammed the telecommunications regulators in every country in the world with a letter (pdf) asking them to coordinate their home ISPs to be ready for the change.
The organization’s comms teams has also been doing a pretty good job getting word of the rollover into the tech press over the last few months.
But, with a flashback to the new gTLD program, that outreach doesn’t seem to have reached out as far as it needed to.
ICANN said last night that a “significant number” of ISPs are still not ready for the rollover.
It seems ICANN only became aware of this problem due to a new feature of DNS that reports back to the root which keys it is configured to use.
Without being able to collate that data, it’s possible it could have been assumed that the situation was hunky-dory and the rollover might have gone ahead.
ICANN still isn’t sure why so many resolvers are not yet ready for the 2017 KSK. It said in a statement:

There may be multiple reasons why operators do not have the new key installed in their systems: some may not have their resolver software properly configured and a recently discovered issue in one widely used resolver program appears to not be automatically updating the key as it should, for reasons that are still being explored.

It’s not clear why the broken resolver software has not been named — one would assume that getting the word out would be a priority unless issues of responsible disclosure were in play.
ICANN said it is “reaching out to its community, including its Security and Stability Advisory Committee, the Regional Internet Registries, Network Operator Groups and others to help explore and resolve the issues.”
The organization is hopeful that it will be able to go ahead with the rollover in Q1 2018, but noted that would be dependent on “more fully understanding the new information and mitigating as many potential failures as possible.”
While it’s excellent news that ICANN is on top of the situation, the delay is unlikely to do anything to help the perception that DNSSEC is mainly just an administrative ball-ache and far more trouble than it’s worth.

Want to be one of the internet’s SEVEN SECRET KEY-HOLDERS? Apply now!

ICANN has put out a call for volunteers, looking for people to become what are sometimes referred to as “the internet’s seven secret key holders”.
Specifically, it needs Trusted Community Representatives, people of standing in the internet community who don’t mind carrying around a small key and getting a free trip to Los Angeles or Virginia once or twice a year.
The TCRs are used in the paranoia-inducing cryptographic key-signing ceremonies that provide DNSSEC at the root of the domain name system.
The ceremonies take place at ICANN data centers four times a year. The ceremonies themselves take hours, involve multiple layers of physical and data security, and the volunteers are expected to hang around for a day or two before and after each.
There’s no compensation involved, but the TCRs are allowed to apply to ICANN for travel reimbursements.
ICANN expects TCRs to stick around for about five years, but the large majority of the 28 people who act as TCRs (yeah, it’s not seven, it’s 28) have been in the role since 2010 and ICANN is probably planning a cull.
Other than knowing what the DNS is and how it works, the primary requirements are “integrity, objectivity, and intelligence, with reputations for sound judgment and open minds”.
If you think you tick those boxes, head here to apply.