KeyTrap ‘the most devastating vulnerability ever found in DNSSEC’

A security vulnerability in the DNSSEC standard that could crash DNS resolution in software such as BIND and services such as Cloudflare and Google Public DNS has been called “the most devastating vulnerability ever found in DNSSEC”.

Named KeyTrap, it enables attackers to overwhelm a DNS resolver’s CPU for as long as 16 hours, forcing it to process up to two million times its usual load, using a single malicious DNS packet, making for a potentially crippling denial-of-service attack.

The flaw was discovered last year by Elias Heftrig, Haya Schulmann, Niklas Vogel, and Michael Waidner from ATHENE, the German National Research Center for Applied Cybersecurity, and publicly disclosed last week after vendors were given time to develop and deploy patches.

While KeyTrap has been present in DNS software for almost a quarter-century, the researchers are not aware of it being exploited in the wild.

The vulnerability is actually baked into the DNSSEC technical standards, developed in the 1990s, rather than being specific to any one implementation of the specs, according to the researchers. In fact, in order to patch the problem vendors had to break with the standard RFCs.

KeyTrap works because DNSSEC is (believe it or not) designed to avoid causing downtime when it fails, so it tries too eagerly to validate cryptographic signatures by checking all the keys available to it. Exploiting this helpfulness, an attacker could trick a resolver into eating up all its CPU resources checking huge numbers of keys.

Schulmann wrote in an article explaining the vulnerability:

Our methods show a low-resource adversary can fully attack a DNS resolver with a Denial-of-Service (DoS) for up to 16 hours with a single DNS request. Members from the 31-participant task force of major operators, vendors, and developers of DNS/DNSSEC, to which we disclosed our research, dubbed our attack ‘the most devastating vulnerability ever found in DNSSEC’.

DNSSEC is designed to mitigate the risk of DNS cache-poisoning and man-in-the-middle attacks, but because its default behavior when the crypto fails is to refuse to resolve the affected domains, it can also lead to availability problems.

It’s not uncommon for entire TLDs to fail for hours when the registry screws up a DNSSEC key rollover. The web site you’re reading right now suffered downtime a few years ago due to a DNSSEC fail at the registrar level.

The KeyTrap researchers believe about 31% of web client devices currently use DNSSEC resolvers.

Russia blames DNSSEC, not Ukraine, for internet downtime

Another ccTLD has blamed DNSSEC after seeing hours of downtime affecting its country’s biggest web services yesterday.

This time it’s Russia’s ccTLD.ru, which confirmed today that it was responsible for the widely reported outages on Tuesday, which had sparked speculation that a cyber-attack related to the war in Ukraine might be the culprit.

It was rather a DNSSEC failure that affected both .ru and the Cyrillic .рф domains, the registry said. It was related to a cryptograpghic key rollover, the registry indicated.

“After the failure was detected, the updated keys were revoked, and the functionality of the .RU zone was fully restored, which took about two hours, including the distribution of data through the DNS system,” the registry said on its web site.

“The investigation into the incident is currently ongoing, but it is already clear that the main cause of the failure was the imperfection of the software used to create the encryption keys,” it added.

The explanation was echoed by Russian government officials on social media, and it’s sadly rather plausible. DNSSEC failures at ccTLDs, and to a lesser extent gTLDs, usually related to fluffed key rollovers, are rather common.

There have been similar outages reported in the last few years in Australia (twice), Namibia, Fiji, and Sweden. And those are just the ones reported on this blog. People who track this kind of thing more closely have recorded hundreds of incidents.

Second DNSSEC screw-up takes down Aussie web sites

.au domains failed to resolve for many internet users for almost an hour on Monday, after the registry operator messed up a DNSSEC update.

ccTLD overseer auDA said the issue was caused by a “key re-signing process that generated an incorrect record”. Users on ISPs that strictly enforce DNSSEC would have returned not-found errors for .au domains during the outage.

.au’s technical back-end is managed by Identity Digital, which reportedly said that the outage lasted from 0005 UTC until 0052 UTC.

With over four million domains, .au is I believe the largest TLD zone to fall victim to DNSSEC-related downtime, but it’s not the first time it has happened to the domain.

In March 2022, thousands of .au domains were affected by a DNSSEC snafu that lasted a few hours.

DNSSEC is meant to make the DNS more secure by reducing the risk of man-in-the-middle attacks, but it’s appears to be easy to screw up, judging by a list of TLD outages. Just this year, Mexico, New Zealand and Venezuela have also suffered downtime.

DNSSEC claims another ccTLD victim

A botched DNSSEC upgrade has been fingered as the source of an outage that made .na domain names inaccessible last Tuesday.

Reports and archived DNS records show that names in the Namibian ccTLD suffered as many as 12 hours of downtime following the glitch, which has been blamed on human error.

When DNSSEC-signed domains, including TLDs, are unable to establish a cryptographic chain of trust, anyone using a DNSSEC-compatible resolver will be unable to access web sites or emails of affected domains.

Namibian Network Information Center boss Eberhard Lisse, talking to The Namibian newspaper, blamed the downtime on an unspecified upstream provider pushing an algorithm upgrade “without all prerequisite steps having been completed”.

It’s the second DNSSEC incident to hit .na following a July 2019 glitch, and one of dozens to affect TLDs since the technology started to become more broadly adopted.

Another DNSSEC screw-up takes down thousands of .au domains

Australia’s ccTLD has become the latest to see a widespread outage that appears to be the result of a DNSSEC misconfiguration.

A reported 15,000 .au domains were affected, though some suspect it could have been more.

Registry overseer auDA said on Twitter that .au “experienced an error” that affected a “small number of domains” and that an investigation was underway.

Earlier today the .au domain experienced an error. We regret that a small number of domains were affected but are pleased to report the .au is back up and operating. An investigation as to the cause of this issue is underway.

— auDA (@auda) March 22, 2022

Donuts subsidiary Afilias, which runs the back-end for .au’s more that 3.4 million domains, has yet to publicly comment.

Network operators and DNS experts took to social media and mailing lists to observe that .au’s DNSSEC was broken, though it appears the problem was fixed rather quickly.

(For those following along from home, .AU DNSSEC key management was broken from 05:48 – 06:18 UTC. Quick repair by AuDA.)https://t.co/jmHAbZdWKo https://t.co/SzOi9U7Kwi

— Bill Woodcock (@woodyatpch) March 22, 2022

DNSSEC creates a chain of cryptographic keys all the way to the DNS root, and when that chain is broken by a misconfiguration such as a missing key, most DNSSEC-enabled resolvers treat the affected domains as if they simply don’t exist.

That means services such as web sites and email addresses stop working until the chain is reestablished. People not using DNSSEC resolvers wouldn’t have seen a problem.

It’s the third TLD to experience a significant outage due to DNSSEC in the last six weeks.

In February, thousands of domains in Sweden’s .se went dark for hours, and Fiji’s entire .fj zone disappeared for DNSSEC users less than two weeks ago.

The outage comes at a particularly unfortunate time in terms of public relations for auDA, which on Thursday will start making direct second-level .au registrations available for the first time.

It’s not immediately clear whether the DNSSEC fluff is related to the SLD launch.

DNSSEC claims another victim as entire TLD disappears

A country’s top-level domain disappeared from the internet for many people yesterday, apparently due to a DNSSEC key rollover gone wrong.

All domains in Fiji’s ccTLD, .fj, stopped resolving for anyone behind a strict DNSSEC resolver in the early hours of the morning UTC, afternoon local time, and stayed down for over 12 hours.

Some domains may still be affected due to caching, according to the registry and others.

The University of the South Pacific, which runs the domain, said that it had to contact ICANN’s IANA people to get the problem fixed, which took a while because it had to wait for IANA’s US-based support desk to wake up.

IANA head Kim Davies said that in fact its support runs 24/7 and in this case IANA took Fiji’s call at 2.47am local time.

Analyses on mailing lists and by Cloudflare immediately pointed to a misconfiguration in the country’s DNSSEC.

It seems Fiji rolled one of its keys for the first time and messed it up, meaning its zone was signed with a non-existent key.

Resolvers that implement DNSSEC strictly view such misconfigurations as a potential attack and nix the entire affected zone.

It happens surprisingly often, though not usually at the TLD level. That said, a similar problem hit thousands of Sweden’s .se domains, despite the registry having a decade’s more DNSSEC experience than Fiji, last month.

Domain Incite had a similar problem recently when its registrar carried on publishing DNSSEC information for the domain long after I’d stopped paying for it.

UPDATE: This post was updated with comment from IANA.

Thousands of domains hit by downtime after DNSSEC error

Sweden saw thousands of domains go down for hours on Friday, after DNSSEC errors were introduced to the .se zone file.

Local ccTLD registry IIS said in a statement that around 8,000 domains had a “technical difficulty” that started around 1530 local time and lasted around seven hours:

On the afternoon of 4/2, a problem was discovered that concerned approximately 8,000 .se domains. The problem meant that services, such as email and web, that are linked to the affected domains in some cases could not be used or reached. In total, there are approximately 1.49 million .se domains, of which approximately 8,000 were affected.

During the afternoon and evening, a thorough work was done with the troubleshooting and the error could be fixed for the affected .se domains at approximately 22.25.

The problem is believed to have been caused by incorrect DNSSEC signatures being published in the .se zone file. Any machine using a DNSSEC-validating resolver would have seen the errors and flat-out refused to resolve the domain.

This is probably the key drawback of DNSSEC — typically resolvers will treat badly signed domains as if they do not exist, rather than fail over to an unsigned, but resolving, response.

Sweden is not a DNSSEC newbie — .se was the first TLD to deploy the technology, all the way back in 2005, with services for domain holders coming a couple of years later.

DNS genius and ICANN key-holder Dan Kaminsky dies at 42

Security researcher Dan Kaminsky, best known for uncovering the so-called “Kaminsky Bug” DNS vulnerability, has reportedly died at the age of 42.

It has been widely reported that Kaminsky’s niece confirmed his death from serious complications from his longstanding diabetes.

On Twitter, she rebutted emerging conspiracy theories that his death was linked to the coronavirus vaccine, which he had received April 12, saying her uncle would “laugh” at such views.

During his career as a white-hat hacker, Kaminsky worked for companies including Cisco, Avaya, and IOActive.

He occasionally spoke at ICANN meetings on security issues, and was since 2010 one of IANA’s seven Recovery Key Share Holders, individuals trusted to hold part of a cryptographic key that would be used to reboot root zone DNSSEC in the case of a massive disaster.

But he was best known for his 2008 discovery of a fundamental flaw in the DNS protocol that allowed cache poisoning, and therefore serious man-in-the middle attacks, across millions of name servers worldwide. He worked with DNS software vendors in private to help them with their patches before the problem was publicly disclosed.

His discoveries led in part to the ongoing push for DNSSEC deployment across the internet.

The vulnerability received widespread attention, even in the mainstream media, and quickly came to bear his name.

For me, my standout memory of Kaminsky is one of his series of annual “Black Ops” talks, at the Defcon 12 conference in Las Vegas in 2004, during which he demonstrated to a rapt audience of hackers how it was possible to stream live radio by caching small chunks of audio data in the TXT fields of DNS records and using DNS queries to quickly retrieve and play them in sequence.

As well as being a bit of a DNS genius, he knew how to work a stage: the crowd went mental and I grabbed him for an interview soon after his talk was over.

His death at such a young age is a big loss for the security community.

Fraud checks coming to .ch as SWITCH renews contract

Swiss ccTLD registry SWITCH has agreed to implement new security measures as part of its contract renewal with the government.

The company said Friday that it has extended its contract to run .ch names with the telecoms regulator OFCOM for five more years, bring it up to December 2026.

But as part of the renewal, SWITCH has agreed to “speed up the adoption and implementation of technical security standards”.

This will involved financial incentives for registrars to adopt DNSSEC, the registry said.

It will also introduce measures to combat fraud at the point of registration, with SWITCH saying “in the event of suspected fraudulent intent, newly registered domain names can be used only after an identity check.”

The policy appears similar to those at other ccTLDs, including .uk, where new regs are flagged under certain circumstances (such as containing coronavirus-related terms) and cannot resolve until further checks are carried out.

Coronavirus could cause “high risk of widespread outages”, ICANN says

There’s a “high risk of widespread outages” in the DNS if ICANN can’t get enough people in the same room for its next root DNSSEC ceremony because of the coronavirus pandemic.

That’s according to ICANN’s own board of directors, which yesterday published a contingency plan that — in the worst case scenario — could see parts of the internet come to a screeching halt in July.

The problem is with the elaborate “ceremonies” that ICANN and its IANA/PTI unit uses to make sure the internet can support DNSSEC — the secure version of the DNS protocol — all the way from the root servers down.

Every quarter, ICANN, Verisign and a select few “Trusted Community Representatives” from all over the world meet in person at one of two secure US-based facilities to generate the public Zone Signing Keys for the root.

In addition to the complex cryptographic stuff happening in the computers, there’s a shedload of physical security, such as retinal scans, PIN-based locks, and reinforced walls.

And the “secret key-holders”, memorably fictionalized in a US spy drama a few years ago, actually have physical keys that they must bring to these ceremonies.

The events are broadcast live and archived on YouTube, where they typically get anything from a few hundred to a few thousand views.

Obviously, with the key-holders dotted all over the globe and most under some form of coronavirus-related lockdown, getting a quorum into the same facility at the same time — originally, Culpeper, Virginia on April 23 — isn’t going to be possible.

So IANA has made the decision to instead move the ceremony to the facility in El Segundo, California, within easy driving distance of ICANN’s headquarters, and have it carried out almost entirely by ICANN staff, wrapped in personal protective equipment and keeping their distance from each other.

The TCRs for El Segundo live in Mauritius, Spain, Russia, Tanzania, Uruguay and on the east-coast of the US, according to ICANN.

Four of these key-holders have mailed their keys to different IANA staff “wrapped in opaque material” and sealed in “tamper-evident bags”. These IANA employees will stand in for the TCRs, who will be watching remotely to verify that nothing fishy is going on.

Verisign and the independent auditors will also be watching remotely.

That’s the current plan, anyway, and I’ve no reason to believe it won’t go ahead, but ICANN’s new contingency plans do provide four alternatives.

It’s already discarded the first two options, so if the current, third, plan for the ceremony can’t go ahead before June 19 for some reason, all that would be left is the nuclear option.

Option D: Suspend signing of the DNS root zone

This is the final option if there is no conceivable way to activate the KSK and perform signing operations. There would need to be a massive education campaign at short notice to have resolver operators disable DNSSEC validation. There is a high risk of widespread outages as it is not possible to ensure global implementation, and high risk this will fatally compromise trust in DNSSEC in general as a technology.

This is considered highly unlikely, but nonetheless the final option. Without exercising the option, in the absence of a successful key signing ceremony, DNSSEC validation would be unsuccessful starting in July 2020.

The reason for this scenario is that DNSSEC keys have a finite time-to-live and after that period expires they stop functioning, which means anyone validating DNSSEC on their network may well stop resolving the signed zones.

ICANN typically generates the keys one quarter in advance, so the current key expires at the start of July.

However, the planned April 23 ceremony will generate three quarters worth of keys in advance, so the root should be good until the end of March 2021, assuming everything goes according to plan.

Clearly, the idea that half the planet might be on the verge of lockdown wasn’t taken into consideration on February 12, the last ceremony, when ICANN’s biggest problem was that it couldn’t get into one of its safes.

If you’re interested in more about the ceremony and the coronavirus-related changes, info can be found here.