IDBS and the Log4j2 Vulnerability

Some people see December as the month that brings lots of gifts, but in 2021 it included several that no one wanted, including a critical security vulnerability in a widely used library - Apache Log4j2. Addressing this vulnerability required pretty much everyone everywhere to make changes. In this article we’d like to share what we did at IDBS to mitigate it.

The Log4j2 vulnerability

Apache Log4j is a widely used java library that provides a logging API. It provides a means to log messages at varying levels of importance, in a fast efficient manner. There are numerous configuration options that can be set at runtime that control the type, appearance & ultimately the location of the log messages.

All of these abilities have really made Log4j the de facto standard for java applications.

So where’s the problem?

In December, researchers discovered a vulnerability within the library where a specific feature (JNDI lookup) did not offer adequate protection against the response from the lookup server (identified as CVE-2021-44228). This meant that a malicious attacker could push data into the system that triggered the lookup, which would contact a server under their control, and execute arbitrary code from that server.

This is known as Remote Code Execution (RCE) and this vulnerability is one of the most severe that can occur (rated Critical: 10.0 on the NIST National Vulnerability Database). One of the more common activities attackers have been using is to install crypto mining software onto vulnerable systems (as this requires little to no knowledge of the targeted application). But if an attacker had the time and experience, they could use this to hack into a system to uncover application specific data (such as usernames and secrets). Active attempts to exploit this vulnerability were seen in the wild by security researchers at least a week before an initial fix was officially made available https://blog.talosintelligence.com/2021/12/apache-log4j-rce-vulnerability.html.

And because log4j is used everywhere, this affected everyone, from Amazon Web Services, all the way through to Minecraft.

The log4j developers responded to this, and as people and companies rushed to understand and address the issue, it drew scrutiny from security experts who discovered additional logging vulnerabilities along the way:

• CVE-2021-45046 - (Log4j2) an incomplete fix of the original vulnerability.
• CVE-2021-45105 - (Log4j2) potential denial of service (DoS) using non default pattern layouts.
• CVE-2021-44832 - (Log4j2) RCE vulnerability using JDBC appenders.
• CVE-2021-4104 - (Log4j1) vulnerability if someone could modify the local configuration file.
• CVE-2021-42550 - (Logback) vulnerability if someone could modify the local configuration file.

The impact on IDBS

IDBS constantly monitors our systems both for known vulnerabilities as they are reported and identifying any malicious infiltration attempts. Very soon after the publication of the Log4j2 vulnerability, these systems did detect attempts from external agents trying to exploit the vulnerability, alerting us to the situation. Due to the design of our systems, these attempts were being blocked before they could reach any potentially vulnerable servers, but we did not want to rely on this alone.

IDBS does use Log4j2 in some of our applications, and following the announcement of the vulnerability, the IDBS Security team initiated several activities:

• Assessing the vulnerability of our SaaS systems to this exploit
• Monitoring our intrusion detection tools for attacks and breaches
• Analysing our software applications
• Analysing our suppliers
• Validating and applying proposed mitigations (a challenge with the continuing discovery of new vulnerabilities)

This allowed us to identify affected systems and apply a mitigation to them. This was a “stop the line” activity for us - the whole engineering department were called in to assist, pausing their other work. They analysed the codebase, monitored our suppliers progress in addressing the issue, and ensured that potential mitigations did not have an adverse impact on our products.

And identifying a suitable mitigation was not as straight-forward as it might sound. While the community was quick to advertise a simple configuration change as a fix, this was later discredited as it did not address all code paths to the vulnerability, requiring a different approach.

Mitigating the vulnerability

At IDBS we believe in a “defence in depth” approach to security. Our SaaS systems are designed and built with this in mind and include multiple layers of protection. This allows us to perform live threat monitoring and prevent attacks like these from reaching our systems. But this wasn’t enough - we wanted to guarantee that should an attacker get through these layers, then the vulnerability was not present. We applied a validated mitigation to all our SaaS systems and ensured that a secure version of the library was included in our next release. At the same time, we worked with our suppliers to make certain their systems were not vulnerable.

As the subsequent vulnerabilities were published, each one was reviewed and we determined that they did not impact IDBS products and systems.

While we’ve talked a lot about SaaS systems, we also have customers with on-premise installations, and while these are generally safer against attack (e.g., being on an internal network) they are still at risk, and so we also worked to provide mitigation advice for those customers as well.

Continuous Improvement

Unfortunately, these kind of situations will never go away. Dependencies on third party products, libraries or services always introduce a risk, but they bring with them benefits that outweigh these risks. We cannot predict everything that will happen, but at IDBS we have processes, procedures and tools in place to minimise the risk and reduce the risk of obvious flaws in our software. We also have security issue/incident processes that deal with these kind of situations - which can be clearly demonstrated here. And finally, IDBS has a continuous improvement mindset - there are always lessons to learn, and room to grow. As incidents such as Log4j2 happen, we’ll take those learnings and improve these processes, ready for next time.