Difference between revisions of "Vulnerability Remediation"

From Lingoport Wiki
Jump to: navigation, search
(Lingoport's Response to Major Software Vulnerabilities)
 
(27 intermediate revisions by 2 users not shown)
Line 11: Line 11:
 
Pending further action, Lingoport has shut down all non-critical systems.
 
Pending further action, Lingoport has shut down all non-critical systems.
   
Critical systems have been patched to remove all copies of log4j 2.x with log4j 2.15 followed by a hard reboot.
+
Critical systems have been patched to remove all copies of log4j 2.x with log4j 2.17.1 followed by a hard reboot.
   
 
=== For Lingoport Clients ===
 
=== For Lingoport Clients ===
   
The below scripts may be used in conjunction to replace all log4j 2.x with log4j 2.15.
+
The below scripts may be used in conjunction to replace all log4j 2.x with log4j 2.17.1.
   
  +
==== 1. Remove previous patch files, ''if any'' ====
0. Check if your system is vulnerable with the following script, which will search all of /var /tomcat /home /opt /lib for the vulnerable libraries.
 
  +
  +
If you applied patches previously, you may have log4j 2.15, 2.16, or 2.17.0 files. They are likely under the /tmp directory, as per our instructions.
  +
* Navigate to /tmp
  +
cd /tmp
  +
* Remove the files. For instance,
  +
ls apache-log4j*.zip
  +
rm apache-log4j*.zip
  +
rm -rf /tmp/apache-log4j-2.15.0-bin/
  +
rm -rf /tmp/apache-log4j-2.16.0-bin/
  +
rm -rf /tmp/apache-log4j-2.17.0-bin/
  +
  +
  +
If previous patches were not applied, this step is not necessary.
  +
  +
==== 2. Check the status of your system before the fix ====
  +
  +
Check if your system is vulnerable with the following script, which will search your system for vulnerable libraries.
   
 
This script is designed to be run as root, or via sudo:
 
This script is designed to be run as root, or via sudo:
Line 24: Line 41:
 
#!/bin/bash
 
#!/bin/bash
   
if [ "$EUID" -ne 0 ]
+
if [ "$EUID" -ne 0 ]
then echo "Please run $0 as root"
+
then echo "Please run $0 as root"
exit
+
exit
 
fi
 
fi
 
   
 
while read -r log4j_jar ; do
 
while read -r log4j_jar ; do
Line 34: Line 50:
 
continue
 
continue
 
fi
 
fi
if [[ "$log4j_jar" == *"-2.15"* ]] ; then
+
if [[ "$log4j_jar" == *"-2.17.1"* ]] ; then
#echo "Up to date: $log4j_jar"
+
echo "Up to date: $log4j_jar"
 
continue
 
continue
 
fi
 
fi
 
if [[ "$log4j_jar" == *"-1."* ]] ; then
 
if [[ "$log4j_jar" == *"-1."* ]] ; then
#echo "1.x - safe: $log4j_jar"
+
echo "1.x - safe: $log4j_jar"
 
continue
 
continue
 
fi
 
fi
 
if unzip -l "$log4j_jar" | grep -q JndiLookup.class ; then
 
if unzip -l "$log4j_jar" | grep -q JndiLookup.class ; then
echo "$log4j_jar"
+
echo "Vulnerable: $log4j_jar"
  +
else
  +
echo "Outdated: $log4j_jar"
 
fi
 
fi
done <<< "$(find / '(' -path '/var*' -o -path '/tomcat*' -o -path '/home*' -o -path '/opt*' -o -path '/lib*' ')' -name 'log4j*.jar')"
+
done <<< "$(find / -name 'log4j*.jar')"
 
</pre>
 
</pre>
   
  +
==== 3. Retrieve log4j 2.17.1 ====
 
1. Retrieve log4j 2.15:
 
   
 
<pre>
 
<pre>
 
cd /tmp/
 
cd /tmp/
curl -O https://dlcdn.apache.org/logging/log4j/2.15.0/apache-log4j-2.15.0-bin.zip
+
curl -O https://dlcdn.apache.org/logging/log4j/2.17.1/apache-log4j-2.17.1-bin.zip
unzip apache-log4j-2.15.0-bin.zip
+
unzip apache-log4j-2.17.1-bin.zip
 
</pre>
 
</pre>
   
2. Replace other log4j instances on your system with 2.15
+
==== 4. Replace other log4j instances on your system with 2.17.1 ====
   
The following script will replace vulnerable log4j libraries with 2.15. It searches all of /var /tomcat /home /opt /lib for the vulnerable libraries, and replace any that are found.
+
The following script will replace vulnerable log4j libraries with 2.17.1. It searches your system for the vulnerable libraries, and replaces any that are found.
   
 
This script is designed to be run as root or via sudo.
 
This script is designed to be run as root or via sudo.
   
 
<pre>
 
<pre>
  +
 
#!/bin/bash
 
#!/bin/bash
   
Line 72: Line 90:
 
exit
 
exit
 
fi
 
fi
 
   
 
strip_version() {
 
strip_version() {
Line 79: Line 96:
 
}
 
}
   
  +
note_not_replaced() {
if [[ ! -d /tmp/apache-log4j-2.15.0-bin/ ]] ; then
 
  +
target="$1"
echo >&2 "Please retrieve apache log4j 2.15 and unzip it in /tmp before running this script"
 
  +
  +
if unzip -l "$log4j_jar" | grep -q JndiLookup.class ; then
  +
echo >&2 "unable to replace vulnerable $log4j_jar"
  +
exit_="1"
  +
else
  +
echo >&2 "unable to replace $log4j_jar (note: vuln not detected)"
  +
fi
  +
}
  +
  +
if [[ ! -d /tmp/apache-log4j-2.17.1-bin/ ]] ; then
  +
echo >&2 "Please retrieve apache log4j 2.17.1 and unzip it in /tmp before running this script"
 
exit 1
 
exit 1
 
fi
 
fi
Line 88: Line 116:
 
continue
 
continue
 
fi
 
fi
if [[ "$log4j_jar" == *"-2.15"* ]] ; then
+
if [[ "$log4j_jar" == *"-2.17.1"* ]] ; then
 
echo "Up to date: $log4j_jar"
 
echo "Up to date: $log4j_jar"
 
continue
 
continue
 
fi
 
fi
if [[ "$log4j_jar" == *"-1."* ]] ; then
+
if [[ "$log4j_jar" =~ .*-1([0-9.]+).jar ]] ; then
 
echo "1.x - safe: $log4j_jar"
 
echo "1.x - safe: $log4j_jar"
 
continue
 
continue
 
fi
 
fi
if unzip -l "$log4j_jar" | grep -q JndiLookup.class ; then
+
if [[ "$log4j_jar" =~ .*\/log4j[^0-9]*(slf4j)?[^0-9]*.jar ]] ; then
echo "$log4j_jar"
+
if unzip -l "$log4j_jar" | grep -q JndiLookup.class ; then
  +
echo >&2 Cannot replace vulnerable versionless "$log4j_jar"
while read -r replace_target ; do
 
user_group="$(stat -c "%U:%G" "$replace_target")"
+
export exit_=1
  +
continue
without_version="$(strip_version "$replace_target")"
 
  +
fi
patched_jar="$(basename "$without_version")2.15.0.jar"
 
echo "$replace_target - $perms - $without_version - $patched_jar"
+
echo "ignoring versionless: $log4j_jar"
set -x
+
continue
cp /tmp/apache-log4j-2.15.0-bin/"$patched_jar" "$(dirname "$replace_target")"
 
chown "$user_group" "$(dirname "$replace_target")/$patched_jar"
 
mv "$replace_target" "$replace_target.orig.vulnerable"
 
set +x
 
done <<< "$(find "$(dirname "$log4j_jar")" -name "log4j*.jar")"
 
#cp "$log4j_jar" "$log4j_jar.orig.vulnerable"
 
#zip -q -d "$log4j_jar" org/apache/logging/log4j/core/lookup/JndiLookup.class
 
 
fi
 
fi
  +
if [[ "$log4j_jar" =~ .*\/log4j[^-]*[0-9.]+.jar ]] ; then
done <<< "$(find / '(' -path '/var*' -o -path '/tomcat*' -o -path '/home*' -o -path '/opt*' -o -path '/lib*' ')' -name 'log4j*.jar')"
 
  +
if unzip -l "$log4j_jar" | grep -q JndiLookup.class ; then
  +
echo >&2 Cannot replace vulnerable strange-versioned "$log4j_jar"
  +
export exit_=1
  +
continue
  +
fi
  +
echo "ignoring strange-versioned: $log4j_jar"
  +
continue
  +
fi
  +
user_group="$(stat -c "%U:%G" "$log4j_jar")"
  +
without_version="$(strip_version "$log4j_jar")"
  +
patched_jar="$(basename "$without_version")2.17.1.jar"
  +
base_dir="$(dirname "$log4j_jar")"
  +
echo "replace $log4j_jar ($user_group) - with $patched_jar"
  +
set -x
  +
cp /tmp/apache-log4j-2.17.1-bin/"$patched_jar" "$base_dir" || {
  +
set +x
  +
note_not_replaced "$log4j_jar"
  +
continue
  +
}
  +
chown "$user_group" "$base_dir/$patched_jar"
  +
mv "$log4j_jar" "$log4j_jar.orig.vulnerable"
  +
set +x
  +
done <<< "$(find / -name 'log4j*.jar')"
  +
  +
if [[ "${exit_}" == "1" ]] ; then
  +
echo "Failed!"
  +
exit 1
  +
fi
   
 
</pre>
 
</pre>
   
  +
==== 5. Check the status after the fix ====
3. You may wish to run the check script from #0 a second time to validate the fix.
 
  +
You may wish to run the check script from #2 a second time to validate the fix.
  +
Please note that the log4j 1.x are being worked on for our upcoming release (codename 'Ireland').
  +
  +
  +
==== 6. Please reboot your system ====
  +
Reboot your system after replacing your libraries. This will ensure that the patch becomes fully effective.
  +
  +
== FATAL: Could not initialize class com.lingoport.common.velocity.LingoportVelocityWriter ==
  +
  +
In versions <b>prior to Ireland</b>, if the console output of a Localyzer project shows the following type of error:
  +
  +
FATAL: Could not initialize class com.lingoport.common.velocity.LingoportVelocityWriter
  +
java.lang.NoClassDefFoundError: Could not initialize class com.lingoport.common.velocity.LingoportVelocityWriter
  +
at com.lingoport.plugins.jenkinsgyzrlrmplugin.BuildLRM.saveProjectSettings(BuildLRM.java:860)
  +
  +
or if the error is like:
  +
  +
FATAL: org/apache/log4j/Logger
  +
java.lang.ClassNotFoundException: org.apache.log4j.Logger
  +
at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1393)
  +
at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1348)
  +
at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1094)
  +
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
  +
Caused: java.lang.NoClassDefFoundError: org/apache/log4j/Logger
  +
at com.lingoport.common.velocity.LingoportVelocityWriter.<clinit>(LingoportVelocityWriter.java:45)
  +
  +
  +
it is likely that a Log4j is actually missing from the classpath. In that case,
  +
  +
copy the lingoport/LRM-server-xx/lib/log4j-x.y.z.jar to /var/lib/jenkins/plugins/jenkins-gyzr-lrm-plugin/WEB-INF/lib/.
   
  +
This should only happen with versions prior to Ireland.
4. Please reboot your system after replacing your libraries. This will ensure that the patch becomes fully effective.
 

Latest revision as of 16:36, 21 January 2022

Lingoport's Response to Major Software Vulnerabilities

Apache Log4j Security Vulnerabilities

A major security vulnerability allowing for remote code execution on affected systems.

See: https://logging.apache.org/log4j/2.x/security.html

Lingoport Response

Pending further action, Lingoport has shut down all non-critical systems.

Critical systems have been patched to remove all copies of log4j 2.x with log4j 2.17.1 followed by a hard reboot.

For Lingoport Clients

The below scripts may be used in conjunction to replace all log4j 2.x with log4j 2.17.1.

1. Remove previous patch files, if any

If you applied patches previously, you may have log4j 2.15, 2.16, or 2.17.0 files. They are likely under the /tmp directory, as per our instructions.

  • Navigate to /tmp
 cd /tmp
  • Remove the files. For instance,
  ls apache-log4j*.zip
  rm apache-log4j*.zip
  rm -rf /tmp/apache-log4j-2.15.0-bin/
  rm -rf /tmp/apache-log4j-2.16.0-bin/
  rm -rf /tmp/apache-log4j-2.17.0-bin/


If previous patches were not applied, this step is not necessary.

2. Check the status of your system before the fix

Check if your system is vulnerable with the following script, which will search your system for vulnerable libraries.

This script is designed to be run as root, or via sudo:

#!/bin/bash

if [ "$EUID" -ne 0  ]
then echo "Please run $0 as root"
     exit
fi

while read -r log4j_jar ; do
    if [[ -z "$log4j_jar" ]] ; then
        continue
    fi
    if [[ "$log4j_jar" == *"-2.17.1"* ]] ; then
        echo "Up to date: $log4j_jar"
        continue
    fi
    if [[ "$log4j_jar" == *"-1."* ]] ; then
        echo "1.x - safe: $log4j_jar"
        continue
    fi
    if unzip -l "$log4j_jar" | grep -q JndiLookup.class ; then
        echo "Vulnerable: $log4j_jar"
    else
        echo "Outdated:   $log4j_jar"
    fi
done <<< "$(find / -name 'log4j*.jar')"

3. Retrieve log4j 2.17.1

cd /tmp/
curl -O https://dlcdn.apache.org/logging/log4j/2.17.1/apache-log4j-2.17.1-bin.zip 
unzip apache-log4j-2.17.1-bin.zip 

4. Replace other log4j instances on your system with 2.17.1

The following script will replace vulnerable log4j libraries with 2.17.1. It searches your system for the vulnerable libraries, and replaces any that are found.

This script is designed to be run as root or via sudo.


#!/bin/bash

set -e

if [ "$EUID" -ne 0 ]
  then echo "Please run $0 as root"
  exit
fi

strip_version() {
    target="$1"
    echo "$target" | sed -E 's|-[0-9.]+.jar|-|'
}

note_not_replaced() {
    target="$1"

    if unzip -l "$log4j_jar" | grep -q JndiLookup.class ; then
        echo >&2 "unable to replace vulnerable $log4j_jar"
        exit_="1"
    else
        echo >&2 "unable to replace $log4j_jar (note: vuln not detected)"
    fi
}

if [[ ! -d /tmp/apache-log4j-2.17.1-bin/ ]] ; then
    echo >&2 "Please retrieve apache log4j 2.17.1 and unzip it in /tmp before running this script"
    exit 1
fi

while read -r log4j_jar ; do
    if [[ -z "$log4j_jar" ]] ; then
        continue
    fi
    if [[ "$log4j_jar" == *"-2.17.1"* ]] ; then
        echo "Up to date: $log4j_jar"
        continue
    fi
    if [[ "$log4j_jar" =~ .*-1([0-9.]+).jar ]] ; then
        echo "1.x - safe: $log4j_jar"
        continue
    fi
    if [[ "$log4j_jar" =~ .*\/log4j[^0-9]*(slf4j)?[^0-9]*.jar  ]] ; then
        if unzip -l "$log4j_jar" | grep -q JndiLookup.class ; then
            echo >&2 Cannot replace vulnerable versionless "$log4j_jar"
            export exit_=1
            continue
        fi
        echo "ignoring versionless:  $log4j_jar"
        continue
    fi
    if [[ "$log4j_jar" =~ .*\/log4j[^-]*[0-9.]+.jar  ]] ; then
        if unzip -l "$log4j_jar" | grep -q JndiLookup.class ; then
            echo >&2 Cannot replace vulnerable strange-versioned "$log4j_jar"
            export exit_=1
            continue
        fi
        echo "ignoring strange-versioned:  $log4j_jar"
        continue
    fi
    user_group="$(stat -c "%U:%G" "$log4j_jar")"
    without_version="$(strip_version "$log4j_jar")"
    patched_jar="$(basename "$without_version")2.17.1.jar"
    base_dir="$(dirname "$log4j_jar")"
    echo "replace $log4j_jar ($user_group) - with $patched_jar"
    set -x
    cp /tmp/apache-log4j-2.17.1-bin/"$patched_jar" "$base_dir" || {
        set +x
        note_not_replaced "$log4j_jar"
        continue
    }
    chown "$user_group" "$base_dir/$patched_jar"
    mv "$log4j_jar" "$log4j_jar.orig.vulnerable"
    set +x
done <<< "$(find / -name 'log4j*.jar')"

if [[ "${exit_}" == "1" ]] ; then
    echo "Failed!"
    exit 1
fi

5. Check the status after the fix

You may wish to run the check script from #2 a second time to validate the fix. Please note that the log4j 1.x are being worked on for our upcoming release (codename 'Ireland').


6. Please reboot your system

Reboot your system after replacing your libraries. This will ensure that the patch becomes fully effective.

FATAL: Could not initialize class com.lingoport.common.velocity.LingoportVelocityWriter

In versions prior to Ireland, if the console output of a Localyzer project shows the following type of error:

 FATAL: Could not initialize class com.lingoport.common.velocity.LingoportVelocityWriter
 java.lang.NoClassDefFoundError: Could not initialize class com.lingoport.common.velocity.LingoportVelocityWriter
    at com.lingoport.plugins.jenkinsgyzrlrmplugin.BuildLRM.saveProjectSettings(BuildLRM.java:860)

or if the error is like:

 FATAL: org/apache/log4j/Logger
 java.lang.ClassNotFoundException: org.apache.log4j.Logger
    at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1393)
    at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1348)
    at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1094)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
 Caused: java.lang.NoClassDefFoundError: org/apache/log4j/Logger
    at com.lingoport.common.velocity.LingoportVelocityWriter.<clinit>(LingoportVelocityWriter.java:45)


it is likely that a Log4j is actually missing from the classpath. In that case,

  copy the  lingoport/LRM-server-xx/lib/log4j-x.y.z.jar to /var/lib/jenkins/plugins/jenkins-gyzr-lrm-plugin/WEB-INF/lib/.

This should only happen with versions prior to Ireland.