-
Notifications
You must be signed in to change notification settings - Fork 186
jakarta taglibs standard update broke Manager (bsc#1044804)
Bug reference: https://bugzilla.suse.com/show_bug.cgi?id=1044804
Description:
On 06/15/2017 09:16 AM, wrote:
not sure if you have seen it, but since yesterday spacewalk-java failed to build in all projects. This includes 3.1 where we did not submit new code.
This means that again a release of a maintenance update caused trouble:-(
But this time I really hope it is only a build time problem.
osc -A https://api.suse.de rbl Devel:Galaxy:Manager:3.1 spacewalk-java SLE_12_SP2 x86_64
With this command you can see the java exception. AFAIK there was a release of the jakarta-taglib library.
Indeed https://www.suse.com/support/update/ shows:
But it was already a problem to prove that this was really the culprit, as symptom did not point directly to this package:
[ 138s] jspcompile:
[ 138s] [mkdir] Created dir: /home/abuild/rpmbuild/BUILD/spacewalk-java-git-0.e665778/build/jsp-temp
[ 138s] [mkdir] Created dir: /home/abuild/rpmbuild/BUILD/spacewalk-java-git-0.e665778/build/apidoc-temp
[ 139s] [jasper2] Jun 16, 2017 7:02:23 AM org.apache.jasper.servlet.TldScanner scanJars
[ 139s] [jasper2] INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
[ 139s]
[ 139s] BUILD FAILED
[ 139s] /home/abuild/rpmbuild/BUILD/spacewalk-java-git-0.e665778/build.xml:857: The following error occurred while executing this line:
[ 140s] /home/abuild/rpmbuild/BUILD/spacewalk-java-git-0.e665778/build.xml:645: org.apache.jasper.JasperException: <h3>Validation error messages from TagLibraryValidator for c in /WEB-INF/decorators/layout_c.jsp</h3><p>null: org.xml.sax.SAXNotRecognizedException: Feature 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.</p>
- SAX and DOM APIs in Java are standardized under the name JAXP (JSR 5), and implementations are pluggable (Apache Xerces being the most common implementation)
- most JDKs vendor one or more JAXP implementations - IBM Java 8 vendors has three, all based on Xerces
- I did a brutal search on all jar files in my system:
for jar in `find . -name "*.jar"`; do echo $jar; unzip -l $jar | grep SAXParser; echo; echo ; done > /tmp/xerces_implementation_list.txt
- JAXP implementations optionally offer support to a number of features defined by URLS (eg. "secure processing" is identified by http://javax.xml.XMLConstants/feature/secure-processing)
- JAXP implementations shipped by JDKs typically lack some features (see History in this StackOverflow page), so people end up adding Xerces to their dependency lists anyway
- reproduce the issue
-
osc build
of spacewalk-java was sufficient to reproduce the problem (even in Ubuntu), but was cumbersome
-
- prove that the issue was due to the update, get an easier reproduction path
- deploying via
manager-build.xml
and visiting any JSP page initially worked, becausetaglibs-core.jar
andtaglibs-standard.jar
came from Ivy, hence our jar cache did not have the patch - deploying via
manager-build.xml
while using the patched version of the jars (extracted from the updated RPM and placed inbuildconf/testjars
), then visiting a JSP page DID trigger the same problem, as JSPs are not precompiled by that script. A similar stack trace could be found inrhn_web_ui.log
- note that Tomcat only attempts to compile JSPs once. A tomcat restart was needed to reproduce
- deploying via
- understand why the update patch lead to that failure
- looking at the patch source, one can see that the "secure processing" feature is set, and the stack trace is about the JAXP implementation not supporting it:
https://build.suse.de/package/view_file/SUSE:SLE-12:Update/jakarta-taglibs-standard/CVE-2015-0254.patch?expand=1 lines 2090, 1757
- understanding which JAXP implementation is in use
- approach 1: look at what the JAXP implementation is doing. We need to add
-Djaxp.debug=1
to thejava
commandline. Inside of our specfile, for example, one could use: export IBM_JAVA_OPTIONS='-Djaxp.debug=1' ant -Dprefix=$RPM_BUILD_ROOT install-tomcat8-suse - approach 2: look at what the Class Loader is loading. We need to add
-verbose:class
to thejava
commandline - with both approaches, it seemed that the JAXP implementation was coming from the JDK
- approach 1: look at what the JAXP implementation is doing. We need to add
- try to use a different implementation
- JAXP implementations can be swapped out in a number of ways (system properties,
META-INF
...) but none seemed to work - the JAXP implementation that ended up being used was always the same
- JAXP implementations can be swapped out in a number of ways (system properties,
export IBM_JAVA_OPTIONS='-Djavax.xml.parsers.SAXParserFactory=org.apache.xerces.jaxp.SAXParserFactoryImpl
- figuring out why swapping the implementation didn't work
- exclude that our version of Xerces did not support the feature. Checked xerces source archive directly and the feature that we missed actually existed since version 2.7.0 (we shipped in SLES 2.8.1)
- try to attach a debugger to analyse behaviour step-by-step
- use the
manager-build.xml
approach above to reproduce in Tomcat - attach a debugger and step: Tomcat sources were missing
- download Tomcat source RPMs, unpack them, attach sources to Eclipse
-
jakarta-taglibs-standard
sources are missing- download the source RPM, unpack it, apply source patches manually (in the order and with the flags given in the specfile), attach sources to Eclipse
-
jakarta-taglibs-standard
jar did not have debug symbols, hence stepping is impossible- branch the package, add one missing option to the Ant configuration in the specfile, rebuild:
- use the
cat > build.properties <<EOBP
build.dir=build
dist.dir=dist
servlet24.jar=$(build-classpath servletapi5)
jsp20.jar=$(build-classpath jspapi)
xalan.jar=$(build-classpath xalan-j2)
+compile.debug=true
EOBP
%build
ant \
-Dfinal.name=%{short_name} \
-Dj2se.javadoc=%{_javadocdir}/java \
-f standard/build.xml \
dist
- looking at the patch again it is now possible to see that it uses a hacked class loader, which is not sensitive to those options (see the
runWithOurClassLoader
method)
-
Feb 2015, a vulnerability is disclosed in Apache Standard Taglibs. Upstream advice: update to version 1.2.3 or higher
-
few days later, it was decided to postpone the fix, because new version requires Maven to build and backporting the patch is difficult
-
Oct 2015, Debian folks have done the heavy lifting, patch is taken in openSUSE
-
June 2017, patch lands in SLE breaking SUSE Manager
-
reason of the breakage is that the Debian patch is very intrusive - among other things it adds a hack to force the way the class loader works for the SAX parser. Due to this hack, the ability of configuring which SAX parser implementation to use is lost (at least when compiling JSP pages) - the implementation bundled in the JRE is always used. Problem is this implementation does not support a certain option ("secure process") which is crucial in the patch itself to fix the vulnerability, leading to an Exception being thrown
-
effect is that spacewalk-java does not build any more: JSP compilation is broken as it relies on the patched Apache Standard Taglibs. The patch on one hand requires a SAX parser implementation that supports "secure process", and on the other hand blocks the selection to the JDK-provided one - but that does not support it!
-
users should be safe to upgrade the package, as all JSP compilation should happen in OBS. SLE customers using Apache Standard Taglibs for their own webapps might see the same problem we have
We followed upstream advice and upgraded to 1.2.3. We created a new tetra package, changed our specfiles and ran the testsuites. No regressions were noted, so solution was accepted.