Fixes #11072 - Jetty 12: CompleteCallbackHandler (#11786)

Introduced StateTrackingHandler.

StateTrackingHandler is a troubleshooting Handler that helps to identify those cases where the Handler/Request/Response APIs are used improperly.

In particular, it tracks the events described in StateTrackingHandler.Listener, such as the Handler callback not completed, or blocking demand callback, or a write callback not completed, etc.

It also provides dump() capabilities, so the current requests and their state is dumped to help troubleshooting.

Signed-off-by: Simone Bordet <simone.bordet@gmail.com>

documentation/jetty-documentation/src/main/asciidoc/operations-guide/modules/module-state-tracking.adoc

@@ -0,0 +1,31 @@
+//
+// ========================================================================
+// Copyright (c) 1995 Mort Bay Consulting Pty Ltd and others.
+//
+// This program and the accompanying materials are made available under the
+// terms of the Eclipse Public License v. 2.0 which is available at
+// https://www.eclipse.org/legal/epl-2.0, or the Apache License, Version 2.0
+// which is available at https://www.apache.org/licenses/LICENSE-2.0.
+//
+// SPDX-License-Identifier: EPL-2.0 OR Apache-2.0
+// ========================================================================
+//
+
+[[og-module-state-tracking]]
+===== Module `state-tracking`
+
+The `state-tracking` Jetty module inserts the `StateTrackingHandler` at the beginning of the Handler chain.
+
+`StateTrackingHandler` is a xref:og-troubleshooting[troubleshooting] `Handler` that tracks usages of `Handler`/`Request`/`Response` asynchronous APIs, and logs at warning level invalid usages of the APIs that may lead to blockages, deadlocks, or missing completion of ``Callback``s.
+
+This module can be enabled to troubleshoot web applications that do not behave as expected, for example:
+
+* That consume a lot of threads (possibly because they block).
+* That do not send responses (or send only partial responses) to clients.
+* That timeout when apparently they have received or have sent all data.
+
+The module properties are:
+
+----
+include::{jetty-home}/modules/state-tracking.mod[tags=documentation]
+----

documentation/jetty-documentation/src/main/asciidoc/operations-guide/modules/modules-standard.adoc

@@ -20,6 +20,7 @@ include::module-console-capture.adoc[]
 include::module-core-deploy.adoc[]
 include::module-cross-origin.adoc[]
 include::module-eeN-deploy.adoc[]
+include::module-eeN-webapp.adoc[]
 include::module-http.adoc[]
 include::module-http2.adoc[]
 include::module-http2c.adoc[]
@@ -34,9 +35,9 @@ include::module-rewrite.adoc[]
 include::module-server.adoc[]
 include::module-ssl.adoc[]
 include::module-ssl-reload.adoc[]
+include::module-state-tracking.adoc[]
 include::module-test-keystore.adoc[]
 include::module-threadpool.adoc[]
 include::module-threadpool-virtual.adoc[]
 include::module-threadpool-virtual-preview.adoc[]
-include::module-eeN-webapp.adoc[]
 include::module-well-known.adoc[]

documentation/jetty-documentation/src/main/asciidoc/operations-guide/troubleshooting/chapter.adoc

@@ -14,7 +14,7 @@
 [[og-troubleshooting]]
 === Troubleshooting
 
-To troubleshoot Jetty when used as a production server, there are two main tools: the Jetty Server Dump and enabling DEBUG level logging.
+To troubleshoot Jetty when used as a standalone server, there are two main tools: the Jetty Server Dump and enabling DEBUG level logging.
 
 Jetty is based on components organized as a tree, with the `Server` instance at the root of the tree.
 
@@ -36,3 +36,4 @@ IMPORTANT: Make sure you read about how to secure the access to Jetty when using
 include::troubleshooting-dump.adoc[]
 include::troubleshooting-logging.adoc[]
 include::troubleshooting-debugging.adoc[]
+include::troubleshooting-handlers.adoc[]

documentation/jetty-documentation/src/main/asciidoc/operations-guide/troubleshooting/troubleshooting-dump.adoc

@@ -14,7 +14,7 @@
 [[og-troubleshooting-dump]]
 ==== Server Dump
 
-The Jetty Server Dump is obtained by invoking, via JMX, the `Server.dump()` operation, as shown below.
+The Jetty Server Dump is obtained by invoking, via JMX, the `Server.dump()` operation, as shown below using link:https://adoptium.net/jmc.html[Java Mission Control (JMC)]:
 
 image::jmc-server-dump.png[]
 

documentation/jetty-documentation/src/main/asciidoc/operations-guide/troubleshooting/troubleshooting-handlers.adoc

@@ -0,0 +1,28 @@
+//
+// ========================================================================
+// Copyright (c) 1995 Mort Bay Consulting Pty Ltd and others.
+//
+// This program and the accompanying materials are made available under the
+// terms of the Eclipse Public License v. 2.0 which is available at
+// https://www.eclipse.org/legal/epl-2.0, or the Apache License, Version 2.0
+// which is available at https://www.apache.org/licenses/LICENSE-2.0.
+//
+// SPDX-License-Identifier: EPL-2.0 OR Apache-2.0
+// ========================================================================
+//
+
+[[og-troubleshooting-handlers]]
+==== Troubleshooting Handlers
+
+[[og-troubleshooting-handlers-state-tracking]]
+===== `StateTrackingHandler`
+
+Jetty's `StateTrackingHandler` (described in xref:og-module-state-tracking[this module]) can be used to troubleshoot problems in web applications.
+
+`StateTrackingHandler` tracks the usages of `Handler`/`Request`/`Response` asynchronous APIs by web applications, emitting events (logged at warning level) when an invalid usage of the APIs is detected.
+
+In conjunction with xref:og-troubleshooting-dump[dumping the Jetty component tree], it dumps the state of current requests, detailing whether they have reads or writes that are pending, whether callbacks have been completed, along with thread stack traces (including virtual threads) of operations that have been started but not completed, or are stuck in blocking code.
+
+You need to enable the `state-tracking` Jetty module, and configure it to track what you are interested in tracking (for more details, see the link:{javadoc-url}/org/eclipse/jetty/server/handler/StateTrackingHandler.html[javadocs]).
+
+// TODO: add a section about DebugHandler.

documentation/jetty-documentation/src/main/asciidoc/programming-guide/server/http/server-http-handler-use.adoc

@@ -456,6 +456,34 @@ If the application deployed in the cross domain requires cookies or authenticati
 
 For more `CrossOriginHandler` configuration options, refer to the link:{javadoc-url}/org/eclipse/jetty/server/handler/CrossOriginHandler.html[`CrossOriginHandler` javadocs].
 
+[[pg-server-http-handler-use-state-tracking]]
+====== StateTrackingHandler
+
+`StateTrackingHandler` is a xref:pg-troubleshooting[troubleshooting] `Handler` that tracks whether `Handler`/`Request`/`Response` asynchronous APIs are properly used by applications.</p>
+
+Asynchronous APIs are notoriously more difficult to troubleshoot than blocking APIs, and may be subject to restrictions that applications need to respect (a typical case is that they cannot perform blocking operations).
+
+For example, a `Handler` implementation whose `handle(\...)` method returns `true` _must_ eventually complete the callback passed to `handle(\...)` (for more details on the `Handler` APIs, see xref:pg-server-http-handler-impl[this section]).
+
+When an application forgets to complete the callback passed to `handle(\...)`, the HTTP response may not be sent to the client, but it will be difficult to troubleshoot why the client is not receiving responses.
+
+`StateTrackingHandler` helps with this troubleshooting because it tracks the callback passed to `handle(\...)` and emits an event if the callback is not completed within a configurable timeout:
+
+[source,java,indent=0]
+----
+include::../../{doc_code}/org/eclipse/jetty/docs/programming/server/http/HTTPServerDocs.java[tags=stateTrackingHandle]
+----
+
+By default, events are logged at warning level, but it is possible to specify a listener to be notified of the events tracked by `StateTrackingHandler`:
+
+[source,java,indent=0]
+----
+include::../../{doc_code}/org/eclipse/jetty/docs/programming/server/http/HTTPServerDocs.java[tags=stateTrackingListener]
+----
+
+Other events tracked by `StateTrackingHandler` are demand callbacks that block, writes that do not complete their callbacks, or write callbacks that block.
+The complete list of events is specified by the `StateTrackingHandler.Listener` class (link:{javadoc-url}/org/eclipse/jetty/server/handler/StateTrackingHandler.Listener.html[javadocs]).
+
 [[pg-server-http-handler-use-default]]
 ====== DefaultHandler
 

documentation/jetty-documentation/src/main/asciidoc/programming-guide/troubleshooting.adoc

@@ -72,22 +72,34 @@ java -Dorg.eclipse.jetty.http2.LEVEL=DEBUG --class-path ...
 === JVM Thread Dump
 TODO
 
+[[pg-troubleshooting-state-tracking]]
+=== `StateTrackingHandler`
+
+`StateTrackingHandler` (described xref:pg-server-http-handler-use-state-tracking[here]) is a troubleshooting `Handler` that can be inserted in the `Handler` chain to track usages of `Handler`/`Request`/`Response` asynchronous APIs.
+
+xref:pg-troubleshooting-component-dump[Dumping the Jetty component tree] will dump the `StateTrackingHandler`, which will dump the state of the current requests.
+
+This will help detecting whether requests are not completed due to callbacks not being completed, or whether callback code is stuck while invoking blocking APIs, etc.
+
+Thread stack traces (including virtual threads) of operations that have been started but not completed, or are stuck in blocking code are provided in the component tree dump.
+
 [[pg-troubleshooting-component-dump]]
-=== Jetty Component Tree Dump
+=== Component Tree Dump
 
 Jetty components are organized in a xref:pg-arch-bean[component tree].
 
 At the root of the component tree there is typically a `ContainerLifeCycle` instance -- typically a `Server` instance on the server and an `HttpClient` instance on the client.
 
-`ContainerLifeCycle` has built-in _dump_ APIs that can be invoked either directly or xref:pg-arch-jmx[via JMX].
+`ContainerLifeCycle` has built-in _dump_ APIs that can be invoked either directly on the `Server` instance, or xref:pg-arch-jmx[via JMX].
+
+You can invoke `Server.dump()` via JMX using a JMX console such as link:https://adoptium.net/jmc.html[Java Mission Control (JMC)]:
 
-// TODO: images from JMC?
-// TODO: Command line JMX will be in JMX section.
+image::jmc-server-dump.png[]
 
 TIP: You can get more details from a Jetty's `QueuedThreadPool` dump by enabling detailed dumps via `queuedThreadPool.setDetailedDump(true)`.
 
 [[pg-troubleshooting-debugging]]
-=== Debugging
+=== Remote Debugging
 
 Sometimes, in order to figure out a problem, enabling xref:pg-troubleshooting-logging[DEBUG logging] is not enough and you really need to debug the code with a debugger.
 

documentation/jetty-documentation/src/main/java/org/eclipse/jetty/docs/programming/server/http/HTTPServerDocs.java

@@ -85,6 +85,7 @@
 import org.eclipse.jetty.server.handler.QoSHandler;
 import org.eclipse.jetty.server.handler.ResourceHandler;
 import org.eclipse.jetty.server.handler.SecuredRedirectHandler;
+import org.eclipse.jetty.server.handler.StateTrackingHandler;
 import org.eclipse.jetty.server.handler.StatisticsHandler;
 import org.eclipse.jetty.server.handler.gzip.GzipHandler;
 import org.eclipse.jetty.unixdomain.server.UnixDomainServerConnector;
@@ -1535,6 +1536,33 @@ public void crossOriginAllowedOrigins()
         // end::crossOriginAllowedOrigins[]
     }
 
+    public void stateTrackingHandle()
+    {
+        // tag::stateTrackingHandle[]
+        StateTrackingHandler stateTrackingHandler = new StateTrackingHandler();
+
+        // Emit an event if the Handler callback is not completed with 5 seconds.
+        stateTrackingHandler.setHandlerCallbackTimeout(5000);
+        // end::stateTrackingHandle[]
+    }
+
+    public void stateTrackingListener()
+    {
+        // tag::stateTrackingListener[]
+        StateTrackingHandler stateTrackingHandler = new StateTrackingHandler(new StateTrackingHandler.Listener()
+        {
+            @Override
+            public void onHandlerCallbackNotCompleted(Request request, StateTrackingHandler.ThreadInfo handlerThreadInfo)
+            {
+                // Your own event handling logic.
+            }
+        });
+
+        // Emit an event if the Handler callback is not completed with 5 seconds.
+        stateTrackingHandler.setHandlerCallbackTimeout(5000);
+        // end::stateTrackingListener[]
+    }
+
     public void defaultHandler() throws Exception
     {
         // tag::defaultHandler[]

jetty-core/jetty-server/src/main/config/etc/jetty-state-tracking.xml

@@ -0,0 +1,16 @@
+<?xml version="1.0"?>
+<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "https://www.eclipse.org/jetty/configure_10_0.dtd">
+
+<Configure id="Server" class="org.eclipse.jetty.server.Server">
+  <Call name="insertHandler">
+    <Arg>
+      <New class="org.eclipse.jetty.server.handler.StateTrackingHandler">
+        <Set name="handlerCallbackTimeout" property="jetty.stateTracking.handlerCallbackTimeout" />
+        <Set name="completeHandlerCallbackAtTimeout" property="jetty.stateTracking.completeHandlerCallbackAtTimeout" />
+        <Set name="demandCallbackTimeout" property="jetty.stateTracking.demandCallbackTimeout" />
+        <Set name="writeTimeout" property="jetty.stateTracking.writeTimeout" />
+        <Set name="writeCallbackTimeout" property="jetty.stateTracking.writeCallbackTimeout" />
+      </New>
+    </Arg>
+  </Call>
+</Configure>

jetty-core/jetty-server/src/main/config/modules/state-tracking.mod

@@ -0,0 +1,39 @@
+[description]
+Enables the StateTrackingHandler as the outermost Handler, to track
+the state of various Handler/Request/Response asynchronous APIs.
+
+[tags]
+server
+debug
+
+[depends]
+server
+
+[before]
+cross-origin
+graceful
+gzip
+secure-redirect
+statistics
+thread-limit
+
+[xml]
+etc/jetty-state-tracking.xml
+
+[ini-template]
+# tag::documentation[]
+## The timeout in ms for the completion of the handle() callback.
+# jetty.stateTracking.handlerCallbackTimeout=0
+
+## Whether the handle() callback is completed in case of timeout.
+# jetty.stateTracking.completeHandlerCallbackAtTimeout=false
+
+## The timeout in ms for the execution of the demand callback.
+# jetty.stateTracking.demandCallbackTimeout=0
+
+## The timeout in ms for the execution of a response write.
+# jetty.stateTracking.writeTimeout=0
+
+## The timeout in ms for the execution of the response write callback.
+# jetty.stateTracking.writeCallbackTimeout=0
+# end::documentation[]

jetty-core/jetty-server/src/main/config/modules/well-known.mod

@@ -19,7 +19,7 @@ etc/well-known.xml
 # tag::documentation[]
 ## Well Known Directory (relative to $JETTY_BASE if relative path, otherwise it is an absolute path).
 # jetty.wellknown.dir=.well-known
-# end::documentation[]
 
 ## Allow contents of the well-known directory to be listed.
 # jetty.wellknown.listDirectories=false
+# end::documentation[]

jetty-core/jetty-server/src/main/java/org/eclipse/jetty/server/Request.java

@@ -696,7 +696,7 @@ default InvocationType getInvocationType()
     /**
      * <p>A wrapper for {@code Request} instances.</p>
      */
-    class Wrapper implements Request, Attributes
+    class Wrapper implements Request
     {
         /**
          * Implementation note: {@link Request.Wrapper} does not extend from {@link Attributes.Wrapper}