Performance regression from Servlet header adapters in 7.0.5 with repeated access of all values for a header

[180254]

Observations on our service overhead tests

I found a problem while conducting "overhead" tests on our service. By "overhead" I mean tests in which the data processing is mocked - it does no actual work and simply returns fake data, but all service architecture blocks are involved.

Upgrading Spring Framework 7.0.4 → 7.0.5 (via Spring Boot 4.0.2 → 4.0.3) caused a ~8x throughput drop on one of our Spring MVC (Servlet, Jetty) services. The problem also occurs on 7.0.8. We bisected it to a single change: reverting org.springframework.http.server.ServletServerHttpRequest and org.springframework.http.server.ServletServerHttpResponse to their 7.0.4 versions (while keeping Spring Framework current (7.0.8) for everything else) fully restores throughput.

A brief overview of our service and how it relates to the problem:

• The service runs on Spring MVC behind Jetty, is served over HTTP/2, and is CPU-constrained (~2 vCPU). It touches HTTP headers many times per request, on both sides of the exchange.
• Request side: every request reads a number of incoming headers (identity/authorization and various metadata supplied by our API management gateway) to make per-request decisions. Some headers are checked multiple times across different subcomponents and filters.
• Response side: responses are produced as ResponseEntity, so the framework pipeline (content negotiation, message conversion, and container-level compression) reads the response headers several times per request. A couple of custom headers are also set early, on the raw servlet response, before serialization.

References

• Optimize request and response header handling in Spring MVC #36334
• Consistent adaptation of HTTP headers on Servlet responses #36343
• Avoid update to request "Content-Type" for characterEncoding in ServletServerHttpRequest #36426

Minimal sample application

https://github.com/180254/spring-framework-issue-37042

What do the "shadowServletServerHttpRequest" and "shadowServletServerHttpResponse" profiles do? These are Maven profiles activated in the sample app. Each profile copies its corresponding class (ServletServerHttpRequest or ServletServerHttpResponse), taken from Spring Framework 7.0.4, onto the classpath, so that it overrides ("shadows") the version otherwise provided. In effect, I'm running a newer version of the Spring Framework but with these specific classes reverted to their 7.0.4 form.

Please see REQ/S for various cases:

$ ./matrix.sh 

===== SUMMARY (req/s; responseHeaderCount=50 responseHeaderReads=50 requestHeaderReads=50, server CPUs=0-1) =====
CONTAINER VERSION                                                                  PROTO    REQ/S      CORES   
--------- ----------------                                                         -----    -----      -----   
tomcat    7.0.4                                                                    h1       2830.94    1.97    
tomcat    7.0.5                                                                    h1       937.20     1.98    
tomcat    7.0.8                                                                    h1       905.66     1.98    
tomcat    7.0.8+shadowServletServerHttpRequest                                     h1       1531.25    1.97    
tomcat    7.0.8+shadowServletServerHttpResponse                                    h1       1229.97    1.99    
tomcat    7.0.8+shadowServletServerHttpRequest,shadowServletServerHttpResponse     h1       2956.34    1.98    
tomcat    7.0.4                                                                    h2       2987.46    1.98    
tomcat    7.0.5                                                                    h2       952.25     1.98    
tomcat    7.0.8                                                                    h2       923.62     1.98    
tomcat    7.0.8+shadowServletServerHttpRequest                                     h2       1628.64    1.98    
tomcat    7.0.8+shadowServletServerHttpResponse                                    h2       1287.94    1.99    
tomcat    7.0.8+shadowServletServerHttpRequest,shadowServletServerHttpResponse     h2       2912.74    1.99    
jetty     7.0.4                                                                    h1       2157.92    1.98    
jetty     7.0.5                                                                    h1       778.80     1.98    
jetty     7.0.8                                                                    h1       818.60     1.99    
jetty     7.0.8+shadowServletServerHttpRequest                                     h1       1001.91    1.99    
jetty     7.0.8+shadowServletServerHttpResponse                                    h1       1373.84    1.98    
jetty     7.0.8+shadowServletServerHttpRequest,shadowServletServerHttpResponse     h1       2513.35    1.97    
jetty     7.0.4                                                                    h2       2407.70    1.99    
jetty     7.0.5                                                                    h2       781.01     1.98    
jetty     7.0.8                                                                    h2       821.94     1.98    
jetty     7.0.8+shadowServletServerHttpRequest                                     h2       1043.09    1.99    
jetty     7.0.8+shadowServletServerHttpResponse                                    h2       1415.36    1.97    
jetty     7.0.8+shadowServletServerHttpRequest,shadowServletServerHttpResponse     h2       2482.93    1.96    

[rstoyanchev]

Thanks for the detailed report.

HttpServletRequestAdapter and HttpServletResponseAdapter avoid the need to automatically copy all headers into an HttpHeaders instance first, and therefore accessing a specific header value is cheaper. However, as the Servlet API uses Enumeration for all values, the get for all values currently copies the values into a new List on each call. It's a trade off between the upfront cost of creating and copying into HttpHeaders and the lower cost then of repeated access to all values for a header.

We can look to optimize the get returning a List by either caching already created lists, or by creating a custom List that accesses values from the underlying Enumeration.

One question, as far as I can see the sample code is accessing all values for all headers, and it does so 50 times. This is probably intentional to exaggerate the difference, but how close is that to what your application actually does?

[180254]

I checked the numbers from the actual application, considering the flow that is invoked in the described test.

To start with, here are a few more details: one client request actually produces two HTTP requests to the front component's embedded server: the client's submission request (held open as a Spring DeferredResult) and an internal callback delivering the result - both pass through the same servlet filter, HandlerInterceptors, and a custom ServerRequestObservationConvention. There is also the backend component's embedded server that receives one internal request but touches no headers in application code.

Totals:

• 44 header operations: 36 reads + 8 writes
• 13 unique request headers read: standard ones (Content-Type, Accept-Encoding, X-Forwarded-Host) plus a set of proprietary headers carrying customer identification, entitlements, tracing, and request classification
• 5 unique response headers written: standard content metadata (Content-Type, Content-Encoding, Content-Length) plus two proprietary ones (billing marker, tracing)

Mechanisms used:

Mechanism	Direction	Calls
jakarta.servlet.http.HttpServletRequest.getHeader() - in interceptors and the observation convention	read	24
org.springframework.http.HttpHeaders.getFirst() - on the @RequestHeader HttpHeaders snapshot in the controller	read	11
org.springframework.http.HttpHeaders.getValuesAsList() - same snapshot	read	1
jakarta.servlet.http.HttpServletResponse.setHeader() - in a filter and an interceptor	write	5
org.springframework.http.HttpHeaders setters on a ResponseEntity	write	3

[rstoyanchev]

That's helpful.

The sample demonstrates the regression through calls to HttpHeaders#get returning the List<String> of all values for the request and the response.

The majority of operations in the actual application (29 out of 44) are through the Servlet API, which is neutral to the Spring Framework upgrade. There are 3 calls to setters on HttpHeaders, which I don't expect to be any slower, and is not what the sample demonstrates in any case. The HttpHeaders#getValuesAsList is expected to be slower and matches the sample, but it's only 1 of 44 calls. The remaining 11 HttpHeaders#getFirst I don't expect to be slower, but I'm also not 100% sure what you mean with @RequestHeader HttpHeaders (controller snippet would help).

All in all if the table is correct, only 1 out of 44 calls is equivalent to what the sample does. So while the sample does demonstrate a performance regression, it does not seem to match what the application does. I still think we should consider an improvement for what your tests uncovered, but it doesn't seem like the actual application has a performance regression related to the header changes in 7.0.5?

[180254]

Thanks for the comment and for pushing back on actual application vs. minimal reproducer.

As I explained in the intro, I managed to narrow down the search for what's causing the test results to degrade to these two classes. I noticed the results go back to normal when I revert those two classes to version 7.0.4 in the front component. The AI-assisted reproducer showed that repeated reads and writes really do cause the performance to drop, which, at first glance, seems closely related to the problem.

That said, I admit there are still some gaps in this story. It seems like a piece of the puzzle is missing to fully explain why the actual application test shows degradation. I'll dig into it further and let you know if I find anything solid.

Here are some example snippets illustrating what I meant in the 'Mechanisms used' tables:

public class CustomerIdentifierHandlerInterceptor implements HandlerInterceptor, AsyncHandlerInterceptor {

  @Override
  public boolean preHandle(final HttpServletRequest request, final HttpServletResponse response,
      final Object handler) {

    Entitlements entitlements = Entitlements.parse(request.getHeader(ENTITLEMENTS_HEADER));

    String customerIdentifierValue = ObjectUtils.firstNonNull(
        request.getHeader(CustomerIdentifier.HEADER_1),
        request.getHeader(CustomerIdentifier.HEADER_2)
    );

   response.setHeader(CORRELATION_VALUE, corellationValue);

@RestController
public class LargeServiceController {

  @RequestMapping(
      path = {"/largeService/{largeId}"},
      method = RequestMethod.GET
  )
  public DeferredResult<ResponseEntity<?>> status(
      @PathVariable("largeId") final String largeId,
      @NonNull @RequestHeader final HttpHeaders requestHeaders,
      @RequestParam(required = false) boolean otherFlag
  ) {

    final Object value1 = operation1(requestHeaders);
    final Object value2 = operation2(requestHeaders, value1);
    
    ...
        .withOption3(extractOption3(requestHeaders))
        .moreHeaderExtractions()

    return 
  }

  private boolean isGzipAccepted(HttpHeaders requestHeaders) {
    ...
 requestHeaders
        .getValuesAsList(HttpHeaders.ACCEPT_ENCODING)
        .stream()
...

public class ServerRequestObservationCustomConvention extends DefaultServerRequestObservationConvention {


  public KeyValues getLowCardinalityKeyValues(ServerRequestObservationContext context) {
    var carrier = Objects.requireNonNull(context.getCarrier());

    String customerIdentifierValue = ObjectUtils.firstNonNull(
        carrier.getHeader(CustomerIdentifier.HEADER_1),
        carrier.getHeader(CustomerIdentifier.HEADER_2)
    );
   
    String otherValue = carrier.getHeader(Metadatas.SOME_OTHER_HEADER);

@ControllerAdvice
public class GlobalControllerExceptionHandler extends ResponseEntityExceptionHandler {

  @ExceptionHandler({Mtt400Exception.class, My404Exception.class})
  public ResponseEntity<ErrorResponse> handleMyException(MyException exception, WebRequest request) {
    
    gatherDetails(request, exception, httpStatus).log();

    ...
  }

  GlobalControllerExceptionHandlerLog gatherDetails(WebRequest webRequest, Exception ex, HttpStatusCode httpStatus) {
  	val value1 = webRequest.getHeader(Metadatas.HEADER_1);

public class ResponseHeaderFilter implements Filter {


  @Override
  public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain)
      throws ServletException, IOException {
    HttpServletResponse response = (HttpServletResponse) servletResponse;
    response.setHeader(HEADER_NAME, HEADER_VALUE);
    filterChain.doFilter(servletRequest, servletResponse);

[rstoyanchev]

I managed to narrow down the search for what's causing the test results to degrade to these two classes. I noticed the results go back to normal when I revert those two classes to version 7.0.4 in the front component.

Yes but that's based on a reproducer that hammers HttpHeaders#get returning a List. Is it possible your application test is also not representative of the methods the application actually uses?

For LargeServiceController the @RequestHeader annotation is unnecessary (and ignored) when used with HttpHeaders, so you can just remove the annotation. That aside the argument resolver populates a new HttpHeaders instance by copying values into it directly from the Servlet request, so that usage is unrelated to the changes in 7.0.5.

That means only 1 of 44 usages in the application is related to the 7.0.5 changes. Unless there are other usages not listed, there must be something else to explain any performance regression of the actual application. Or perhaps the test itself is a false positive.

[180254]

I ran further tests and have some new key observations and findings:

• To restore performance in my overhead test, it is enough to revert ServletServerHttpResponse to its 7.0.4 state. This confirms that first reproducer does indicate some degradation, but it has little connection to what my application is actually hitting.
• I compared ServletServerHttpResponse 7.0.4 with later versions and noticed that writeHeaders() used to execute the code below, but 7.0.5+ no longer does:

long contentLength = getHeaders().getContentLength();
if (contentLength != -1) {
    this.servletResponse.setContentLengthLong(contentLength);
}

• Using the 7.0.8 ServletServerHttpResponse with only this fragment restored also fixes the overhead test.

I gathered ingress proxy logs, service logs, CPU and wall-clock profiles, established facts, etc and then a coding agent did much of the analysis and wrote reproducer 2. Several conditions must be met to observe the failure, but the Spring Framework change is the only one of them that changed. The complete analysis is in the reproduction repository: part2 - please take a look.

It seems to me that not calling setContentLengthLong is an oversight. Could you please check this on your end? Thank you for your help.