Problem found

The online service was restarted, and it was good to get the dump file of the online service and download it locally for analysis. Opened the snapshot file using MAT, omitted the process of using MAT here, and found that there was a large amount of com.netflix.servo.monitor.BasicTimer that was not released and was occupied by org.springframework.cloud.netflix.metrics.servo. ServoMonitorCache is occupied.

Analysis

Find the ServoMonitorCache class in the project, found under the spring-cloud-netflix-core package, then open the jar package, check its spring.facts to see where the class is automatically configured, find org.springframework ServoMetricsAutoConfiguration, and then search for where the class is used, in org.springframework.cloud.netflix.metrics. MetricsInterceptorConfiguration found the use of the ServoMonitorCache object. When you see metrics you understand that it is a monitoring object for the service. The code is as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

@Configuration
@ConditionalOnProperty(value = "spring.cloud.netflix.metrics.enabled", havingValue = "true", matchIfMissing = true)
@ConditionalOnClass({ Monitors.class, MetricReader.class })
public class MetricsInterceptorConfiguration {

    @Configuration
    @ConditionalOnWebApplication
    @ConditionalOnClass(WebMvcConfigurerAdapter.class)
    static class MetricsWebResourceConfiguration extends WebMvcConfigurerAdapter {
        @Bean
        MetricsHandlerInterceptor servoMonitoringWebResourceInterceptor() {
            return new MetricsHandlerInterceptor();
        }

        @Override
        public void addInterceptors(InterceptorRegistry registry) {
            registry.addInterceptor(servoMonitoringWebResourceInterceptor());
        }
    }

    @Configuration
    @ConditionalOnClass({ RestTemplate.class, JoinPoint.class })
    @ConditionalOnProperty(value = "spring.aop.enabled", havingValue = "true", matchIfMissing = true)
    static class MetricsRestTemplateAspectConfiguration {

        @Bean
        RestTemplateUrlTemplateCapturingAspect restTemplateUrlTemplateCapturingAspect() {
            return new RestTemplateUrlTemplateCapturingAspect();
        }

    }

    @Configuration
    @ConditionalOnClass({ RestTemplate.class, HttpServletRequest.class })   // HttpServletRequest implicitly required by MetricsTagProvider
    static class MetricsRestTemplateConfiguration {

        @Value("${netflix.metrics.restClient.metricName:restclient}")
        String metricName;
                /*
                  * This is the key code
                  * No.1
                  */
        @Bean
        MetricsClientHttpRequestInterceptor spectatorLoggingClientHttpRequestInterceptor(
                Collection<MetricsTagProvider> tagProviders,
                ServoMonitorCache servoMonitorCache) {
            return new MetricsClientHttpRequestInterceptor(tagProviders,
                    servoMonitorCache, this.metricName);
        }

        @Bean
        BeanPostProcessor spectatorRestTemplateInterceptorPostProcessor() {
            return new MetricsInterceptorPostProcessor();
        }
                //No.2
        private static class MetricsInterceptorPostProcessor
                implements BeanPostProcessor, ApplicationContextAware {
            private ApplicationContext context;
            private MetricsClientHttpRequestInterceptor interceptor;

            @Override
            public Object postProcessBeforeInitialization(Object bean, String beanName) {
                return bean;
            }

            @Override
            public Object postProcessAfterInitialization(Object bean, String beanName) {
                if (bean instanceof RestTemplate) {
                    if (this.interceptor == null) {
                        this.interceptor = this.context
                                .getBean(MetricsClientHttpRequestInterceptor.class);
                    }
                    RestTemplate restTemplate = (RestTemplate) bean;
                    // create a new list as the old one may be unmodifiable (ie Arrays.asList())
                    ArrayList<ClientHttpRequestInterceptor> interceptors = new ArrayList<>();
                    interceptors.add(interceptor);
                    interceptors.addAll(restTemplate.getInterceptors());
                    restTemplate.setInterceptors(interceptors);
                }
                return bean;
            }

            @Override
            public void setApplicationContext(ApplicationContext context)
                    throws BeansException {
                this.context = context;
            }
        }
    }
}

In the above code No.1, the automatic configuration generated MetricsClientHttpRequestInterceptor interceptor, and then ServoMonitorCache using the constructor injection passed into the interceptor; then code No.2 postProcessAfterInitialization function, the interceptor assigned to RestTemplate, which is a very familiar object.

Then enter the MetricsClientHttpRequestInterceptor, the core code is as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

@Override
    public ClientHttpResponse intercept(HttpRequest request, byte[] body,
            ClientHttpRequestExecution execution) throws IOException {
        long startTime = System.nanoTime();

        ClientHttpResponse response = null;
        try {
            response = execution.execute(request, body);
            return response;
        }
        finally {
            SmallTagMap.Builder builder = SmallTagMap.builder();
                        //No.3
            for (MetricsTagProvider tagProvider : tagProviders) {
                for (Map.Entry<String, String> tag : tagProvider
                        .clientHttpRequestTags(request, response).entrySet()) {
                    builder.add(Tags.newTag(tag.getKey(), tag.getValue()));
                }
            }
                        //No.4
            MonitorConfig.Builder monitorConfigBuilder = MonitorConfig
                    .builder(metricName);
            monitorConfigBuilder.withTags(builder);

            servoMonitorCache.getTimer(monitorConfigBuilder.build())
                    .record(System.nanoTime() - startTime, TimeUnit.NANOSECONDS);
        }
    }

No.3 code, found the object tagProviders, back to see the code is also the interceptor construction parameters passed in; now go to see what this object is, because the object is the constructor injection, that is also generated by the spring container configuration, so continue to look in the autoconfig file, found in the org. springframework.cloud.netflix.metrics.servo.ServoMetricsAutoConfiguration in the automatic configuration generated by.

1
2
3
4
5
6
7
8

@Configuration
    @ConditionalOnClass(name = "javax.servlet.http.HttpServletRequest")
    protected static class MetricsTagConfiguration {
        @Bean
        public MetricsTagProvider defaultMetricsTagProvider() {
            return new DefaultMetricsTagProvider();
        }
    }

Go to DefaultMetricsTagProvider and the core code is as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

public Map<String, String> clientHttpRequestTags(HttpRequest request,
           ClientHttpResponse response) {
       String urlTemplate = RestTemplateUrlTemplateHolder.getRestTemplateUrlTemplate();
       if (urlTemplate == null) {
           urlTemplate = "none";
       }

       String status;
       try {
           status = (response == null) ? "CLIENT_ERROR" : ((Integer) response
                   .getRawStatusCode()).toString();
       }
       catch (IOException e) {
           status = "IO_ERROR";
       }

       String host = request.getURI().getHost();
       if( host == null ) {
           host = "none";
       }

       String strippedUrlTemplate = urlTemplate.replaceAll("^https?://[^/]+/", "");

       Map<String, String> tags = new HashMap<>();
       tags.put("method",   request.getMethod().name());
       tags.put("uri",     sanitizeUrlTemplate(strippedUrlTemplate));
       tags.put("status",   status);
       tags.put("clientName", host);

       return Collections.unmodifiableMap(tags);
   }

Found that it is the decomposition of the Http client request, where the key is the method (get, post, delete and other http methods), status status, clientName access to the service domain, uri access path (containing parameters).

Then, go back to the code No.4, generated an object com.netflix.servo.monitor.MonitorConfig, mainly name and tags, name default is restclient (can be modified in the properties file); tags is DefaultMetricsTagProvider in those tag tags.

Then go to the ServoMonitorCache.getTimer function.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

public synchronized BasicTimer getTimer(MonitorConfig config) {
        BasicTimer t = this.timerCache.get(config);
        if (t != null)
            return t;

        t = new BasicTimer(config);
        this.timerCache.put(config, t);

        if (this.timerCache.size() > this.config.getCacheWarningThreshold()) {
            log.warn("timerCache is above the warning threshold of " + this.config.getCacheWarningThreshold() + " with size " + this.timerCache.size() + ".");
        }

        this.monitorRegistry.register(t);
        return t;
    }

Here is very simple, first in the cache to find the MonitorConfig object has no, no then add a BasicTimer, if there is to update the BasicTimer parameters, BasicTimer stores the maximum time, minimum time, average time of each interface access, etc..

It is clear from the analysis here that if each interface access url is different, then the uri resolved in DefaultMetricsTagProvider is also different, which eventually leads to a different MonitorConfig object, so the interface is called once to generate a BasicTimer object, which over time also explodes the Jvm heap memory.

And our online services, as many are called by query parameters to internal or external interfaces.

Solution

  • Modify the call method to use POST to pass parameters (for our services, especially the three-party ones, this way is obviously not suitable.)
  • Remove the interceptor

Go back to MetricsInterceptorConfiguration and see the following code.

1
2
3
4

@Configuration
@ConditionalOnProperty(value = "spring.cloud.netflix.metrics.enabled", havingValue = "true", matchIfMissing = true)
@ConditionalOnClass({ Monitors.class, MetricReader.class })
public class MetricsInterceptorConfiguration {

If you are familiar with springboot, you just need to set the property spring.cloud.netflix.metrics.enabled to false to turn off the automatic profile class.

Reference https://www.lifengdi.com/archives/article/3770