Upload
alexander-lee
View
5.970
Download
7
Embed Size (px)
DESCRIPTION
Citation preview
Axis2 client memory leak
Outline
• Issue & diagnosis• Postmortem from technique point– Axis2 1.4/1.4.1/1.5.6 client– Implement & design issue
Issue
CServer exhaust 8G memory -> memory leak
diagnosis
jmap –histo pid
num #instances #bytes class name---------------------------------------------- 1: 1001744 246906600 [C //char array 2: 2855952 137085696 edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$Segment 3: 282338 115692192 [I // int array 4: 677362 102508648 [Ljava.util.HashMap$Entry; 5: 192117 99164392 [B // byte array 6: 2856122 91732120 [Ledu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$HashEntry; 7: 2855952 91390464 edu.emory.mathcs.backport.java.util.concurrent.locks.ReentrantLock$NonfairSync 8: 1337498 53499920 java.lang.String 9: 433634 43130120 [Ljava.lang.Object; 10: 597908 38266112 java.util.HashMap
1) The main change in this release is using web service client (axis2 1.4.1)
From the histogram, the memory leak is caused by Axis2.
2) char[] / edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap
3) Google ‘axis2 1.4.1 concurrentHashMap memory leak‘
diagnosis
• We got a 2.39G memory heap dump– Very fast, less than 30 seconds (because 8G
memory in production)• Problem : too big to open in dev box– Need enough physical memory– (May) need 64 bit OS– Try Jhat/Jprofile/YJP/Jmat eclipse plugin
(windows/2G)
jmap –dump:file=cdump pid
Open by Jmat standalone application
diagnosis
AxisConfiguration retain 540.9m heap
diagnosis
Hashtable allEndPoints holds more than 2k axis2 endpoint instances
diagnosis
All of the endpoint instances are about FavoriteService
1) Confirm in the code, only FavoriteService related did not call cleanup
Axis2 1.4 memory leak
• Reproduce
for( int i=0;i<count; i++) { VersionStub stub = new VersionStub(configContext,null); GetVersion request = new GetVersion(); stub.getVersion(request); }
Axis2 1.4 memory leak
• Informal Model (OO design)
AxisConfiguration
Map allServices = new Hashtable();Map allEndpoints = new Hashtable();
Axis2 1.4 memory leak
• Cause : Stub Initiation, add into AxisConfiguration
//Init, AddallServices.put(serviceName, axisService);..allEndpoints.put(serviceName + "." + endpointName, axisService);
Axis2 1.4 memory leak
• Cause : Stub Finalize
//Stubprotected void finalize() throws Throwable { super.finalize(); cleanup();}
public void cleanup() throws AxisFault { _service.getAxisConfiguration().removeService(_service.getName());}
//AsixConigurationpublic synchronized void removeService(String name) throws AxisFault { AxisService service = (AxisService) allServices.remove(name); if (service != null) { AxisServiceGroup serviceGroup = service.getAxisServiceGroup(); serviceGroup.removeService(name); log.debug(Messages.getMessage("serviceremoved", name)); } }
Axis2 1.4 memory leak
• Cause : Client Finalize
//ServiceClientprotected void finalize() throws Throwable { super.finalize(); cleanup();}
public void cleanup() throws AxisFault { … axisConfiguration.removeServiceGroup(serviceGroupName); …}
//AsixConigurationpublic AxisServiceGroup removeServiceGroup(String serviceGroupName) throws AxisFault { … Iterator services = axisServiceGroup.getServices();
while (services.hasNext()) { AxisService axisService = (AxisService) services.next(); allServices.remove(axisService.getName());
… } …}
Axis2 1.4 memory leak
• Cause
• Memory Leak : allEndpoints
//Init, AddallServices.put(serviceName, axisService);..allEndpoints.put(serviceName + "." + endpointName, axisService);
//Cleanup(Finalize), RemoveallServices.put(serviceName, axisService);
Axis2 1.4.1 fix
• Fix the bug AXIS2-3870
//AsixConiguration.removeServiceGroup
//removes the endpoints to this service String serviceName = axisService.getName(); String key = null; for (Iterator iter = axisService.getEndpoints().keySet().iterator(); iter.hasNext();){ key = serviceName + "." + (String)iter.next(); this.allEndpoints.remove(key); }
Axis2 1.4.1 memory leak
• Reproduce
for( int i=0;i<count; i++) { VersionStub stub = new VersionStub(configContext,null); GetVersion request = new GetVersion(); stub.getVersion(request); stub.cleanup(); }
Programmer: 1.4 has memory leak issue, so call cleanup
Axis2 1.4.1 memory leak
• Cause : Stub Finalize (no change)
//Stubprotected void finalize() throws Throwable { super.finalize(); cleanup();}
public void cleanup() throws AxisFault { _service.getAxisConfiguration().removeService(_service.getName());}
//AsixConigurationpublic synchronized void removeService(String name) throws AxisFault { AxisService service = (AxisService) allServices.remove(name); if (service != null) { AxisServiceGroup serviceGroup = service.getAxisServiceGroup(); serviceGroup.removeService(name); log.debug(Messages.getMessage("serviceremoved", name)); } }
Axis2 1.4.1 memory leak
• Cause: Client Finalize (no change)
//ServiceClientprotected void finalize() throws Throwable { super.finalize(); cleanup();}
public void cleanup() throws AxisFault { … axisConfiguration.removeServiceGroup(serviceGroupName); …}
Axis2 1.4.1 memory leak
• Cause: Client Finalize (no change)
//AsixConigurationpublic AxisServiceGroup removeServiceGroup(String serviceGroupName) throws AxisFault { … Iterator services = axisServiceGroup.getServices();
while (services.hasNext()) { AxisService axisService = (AxisService) services.next(); allServices.remove(axisService.getName());
… for (Iterator iter = axisService.getEndpoints().keySet().iterator(); iter.hasNext();){ key = serviceName + "." + (String)iter.next(); this.allEndpoints.remove(key); } …
}
…}
Servce are already removed from ServiceGroup in Stub cleanupThe while loop would never enter
Axis2 1.4.1 memory leak
• Cause– Stub cleanup and ServiceClient cleanup have
dependency– Can not call in below order
• Two are two memory leak bugs in 1.4, 1.4.1 only fix 1 bug
Stub.cleanup ServiceClient.cleanup
Axis2 1.5(.6) fix
• Fix the bug AXIS2-4007 AXIS2-4163
//Stubprotected void finalize() throws Throwable { super.finalize(); cleanup();}
public void cleanup() throws AxisFault { // _service.getAxisConfiguration().removeService(_service.getName()); _serviceClient.cleanup();}
Implement issue
• Forget to cleanup– Container object: add only, no remove– Resource object: apply only, no return/close
• Cleanup dependency– One object cleanup depend on other objects– Two object cleanup/two method has order
dependency
Design Issue
• AxisConfiguration is a global shared object– Usually only 1 instance even in client side.
• Purpose for put service/endpoint map in this global object AxisConfiguration ?
Design Issue
• Message Dispatch in server side // RequestURIBasedServiceDispatcher public AxisService findService(MessageContext messageContext) throws AxisFault {
AxisConfiguration registry = configurationContext.getAxisConfiguration();
AxisService axisService = registry.getService(values[0]);
// If the axisService is not null we get the binding that the request came to add // add it as a property to the messageContext if (axisService != null) { Map endpoints = axisService.getEndpoints(); if (endpoints != null) { if (endpoints.size() == 1) { messageContext.setProperty(WSDL2Constants.ENDPOINT_LOCAL_NAME, endpoints.get(axisService.getEndpointName())); } else { String endpointName = values[0].substring(values[0].indexOf(".") + 1); messageContext.setProperty(WSDL2Constants.ENDPOINT_LOCAL_NAME, endpoints.get(endpointName)); } } }
return axisService; }
Design Issue
• Service is ‘singleton’ in server side // AxisConfiguration public AxisService getService(String name) throws AxisFault { AxisService axisService = (AxisService) allServices.get(name); if (axisService != null) { if (axisService.isActive()) { return axisService; } else { throw new AxisFault(Messages .getMessage("serviceinactive", name)); } } else { axisService = (AxisService) allEndpoints.get(name); if (axisService != null) { if (axisService.isActive()) { return axisService; } else { throw new AxisFault(Messages .getMessage("serviceinactive", name)); } } } return null; }
Design Issue
• Client side– (Usually) New instance every method invocation– No need for message routing• Why still register in Axi2Configuration ?
axisConfig = configContext.getAxisConfiguration(); if (axisService == null) { axisService = createAnonymousService(); } this.axisService = axisService; if (axisConfig.getService(axisService.getName()) == null) { axisService.setClientSide(true); axisConfig.addService(axisService); } else { … }
…