Problem
When starting Bitbucket Server, the application fails to start and the following appears in the atlassian-bitbucket.log
2016-05-01 11:05:22,082 ERROR [localhost-startStop-2] c.a.e.i.AsynchronousAbleEventDispatcher There was an exception thrown trying to dispatch event [com.atlassian.config.lifecycle.events.ApplicationStoppingEvent@74c57907] from the invoker [SingleParameterMethodListenerInvoker{method=public void com.atlassian.analytics.client.listener.BitbucketEventListener.onEvent(java.lang.Object), listener=com.atlassian.analytics.client.listener.BitbucketEventListener@605b0ae1}]
java.lang.RuntimeException: Hazelcast instance is not active!
at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:54) ~[atlassian-event-3.0.0.jar:na]
at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:46) ~[atlassian-event-3.0.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299) [guava-18.0.jar:na]
at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:105) [atlassian-event-3.0.0.jar:na]
at com.atlassian.event.internal.LockFreeEventPublisher$Publisher.dispatch(LockFreeEventPublisher.java:260) [atlassian-event-3.0.0.jar:na]
at com.atlassian.event.internal.LockFreeEventPublisher.publish(LockFreeEventPublisher.java:102) [atlassian-event-3.0.0.jar:na]
at com.atlassian.stash.internal.event.TransactionAwareEventPublisher.publish(TransactionAwareEventPublisher.java:84) [bitbucket-platform-4.3.2.jar:na]
at com.atlassian.stash.internal.config.ConfigLifecycle.stop(ConfigLifecycle.java:45) [bitbucket-service-impl-4.3.2.jar:na]
at com.atlassian.stash.internal.spring.AbstractSmartLifecycle.stop(AbstractSmartLifecycle.java:36) [bitbucket-platform-4.3.2.jar:na]
at org.springframework.context.support.DefaultLifecycleProcessor.doStop(DefaultLifecycleProcessor.java:229) [spring-context-4.2.2.RELEASE.jar:na]
at org.springframework.context.support.DefaultLifecycleProcessor.access$300(DefaultLifecycleProcessor.java:51) [spring-context-4.2.2.RELEASE.jar:na]
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.stop(DefaultLifecycleProcessor.java:363) [spring-context-4.2.2.RELEASE.jar:na]
at org.springframework.context.support.DefaultLifecycleProcessor.stopBeans(DefaultLifecycleProcessor.java:202) [spring-context-4.2.2.RELEASE.jar:na]
at org.springframework.context.support.DefaultLifecycleProcessor.onClose(DefaultLifecycleProcessor.java:118) [spring-context-4.2.2.RELEASE.jar:na]
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:969) [spring-context-4.2.2.RELEASE.jar:na]
at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:928) [spring-context-4.2.2.RELEASE.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_51]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_51]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
... 11 frames trimmed
Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.AbstractDistributedObject.getService(AbstractDistributedObject.java:93) ~[hazelcast-3.5.2-atlassian-24.jar:na]
at com.hazelcast.map.impl.proxy.MapProxySupport.toData(MapProxySupport.java:1143) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:81) ~[hazelcast-3.5.2-atlassian-24.jar:3.5.2-atlassian-24]
at com.atlassian.hazelcast.quartz2.HazelcastJobStore.retrieveTrigger(HazelcastJobStore.java:498) ~[atlassian-hazelcast-extras-quartz2-1.3.3.jar:na]
at org.quartz.core.QuartzScheduler.getTrigger(QuartzScheduler.java:1530) ~[quartz-2.2.1.jar:na]
at org.quartz.impl.StdScheduler.getTrigger(StdScheduler.java:508) ~[quartz-2.2.1.jar:na]
at com.atlassian.scheduler.quartz2.Quartz2SchedulerFacade.getTrigger(Quartz2SchedulerFacade.java:191) ~[atlassian-scheduler-quartz2-1.6.0.jar:na]
at com.atlassian.scheduler.quartz2.Quartz2SchedulerFacade.unscheduleJob(Quartz2SchedulerFacade.java:313) ~[atlassian-scheduler-quartz2-1.6.0.jar:na]
at com.atlassian.scheduler.quartz2.Quartz2SchedulerService.unscheduleJob(Quartz2SchedulerService.java:122) ~[atlassian-scheduler-quartz2-1.6.0.jar:na]
at com.atlassian.stash.internal.sal.scheduling.PluginSchedulerAdapter.unscheduleJob(PluginSchedulerAdapter.java:75) ~[bitbucket-platform-4.3.2.jar:na]
at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:30) ~[atlassian-plugins-core-4.0.4.jar:na]
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56) ~[na:na]
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60) ~[na:na]
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70) ~[na:na]
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53) ~[na:na]
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57) ~[na:na]
at com.atlassian.scheduler.compat.local.LocalOnlyCompatibilityPluginScheduler.unscheduleClusteredJobImpl(LocalOnlyCompatibilityPluginScheduler.java:126) ~[na:na]
at com.atlassian.scheduler.compat.local.LocalOnlyCompatibilityPluginScheduler.unscheduleClusteredJob(LocalOnlyCompatibilityPluginScheduler.java:118) ~[na:na]
at com.atlassian.analytics.client.upload.PeriodicEventUploaderScheduler.unscheduleJob(PeriodicEventUploaderScheduler.java:113) ~[na:na]
at com.atlassian.analytics.client.upload.PeriodicEventUploaderScheduler.unscheduleJobs(PeriodicEventUploaderScheduler.java:156) ~[na:na]
at com.atlassian.analytics.client.listener.BitbucketEventListener.destroy(BitbucketEventListener.java:64) ~[na:na]
at com.atlassian.analytics.client.listener.BitbucketEventListener.onEvent(BitbucketEventListener.java:47) ~[na:na]
at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:36) ~[atlassian-event-3.0.0.jar:na]
... 19 common frames omittedDiagnosis
Environment
- This issue may have been preceded by a restart, OS or Java upgrade.
Cause
JAVA_HOME is not set correctly, and points to an old or incorrect JRE location.
Note: if you are seeing this error in a different scenario, where Bitbucket does start up but later crashes with an OutOfMemory error, you may be running into a different issue: Bitbucket Server become unresponsive with «Hazelcast instance is not active»
Resolution
Set JAVA_HOME to the correct value.
JAVA_HOME may be set as an environment variable for the user that runs Bitbucket (e.g. atlbitbucket) or it may be set inside the $BITBUCKET_INSTALL/bin/setenv.sh file. Check both places and ensure that this is pointing to a valid JRE location.
setenv and environment variable changes in Bitbucket Server 5.0+
Starting with Bitbucket Server 5.0, setenv.sh and setenv.bat have been removed. The options that were set in this file can now be set via environment variables. Where to set the environment variable depends on which Operating System you’re running on.
Linux
When using the atlbitbucket service on Linux, the environment variables are ignored. You must set the parameters in _start-webapp.sh (or start-bitbucket.sh). These values will be read when the service starts.
As an example, to set JVM_SUPPORT_RECOMMENDED_ARGS, you would add this line to the file:
Example
JVM_SUPPORT_RECOMMENDED_ARGS=-XX:+HeapDumpOnOutOfMemoryErrorWindows
Set the parameter as an environment variable for the user running Bitbucket Server. For example, if you want to set JVM_SUPPORT_RECOMMENDED_ARGS, create it as an environment variable and assign the appropriate value to it. When Bitbucket Server starts using the startup scripts or service, it will pick up and apply this value.
My application is running for months and working very well. Then suddenly I get the following error:
com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.impl.ProxyServiceImpl$ProxyRegistry.<init>(ProxyServiceImpl.java:220)
at com.hazelcast.spi.impl.ProxyServiceImpl$ProxyRegistry.<init>(ProxyServiceImpl.java:207)
at com.hazelcast.spi.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:69)
at com.hazelcast.spi.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:67)
at com.hazelcast.util.ConcurrencyUtil.getOrPutIfAbsent(ConcurrencyUtil.java:47)
at com.hazelcast.spi.impl.ProxyServiceImpl.getDistributedObject(ProxyServiceImpl.java:101)
at com.hazelcast.instance.HazelcastInstanceImpl.getDistributedObject(HazelcastInstanceImpl.java:285)
at com.hazelcast.instance.HazelcastInstanceImpl.getLock(HazelcastInstanceImpl.java:183)
at com.hazelcast.instance.HazelcastInstanceProxy.getLock(HazelcastInstanceProxy.java:77)
at br.com.xyz.lock.hazelcast.HazelcastLockManager.lock(HazelcastLockManager.java:37)
at br.com.xyz.lock.hazelcast.LockManagerFacade.lock(LockManagerFacade.java:24)
at br.com.xyz.recebe.negocio.NProcessadorMensagemRecebida.processamentoLock(NProcessadorMensagemRecebida.java:85)
at br.com.xyz.recebe.negocio.NProcessadorMensagemRecebida.processaArquivo(NProcessadorMensagemRecebida.java:74)
at br.com.xyz.recebe.processador.ProcessadorBase.processaArquivo(ProcessadorBase.java:75)
at br.com.xyz.recebe.processador.ProcessadorXml.processaArquivo(ProcessadorXml.java:16)
at br.com.xyz.recebe.processador.ProcessadorFacade.processaArquivo(ProcessadorFacade.java:34)
at br.com.xyz.recebe.mail.pdes.ProcessadorPDESMeRecebida.processar(ProcessadorPDESMeRecebida.java:77)
at gov.sefaz.util.pdes.ProcessadorDiretorioEntradaSaidaDaemon.processar(ProcessadorDiretorioEntradaSaidaDaemon.java:575)
at gov.sefaz.util.pdes.ProcessadorDiretorioEntradaSaidaDaemon.varrerDiretorioUsingStrategy(ProcessadorDiretorioEntradaSaidaDaemon.java:526)
at gov.sefaz.util.pdes.ProcessadorDiretorioEntradaSaidaDaemon.run(ProcessadorDiretorioEntradaSaidaDaemon.java:458)
at java.lang.Thread.run(Unknown Source)
I found some issues on Google that says it shutdown because other errors. But in my case there isn’t any.
It shutdown without reason.
Has anyone seen this before?
asked Apr 25, 2014 at 12:22
leocborgesleocborges
4,7895 gold badges32 silver badges38 bronze badges
10
The problem occurs if a hazelcast member doesn’t shutdown normally (terminate), When you stop HazelcastInstance which queue proxy is bound to; then any operation on that queue after instance stopped should throw HazelcastInstanceNotActiveException.
answered Apr 25, 2014 at 12:32
Asif BhuttoAsif Bhutto
3,8961 gold badge23 silver badges21 bronze badges
3
You just need to wait for a few minutes and rerun your job.
answered Jan 17, 2017 at 10:07
王子1986王子1986
2,9294 gold badges30 silver badges43 bronze badges
Hello,
I have 2 nodes cluster managed by hazelcast v. 3.8. activeMQ, executor service and hazelcast cache map which has large amount of locking and unlocking activities. e.g.
getLocalCache().lock(changelog.getBranch()); updateLocalCache(changelog, crw, getLocalCache()); getLocalCache().unlock(changelog.getBranch());
after couple of days working nicely following error starts popping continously
11:57:34.097 [pool-24-thread-16] ERROR c.t.c.h.j.ChangelogGapFillerThread1 - java.lang.InterruptedException: LockOperation was interrupted. Invocation{op=com.hazelcast.concurrent.lock.operations.LockOperation{serviceName='hz:impl:lockService', identityHash=639755771, partitionId=139, replicaIndex=0, callId=-1141578, invocationTime=1515585172356 (2018-01-10 11:52:52.356), waitTimeout=-1, callTimeout=60000, namespace=DefaultObjectNamespace{service='hz:impl:mapService', objectName=distributed_cache}, threadId=810}, tryCount=250, tryPauseMillis=500, invokeCount=0, callTimeoutMillis=60000, firstInvocationTimeMs=1515584888830, firstInvocationTime='2018-01-10 11:48:08.830', lastHeartbeatMillis=1515585243068, lastHeartbeatTime='2018-01-10 11:54:03.068', target=[172.29.238.122]:5701, pendingResponse={VOID}, backupsAcksExpected=0, backupsAcksReceived=0, connection=null}
11:57:34.121 [pool-24-thread-8] ERROR c.t.c.h.j.ChangelogGapFillerThread1 - UnlockOperation got rejected before execution due to not starting within the operation-call-timeout of: 60000 ms. Current time: 2018-01-10 11:56:25.038. Start time: 2018-01-10 11:54:19.443. Total elapsed time: 125595 ms. Invocation{op=com.hazelcast.concurrent.lock.operations.UnlockOperation{serviceName='hz:impl:lockService', identityHash=1453843575, partitionId=139, replicaIndex=0, callId=-1141603, invocationTime=1515585259441 (2018-01-10 11:54:19.441), waitTimeout=-1, callTimeout=60000, namespace=DefaultObjectNamespace{service='hz:impl:mapService', objectName=local_cache}, threadId=802}, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=60000, firstInvocationTimeMs=1515585259443, firstInvocationTime='2018-01-10 11:54:19.443', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 00:00:00.000', target=[172.29.238.122]:5701, pendingResponse={VOID}, backupsAcksExpected=0, backupsAcksReceived=0, connection=null}
Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.impl.operationservice.impl.InvocationRegistry.shutdown(InvocationRegistry.java:192)
at com.hazelcast.spi.impl.operationservice.impl.OperationServiceImpl.shutdownInvocations(OperationServiceImpl.java:479)
at com.hazelcast.spi.impl.NodeEngineImpl.shutdown(NodeEngineImpl.java:487)
at com.hazelcast.instance.Node.shutdownServices(Node.java:489)
at com.hazelcast.instance.Node.shutdown(Node.java:463)
at com.hazelcast.instance.OutOfMemoryHandlerHelper.tryShutdown(OutOfMemoryHandlerHelper.java:61)
at com.hazelcast.instance.DefaultOutOfMemoryHandler.onOutOfMemory(DefaultOutOfMemoryHandler.java:87)
at com.hazelcast.instance.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:189)
at com.hazelcast.instance.OutOfMemoryErrorDispatcher.inspectOutOfMemoryError(OutOfMemoryErrorDispatcher.java:158)
at com.hazelcast.spi.impl.packetdispatcher.impl.PacketDispatcherImpl.dispatch(PacketDispatcherImpl.java:84)
at com.hazelcast.nio.tcp.MemberReadHandler.handlePacket(MemberReadHandler.java:77)
at com.hazelcast.nio.tcp.MemberReadHandler.onRead(MemberReadHandler.java:60)
at com.hazelcast.internal.networking.nonblocking.NonBlockingSocketReader.handle(NonBlockingSocketReader.java:161)
at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.handleSelectionKey(NonBlockingIOThread.java:350)
at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.handleSelectionKeys(NonBlockingIOThread.java:335)
at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.selectLoop(NonBlockingIOThread.java:253)
at com.hazelcast.internal.networking.nonblocking.NonBlockingIOThread.run(NonBlockingIOThread.java:206)
at ------ submitted from ------.(Unknown Source)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolve(InvocationFuture.java:114)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:75)
at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:155)
at com.hazelcast.map.impl.proxy.MapProxySupport.containsKeyInternal(MapProxySupport.java:559)
at com.hazelcast.map.impl.proxy.MapProxyImpl.containsKey(MapProxyImpl.java:245)
at com.tomtom.cpp.herald.BaselineChangelogProcessor.isCLogNewerAndAheadOfHeadInDCache(BaselineChangelogProcessor.java:222)
at com.tomtom.cpp.herald.BaselineChangelogProcessor.process(BaselineChangelogProcessor.java:67)
at com.tomtom.cpp.herald.jms.ChangelogListener.handle(ChangelogListener.java:22)
at sun.reflect.GeneratedMethodAccessor89.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:180)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:112)
at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:104)
I have implemented an service application in java with MuleSoft 3.5.1 that it works in cluster server, and sometimes I get an error «Hazelcast instance is not active!» and the all service are bloked, so I must restart both server to resolve the problem. The hazelcast-3.1.6.jar is in my project.
ERROR 2015-06-22 09:39:00,425 [[gs_p_api_QA-vGS_P_0_1_18].http1_connector.receiver.06] org.mule.exception.DefaultMessagingExceptionStrategy:
*****************************************************************************
Message : Hazelcast instance is not active! (com.hazelcast.core.HazelcastInstanceNotActiveException). Message payload is of type: HashMap
Code : MULE_ERROR-29999
--------------------------------------------------------------------------------
Exception stack is:
1. Hazelcast instance is not active! (com.hazelcast.core.HazelcastInstanceNotActiveException)
com.hazelcast.spi.impl.ProxyServiceImpl$ProxyRegistry:220 (null)
2. Hazelcast instance is not active! (com.hazelcast.core.HazelcastInstanceNotActiveException). Message payload is of type: HashMap (org.mule.api.MessagingException)
org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor:32 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/MessagingException.html)
--------------------------------------------------------------------------------
Root Exception stack trace:
com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.impl.ProxyServiceImpl$ProxyRegistry.<init>(ProxyServiceImpl.java:220)
at com.hazelcast.spi.impl.ProxyServiceImpl$ProxyRegistry.<init>(ProxyServiceImpl.java:207)
at com.hazelcast.spi.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:69)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************
How can I resolve this problem definitively?
asked Jun 22, 2015 at 8:26
StefanoStefano
1,4274 gold badges22 silver badges37 bronze badges
3
I hit the similar exception of «com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!» when running a web app. In my case, it was caused by OutOfMemory, though it was not obvious initially.
I reckon you may have a memory issue, which causes the HashMap getting too large perhaps.
answered Jan 1, 2016 at 10:14
2
#java #hazelcast
Вопрос:
Я запускаю приложение, которое использует Hazelcast во встроенном режиме. Периодически наблюдая эту проблему при доступе к кэшу , приложение выдает ошибку «Экземпляр Hazelcast не активен».
Файлы кода выглядят следующим образом :
public DistributedServiceProvider() {
LOGGER.info("DistributedServiceProvider: " type);
if ("hazelcast".equalsIgnoreCase(type)) {
hazelcastInstance = HazelcastConfig.getInstance();
} else {
throw new Exception(NOT_CONFIGURED);
}
}
public IMap getMap(String name) {
if (hazelcastInstance != null) {
return hazelcastInstance.getMap(name);
}
throw new RuntimeException(NOT_INITIALIZED);
}
При запуске приложения и выполнении некоторых вариантов использования периодически возникает следующая ошибка.
An exception has occurred. Method:: compareIRTrackerRedemptions[null, Fri Apr 30 00:00:00 EDT 2021, Mon May 31 00:00:00 EDT 2021, true]
Root Cause: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.instance.impl.HazelcastInstanceProxy.getOriginal(HazelcastInstanceProxy.java:315)
at com.hazelcast.instance.impl.HazelcastInstanceProxy.getMap(HazelcastInstanceProxy.java:95)
at deshaw.descap.server.cache.provider.DistributedServiceProvider.getMap(DistributedServiceProvider.java:52)
at deshaw.descap.server.cache.BasicCache.getData(BasicCache.java:100)
at deshaw.descap.server.cache.FundCaches$FundCache.get(FundCaches.java:185)
Комментарии:
1. Есть ли еще какие-либо
com.hazelcast.*сообщения в журналах, предшествующих исключению ? Поскольку вашhazelcastInstanceне равен нулю, вы обычно ожидаете этого только в том случае, если что-то отключит его с помощьюHazelcast.shutdownAll()или одного из других механизмов выключения.2. @NeilStevenson Я действительно исследовал это под углом , но, к сожалению, вокруг этого ничего не было. Это только журналы вокруг com.hazelcast.* , остальные-трассировка стека приложений. все, что вы могли бы предложить в конфигурации hazelcast, что могло бы прояснить ситуацию.
3. Попробуйте настроить системное свойство
hazelcast.logging.typeв соответствии с используемым вами РЕГИСТРАТОРОМ. Если уровень веденияcom.hazelcastжурнала-это ИНФОРМАЦИЯ , вы получите более подробную информацию.