hbase client + ScannerTimeoutException
Bonjour à tous,
J'utilise une application développée en java qui traite de gros volume de données grâce à hbase.
Mon but est de configurer correctement hbase afin de faire fonctionner cette application chez nous.
Une nouvelle version de cette dernière, qui optimise grandement les temps de traitements, me pose problème.
J'ai l'erreur suivante qui intervient au bout de 10 à 15 minutes de traitement :
Code:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
| [2016-02-17 09:35:42.751] ERROR PolarizationMapper businessRule 306122407
org.apache.hadoop.hbase.client.ScannerTimeoutException: 74140ms passed since the last invocation, timeout is currently set to 60000
at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:432)
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:364)
at gde.sphere.snapshot.hbase.PolarizationMapper.fill(PolarizationMapper.java:231)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:229)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
at gde.sphere.snapshot.CalculationService$_fillTempRecords_closure10_closure21.doCall(CalculationService.groovy:682)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:88)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
at groovy.lang.Closure.call(Closure.java:276)
at groovy.lang.Closure.call(Closure.java:289)
at groovy.lang.Closure$call$0.call(Unknown Source)
at groovyx.gpars.GParsPoolUtil$_eachParallel_closure3.doCall(GParsPoolUtil.groovy:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:88)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
at groovy.lang.Closure.call(Closure.java:276)
at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:51)
at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:79)
at com.sun.proxy.$Proxy25.map(Unknown Source)
at jsr166y.forkjoin.ParallelArray$WithBoundedMapping.leafTransfer(ParallelArray.java:1960)
at jsr166y.forkjoin.PAS$FJRMap.atLeaf(PAS.java:439)
at jsr166y.forkjoin.PAS$FJBase.compute(PAS.java:299)
at jsr166y.forkjoin.RecursiveAction.exec(RecursiveAction.java:247)
at jsr166y.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:286)
Caused by: org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: 38, already closed?
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2223)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:322)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:255)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:369)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:343)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Name: 38, already closed?
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2223)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1206)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:213)
... 9 more
[2016-02-17 09:35:42.754] ERROR PolarizationMapper org.apache.hadoop.hbase.client.ScannerTimeoutException: 74140ms passed since the last invocation, timeout is currently set to 60000 |
J'ai fais plusieurs recherches, et ce lien m'a fait modifier des variables dans mon hbase-site.xml :
http://stackoverflow.com/questions/1...apreduce-error
Code:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
<property>
<name>hbase.master.lease.period</name>
<value>900000</value>
</property>
<property>
<name>hbase.regionserver.lease.period</name>
<value>450000</value>
</property>
<property>
<name>hbase.rpc.timeout</name>
<value>450000</value>
</property>
<property>
<name>zookeeper.session.timeout</name>
<value>80000</value>
</property> |
Ma question : est-ce que quelqu'un aurait une idée de pourquoi mon timeout est toujours setté à 60 secondes ?
J'ai contrôlé dans ma config hbase, hdfs et zookeeper, plus aucune trace d'une variable qui serait settée à 60s.
J'ai relancé hbase, cleané, modifié de nouveau, mais je ne trouve pas de solution à ce problème.
Merci d'avance pour vos éclaircissements et bonne journée à tous.
