Bonjour à tous,
J'utilise une application développée en java qui traite de gros volume de données grâce à hbase.
Mon but est de configurer correctement hbase afin de faire fonctionner cette application chez nous.
Une nouvelle version de cette dernière, qui optimise grandement les temps de traitements, me pose problème.
J'ai l'erreur suivante qui intervient au bout de 10 à 15 minutes de traitement :
J'ai fais plusieurs recherches, et ce lien m'a fait modifier des variables dans mon hbase-site.xml :
Code : Sélectionner tout - Visualiser dans une fenêtre à part
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85 [2016-02-17 09:35:42.751] ERROR PolarizationMapper businessRule 306122407 org.apache.hadoop.hbase.client.ScannerTimeoutException: 74140ms passed since the last invocation, timeout is currently set to 60000 at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:432) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:364) at gde.sphere.snapshot.hbase.PolarizationMapper.fill(PolarizationMapper.java:231) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:229) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52) at gde.sphere.snapshot.CalculationService$_fillTempRecords_closure10_closure21.doCall(CalculationService.groovy:682) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:88) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058) at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886) at groovy.lang.Closure.call(Closure.java:276) at groovy.lang.Closure.call(Closure.java:289) at groovy.lang.Closure$call$0.call(Unknown Source) at groovyx.gpars.GParsPoolUtil$_eachParallel_closure3.doCall(GParsPoolUtil.groovy:123) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:88) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058) at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886) at groovy.lang.Closure.call(Closure.java:276) at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:51) at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:79) at com.sun.proxy.$Proxy25.map(Unknown Source) at jsr166y.forkjoin.ParallelArray$WithBoundedMapping.leafTransfer(ParallelArray.java:1960) at jsr166y.forkjoin.PAS$FJRMap.atLeaf(PAS.java:439) at jsr166y.forkjoin.PAS$FJBase.compute(PAS.java:299) at jsr166y.forkjoin.RecursiveAction.exec(RecursiveAction.java:247) at jsr166y.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:286) Caused by: org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: 38, already closed? at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2223) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:322) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:255) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:369) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:343) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Name: 38, already closed? at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2223) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1206) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:213) ... 9 more [2016-02-17 09:35:42.754] ERROR PolarizationMapper org.apache.hadoop.hbase.client.ScannerTimeoutException: 74140ms passed since the last invocation, timeout is currently set to 60000
http://stackoverflow.com/questions/1...apreduce-error
Ma question : est-ce que quelqu'un aurait une idée de pourquoi mon timeout est toujours setté à 60 secondes ?
Code : Sélectionner tout - Visualiser dans une fenêtre à part
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17 <property> <name>hbase.master.lease.period</name> <value>900000</value> </property> <property> <name>hbase.regionserver.lease.period</name> <value>450000</value> </property> <property> <name>hbase.rpc.timeout</name> <value>450000</value> </property> <property> <name>zookeeper.session.timeout</name> <value>80000</value> </property>
J'ai contrôlé dans ma config hbase, hdfs et zookeeper, plus aucune trace d'une variable qui serait settée à 60s.
J'ai relancé hbase, cleané, modifié de nouveau, mais je ne trouve pas de solution à ce problème.
Merci d'avance pour vos éclaircissements et bonne journée à tous.
Partager