Discussion :

[bordi]

J'ai installé sur mon cluster hadoop 2.6.0, solr 4.10.2 et manifold cf 1.8.2, zookeeper 3.4.6, pris individuellement ils marchent très bien,
j'en suis encore à apprendre comment fonctionne solr et manifold cf.

l'installation de solr par défaut était relativement simple, meme en le faisant pointer sur hadoop, j'ai fait ca a l'arrache par manque de temps,
il utilise une instance collection1 par défaut dans son dossier example, après démarage des mes noeuds, il voit bien mon cluster.

./solr restart -c -m 1g -z stargate:2181,itx01:2181,itx02:2181 -a "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=1044 -Dsolr.directoryFactory=HdfsDirectoryFactory -Dsolr.lock.type=hdfs -Dsolr.data.dir=hdfs://stargate:9000/solr -Dsolr.updatelog=hdfs://stargate:9000/solr"

j'ai essayé de configurer zookeeper en mode external avec les instances en mode multiserver, mais rien a faire, il veut pas,

la plupart des exemples "multi" sont fait sur une seule machine, ce qui m'arrange pas, je cherche un tuto avec une vrai config de multi serveur, ou un exemple de config réel.
je n'ai pas de probèeme réseau,port,firewall. puisque j'ai déjà deux clusters qui marche hadoop et spark.

https://cwiki.apache.org/confluence/...eeper+Ensemble


j'ai mis le fichier zoo.cfg sur les 3 serveurs et j'ai seulement changer le dataDir data/1 en data/2 et data/3, et créer le myId avec la valeur attendue dans chaque dossier.

dataDir=/var/local/zookeeper/data/1
clientPort=2185
initLimit=5
syncLimit=2
server.1=stargate:2188:3188
server.2=itx01:2189:3189
server.3=itx02:2190:3190


quand je le lance mon zkserver, j'ai un planton, sauf si je lance qu'une seule instance sur les 3 à exécuter, avec deux ou trois je plante en connection refusée,
le netstat plnt ne donne rien, les ports ne sont pas occuper, j'ai même changer les no port pour vérifier. la tache est lancée, mais ce n'est pas utilisable

./zkServer.sh start ../conf/zoojbe.cfg


logs

2015-09-13 18:17:29,356 [myid:] - INFO [main:QuorumPeerConfig@103] - Reading configuration from: ../conf/zoojbe.cfg
2015-09-13 18:17:29,360 [myid:] - INFO [main:QuorumPeerConfig@340] - Defaulting to majority quorums
2015-09-13 18:17:29,363 [myid:1] - INFO [mainatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2015-09-13 18:17:29,363 [myid:1] - INFO [mainatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2015-09-13 18:17:29,364 [myid:1] - INFO [mainatadirCleanupManager@101] - Purge task is not scheduled.
2015-09-13 18:17:29,374 [myid:1] - INFO [main:QuorumPeerMain@127] - Starting quorum peer
2015-09-13 18:17:29,384 [myid:1] - INFO [main:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:2185
2015-09-13 18:17:29,399 [myid:1] - INFO [main:QuorumPeer@959] - tickTime set to 3000
2015-09-13 18:17:29,400 [myid:1] - INFO [main:QuorumPeer@979] - minSessionTimeout set to -1
2015-09-13 18:17:29,400 [myid:1] - INFO [main:QuorumPeer@990] - maxSessionTimeout set to -1
2015-09-13 18:17:29,400 [myid:1] - INFO [main:QuorumPeer@1005] - initLimit set to 5
2015-09-13 18:17:29,415 [myid:1] - INFO [Thread-1:QuorumCnxManager$Listener@504] - My election bind port: stargate/192.168.0.11:3188
2015-09-13 18:17:29,425 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2185:QuorumPeer@714] - LOOKING
2015-09-13 18:17:29,427 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2185:FastLeaderElection@815] - New election. My id = 1, proposed zxid=0x0
2015-09-13 18:17:29,428 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
2015-09-13 18:17:29,431 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@382] - Cannot open channel to 2 at election address itx01/192.168.0.31:3189
java.net.ConnectException: Connexion refusée
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:341)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:449)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:430)
at java.lang.Thread.run(Thread.java:745)
2015-09-13 18:17:29,437 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@382] - Cannot open channel to 3 at election address itx02/192.168.0.32:3190
java.net.ConnectException: Connexion refusée
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:341)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:449)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:430)
at java.lang.Thread.run(Thread.java:745)
2015-09-13 18:17:29,630 [myid:1] - WARN [QuorumPeer[myid=1]/0.0.0.0:2185:QuorumCnxManager@382] - Cannot open channel to 2 at election address itx01/192.168.0.31:3189
java.net.ConnectException: Connexion refusée
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
2015-09-13 18:17:29,632 [myid:1] - WARN [QuorumPeer[myid=1]/0.0.0.0:2185:QuorumCnxManager@382] - Cannot open channel to 3 at election address itx02/192.168.0.32:3190
java.net.ConnectException: Connexion refusée
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
2015-09-13 18:17:29,633 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2185:FastLeaderElection@849] - Notification time out: 400
2015-09-13 18:17:30,034 [myid:1] - WARN [QuorumPeer[myid=1]/0.0.0.0:2185:QuorumCnxManager@382] - Cannot open channel to 2 at election address itx01/192.168.0.31:3189
java.net.ConnectException: Connexion refusée
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
2015-09-13 18:17:30,035 [myid:1] - WARN [QuorumPeer[myid=1]/0.0.0.0:2185:QuorumCnxManager@382] - Cannot open channel to 3 at election address itx02/192.168.0.32:3190
java.net.ConnectException: Connexion refusée
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
2015-09-13 18:17:30,036 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2185:FastLeaderElection@849] - Notification time out: 800
2015-09-13 18:17:30,837 [myid:1] - WARN [QuorumPeer[myid=1]/0.0.0.0:2185:QuorumCnxManager@382] - Cannot open channel to 2 at election address itx01/192.168.0.31:3189
java.net.ConnectException: Connexion refusée
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
2015-09-13 18:17:30,839 [myid:1] - WARN [QuorumPeer[myid=1]/0.0.0.0:2185:QuorumCnxManager@382] - Cannot open channel to 3 at election address itx02/192.168.0.32:3190
java.net.ConnectException: Connexion refusée
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
2015-09-13 18:17:30,840 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2185:FastLeaderElection@849] - Notification time out: 1600
2015-09-13 18:17:32,441 [myid:1] - WARN [QuorumPeer[myid=1]/0.0.0.0:2185:QuorumCnxManager@382] - Cannot open channel to 2 at election address itx01/192.168.0.31:3189
java.net.ConnectException: Connexion refusée
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
2015-09-13 18:17:32,442 [myid:1] - WARN [QuorumPeer[myid=1]/0.0.0.0:2185:QuorumCnxManager@382] - Cannot open channel to 3 at election address itx02/192.168.0.32:3190
java.net.ConnectException: Connexion refusée
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
2015-09-13 18:17:32,443 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2185:FastLeaderElection@849] - Notification time out: 3200
2015-09-13 18:17:35,644 [myid:1] - WARN [QuorumPeer[myid=1]/0.0.0.0:2185:QuorumCnxManager@382] - Cannot open channel to 2 at election address itx01/192.168.0.31:3189
java.net.ConnectException: Connexion refusée
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
2015-09-13 18:17:35,646 [myid:1] - WARN [QuorumPeer[myid=1]/0.0.0.0:2185:QuorumCnxManager@382] - Cannot open channel to 3 at election address itx02/192.168.0.32:3190
java.net.ConnectException: Connexion refusée
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
2015-09-13 18:17:35,646 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2185:FastLeaderElection@849] - Notification time out: 6400

avec le crawler manifold, j'ai configuré solr en connexion output pour poster sur solr puis sur l'index de l'hdfs. mais ca ne fait rien, j'ai le message qui indique
que tout s'est bien dérouler. sauf qu'il n'y a rien dans l'index, et qu'il n y a pas d'erreur ni d'exception. alors qu'en postage manuel ca fonctionne.

Je suis à la recherche de nouvelle idées. reste plus qu'a sortir la pelle et la pioche pour casser ce nouveau mur.

[bordi]

bon je progresse, comme je ne voyais rien dans les traces entre le crawler apache manifold et solr, j'ai mis en place tcpdump pour suivre les trame IP sur le port de solr, je vois
bien une trame partir en direction de solr, puis plus rien, en recherchant une compatibiité matrix, que je n'ai pas trouvé, par recoupement j'ai fini
par trouver les bonnes version, j'ai manifold 1.8.2, ca me semble bon, j'ai vu que la version solr n'était pas la bonne version pour fonctionner avec,
j'ai installé la version 4.10.4 de solr.

Maintenant je vois un petit peu mieux, ca cause enfin, mais le crawler envoi des fichier xml pour poster dans l'index de solr, je plante méchament sur
le hdfs hadoop de solr, grouink ! faut encore chercher, mais ca cause, c'est bien, je progresse. faut que je comprenne la nature de l'erreur et comment je peux
regler ca. après j'aurais mon index arf!

log de solr sur les fichiers envoyé par le crawler sur la mise a jour de l'index stocké sur hdfs

INFO - 2015-09-20 10:56:39.670; org.apache.solr.cloud.DistributedQueue$LatchWatcher; NodeChildrenChanged fired on path /overseer/queue state SyncConnected
INFO - 2015-09-20 10:56:39.687; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
INFO - 2015-09-20 10:56:39.696; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/solr path=/update params={wt=xml&version=2.2} {delete=[file:/usr/local/solr-4.10.2/example/exampledocs/manufacturers.xml (-1512829458511298560)]} 0 11
INFO - 2015-09-20 10:56:50.780; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO - 2015-09-20 10:56:50.815; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/solr path=/update params={waitSearcher=true&distrib.from=http://192.168.0.11:8983/solr/collection1/&update.distrib=FROMLEADER&openSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {} 0 35
ERROR
ERROR - 2015-09-20 10:56:50.817; org.apache.solr.common.SolrException; null:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /solr/index/segments_8 (inode 66950): File does not exist. Holder DFSClient_NONMAPREDUCE_86662280_30 does not have any open files.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3516)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3604)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3574)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:700)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:526)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)

at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy8.complete(Unknown Source)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy8.complete(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:371)
at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:1894)
at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1881)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:71)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:104)
at java.io.FilterOutputStream.close(FilterOutputStream.java:160)
at java.io.FilterOutputStream.close(FilterOutputStream.java:160)
at org.apache.lucene.store.OutputStreamIndexOutput.close(OutputStreamIndexOutput.java:63)
at org.apache.solr.store.blockcache.CachedIndexOutput.closeInternal(CachedIndexOutput.java:56)
at org.apache.solr.store.blockcache.ReusedBufferedIndexOutput.close(ReusedBufferedIndexOutput.java:88)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:77)
at org.apache.lucene.store.NRTCachingDirectory.unCache(NRTCachingDirectory.java:253)

[bordi]

ca y'est, j'ai fini par trouver une version qui marche partout, sans que cela plante quelque part. solr 5.2.1 semble plus abouti que la version 4.x pour ce qui concerne hadoop

je peux poster automatiquement les données de mes documents xml contenus dans mon file système puis envoyés par le crawer dans l'index dans hadoop via solr.

apache manifold cf 1.8.2 : le crawler qui envoit à solr l'indexer
solr 5.2.1: l'indexeur et qui stocke ses données sur l'hdfs hadoop

j'ai pas osé la toute dernière, la 5.3, récemment sortie

une présentation sympa sur les fonctionnalités de l'indexeur, solr+manifold, en français

il me reste plus qu'à explorer les commandes d'administration de solr/manifold et de rendre externe la conf de zookeeper à solrcloud

Maintenant que cela communique de bout en bout, je vais pouvoir faire quelques parties de crawler/indexeur, faut que je trouve ou que je fasse une gui client de recherche documentaire.

affaire à suivre