ERROR DiskBlockObjectWriter: Uncaught exception while reverting partial writes to file spark-submit
Bonjour
Je rencontre une erreur "DiskBlockObjectWriter" lors du lancement d'une commande spark-submit
Code:
1
2
3
4
5
6
7
8
9
10
11
|
=============> Select sales : SELECT AZR_CRR_COD_PLN, substring(AZR_CRR_COD_PLN, 1, 2)||substring(AZR_FLI_NBR_PLN, 3, 4), substring\
(AZR_MKT_CRR_PLN, 1, 2)||substring(AZR_FLI_NBR_MKT, 3, 4), AZR_ORI_PLN_CIT, AZR_DST_PLN_CIT, ( ( SUM(CASE WHEN (FZW_RMB_ECG=0 AND (\
COD_RMS IS NULL OR TRIM(COD_RMS)='' OR COD_RMS = 'R')) THEN NBR_AZR_EMS ELSE 0 END ) ) - ( sum(CASE WHEN FZW_RMB_ECG in ( 1,2,3,4,5\
,6 ) THEN NBR_AZR_EMS ELSE 0 END) ) ) SUM_AZR from sales WHERE ( AZR_CRR_COD_PLN != 'AF' AND AZR_MKT_CRR_PLN = 'AF' AND DAT_AZR_\
FLI_PLN > '2022-08-30' AND DAT_AZR_FLI_PLN < '2023-03-22' ) GROUP BY 1, 2, 3, 4, 5 HAVING SUM_AZR != 0
sales Count return result : 8838
23/03/08 11:14:03 ERROR DiskBlockObjectWriter: Uncaught exception while reverting partial writes to file /tmp/blockmgr-81cbaffe-fd2\
6-47d7-b8c9-3270f713f2db/35/temp_shuffle_0f9f96aa-5207-49b4-8f5c-82c973aa6447
java.io.FileNotFoundException: /tmp/blockmgr-81cbaffe-fd26-47d7-b8c9-3270f713f2db/35/temp_shuffle_0f9f96aa-5207-49b4-8f5c-82c973aa6\
447 (No such file or directory) |
Memoire du linux:
Mem: 3851376 3703388 147988 496 32968 634536
-/+ buffers/cache: 3035884 815492
Swap: 2097148 281616 1815532
commande spark executé depuis un bash sur un serveur Linux avec spark 2.6.5
spark-submit --driver-memory 3g --num-executors 1 --executor-cores 1 --executor-memory 3g /app/ZZCOS/prod/bin/scripts/launcher.py $1 $2 $3 Analysis.xlsx CS $salesdir $4
La commande de départ sans les options posait aussi probleme
spark-submit /app/ZZCOS/prod/bin/scripts/launcher.py $1 $2 $3 Analysis.xlsx CS $salesdir $4
J'ai egalement sans succes essayé --master yarn
mes recherches m'ont indiqué ceci, mais la valeur est déja tres elevée
https://github.com/broadinstitute/gatk/issues/6293
Merci de votre aide
