SAP Knowledge Base Article - Preview

2819085 - Execute pyspark job on Hadoop cluster errors using python 3 SAP

Symptom

  • Executing a python 3 script which is intended to execute on the Hadoop Cluster as if it were a spark-scala job with the following command:
     
    /opt/spark/bin/spark-submit --master yarn --deploy-mode cluster --num-executors 23 --executor-cores 5 --executor-memory 35GB profiling.py
     
  • The job is submitted but fails with the following error message:

    ERROR org.apache.spark.deploy.yarn.ApplicationMaster (Logging.scala:logError(91)) - User class threw exception: java.io.IOException: Cannot run /usr/miniconda3/bin/python - execve failed (ENOENT - No such file or directory) 


Read more...

Environment

  • SAP Cloud Platform Big Data Services (BDS) 4.3
  • SAP Cloud Platform Big Data Services (BDS) 5.0

Product

SAP BusinessObjects Predictive Analytics 3.0

Keywords

Altiscale, Portal, HDFS

Hive, Spark, Pig, Tez, Oozie, Alation, MapReduce, SparkContext, YARN , KBA , BC-NEO-BDS , HCP Big Data Service , Problem

About this page

This is a preview of a SAP Knowledge Base Article. Click more to access the full version on SAP for Me (Login required).

Search for additional results

Visit SAP Support Portal's SAP Notes and KBA Search.