A session represents an interactive shell.
Starting with version 0.5.0-incubating, each session can support all four Scala, Python and R interpreters with newly added SQL interpreter. The kind field in session creation is no longer required, instead users should specify code kind (spark, pyspark, sparkr or sql) during statement submission.
To be compatible with previous versions, users can still specify kind in session creation, while ignoring kind in statement submission. Livy will then use this session kind as default kind for all the submitted statements.
If users want to submit code other than default kind specified in session creation, users need to specify code kind (spark, pyspark, sparkr or sql) during statement submission.
To change the Python executable the session uses, Livy reads the path from environment variable PYSPARK_PYTHON (Same as pyspark).
Starting with version 0.5.0-incubating, session kind “pyspark3” is removed, instead users require to set PYSPARK_PYTHON to python3 executable.
Like pyspark, if Livy is running in local mode, just set the environment variable. If the session is running in yarn-cluster mode, please set spark.yarn.appMasterEnv.PYSPARK_PYTHON in SparkConf so the environment variable is passed to the driver.
A statement represents the result of an execution statement.