Skip to main content
Version: v2.10.0

Python

Python jobs provide flexibility for data transformations, analytics, automation, machine learning, and ETL pipelines. Python jobs allow users to run Python scripts or Spark-based PySpark workloads.

Python Job

Required Fields​

  • Arguments: Arguments required by the Python script
  • Cluster: Cluster where the job will execute
  • Path: Python script location in the repository

Advanced Options​

Supports configurations such as:

  • Python Packages – Install required dependencies
  • Additional Files – Include supporting files
  • Archives – Add compressed resources
  • Spark Configs – Configure Spark properties
  • PyFiles – Distribute Python libraries
  • Driver Configs – Configure driver settings

Advanced Options

When to Use Python Jobs​

Python jobs are ideal for:

  • Machine learning workflows
  • Data wrangling scripts
  • PySpark ETL