SubmitRun SubmitRun

yaml
type: "io.kestra.plugin.databricks.job.SubmitRun"

Submit a Databricks run. Optionally, set waitForCompletion to a desired maximum duration to wait for the run completion.

Examples

Submit a Databricks run and wait up to 5 minutes for its completion.

yaml
id: databricks_job_submit_run
namespace: company.team

tasks:
  - id: submit_run
    type: io.kestra.plugin.databricks.job.SubmitRun
    authentication:
      token: <your-token>
    host: <your-host>
    runTasks:
      - existingClusterId: <your-cluster>
        taskKey: taskKey
        sparkPythonTask:
          pythonFile: /Shared/hello.py
          sparkPythonTaskSource: WORKSPACE
    waitForCompletion: PT5M

Properties

runTasks

The run tasks, if multiple tasks are defined you must set dependsOn on each task.

accountId

  • Type: string
  • Dynamic: ✔️
  • Required:

Databricks account identifier.

authentication

Databricks authentication configuration.

This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.

configFile

  • Type: string
  • Dynamic: ✔️
  • Required:

Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one.

host

  • Type: string
  • Dynamic: ✔️
  • Required:

Databricks host.

runName

  • Type: string
  • Dynamic: ✔️
  • Required:

The name of the run.

waitForCompletion

  • Type: string
  • Dynamic:
  • Required:
  • Format: duration

If set, the task will wait for the run completion.

Outputs

runId

  • Type: integer
  • Required:

The run identifier.

runURI

  • Type: string
  • Required:
  • Format: uri

The run URI on the Databricks console.

Definitions

io.kestra.plugin.databricks.job.task.NotebookTaskSetting

Properties

baseParameters
  • Type:
    • string
    • object
  • SubType: string
  • Dynamic: ✔️
  • Required:

Map of task base parameters.

Can be a map of string/string or a variable that binds to a JSON object.

notebookPath
  • Type: string
  • Dynamic: ✔️
  • Required:
source
  • Type: string
  • Dynamic:
  • Required:
  • Possible Values:
    • GIT
    • WORKSPACE

io.kestra.plugin.databricks.job.task.PythonWheelTaskSetting

Properties

entryPoint
  • Type: string
  • Dynamic: ✔️
  • Required:
namedParameters
  • Type:
    • string
    • object
  • SubType: string
  • Dynamic: ✔️
  • Required:

Map of task named parameters.

Can be a map of string/string or a variable that binds to a JSON object.

packageName
  • Type: string
  • Dynamic: ✔️
  • Required:
parameters
  • Type:
    • string
    • array
  • Dynamic: ✔️
  • Required:

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

io.kestra.plugin.databricks.job.task.LibrarySetting-CranSetting

Properties

_package
  • Type: string
  • Dynamic: ✔️
  • Required:
repo
  • Type: string
  • Dynamic: ✔️
  • Required:

io.kestra.plugin.databricks.job.task.SparkSubmitTaskSetting

Properties

parameters
  • Type:
    • string
    • array
  • Dynamic: ✔️
  • Required:

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

io.kestra.plugin.databricks.job.SubmitRun-RunSubmitTaskSetting

Properties

dependsOn
  • Type: array
  • SubType: string
  • Dynamic:
  • Required:

Task dependencies, set this if multiple tasks are defined on the run.

existingClusterId
  • Type: string
  • Dynamic: ✔️
  • Required:
libraries

Task libraries.

notebookTask

Notebook task settings.

pipelineTask

Pipeline task settings.

pythonWheelTask

Python Wheel task settings.

runJobTask

Run job task settings.

sparkJarTask

Spark JAR task settings.

sparkPythonTask

Spark Python task settings.

sparkSubmitTask

Spark Submit task settings.

taskKey
  • Type: string
  • Dynamic: ✔️
  • Required:
timeoutSeconds
  • Type: integer
  • Dynamic:
  • Required:

io.kestra.plugin.databricks.AbstractTask-AuthenticationConfig

Properties

authType
  • Type: string
  • Dynamic: ✔️
  • Required:
azureClientId
  • Type: string
  • Dynamic: ✔️
  • Required:
azureClientSecret
  • Type: string
  • Dynamic: ✔️
  • Required:
azureTenantId
  • Type: string
  • Dynamic: ✔️
  • Required:
clientId
  • Type: string
  • Dynamic: ✔️
  • Required:
clientSecret
  • Type: string
  • Dynamic: ✔️
  • Required:
googleCredentials
  • Type: string
  • Dynamic: ✔️
  • Required:
googleServiceAccount
  • Type: string
  • Dynamic: ✔️
  • Required:
password
  • Type: string
  • Dynamic: ✔️
  • Required:
token
  • Type: string
  • Dynamic: ✔️
  • Required:
username
  • Type: string
  • Dynamic: ✔️
  • Required:

io.kestra.plugin.databricks.job.task.RunJobTaskSetting

Properties

jobId
  • Type: string
  • Dynamic: ✔️
  • Required:
jobParameters
  • Type: object
  • Dynamic: ✔️
  • Required:

io.kestra.plugin.databricks.job.task.SparkPythonTaskSetting

Properties

pythonFile
  • Type: string
  • Dynamic: ✔️
  • Required: ✔️
sparkPythonTaskSource
  • Type: string
  • Dynamic:
  • Required: ✔️
  • Possible Values:
    • GIT
    • WORKSPACE
parameters
  • Type:
    • string
    • array
  • Dynamic: ✔️
  • Required:

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

io.kestra.plugin.databricks.job.task.PipelineTaskSetting

Properties

fullRefresh
  • Type: boolean
  • Dynamic:
  • Required:
pipelineId
  • Type: string
  • Dynamic: ✔️
  • Required:

io.kestra.plugin.databricks.job.task.LibrarySetting

Properties

cran
egg
  • Type: string
  • Dynamic: ✔️
  • Required:
jar
  • Type: string
  • Dynamic: ✔️
  • Required:
maven
pypi
whl
  • Type: string
  • Dynamic: ✔️
  • Required:

io.kestra.plugin.databricks.job.task.LibrarySetting-PypiSetting

Properties

_package
  • Type: string
  • Dynamic: ✔️
  • Required:
repo
  • Type: string
  • Dynamic: ✔️
  • Required:

io.kestra.plugin.databricks.job.task.SparkJarTaskSetting

Properties

jarUri
  • Type: string
  • Dynamic: ✔️
  • Required:
mainClassName
  • Type: string
  • Dynamic: ✔️
  • Required:
parameters
  • Type:
    • string
    • array
  • Dynamic: ✔️
  • Required:

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

io.kestra.plugin.databricks.job.task.LibrarySetting-MavenSetting

Properties

coordinates
  • Type: string
  • Dynamic: ✔️
  • Required:
exclusions
  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required:
repo
  • Type: string
  • Dynamic: ✔️
  • Required:

Was this page helpful?