Home » Use System Variables windowStartTime and windowEndTime in a Synapse Pipeline

Use System Variables windowStartTime and windowEndTime in a Synapse Pipeline

  • by
  • 3 min read

In this article, you’ll learn how you can get metadata from your pipeline run inside your pipeline’s notebooks (or other activities). The documentation on this subject is relatively limited, so I hope it will prevent a lot of wasted time.

In Airflow, you have macros, to refer to the metadata of a pipeline run. In Kubeflow, you have variables on various levels. In Azure Synapse (and in Data Factory), you have the windowEndTime and windowStartTime system variables of the Tumbling Window trigger. To get them from the pipeline into a Synapse notebook, you should take the following steps.

Set pipeline default parameters

First, specify the default values for both parameters, on the level of the pipeline. You can choose the names of both parameters, but keep them short and consistent as they will be referred to in a later step.

Set trigger run parameters

Next, create the trigger which will start the pipeline.

  • Click Trigger > New/Edit
  • Click the dropdown > Click ‘+ New’
  • Select Tumbling Window and adequate recurrence
  • When you submit, the next window will appear. Add the proper variables.

Set activity parameters

Finally, on the level of the (notebook) activity, you can set the base parameters. On the left hand side, you’ll set the names of your Python variables. On the right hand side, you can load them with the parameters from the pipeline level.

Use the parameters inside a notebook

Inside the notebook, you should create a cell where you add all the variables in which you want to store the window start and window end time. You should also specify a default value for when the parameters aren’t loaded (I set them to None).

window_start_time = None
window_end_time = None

Finally, you should set the cell as a parameter cell. You can do this by clicking the three dots on the left hand side of the cell.

⚠️If you simply want to extract a date from your newly created variables, you should take the first 10 characters of these strings.

window_start_time = window_start_time[0:10]
window_end_time = window_end_time[0:10]

Great success!

Say thanks, ask questions or give feedback

Technologies get updated, syntax changes and honestly… I make mistakes too. If something is incorrect, incomplete or doesn’t work, let me know in the comments below and help thousands of visitors.

Leave a Reply

Your email address will not be published.