Every since I’ve started using Synapse for an assignment, I’ve preferred using Spark Notebooks to get anything done. However, they take time to spin up, something I wanted to mitigate by using Synapse-native components. In this article, we’ll generate a date range without Spark notebooks. It’s unnecessarily complicated, but you…
In this article, you’ll learn how you can get metadata from your pipeline run inside your pipeline’s notebooks (or other activities). The documentation on this subject is relatively limited, so I hope it will prevent a lot of wasted time. In Airflow, you have macros, to refer to the metadata…
I recently bought a Google Chromebook. It’s light, it’s fast, great battery, but it doesn’t support all the development tools that I’m used to when working on a Windows computer. There’s no Visual Studio Code (Python) and no RStudio (R). Me being a data scientist, that hurt in the beginning.…