![]() Without further to say, let’s get to it.įor simplicity let’s design a child notebook that takes a number as an input and then print the multiplication of this number by 10. The idea would be that the parent notebook will pass along a parameter for the child notebook and the child notebook will use that parameter and execute a given task. The parent notebook orchestrates the parallelism process and the child notebook will be executed in parallel fashion. To follow along, you need to have databricks workspace, create a databricks cluster and two notebooks. But there are times where you need to implement your own parallelism logic to fit your needs. Noting that the whole purpose of a service like databricks is to execute code on multiple nodes called the workers in parallel fashion. In this blog, I would like to discuss how you will be able to use Python to run a databricks notebook for multiple times in a parallel fashion.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |