Use Spark with Livy along with Application Insights. Learn to host your external dependencies in data lake.