Hi there, I understand a Databricks notebook on Azure is taking 70+ minutes to complete; I will profile Spark stages, data shuffles and cluster sizing to cut runtime. My experience optimizing PySpark jobs, partitioning, caching and Delta Lake on Azure Databricks makes me a fit for this task. - Run Spark UI / Ganglia profiling, identify slow stages, fix expensive shuffles and optimize joins (broadcast, repartition) in the notebook - Implement code changes: vectorized PySpark/SQL, caching, partition pruning, Delta optimization and tuned cluster configuration (autoscaling, executor size) - Optional: add CI checks (Git + pipeline) to run lightweight performance tests before merge - Risk/quality-control: backup checkpoint + staged deployment and post-fix validation Skills: ✅ Databricks ✅ Azure Databricks / Azure ✅ PySpark / Spark optimization ✅ Git / CI pipeline integration ✅ Performance tuning / monitoring ✅ Delta Lake / partitioning Certificates: ✅ Microsoft® Certified: MCSA | MCSE | MCT ✅ cPanel® & WHM Certified CWSA-2 Is this running on a live production Azure Databricks cluster? Best regards,

₹2,872 INR in 1 day

5.0

(29 reviews)

5.2

@Vsion2

Welcome to professional Python development services! Hi there, I'm Alema, a Python expert programmer who strives for clear code in atmospheric, numerical weather prediction, physics, and all other seminal fields. I'm ready to provide you with high-quality services. I have completed 350+ projects with a 100% Positive Rating. If you are looking for Quality work, look no further. Also, we are a team of professional workers, and we are always available 24/7 to help employers without limitations, and delivery is guaranteed on time. Your faithfully. Eng. Alema Akter

₹3,500 INR in 1 day

5.0

(11 reviews)

3.4

@MauricioB30

Hi! I'm a DevSecOps engineer with over 4 years of experience working with clients such as EY, Globant, and MercadoLibre. Before proposing a solution, could you share more details about the notebook? Specifically: What's the data volume it's processing? Is it mostly SQL queries, Python or a mix? Are you using a shared or dedicated cluster? This will help me identify whether the bottleneck is in the code, the cluster config, or the data pipeline itself. Best regards, Ing. Mauricio Batista

₹35,000 INR in 7 days

5.0

(2 reviews)

2.1

@karthikresonite

Hello, Resonite Technologies has a proven data engineering team experienced in Databricks, PySpark, Spark SQL, Delta Lake, and performance tuning. We can optimize your notebook currently running 70+ minutes by identifying bottlenecks and improving execution time without affecting output accuracy. We will review: • Spark execution plan and job stages • Joins, shuffles, partitions, cache usage • Delta table reads/writes and file sizes • SQL/PySpark logic optimization • Cluster configuration recommendations We have optimized slow ETL/data processing pipelines and reduced runtime significantly through query tuning, partitioning, broadcast joins, caching strategy, and Delta best practices. Please share the notebook, sample data structure, current runtime logs, and expected output. We can start with a quick assessment and provide exact optimization steps. Warm Regards, Karthik B Resonite Technologies

₹3,450 INR in 7 days