Job Description
Expereince: 10 to 15 Years
Role Summary:
The primary responsibility of this role is to implement and manage comprehensive observability and monitoring for Databricks and Apache Spark workloads. The objective is to ensure high platform reliability, optimal performance, and transparent cost tracking.
Key Responsibilities:-
Continuously monitor Spark jobs, clusters, SQL warehouses, and data pipelines to identify performance issues, failures, or breaches of service level agreements (SLAs).- Develop intuitive dashboards and automated alerts by leveraging Databricks system tables, including those for jobs, clusters, billing, audits, and query history.- Proactively address operational challenges through troubleshooting, root cause analysis, and performance tuning of Spark workloads.- Facilitate cost, usage, and capacity transparency across Databricks environments, empowering teams to optimise resource allocation.- Seamlessly integrate Databricks monitoring capabilities with enterprise-grade tools such as Azure Monitor, Log Analytics, or similar platforms.
Required Skills:-
Demonstrated, hands-on expertise in working with Databricks and Apache Spark for both batch and streaming processes.- In-depth understanding of Databricks system tables, job orchestration, cluster management, and workflow automation.- Experience with data and platform observability frameworks, as well as advanced alerting and monitoring solutions.- Proficiency in SQL, PySpark, and dashboard creation to deliver actionable operational insights.
͏
Areas of responsibility
Triaging and resolution of tickets-Establish incident monitoring protocols and provide technical guidance to enable permanent fixes for recurring issues.
SLA Monitoring-Review SLA performance across the team, represent the function in meetings, and ensure consistent delivery quality.
Performance tuning and optimization-Collaborate with technical teams in managing production support activities regarding incident resolution, efficient deployments, and implementation of changes.
Perform system Improvements-Collaborate with technical teams to review mitigation measures and proposed improvement actions.
Handling disaster recovery-Develop disaster recovery and data backup plans to ensure system resilience and quick restoration in case of disruptions.
Reporting and stakeholder engagement-Participate in Quarterly Business Reviews (QBRs), manage client expectations, and ensure alignment with business objectives.
Team Management-Supervise team members by overseeing daily operations, conducting performance appraisals, and providing feedback.
Knowledge base and continuous learning Update and maintain documentation for complex issues, share best practices within the team, and support ongoing knowledge enhancement to improve service efficiency.
͏
͏
͏
Mandatory Skills: Cloud Azure Admin .
Experience: 8-10 Years .
Reinvent your world. We are building a modern Wipro. We are an end-to-end digital transformation partner with the boldest ambitions. To realize them, we need people inspired by reinvention. Of yourself, your career, and your skills. We want to see the constant evolution of our business and our industry. It has always been in our DNA - as the world around us changes, so do we. Join a business powered by purpose and a place that empowers you to design your own reinvention.