|
• Bachelor's or master's degree in computer science, information technology, engineering, mathematics, data science or a related discipline, or equivalent practical experience. • Ideally 5+ years of relevant experience in data engineering, big data engineering or a closely related role, including delivery of production data pipelines. • Strong Python skills covering modular design, object-oriented programming, packaging, dependency management, exception handling, logging, testing and performance-aware coding. • Strong PySpark and Apache Spark expertise, including DataFrame and SQL APIs, distributed processing concepts, complex transformations, joins, window operations, troubleshooting and performance tuning. • Advanced SQL skills, including complex joins, window functions, CTEs, query tuning and data reconciliation. • Hands-on Azure and Databricks experience, including ADLS Gen2, Entra ID, Key Vault, notebooks, compute, Jobs/Workflows, Delta Lake and Unity Catalog. • Hands-on experience using Apache Airflow to develop, schedule, orchestrate, monitor and troubleshoot production workflows. • Experience with ETL/ELT pipelines, lakehouse and dimensional data models, data contracts, metadata, lineage and data-quality controls. • Experience with Git, CI/CD, automated testing and software engineering practices for production-grade data solutions. • Understanding of security, privacy, governance, auditability and operational resilience in a controlled environment. • Experience working in Agile teams and collaborating with analysts, architects, platform teams, product owners and business stakeholders. • Strong analytical, problem-solving and communication skills, with fluent written and verbal English.
Good to have • Exposure to AI concepts and agentic frameworks. • Working knowledge of Scala and experience developing or maintaining Spark workloads in Scala. • Experience with reporting and visualization tools such as Power BI, Tableau or equivalent. • Knowledge of Docker, Kubernetes, Terraform, Linux, shell scripting and REST APIs. • Experience in financial services, compliance, trade surveillance, market conduct, operational risk or regulatory data. • Relevant Azure, Databricks, Spark or data-engineering certifications.
|
|