Data Engineer

Job Summary: Join our team as a Data Engineer with the opportunity to be at the forefront of our data operations, leveraging your expertise in Spark and the Big Data ecosystem to drive meaningful insights and solutions.

• We are looking for engineers who possess 5-7 years of hands-on experience in data engineering within a robust Big Data environment, primarily utilizing Spark as a foundational skill.
• You will be proficient in data integration expertises, such as ETL/ELT development, constructing scalable, and maintainable data pipelines while supporting diverse integration patterns including batch processing, replication, and event streaming.
• In-depth knowledge of NoSQL technologies, particularly HBase, will be essential in your role.
• An understanding of Big Data cluster management and operations will be beneficial.
• You will demonstrate proficiency in manipulating and transforming data across varied datasets and technologies to meet project goals.
• Ideal candidates will have experience with medium to large scale enterprise projects, ideally within the financial services sector.
• A proactive attitude towards learning new technologies and experimenting with different approaches is valued.
• Excellent communication skills, both written and verbal, are crucial as collaboration is key in our data-driven environment.

͏

Key Responsibilities:

• Establish and create data pipelines for retention, cleanup, persistence, and maintenance of data on NoSQL datastores.
• Implement data consistency checks and rules to ensure data meets business standards and needs.
• Optimize data engineering pipelines and queries to ensure efficiency, reliability and scalability of data processing and analysis.
• Perform root cause analysis on issues and processes to answer specific business questions and identify opportunities for improvement.

͏

Skills Required: To excel in this role, candidates should bring a blend of technical expertise and problem-solving capabilities.

• Strong knowledge and hands-on experience with Apache Spark frameworks, including Core Spark, Spark DataFrames, and Spark Streaming functionalities are required.
• Candidates should be proficient in at least one programming language, specifically Java or Scala, to develop and maintain processing workflows.
• A solid understanding of distributed programming concepts will be essential to navigate through our extensive data systems.
• Experience in optimizing Spark Directed Acyclic Graphs (DAG) and Hive queries utilizing Tez will be significant advantages.
• Proficiency in tools such as Git, Autosys, Bitbucket, and Jira will support streamlined workflows and team collaboration.
• Ability to effectively apply Data Warehouse principles in the context of Hadoop environments and NoSQL databases.

͏

Good to Have: While not mandatory, additional skills and experiences can elevate your candidacy.

• Basic experience with Data Preparation Tools will enhance your ability to preprocess and structure data effectively.
• Experience in exploratory data analysis through SQL is highly regarded, facilitating insightful data discovery and evaluation.
• Familiarity with Big Data cluster operations can greatly support your understanding of data processing environments.
• Knowledge or experience with PySpark can provide an added layer of versatility in project approaches.
• Utilization of GitHub Copilot along with prompt engineering knowledge to enhance development processes will be an asset.
• A background in Operating Systems, accompanied by shell scripting proficiency, will further complement your technical toolkit.