Leading development activities for Data engineering team.
Working in collaboration with other teams like application management and product delivery.
Work alongside with technical leads, product managers and support team.
Provide guidance to development, support, and product delivery team.
Provide leadership in implementing tools and technologies to drive cost efficient architecture and infrastructure.
For Azure Data Engineer:
Create and maintain optimal data pipeline.
Assemble large, complex data sets that meet functional / non-functional business requirements. Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.
Build analytics tools that utilize the data pipeline to provide actionable insights into key business performance metrics.
Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
Work with data and analytics experts to strive for greater functionality in our data systems.
Required Skills:
For AWS Data Engineer: Have used AWS services like s3, Glue, SNS, SQS, Lambda, Redshift and RDS. Good experience in programming specially in Python. Have strong experience in designing complex SQL queries and optimizing data fetch. Good knowledge of spark, Pyspark, Hadoop, Hive and spark-Sql.
For Azure Data Engineer: Experience with Azure cloud services. Experience in developing Big Data applications using Spark, Hive, Sqoop, Kafka, and Map Reduce. Experience with stream-processing systems: Spark-Streaming, Strom etc.
