Sr. Big data engineer (Must be local to NYC) -Hybrid role 3 days onsite/Week

EdgeAll - New York, NY

Full-time - Mid Level

New York, NY

About the position

The Senior Big Data Engineer role is focused on leveraging expertise in the Big Data ecosystem to manage and analyze large datasets efficiently. This hybrid position requires a strong background in AWS services and experience in migrating legacy systems to modern data warehouse environments. The role involves creating complex data ingestion pipelines and collaborating with diverse teams to deliver insights through data analysis and visualization.

Responsibilities

• Expertise in all components of the Big Data Ecosystem including Spark, Hive, Sqoop, Oozie, Impala, and Hue.
• Manage and analyze data using AWS services, ensuring efficient data management.
• Lead the migration of legacy systems to Data Warehouse environments and Cloud using Informatica, Snowflake, and AWS technologies.
• Create complex data ingestion pipelines and manage data transformations, governance, and management in a centralized enterprise data hub.
• Perform data analysis and interpretation using Databricks features to run queries and build visualizations.
• Collaborate with diverse teams to deliver insights in the Analytics platform.

Requirements

• Extensive experience with AWS services including S3, CloudWatch, SQS, SNS, Step Functions, and IAM roles.
• Knowledge of data serialization techniques such as Avro, Sequence Files, Parquet, JSON, and ORC.
• Experience handling databases like MySQL, SQL Server, and Teradata.
• Expertise in all phases of the SDLC including requirements gathering, system design, development, testing, and deployment.
• Working knowledge of the Kubernetes ecosystem.

Nice-to-haves

• Strong problem-solving skills.
• Strong team player with good communication, analytical, presentation, and inter-personal skills.