Senior Data Engineer

$140,000 - $170,000/Yr

Compass, Inc. - Boston, MA

Full-time - Senior

Boston, MA

Real Estate

About the position

The Senior Data Engineer at Compass is responsible for developing and operationalizing scalable data pipelines, managing complex datasets, and improving data quality. This role involves collaboration with various teams to enhance data systems and requires expertise in database management and scripting languages. The position is crucial for supporting the company's mission to revolutionize the real estate industry through data-driven solutions.

Responsibilities

• Develop & operationalize scalable and reliable data pipelines.
• Assemble large, complex data sets that meet functional/non-functional business requirements.
• Implement database management techniques including logical data modeling, physical database optimization, and security.
• Improve the integrity of data pipelines by setting up data quality checks.
• Collaborate with Data and Analytics experts to enhance functionality in data systems.
• Provide on-call support to resolve user issues based on severity.

Requirements

• 7+ years of experience in engineering and operationalizing data pipelines with large and complex datasets.
• 3+ years of experience working with unstructured data (JSON).
• Experience with batch and stream processing.
• Extensive experience in advanced SQL.
• 3+ years of experience in Python/PySpark.
• 3+ years of experience with transforming data using Databricks.
• 3+ years of experience with AWS.
• 2+ years of experience in orchestrating data pipelines using Airflow.
• Experience with GIT (CI/CD).

Nice-to-haves

• Good to have DBT experience.
• Good to have Tableau experience.

Benefits

• Participation in incentive programs (cash, equity, or commissions).
• Paid vacation, holidays, sick time, parental leave, marriage leave, and recharge leave.
• Medical, tele-health, dental, and vision benefits.
• 401(k) plan.
• Flexible spending accounts (FSAs).
• Commuter program.
• Life and disability insurance.
• Maven (support system for new parents).
• Carrot (fertility benefits).
• UrbanSitter (caregiver referral network).
• Employee Assistance Program.
• Pet insurance.