Tech stack:

• SQL
• Python
• Spark / PySpark
• Databricks
• ML workflows

Requirements:

- Advanced proficiency in:

• SQL
• Python
• Spark / PySpark
• Hands-on experience with Databricks

- Solid understanding of ML workflows (focus on data preparation; this is not an ML engineering role)
- Bachelor’s degree in Computer Science or related field (preferred)
- Minimum 4 years of professional experience in data engineering or analytics

Project description:

This long-term engagement supports simulation data processing for Autonomous Vehicle (AV) development. Key scenarios include obstacle detection, path planning, and complex traffic situations (e.g., tunnels, unusual vehicles, or temporary network issues).

The candidate will work with high-volume sensor data from a test AV fleet (8–12 cameras, LiDAR, radar), generating up to ~1TB/hour. Familiarity with the AV domain and real-world edge cases will be valuable. The team collaborates closely with a leading global automotive OEM to develop and validate safety-critical AV features.

The scope includes full-cycle data curation—from raw sensor input to simulation-ready datasets—and close cooperation with engineers and researchers.

Main responsibilities:

• Analyze real-world sensor data to identify edge cases (e.g., hard braking, nearby vehicles) • Create advanced SQL, Python, and Spark/PySpark queries for data filtering and transformation

• Work with internal tools for data search and auto-labeling workflows

• Process structured/semi-structured data (e.g., object detection output)

• Identify relevant data for AV simulations and ML pipelines

• Suggest and validate improvements in data discovery processes

• Build and maintain data mining scripts and ETL processes

• Develop tools to enhance analytics and streamline workflows