Job Brief
We are looking for a Senior Data Engineer to support the migration of ETL pipelines from Talend to a modern data engineering stack comprising Python, dbt, and Airflow or Dagster. This is a remote consulting role that involves hands-on engineering work, tactical support for both legacy and new pipelines, and seamless integration into an agile team working toward modernization. The ideal candidate is an expert in data pipeline development, highly experienced with Kafka, Apache NiFi, Python, dbt, and orchestrators like Airflow or Dagster. A strong understanding of modern data storage formats (Parquet, Delta Lake, Iceberg), databases (RDBMS, in-memory DBs), and agile/scrum practices is critical to success in this role.

VentureDive Overview
Founded in 2012 by veteran technology entrepreneurs from MIT and Stanford, VentureDive is the fastest-growing technology company in the region that develops and invests in products and solutions that simplify and improve the lives of people worldwide. We aspire to create a technology organization and an entrepreneurial ecosystem in the region that is recognized as second to none in the world.

Key Responsibilities:
ETL Migration & Development

• Lead the re-engineering of legacy Talend pipelines into modern, scalable ETL/ELT workflows using Python, dbt, and Airflow/Dagster.
• Ensure smooth transition and backward compatibility where necessary during the migration period.

Support & Maintenance

• Provide ongoing support for existing Talend workflows until decommissioning.
• Handle ad-hoc development requests and change requests across legacy and new pipelines to meet evolving business needs.

Data Flow Design & Implementation

• Design, build, and maintain data ingestion and streaming pipelines using Kafka and NiFi.
• Integrate diverse data sources including structured, semi-structured, and unstructured data across filesystems and real-time streams.

Data Storage & Processing

• Work with various storage formats such as Parquet, Delta Lake, and Apache Iceberg to ensure data quality, schema management, and performance optimization.
• Leverage in-memory databases and traditional RDBMSs for data processing, caching, and analytical queries.

Process Optimization & Best Practices

• Apply engineering best practices in modularization, testing, logging, and monitoring of ETL pipelines.
• Contribute to code reviews, CI/CD practices, and infrastructure improvements aligned with the overall data strategy.

Collaboration & Agile Delivery

• Actively participate in Scrum ceremonies—stand-ups, sprint planning, reviews, and retrospectives.
• Collaborate with Data Architects, Analysts, and Platform Engineers to deliver high-quality, business-aligned data solutions.

Required Qualifications & Experience:

• 6+ years of experience in data engineering with a focus on pipeline development and orchestration.
• Strong, hands-on experience with:
     - Python (including modules like pandas, pyarrow, sqlalchemy)
     - dbt (data modeling, testing, documentation)
     - Airflow or Dagster (DAG design, scheduling, monitoring)
     - Kafka and Apache NiFi (streaming and ingestion)
• Deep knowledge of data formats and file systems: Parquet, Delta Lake, Iceberg.
• Proven experience working with relational and in-memory databases (e.g., PostgreSQL, Redis, MemSQL, etc.).
• Solid grasp of software engineering practices: Git, CI/CD, logging, and exception handling.
• Familiarity with data governance, lineage, and metadata management.
• Demonstrated understanding of Agile/Scrum processes with a focus on quality and delivery.

Preferred Qualifications:

• Experience in cloud data ecosystems (AWS, GCP, or Azure).
• Familiarity with containerization and deployment tools (e.g., Docker, Kubernetes) is a plus.
• Exposure to DataOps and MLOps concepts and tooling.

What we look for beyond required skills
In order to thrive at VentureDive, you
…are intellectually smart and curious
…have the passion for and take pride in your work
…deeply believe in VentureDive’s mission, vision, and values
…have a no-frills attitude
…are a collaborative team player
…are ethical and honest

Are you ready to put your ideas into products and solutions that will be used by millions?
You will find VentureDive to be a quick pace, high standards, fun and a rewarding place to work at. Not only will your work reach millions of users world-wide, you will also be rewarded with competitive salaries and benefits. If you think you have what it takes to be a VenDian, come join us ... we're having a ball!    

#LI-Hybrid