Web Scraping Data Engineering Manager

As the Data Engineering Manager for web scraping at COMPLY, you will play a pivotal role in the design and implementation of advanced web scraping tools and services. You will lead the development of workflows that enhance our data extraction capabilities, ensuring high accuracy and quality through diligent monitoring and testing. Under your guidance, our team will efficiently scale operations to meet increasing data demands. Your leadership will extend to mentoring in-house staff and managing offshore contractors, ensuring all activities align with COMPLY’s strategic goals.

Reporting directly to the Head of Data & Analytics, you will spearhead efforts to refactor our web scraping frameworks, significantly improving the speed, accuracy, and durability of our data processing. Your insights and deep technical expertise will be key in sustaining COMPLY’s competitive advantage and establishing new benchmarks in data excellence.

Job Responsibilities:

• Robust System Design: Develop and refine advanced web scraping tools and services to efficiently extract and ingest data from diverse web sources.
• Proactive Monitoring: Implement comprehensive monitoring systems to track the performance and health of web scraping operations, enabling early detection of issues before they affect data quality.
• Quality Assurance: Conduct rigorous testing partnering with QA to ensure the accuracy and reliability of data collected through web scraping. Implement quality control measures to maintain high standards.
• Leadership: Lead a team of geographically distributed data engineers, providing mentorship and direction to enhance skills and foster a collaborative, innovative work environment.
• Issue Resolution: Quickly identify and resolve technical issues and discrepancies in the data collection process, ensuring minimal disruption to operations.
• Continuous Improvement and Innovation: Continuously evaluate and improve scraping processes and methodologies to increase efficiency and adapt to changing technological landscapes.
• Strategic Planning: Collaborate with cross-functional teams to align web scraping activities with overall business objectives and strategic goals.
• Project Execution: Ensure timely execution of project deliverables by overseeing project schedules and resource allocation.
• Continuous Learning: Proactively stay current with advancements and trends in web scraping technologies to keep our practices in line with top industry standards.

Qualifications:

• Education: Bachelor’s or higher in Computer science, engineering, analytics, or information technology or similar.
• Experience: 10+ years in data engineering with 2+ years focused on web scraping. 3+ years managing a team.
• Technical Expertise: High Proficiency in Python a must! Experience with Celery, RabbitMQ, and Django to manage existing web scraping processes. Understanding of Elasticsearch indices, complex queries, and search optimization
• Web Scraping Framework: Deep knowledge in Scrapy, Beautiful Soup, Selenium or similar framework
• Leadership: 3+ Experience in senior or lead roles, demonstrating growth and increased responsibility in data handling.
• Web Technologies: Thorough understanding of HTML, CSS, and JavaScript to navigate complex websites.

Nice to Have:

• Industry Knowledge: Experience in financial services, political contributions or other relevant industries
• Certifications: AWS (preferred), Azure, or GCP. Data specific certification a plus.
• Other Technologies: Kubernetes, Redis, Jenkins, Postgres, ETL, dbt, NoSQL, SQL

$120,000 - $140,000 a year