Senior Data Engineer – Big Data & Cloudera

• Design, develop and maintain scalable Big Data and ETL data pipelines.
• Work extensively with the Cloudera Data Platform (CDP) and Hadoop ecosystem.
• Develop and optimize data processing solutions using Apache Spark and PySpark.
• Build and manage data ingestion pipelines using Apache NiFi and Sqoop.
• Work with HDFS, Hive and Impala for large-scale data storage, processing and querying.
• Develop complex and optimized SQL queries for data extraction, transformation and analysis.
• Develop data engineering solutions using Python and Shell scripting.
• Integrate and process data from enterprise data sources, including Oracle.
• Develop, maintain and optimize ETL processes to support business and analytical requirements.
• Monitor data pipelines and scheduled workloads using Control-M.
• Perform troubleshooting, performance tuning and root-cause analysis across data processing environments.
• Work within Linux/Unix environments to administer, troubleshoot and automate data engineering processes.
• Support data quality, data integrity and data availability across enterprise data platforms.
• Collaborate with Data Analysts, Developers, Architects, Business Analysts and other technology teams.
• Contribute to the continuous improvement of data engineering standards, processes and platforms.

Requirements

• 7–8 years of solid hands-on experience as a platform and data engineer (intermediate to senior level).
• Design, develop and maintain scalable Big Data and ETL data pipelines.
• Work extensively with the Cloudera Data Platform (CDP) and Hadoop ecosystem.
• Develop and optimize data processing solutions using Apache Spark and PySpark.
• Build and manage data ingestion pipelines using Apache NiFi and Sqoop.
• Work with HDFS, Hive and Impala for large-scale data storage, processing and querying.
• Develop complex and optimized SQL queries for data extraction, transformation and analysis.
• Develop data engineering solutions using Python and Shell scripting.
• Integrate and process data from enterprise data sources, including Oracle.
• Develop, maintain and optimize ETL processes to support business and analytical requirements.
• Monitor data pipelines and scheduled workloads using Control-M.
• Perform troubleshooting, performance tuning and root-cause analysis across data processing environments.
• Work within Linux/Unix environments to administer, troubleshoot and automate data engineering processes.
• Support data quality, data integrity and data availability across enterprise data platforms.
• Collaborate with Data Analysts, Developers, Architects, Business Analysts and other technology teams.
• Contribute to the continuous improvement of data engineering standards, processes and platforms.