Job Description

Design, develop, and maintain robust data pipelines using Databricks to process and transform large volumes of data. ETL Process Management: Implement ETL (Extract, Transform, Load) processes to integrate data from various sources into Databricks, ensuring data quality and integrity. Data Integration: Integrate Databricks with other data storage solutions and data lakes, ensuring seamless data flow and accessibility. Performance Optimization: Optimize data processing and query performance within Databricks to ensure efficient data retrieval and processing. Data Analysis and Visualization: Utilize Databricks to perform complex data analysis and create visualizations to support data-driven decision-making. Collaborate with Data Scientists and Analysts: Work closely with data scientists and analysts to understand their requirements and provide the necessary infrastructure and tools within Databricks. Security and Compliance: Ensure that data processing within Databricks complies with organizational security policies and industry regulations, implementing necessary security measures. This includes setting up encryption, managing network security configurations, and performing regular security audits. Monitoring and Troubleshooting: Monitor data pipelines and workflows for performance issues or errors, and troubleshoot any problems that arise to maintain smooth operations. Manage the creation, configuration, and scaling of Databricks clusters to ensure optimal performance and cost-efficiency. This includes monitoring cluster usage, resource allocation, and ensuring high availability. User and Access Management: Implement and manage user access controls, ensuring that only authorized personnel have access to Databricks resources. This involves setting up role-based access controls (RBAC), managing permissions, and integrating with identity management systems. Backup and Disaster Recovery: Develop and implement backup and disaster recovery plans for Databricks environments. Ensure that data and configurations are regularly backed up and that there are clear procedures in place for restoring services in the event of a failure. Required Qualifications: Technical Skills Experience with Databricks: Hands-on experience with Databricks, including familiarity with its architecture, features, and services. Proficiency in Spark: Strong knowledge of Apache Spark, including Spark SQL, Spark Streaming, and Spark MLlib, as Databricks is built on Spark. Programming Languages: Proficiency in programming languages commonly used in data engineering such as Python, Scala, SQL, and Java. Data Warehousing and ETL: Experience with data warehousing concepts, ETL processes, and tools like Apache Airflow, Talend, or Informatica. Database Management: Knowledge of relational and NoSQL databases, data modeling, and query optimization. Familiarity with big data technologies and ecosystems, including Hadoop, Hive, and Kafka. Analytical and Problem-Solving Skills Data Analysis: Ability to perform complex data analysis and create data visualizations to support business decisions. Problem-Solving: Strong analytical and problem-solving skills to troubleshoot and resolve issues in data pipelines and workflows. Soft Skills Communication Skills: Excellent verbal and written communication skills to collaborate with data scientists, analysts, and other stakeholders. Ability to work effectively in a team environment and contribute to cross-functional projects. Certifications (Optional but Beneficial) Databricks Certifications: Certifications such as Databricks Certified Associate Developer for Apache Spark or Databricks Certified Professional Data Scientist can demonstrate expertise and enhance job prospects. Cloud Certifications: Certifications from cloud providers (e.g., Azure Certified Solutions Architect, Azure Data Engineer) can be advantageous. Work Experience Prior experience working in data engineering, data analytics, or a related field is often required. This includes experience in building and maintaining data pipelines, ETL processes, and data integration. #J-18808-Ljbffr Compunnel, Inc.

Job Tags

Similar Jobs