Data Engineer

PURPOSE OF THE POSITION: To develop and maintain the architecture (pipelines and systems) that allow for the collection, storage, processing and analysing of large volumes of data across the business.

KEY RESPONSIBILITIES

• Develop and implement databases, data collection systems, data analytics and other strategies that optimise statistical efficiency and quality
• Data exploration and data quality assurance
• Acquire data from primary or secondary data sources and maintain databases 
• Filter and “clean” data by reviewing other reports and performance indicators to locate and correct code problems
• Development, construction, and maintenance of data architectures
• Conducting testing on large scale data platforms
• Handling error logs and building robust data pipelines
• Provide recommendations for data improvement, quality, and efficiency of data
• Development of data processes for data modeling, mining, and data production
• Analyse data through descriptive statistics
• Communicating results with the team and business using data visualization
• Design and build products such as dashboards and published notebooks for users
• Feature engineering
• Standardisation or scaling of the data
• Create Value Transformation like Square or Cube
• Design and build the feature stores for model training
• Operationalize feature stores for model training
• Gather information on resources, constraints, assumptions, risks to identify priorities.
• Drive best practice, continuous improvement and innovation within the ES function
• Ad hoc duties and projects

JOB INCUMBENT REQUIREMENTS

• BSc in Computer Science/ Data Science or Information Technology 
• 4+ years of relevant experience within the field of data engineering/software development (including programming experience)
• Knowledge of core computing algorithms to build robust data systems
• Proven ability to handle raw and unstructured data
• Knowledge of application development and working of APIs
• Proficient in cloud tools (preferably on GCP): Big Query, Composer/Airflow, AI Platform, Kubernetes, Dataflow, Pub/Sub, Cloud (Functions, Scheduler, SQL, Storage) and Computer Engine monitoring
• Knowledge of the following systems will be advantageous: Git workflow, Agile management (JIRA, Shortcut, Trello), Slack, CI/CD, Docker, Python, JavaScript, Google Workspace, Visualization tools: QlikView / LookerStudio / Grafana
• SA citizen 

Competencies required: Honesty; Passion & Resourcefulness; Problem assessment;
Detail orientation; Planning, Organizing and control; Service orientation; Stress Tolerance;
Confidence and decisiveness; Building and maintaining relationships: Able to function independently; Good communication skills