Hadoop Jobs

We are looking for a Senior Java Backend Developer with deep expertise in high-performance data architectures to join our team. As we scale our career pathway platform, your primary focus will be optimizing how we store, retrieve, and search complex datasets. You will be responsible for bridging our robust Java/Spring services with a modern data stack centered around MongoDB and Elasticsearch to provide our users with lightning-fast, relevant results. Key Responsibilities * Data Architecture: Design and implement scalable data models in MongoDB, ensuring efficient document structures for high-speed read/write operations. * Search Optimization: Architect and maintain Elasticsearch clusters, including index management, custom mapping, and complex query DSL optimization to power advanced s...

I’m upgrading our analytics stack and need an expert who can own the Hadoop side and turn raw, high-volume feeds into analysis-ready datasets. The core objective is to design and build end-to-end data pipelines on a Hadoop cluster—this is where I believe Hadoop will be most valuable for the project. Here’s what I need from you: • An architecture that takes terabyte-scale log files, lands them in HDFS, applies basic cleansing, and outputs partitioned Parquet tables queryable from Hive or Spark • All scripts, configs, and scheduling (Oozie, Airflow, or your preferred orchestrator) committed to Git with clear documentation • A deployment guide plus a brief hand-over session so I can reproduce the setup on another cluster Acceptance criteria: the pipeline...

My daughter is making a career switch into data engineering from a completely non-science background. She already writes basic Python and SQL, but everything else—cloud services, robust pipelines, modelling, and modern storage patterns—remains a mystery to her. I want a personalised training programme that: • Builds solid foundations in data pipelines, cloud infrastructure, data modelling, and database & storage concepts. • Uses both Microsoft Azure and AWS so she can compare services and deploy end-to-end solutions on each. The format, pace, and depth are open to your professional guidance; however, by the end of the engagement she should be comfortable designing, building, and operating production-ready workflows on either platform. A brief outline of ...

I am ready to dive into natural language processing and would like structured, hands-on coaching that takes me from the basics to building reliable text-classification models. My preferred language is Python, so examples should rely on common stacks such as Jupyter Notebook, Pandas, scikit-learn or, when it makes sense, PyTorch and Hugging Face Transformers. Java and R are not in scope for this engagement. Here is what I need from you: • A clear learning roadmap that starts with data cleaning and exploratory analysis, then walks through feature engineering (tokenisation, embeddings, etc.), model selection, training, evaluation and deployment. • Well-commented Python notebooks and sample datasets so I can reproduce every step on my own machine. • Short explanations ...

We are conducting a research project in Geospatial Artificial Intelligence and Remote Sensing that supports an academic manuscript submission to journals such as: ISPRS International Journal of Geo-Information IEEE Journal of Selected Topics in Applied Earh Observations and Remote Sensing The system integrates: Satellite image processing Computer vision object detection Large Language Model (LLM) enrichment Retrieval-Augmented Generation (RAG) The engineering system already exists. Your role is to design and execute rigorous ML experiments and evaluation pipelines to support publication-quality results. Infrastructure and deployment will be handled by a DevOps engineer. System Architecture The pipeline is based on the AWS open-source geospatial processing framework: OSML ModelRunner Refer...

Project Brief – Customer Churn Prediction for Telecom Project Goal: The project aims to predict customer churn in the telecommunications industry. By identifying customers likely to leave, telecom companies can take proactive retention actions, reduce revenue loss, and improve customer satisfaction. Data Used: The dataset includes customer demographics, service subscriptions, contract details, billing information, and payment methods. Project Steps: Data Cleaning: Handling missing values and correcting inconsistencies. Exploratory Data Analysis (EDA): Understanding patterns and key factors affecting churn. Feature Engineering: Creating new variables to improve model performance. Model Building: Developing a machine learning classification model to predict churn. Technologies & To...

Project Brief – Customer Churn Prediction for Telecom Project Goal: The project aims to predict customer churn in the telecommunications industry. By identifying customers likely to leave, telecom companies can take proactive retention actions, reduce revenue loss, and improve customer satisfaction. Data Used: The dataset includes customer demographics, service subscriptions, contract details, billing information, and payment methods. Project Steps: Data Cleaning: Handling missing values and correcting inconsistencies. Exploratory Data Analysis (EDA): Understanding patterns and key factors affecting churn. Feature Engineering: Creating new variables to improve model performance. Model Building: Developing a machine learning classification model to predict churn. Technologies & To...

We are conducting a research project in Geospatial Artificial Intelligence and Remote Sensing that supports an academic manuscript submission to journals such as: ISPRS International Journal of Geo-Information IEEE Journal of Selected Topics in Applied Earh Observations and Remote Sensing The system integrates: Satellite image processing Computer vision object detection Large Language Model (LLM) enrichment Retrieval-Augmented Generation (RAG) The engineering system already exists. Your role is to design and execute rigorous ML experiments and evaluation pipelines to support publication-quality results. Infrastructure and deployment will be handled by a DevOps engineer. System Architecture The pipeline is based on the AWS open-source geospatial processing framework: OSML ModelRunner Refer...