Ilios Digital

Government of Tamil Nadu Water Resources Department

Project Title: AWS-Based IoT Data Ingestion and Transmission Platform

Description

The Tamil Nadu Water Project implemented an AWS-based IoT solution to collect weather sensor data every 15 minutes via FTP. The data is processed in AWS and securely sent to the State Data Center for water management. Designed for high availability and security, the solution uses AWS EC2, Lambda, DynamoDB, API Gateway, WAF and Application Load Balancer. This project demonstrates effective cloud use for real-time environmental monitoring.

Problem Statement

.slidetitle, .slidedesc { color: #000000; }

Government of Tamil Nadu Water Resources Department required a scalable and secure cloud-based solution to collect, process, and transmit environmental sensor data from remote locations across the state.

Existing weather sensors were already installed and generating data, but there was no infrastructure in place to efficiently ingest, process, and deliver this data to the State Data Center (SDC) for further analysis.

The challenge was to rapidly implement a cost-effective system capable of:
• Handling periodic data uploads via FTP
• Transforming the data for analysis
• Securely transmitting the data via HTTPS APIs to the SDC

The solution had to ensure:
• System reliability and fault tolerance
• Minimal latency for near real-time updates
• Compliance with public sector IT and data protection standards

Project Outcomes

Real-Time Data Collection

Enabled seamless collection of weather sensor data every 15 minutes from remote locations across Tamil Nadu using FTP uploads to AWS.

Efficient Data Pipeline

Built a serverless ingestion and processing pipeline using AWS EC2, S3, Lambda, and DynamoDB to manage sensor data effectively.

Secure and Compliant Data Delivery

Successfully transferred processed data to the State Data Center (SDC) via a secure API Gateway, ensuring data confidentiality and integrity.

Operational Simplicity and Speed

Deployed the cloud infrastructure quickly in response to the project's high urgency, allowing immediate utilization of already-installed sensors.

Use Case:

The Government of Tamil Nadu needed a scalable and secure cloud-based solution to collect, process, and transmit environmental sensor data from remote locations. Weather sensors were already installed, but lacked infrastructure for efficient ingestion and delivery to the State Data Center (SDC).

Key Components:

  • EC2 (FTP) Server: An EC2 Linux instance configured with an FTP server to receive weather sensor data uploads via FTP protocol.
  • Amazon S3: Primary storage for ingested files.
  • AWS Lambda: Transforms and processes data.
  • Amazon DynamoDB: Stores sensor metadata.
  • Amazon API Gateway: Delivers data to the SDC.
  • AWS WAF & ALB: Ensures security and load balancing.
  • Amazon CloudWatch: Monitors system performance and triggers alerts.

Goals:

  • Provide a scalable pipeline for IoT sensor data.
  • Enable real-time access for analysis and hydro-modeling.
  • Ensure high availability and low operational overhead.
  • Maintain public sector security compliance.

Solution Flow:

  1. Sensor Upload: FTP transmission via AWS EC2.
  2. Storage & Trigger: Files land in S3 and trigger Lambda.
  3. Processing: Lambda enriches and stores data in DynamoDB.
  4. Secure Push: Data is sent to SDC via API Gateway.
  5. Monitoring: CloudWatch logs metrics and alerts anomalies.

Current Architecture

Notes

Initially, the customer’s application will support only FTP, which will require the use of a Linux-based EC2 instance configured as an FTP server for ingesting sensor data. However, the customer is expected to upgrade their application to support SFTP during the course of the project. Once this upgrade is complete, the ingestion pipeline will be migrated from the EC2 FTP server to AWS Transfer Family leveraging its managed SFTP capabilities for improved scalability, security, and ease of maintenance. The rest of the architecture will remain unchanged, enabling a smooth transition with minimal impact on downstream components.

Proposed Architecture

Conclusion

The Tamil Nadu Water Project successfully demonstrated the viability of using an AWS EC2-hosted FTP server as a secure and scalable ingestion point for weather sensor data. This solution enabled real-time data collection every 15 minutes from distributed sensors, which was essential for effective water flow monitoring. By leveraging a hybrid architecture that combined EC2-based FTP ingestion with serverless AWS services such as Lambda, S3, and DynamoDB, the project achieved a balance of operational speed, data integrity, and cost efficiency.