Skip to content

basit-devBE/Secure-FIle-Upload

BranchesTags

Repository files navigation

Secure File Upload Service with AWS Integration

A comprehensive Node.js/TypeScript application that provides secure file upload capabilities with role-based access control, multipart upload support, and CloudWatch monitoring.

Architecture Overview

This project demonstrates enterprise-grade cloud architecture principles:

  1. Security First: Multi-layer security with AWS Cognito, IAM roles, and KMS encryption
  2. Scalability: Supports files up to 5GB with intelligent multipart upload
  3. Observability: Comprehensive monitoring with CloudWatch metrics and logs
  4. Role-Based Access: Fine-grained permissions for different user types
  5. Cloud-Native: Leverages AWS managed services for reliability and cost optimization

Project Structure

app/
├── config/
│   └── aws.config.ts              # AWS service configurations and validation
├── controllers/
│   ├── auth.controller.ts         # Authentication endpoint handlers
│   └── file.controller.ts         # File management endpoint handlers
├── middlewares/
│   ├── auth.middleware.ts         # JWT validation and role checking
│   └── cloudwatch.middleware.ts   # Performance tracking and metrics
├── routes/
│   ├── auth.route.ts             # Authentication route definitions
│   └── file.routes.ts            # File management route definitions
├── services/
│   ├── auth.service.ts           # AWS Cognito integration
│   ├── cloudwatch.service.ts     # CloudWatch metrics and logging
│   └── file.service.ts           # S3 operations and file management
├── utils/
│   └── logger.ts                 # Structured logging utilities
├── server.ts                     # Express application entry point
├── create-test-user.ts           # User creation and management script
├── package.json                  # Dependencies and scripts
├── tsconfig.json                 # TypeScript configuration
└── .env                          # Environment variables

Project Objectives

The application integrates multiple AWS services to create a production-ready file upload system:

  • AWS Cognito for user authentication and management
  • Amazon S3 for secure file storage with encryption
  • AWS IAM for fine-grained access control
  • AWS KMS for encryption key management
  • CloudWatch for comprehensive monitoring and alerting
  • AWS STS for temporary credential management

Features

Authentication & Authorization

  • AWS Cognito Integration: User pools with MFA support
  • Role-Based Access Control: Admin, Uploader, Viewer roles
  • JWT Token Management: Secure token-based authentication
  • Temporary IAM Credentials: AssumeRole for enhanced security

File Upload Capabilities

  • Single File Upload: Direct S3 pre-signed URLs
  • Multipart Upload: For files over 100MB (up to 5GB)
  • Progress Tracking: Real-time upload progress monitoring
  • File Validation: Size, type, and security validation
  • Metadata Preservation: Original filename, user context, timestamps

Monitoring & Observability

  • CloudWatch Metrics: Upload performance, API response times
  • Structured Logging: JSON logs with contextual information
  • Performance Tracking: Upload speeds, success rates, error rates
  • System Metrics: CPU, memory, active connections
  • Alert Capabilities: Configurable thresholds for performance issues

File Management

  • Organized Storage: Date-based folder structure in S3
  • File Listing: Role-based file access with metadata
  • Secure Downloads: Pre-signed download URLs
  • Encryption at Rest: KMS integration for data protection

Quick Start

Prerequisites

  • Node.js 18+
  • TypeScript
  • AWS Account with appropriate permissions
  • AWS CLI configured

Installation

  1. Clone and Install Dependencies
cd "project 3/app"
npm install
  1. Environment Configuration
# Copy and configure environment variables
cp .env.example .env

Required environment variables:

# AWS Configuration
AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key
AWS_REGION=eu-north-1
AWS_ACCOUNT_ID=your_account_id
S3_BUCKET_NAME=project3tmp
KMS_KEY_ID=your_kms_key_id

# IAM Roles
ADMIN_ROLE_ARN=arn:aws:iam::account:role/SecureUpload-AdminRole
UPLOADER_ROLE_ARN=arn:aws:iam::account:role/SecureUpload-UploaderRole
VIEWER_ROLE_ARN=arn:aws:iam::account:role/SecureUpload-ViewerRole

# Cognito Configuration
COGNITO_USER_POOL_ID=your_user_pool_id
COGNITO_CLIENT_ID=your_client_id
COGNITO_CLIENT_SECRET=your_client_secret

# Application Configuration
PORT=3000
NODE_ENV=development
JWT_SECRET=your_jwt_secret

# CloudWatch Configuration
CLOUDWATCH_LOG_GROUP_NAME=my-app-log-group
CLOUDWATCH_NAMESPACE=my-app-namespace
  1. Create Test Users
# Create users with different roles
npx ts-node create-test-user.ts
  1. Start the Application
npm run dev

API Documentation

Authentication Endpoints

Login

POST /auth/login
Content-Type: application/json

{
  "email": "admin@example.com",
  "password": "SecurePass123!"
}

Response:

{
  "success": true,
  "data": {
    "token": "jwt_token_here",
    "user": {
      "email": "admin@example.com",
      "role": {
        "role": "admin",
        "roleArn": "arn:aws:iam::account:role/SecureUpload-AdminRole"
      }
    }
  }
}

File Upload Endpoints

Generate Upload URL (Single File)

POST /files/upload-url
Authorization: Bearer <jwt_token>
Content-Type: application/json

{
  "fileName": "document.pdf",
  "fileSize": 1048576,
  "contentType": "application/pdf",
  "metadata": {
    "description": "Important document",
    "category": "business"
  }
}

Response:

{
  "success": true,
  "data": {
    "uploadUrl": "https://project3tmp.s3.eu-north-1.amazonaws.com/...",
    "fileId": "uuid-here",
    "expiresIn": 900,
    "s3Key": "uploads/2025-09-26/uuid_document.pdf"
  },
  "message": "Upload URL generated successfully"
}

Initialize Multipart Upload

POST /files/multipart/initiate
Authorization: Bearer <jwt_token>
Content-Type: application/json

{
  "fileName": "large-file.zip",
  "fileSize": 524288000,
  "contentType": "application/zip",
  "partSize": 104857600,
  "metadata": {
    "description": "Large archive file",
    "category": "backup"
  }
}

Response:

{
  "success": true,
  "data": {
    "uploadId": "multipart_upload_id",
    "fileId": "uuid-here",
    "partUrls": [
      {
        "partNumber": 1,
        "uploadUrl": "https://s3-presigned-url-part-1"
      },
      {
        "partNumber": 2,
        "uploadUrl": "https://s3-presigned-url-part-2"
      }
    ],
    "completeUrl": "https://s3-complete-url",
    "abortUrl": "https://s3-abort-url",
    "expiresIn": 3600,
    "s3Key": "uploads/2025-09-26/uuid_large-file.zip",
    "totalParts": 5
  }
}

Complete Multipart Upload

POST /files/multipart/complete
Authorization: Bearer <jwt_token>
Content-Type: application/json

{
  "uploadId": "multipart_upload_id",
  "fileId": "uuid-here",
  "s3Key": "uploads/2025-09-26/uuid_filename.ext",
  "parts": [
    {"partNumber": 1, "etag": "\"etag1\""},
    {"partNumber": 2, "etag": "\"etag2\""}
  ]
}

List Files

GET /files/list?withMetadata=true&maxKeys=50&prefix=uploads/2025-09-26
Authorization: Bearer <jwt_token>

Response:

{
  "success": true,
  "data": {
    "files": [
      {
        "key": "uploads/2025-09-26/uuid_document.pdf",
        "fileName": "document.pdf",
        "fileId": "uuid-here",
        "size": 1048576,
        "lastModified": "2025-09-26T10:30:00.000Z",
        "contentType": "application/pdf",
        "uploadedBy": "user@example.com",
        "userRole": "uploader",
        "etag": "d41d8cd98f00b204e9800998ecf8427e",
        "storageClass": "STANDARD"
      }
    ],
    "nextContinuationToken": null,
    "isTruncated": false,
    "totalCount": 1
  },
  "message": "Found 1 files"
}

Security Implementation

Multi-Layer Security Architecture

  1. Authentication Layer

    • AWS Cognito User Pools with configurable password policies
    • JWT token validation with expiration
    • Optional MFA support for enhanced security

  2. Authorization Layer

    • Role-based access control with three distinct roles
    • AWS IAM temporary credentials via AssumeRole
    • Principle of least privilege enforcement

  3. Data Protection

    • S3 server-side encryption with customer-managed KMS keys
    • Pre-signed URLs with configurable expiration times
    • Input validation and sanitization for all endpoints

  4. Network Security

    • HTTPS enforcement for all communications
    • CORS configuration for web applications
    • Request rate limiting to prevent abuse

Role Definitions

Admin Role - Full System Access

{
  "role": "admin",
  "permissions": [
    "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket",
    "s3:GetBucketLocation", "iam:ListUsers", "iam:CreateUser", "iam:DeleteUser",
    "kms:Encrypt", "kms:Decrypt", "kms:GenerateDataKey",
    "cloudwatch:GetMetricStatistics", "logs:CreateLogGroup",
    "logs:CreateLogStream", "logs:PutLogEvents"
  ]
}

Uploader Role - Upload and Manage Own Files

{
  "role": "uploader",
  "permissions": [
    "s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:ListBucket",
    "kms:Encrypt", "kms:Decrypt", "kms:GenerateDataKey"
  ]
}

Viewer Role - Read-Only Access

{
  "role": "viewer",
  "permissions": [
    "s3:GetObject", "s3:ListBucket"
  ]
}

Monitoring & Observability

CloudWatch Metrics Tracked

Metric Name Description Dimensions Unit
ApiRequestDuration API endpoint response time Endpoint, Method, StatusCode Milliseconds
ApiRequestCount Number of API requests Endpoint, Method, StatusCode Count
UploadDuration File upload completion time UserRole, UploadType, Success Milliseconds
UploadSpeed Upload throughput UserRole, UploadType Bytes/Second
FileSize Distribution of uploaded file sizes UserRole, UploadType Bytes
UploadCount Success and failure counts UserRole, UploadType, Success Count
MultipartPartCount Number of parts in multipart uploads UserRole Count

Structured Logging Format

{
  "timestamp": "2025-09-26T10:30:00.000Z",
  "level": "INFO",
  "message": "Upload completed successfully",
  "metadata": {
    "fileId": "abc-123",
    "fileName": "document.pdf",
    "fileSize": 1048576,
    "uploadSpeed": "2.5 MB/s",
    "userEmail": "user@example.com",
    "userRole": "uploader",
    "duration": 412,
    "s3Key": "uploads/2025-09-26/abc-123_document.pdf"
  }
}

Performance Monitoring Features

  • Automatic alerting for API requests exceeding 5 seconds
  • System resource tracking with CPU and memory usage
  • Upload success rate monitoring with configurable thresholds
  • Error rate tracking with detailed error categorization
  • Real-time metrics available in CloudWatch console

Configuration

AWS Services Setup

  1. S3 Bucket Configuration
# Create bucket with proper configuration
aws s3 mb s3://project3tmp --region eu-north-1

# Enable versioning for data protection
aws s3api put-bucket-versioning \
  --bucket project3tmp \
  --versioning-configuration Status=Enabled

# Configure server-side encryption
aws s3api put-bucket-encryption \
  --bucket project3tmp \
  --server-side-encryption-configuration '{
    "Rules": [{
      "ApplyServerSideEncryptionByDefault": {
        "SSEAlgorithm": "aws:kms",
        "KMSMasterKeyID": "your-kms-key-id"
      }
    }]
  }'
  1. IAM Roles and Policies
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl", 
        "s3:PutObjectTagging",
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::project3tmp/*",
        "arn:aws:s3:::project3tmp"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:GenerateDataKey",
        "kms:DescribeKey"
      ],
      "Resource": "arn:aws:kms:eu-north-1:account:key/your-kms-key-id"
    }
  ]
}
  1. CloudWatch Permissions
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams",
        "cloudwatch:PutMetricData",
        "cloudwatch:ListMetrics",
        "cloudwatch:GetMetricStatistics"
      ],
      "Resource": "*"
    }
  ]
}

Testing

File Upload Testing

  1. Single File Upload Test
# Create test file
echo "Test content" > test-document.txt

# Get upload URL via API
curl -X POST http://localhost:3000/files/upload-url \
  -H "Authorization: Bearer your-jwt-token" \
  -H "Content-Type: application/json" \
  -d '{
    "fileName": "test-document.txt",
    "fileSize": 13,
    "contentType": "text/plain"
  }'

# Upload file using returned pre-signed URL
curl -X PUT "pre-signed-url-here" \
  -H "Content-Type: text/plain" \
  -H "Content-Length: 13" \
  --data-binary "@test-document.txt"
  1. Multipart Upload Test
# Create large test file (500MB)
dd if=/dev/zero of=test-500mb.bin bs=1024 count=512000

# Split into parts for multipart upload
dd if=test-500mb.bin of=part1.bin bs=1048576 count=100 skip=0
dd if=test-500mb.bin of=part2.bin bs=1048576 count=100 skip=100
dd if=test-500mb.bin of=part3.bin bs=1048576 count=100 skip=200
dd if=test-500mb.bin of=part4.bin bs=1048576 count=100 skip=300
dd if=test-500mb.bin of=part5.bin bs=1048576 count=100 skip=400

Monitoring Verification

  1. Check CloudWatch Metrics

    • Navigate to AWS CloudWatch Console
    • Browse metrics under "my-app-namespace" namespace
    • Verify API request metrics are being recorded

  2. Verify Logs

    • Check CloudWatch Logs for "my-app-log-group"
    • Confirm structured JSON log entries
    • Validate error logging for failed requests

  3. Performance Testing

    • Monitor upload speeds for different file sizes
    • Verify multipart upload performance improvements
    • Check API response time metrics

Performance Characteristics

Upload Performance Benchmarks

  • Small files (< 10MB): Single upload recommended, typical speed 5-15 MB/s
  • Medium files (10-100MB): Single upload acceptable, speeds 10-25 MB/s
  • Large files (> 100MB): Multipart upload required, speeds 15-50 MB/s
  • Maximum file size: 5GB (AWS S3 limit for single operation)

API Response Time Targets

  • Authentication: < 500ms for login operations
  • Upload URL generation: < 1000ms for single file URLs
  • Multipart initialization: < 2000ms for complex operations
  • File listing: < 3000ms depending on result set size

Scalability Considerations

  • Concurrent uploads: Limited by AWS service quotas and IAM role limits
  • Storage capacity: Unlimited with S3, monitored for cost optimization
  • Request rate limits: Configurable via middleware, default 100 req/min per user
  • Memory usage: Optimized for minimal memory footprint during uploads

Error Handling

Common Error Scenarios and Solutions

  1. Authentication Failures

    • Invalid JWT Token: Token expired or malformed
    • Missing Authorization: Header not provided
    • Cognito Errors: User not found or password incorrect

  2. Authorization Errors

    • Insufficient Permissions: User role lacks required S3 permissions
    • AssumeRole Failures: IAM role trust relationship issues
    • Resource Access Denied: Attempting to access unauthorized files

  3. File Upload Errors

    • File Too Large: Exceeds configured size limits
    • Invalid Content Type: Unsupported file format
    • S3 Service Errors: Network timeouts or service unavailability
    • Multipart Upload Failures: Part upload failures or completion errors

Error Response Format

{
  "success": false,
  "error": "Detailed error message for debugging",
  "code": "UPLOAD_FILE_TOO_LARGE", 
  "requestId": "uuid-for-tracing",
  "timestamp": "2025-09-26T10:30:00.000Z",
  "details": {
    "maxFileSize": 5368709120,
    "providedFileSize": 6442450944
  }
}

Error Recovery Strategies

  • Retry Logic: Automatic retry for transient errors
  • Circuit Breakers: Prevent cascading failures
  • Graceful Degradation: Fallback to basic functionality
  • Dead Letter Queues: For processing failed operations

Deployment and Operations

Production Deployment Checklist

  • Environment variables configured for production
  • AWS IAM roles and policies properly configured
  • S3 bucket created with encryption enabled
  • CloudWatch log groups and metric alarms configured
  • SSL/TLS certificates installed and verified
  • Security groups and network ACLs configured
  • Backup and disaster recovery procedures documented
  • Monitoring dashboards and alerting rules established

Infrastructure as Code Example

# CloudFormation template snippet
Resources:
  FileUploadBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub '${Environment}-secure-file-upload'
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: aws:kms
              KMSMasterKeyID: !Ref UploadKMSKey
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      
  UploadKMSKey:
    Type: AWS::KMS::Key
    Properties:
      Description: 'Encryption key for file uploads'
      KeyPolicy:
        Statement:
          - Effect: Allow
            Principal:
              AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
            Action: 'kms:*'
            Resource: '*'

Operational Monitoring

  1. Daily Monitoring Tasks

    • Review CloudWatch metrics for anomalies
    • Check error rates and response times
    • Verify successful upload completion rates
    • Monitor storage usage and costs

  2. Weekly Maintenance

    • Review security logs for unauthorized access attempts
    • Analyze performance trends and optimization opportunities
    • Update dependencies and security patches
    • Backup configuration and disaster recovery testing

  3. Monthly Reviews

    • Cost optimization analysis
    • Security audit and access review
    • Performance benchmark updates
    • Documentation and procedure updates

Troubleshooting Guide

Common Issues and Resolutions

  1. Upload Failures

    # Check S3 bucket permissions
aws s3api get-bucket-policy --bucket project3tmp

# Verify KMS key access
aws kms describe-key --key-id your-kms-key-id

# Test pre-signed URL generation
aws s3 presign s3://project3tmp/test-file --expires-in 3600

  2. Authentication Problems

    # Verify Cognito user pool configuration
aws cognito-idp describe-user-pool --user-pool-id your-pool-id

# Check user status
aws cognito-idp admin-get-user --user-pool-id your-pool-id --username user@example.com

  3. Performance Issues

    • Review CloudWatch metrics for bottlenecks
    • Check network connectivity and latency
    • Analyze database query performance
    • Monitor system resource utilization

Support and Maintenance

Monitoring Best Practices:

  • Set up automated alerts for critical metrics
  • Implement log aggregation and analysis
  • Regular security assessment and penetration testing
  • Maintain documentation and runbooks

Scaling Considerations:

  • Monitor AWS service quotas and request increases as needed
  • Implement caching layers for frequently accessed data
  • Consider multi-region deployment for global users
  • Plan for disaster recovery and business continuity

Contributing

This project serves as a reference implementation for secure file upload systems using AWS services. Key areas for enhancement include:

  • Advanced Security Features: Implement additional security controls
  • Performance Optimization: Fine-tune upload algorithms and caching
  • Multi-Region Support: Extend to support global deployments
  • Cost Optimization: Implement intelligent storage tiering
  • Integration Extensions: Add support for additional AWS services

License

This project is designed for educational and demonstration purposes, showcasing enterprise-grade cloud architecture patterns and AWS service integration best practices.

Technical Stack: Node.js, TypeScript, Express.js, AWS SDK v3, AWS Cognito, Amazon S3, AWS KMS, CloudWatch

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages