Resource-Efficient CI/CD Pipelines

This project introduces a CI/CD pipeline that tracks resource efficiency (CPU, memory, disk, and network) over time. It provides a framework for analyzing and visualizing the resource consumption of a Java application, helping to identify performance regressions and improve resource utilization.

Features

• Resource Monitoring: Tracks CPU, memory, disk, and network usage of a Java application.
• Two Monitoring Approaches: Implements both "Direct Instrumentation" and "External Monitoring" for comprehensive analysis.
• Data Visualization: Generates graphs to visualize resource consumption trends over time.
• Case Study: Includes a case study on the resource consumption of the GSON library over six major releases.
• Extensible: The framework can be extended to monitor other applications and resources.

System Under Test

The system under test is a Spring Boot web application with four REST endpoints, each designed to stress a specific resource:

• GET /cpu: Consumes CPU by performing intensive hashing operations.
• GET /memory: Consumes memory by creating large objects.
• GET /disk: Consumes disk I/O by writing a large file.
• GET /network: Consumes network bandwidth by downloading a large file.

The intensity of the resource load can be configured in the application.yml file.

Methodology

The project employs two different methods for monitoring resource consumption:

Direct Instrumentation

This approach involves measuring resource usage from within the application code itself. The application records the time taken for certain operations, memory consumed, and other metrics, and saves this data to text files. This method provides accurate and detailed information but requires modifying the application code.

External Monitoring

This approach uses external tools to monitor the application's resource usage. The application's process ID (PID) is used to track its CPU, memory, disk, and network consumption. This method is less intrusive than direct instrumentation but may be less accurate due to the overhead of the monitoring tools.

Case Study: GSON Library

The project includes a case study on the resource consumption of the GSON library, a popular open-source Java library for JSON processing. The case study analyzes the CPU, memory, and throughput of GSON over six major releases, providing insights into the library's performance evolution.

The results of the case study are available in the results/case study directory, and the corresponding graphs are in the results/graphs directory.

How to Run

Prerequisites

• Java 17
• Maven
• Python 3
• matplotlib library (pip install matplotlib)

1. Build and Run the Application

1. Navigate to the app directory:

   cd app

2. Build the application using Maven:

   ./mvnw clean install

3. Run the application:

   java -jar target/test-rsc-ci-0.0.1-SNAPSHOT.jar

2. Generate Graphs

1. Navigate to the results directory:

   cd results

2. Run the Python scripts to generate the graphs:
   • For the GSON case study:

     python case_study_graph_plotter.py

   • For the direct instrumentation and external monitoring comparison:

     python display_usage.py

The generated graphs will be saved in the results/graphs directory.

Directory Structure

.
├── app
│   ├── src/main/java/com/nishant/testrscci/TestRscCiApplication.java  # Main application file
│   ├── pom.xml                                                     # Maven build configuration
│   └── ...
├── results
│   ├── case study                                                  # Data for the GSON case study
│   ├── direct instrumentation                                      # Data from the direct instrumentation approach
│   ├── ext monitoring                                              # Data from the external monitoring approach
│   ├── graphs                                                      # Generated graphs
│   ├── case_study_graph_plotter.py                                 # Python script for the GSON case study graphs
│   └── display_usage.py                                            # Python script for comparing the two monitoring approaches
└── ...

Contributing

Contributions are welcome! Please feel free to submit a pull request or open an issue if you have any suggestions or improvements.