AdaptiveFeed

A personalised news ranking system that learns and adapts to each user's preferences through interaction feedback such as clicks, dwell-time, likes, shares, and bookmarks.

Documentation: DOCUMENTATION.md.

Directory Structure

.
├── code
│   ├── elastic_search
│   │   └── ...
│   ├── lambda_mart
│   │   └── ...
│   ├── logistic_regression
│   │   └── ...
│   ├── rank_net
│   │   └── ...
│   ├── reinforcement_learning
│   │   └── ...
│   ├── shared
│   │   ├── constants.py
│   │   └── utils.py
│   ├── tools
│   │   ├── test.py
│   │   └── train.py
│   ├── ab_testing.py
│   ├── build_features.py
│   ├── docker-compose.yml
│   ├── generate_baseline_logs.py
│   ├── plot_ab_results.py
│   ├── plot_evaluation_results.py
│   ├── setup_es_index.py
│   ├── setup_train_data.sh
│   ├── start_servers.sh
│   └── stop_servers.sh
├── configs
│   ├── lambda_mart_config.yaml
│   ├── logistic_regression_config.yaml
│   ├── rank_net_config.yaml
│   └── reinforcement_learning_config.yaml
├── data
│   └── articles.jsonl
├── features
│   ├── article_features.parquet
│   └── user_profiles.parquet
├── models
│   └── ...                         # Trained models (.joblib, .pth)
├── output
│   └── ...                         # Output plots and user interaction logs
├── results
│   └── ...                         # Evaluation + AB testing results (.json)
├── servers
│   └── user_simulation_server.tar  # Download from provided cloud link
├── .gitattributes
├── .gitignore
├── config.yaml
├── DOCUMENTATION.md
├── README.md
└── requirements.txt

Setup

Setup Repository (Run Once)

Before cloning, make sure Git LFS is installed:

# Install Git LFS (Mac)
brew install git-lfs

# Linux
sudo apt install git-lfs

# Windows
choco install git-lfs

Then initialise:

git lfs install

Then clone: (Might take some time due to large files)

# Clone the repository
git clone https://github.com/Vinit2244/AdaptiveFeed.git

# Move into the repository folder
cd AdaptiveFeed

Env Setup (Run Once)

Create env:

# Mac/Linux
python3 -m venv adaptivefeed_env

# Windows
python -m venv adaptivefeed_env

Initialise env:

# Mac/Linux
source ./adaptivefeed_env/bin/activate

# Windows (Powershell)
adaptivefeed_env\Scripts\Activate.ps1

# Windows (Cmd)
adaptivefeed_env\Scripts\activate.bat

Install requirements:

# Mac/Linux
pip3 install -r requirements.txt

# Windows
pip install -r requirements.txt

---

Usage

Please make sure to update all the config files (config.yaml and model config files) file according to your settings before moving forward. All paths passed as arguments should be relative to project root folder: Adaptivefeed/

Move into code dir:

cd code

Start the servers:

sh start_servers.sh

Now open a new terminal window.

Create ES Index, Generate baseline logs and extract features:

# Update setup_train_data.sh code for flags
    # setup_es_index.py --force_recreate: forcefully recreate the index if index with same name already exists
    # generate_baseline_logs.py --filtered: only store the responses which have some user interaction
# Update config.yaml for training data generation settings
sh setup_train_data.sh

Train models:

python3 tools/train.py --model <MODEL_TYPE> --cnfg <PATH_TO_CONFIG_FILE>
# ex: python3 tools/train.py --model lr --cnfg configs/logistic_regression_config.yaml
# ex: python3 tools/train.py --model rl --cnfg configs/reinforcement_learning_config.yaml
# ex: python3 tools/train.py --model rn --cnfg configs/rank_net_config.yaml
# ex: python3 tools/train.py --model lm --cnfg configs/lambda_mart_config.yaml

Evaluate models:

python3 tools/test.py --model <MODEL_TYPE> --cnfg <PATH_TO_CONFIG_FILE>
# ex: python3 tools/test.py --model lr --cnfg configs/logistic_regression_config.yaml
# ex: python3 tools/test.py --model rl --cnfg configs/reinforcement_learning_config.yaml
# ex: python3 tools/test.py --model rn --cnfg configs/rank_net_config.yaml
# ex: python3 tools/test.py --model lm --cnfg configs/lambda_mart_config.yaml

Generate visualisation for outputs for individual evaluations of models:

# -r: Where the output logs (.json) file from model evaluation is stored
# -o: Where to store the output plots
python3 plot_evaluation_results.py -r results/individual_test -o output/plots

Perform A/B Testing:

python3 ab_testing.py --model <MODEL_TYPE> --cnfg <PATH_TO_CONFIG_FILE>
# ex: python3 ab_testing.py --model lr --cnfg configs/logistic_regression_config.yaml
# ex: python3 ab_testing.py --model rl --cnfg configs/reinforcement_learning_config.yaml
# ex: python3 ab_testing.py --model rn --cnfg configs/rank_net_config.yaml
# ex: python3 ab_testing.py --model lm --cnfg configs/lambda_mart_config.yaml

Generate visualisation for ab testing results:

# -r: Where the output logs (.json) file from AB testing is stored
# -o: Where to store the output plots
python3 plot_ab_test_results.py -r results/ab_test -o output/plots

Stop all the servers and deactivate environment:

sh stop_servers.sh

# Deactivate python env
deactivate