Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
134 commits
Select commit Hold shift + click to select a range
1c3719f
Metrics that I started modifying
kf-cuanschutz Apr 25, 2024
e2db3a3
[benchmark]: Add simple benchmarks
haoyu-zc May 20, 2024
cf21b48
[test]: Create a new jupyter testing folder
haoyu-zc May 30, 2024
a7e54c7
[metrics]: Complete working version of get_contingency_matrix
haoyu-zc May 30, 2024
52d22f3
[metrics]: Complete working version of get_contingency_matrix
haoyu-zc May 30, 2024
b8b50a2
Clean up code
haoyu-zc Jun 4, 2024
ef1d075
[bench]: Add reference CPU benchmarks
haoyu-zc Jun 4, 2024
a91ff20
[bench]: Run single-threaded and multi-threading configs
haoyu-zc Jun 4, 2024
fec6aa6
[bench]: Test ccc-gpu under different configs
haoyu-zc Jun 11, 2024
8fcfefd
bench]: Move test results
haoyu-zc Jun 20, 2024
c250808
[bench]: Add template notebook
haoyu-zc Jun 20, 2024
1c4ddf1
[bench]: Add pyinstrument metrics
haoyu-zc Jun 28, 2024
1e02f13
[bench]: Add pyinstrument metrics for different cores
haoyu-zc Jun 28, 2024
c55ca5a
[bench]: Use pyinstrument to profile ari_gpu
haoyu-zc Jul 2, 2024
5a95636
[cuda/coef]: Refactor functions for future use
haoyu-zc Aug 6, 2024
855534f
[cuda/coef]: Factor out more functions
haoyu-zc Aug 8, 2024
d424a40
[coef]: Add tests for functions
haoyu-zc Aug 8, 2024
0eeefad
[coef]: Apply grid-stride loops
haoyu-zc Aug 11, 2024
42faf9a
[build]: Update dependencies
haoyu-zc Aug 12, 2024
ef4132b
[coef]: Test cupy routines
haoyu-zc Aug 12, 2024
29f0c88
[coef]: Finish get_parts cuda version
haoyu-zc Aug 12, 2024
9bb2af8
[test/coef]: Add tests for get_parts
haoyu-zc Aug 12, 2024
88e3fc9
[coef]: Return gpu reference for get_parts
haoyu-zc Aug 12, 2024
550c663
[coef]: Prepare to do matrix transformation
haoyu-zc Aug 13, 2024
074a6a6
[coef]: Finish matrix transformation for d_parts
haoyu-zc Aug 13, 2024
15b3531
[coef]: Pass compilation for compute_coef
haoyu-zc Aug 13, 2024
8c8486d
[coef]: Need to adapt the ari function
haoyu-zc Aug 13, 2024
6fc442b
[build]: Upgrade scipy
haoyu-zc Aug 15, 2024
b538020
[test]: Fail on previous test
haoyu-zc Aug 16, 2024
61ad1cb
[metrics]: Eliminate in-kernel memory allocations
haoyu-zc Aug 23, 2024
1cfc7a5
[get_parts]: GPU version fail on n_cluster = 6
haoyu-zc Aug 25, 2024
690f663
[test/get_parts]: Located the cause of the partitioning issue
haoyu-zc Aug 26, 2024
e32f03a
[test]: Add a debugging scrit
haoyu-zc Sep 5, 2024
6232726
[test]: Use all-close to take CPU-GPU FP errors into consideration
haoyu-zc Sep 5, 2024
f4ef165
[get_parts]: Fix many edge cases
haoyu-zc Sep 6, 2024
2988cde
[test/get_parts]: Add different distributions to generate features
haoyu-zc Sep 6, 2024
6b06a3f
Clean up code
haoyu-zc Sep 6, 2024
e5cddca
[test/regress]: Add parameters
haoyu-zc Sep 12, 2024
d93d15a
[test/coef]: Rewrite tests to introduce error tolerance
haoyu-zc Sep 13, 2024
40ccec5
[test]: Comment out unnecessary tests
haoyu-zc Sep 13, 2024
eca5bf9
[ari]: Introduce new libraries
haoyu-zc Sep 14, 2024
7d8da5b
[test/coef]: Allow errors in GPU vs CPU results
haoyu-zc Sep 15, 2024
c965b6e
[coef]: Try out cupy raw kernel with thrust
haoyu-zc Sep 22, 2024
1aae269
[coef]: Precompute number of unique partition values
haoyu-zc Sep 23, 2024
2d9a995
[coef]: Add tests for raw device function
haoyu-zc Sep 23, 2024
00ecdc9
[coef]: Add device function to unravel 2D index
haoyu-zc Sep 24, 2024
9604af4
[test]: Fix calls to get_parts
haoyu-zc Sep 24, 2024
98e07da
[test]: Add test skeleton
haoyu-zc Sep 29, 2024
1349d99
[cccl]: Compile cub raw kernel successfully
haoyu-zc Sep 29, 2024
0ae0ac5
[cccl]: Try out cupy shared_mem dynamic allocation
haoyu-zc Sep 29, 2024
6a69312
[cccl]: Hack non-type template parameters
haoyu-zc Sep 29, 2024
1858f6d
[kernel]: Test fail on partition selection
haoyu-zc Oct 1, 2024
5b19b2f
[ari/kernel]: Make sure variable "parts" is correctly loaded
haoyu-zc Oct 1, 2024
a3ce150
[kernel]: Add cuda cppp source code
haoyu-zc Oct 5, 2024
386e7e2
[kernel/cu]: Clean up code
haoyu-zc Oct 5, 2024
bccf7a0
[coef]: Precompute the max for each partition
haoyu-zc Oct 5, 2024
a8d3d68
[metrics/kennel]: Add device function to compute the contingency matrix
haoyu-zc Oct 9, 2024
fc02525
[metrics/kernel]: Fix test
haoyu-zc Oct 9, 2024
e148320
[metrics/kernel]: Parameterize tests
haoyu-zc Oct 9, 2024
951b004
[metrics/kernel]: Update C++ CUDA code
haoyu-zc Oct 9, 2024
945de79
[metrics/kernel]: Add initial version of pair_confusion_matrix
haoyu-zc Oct 11, 2024
f2bc8cf
[metric/device]: Add logic to perform matrix sum
haoyu-zc Oct 15, 2024
39d06db
[metrics/device]: Finish first version of get_pair_conf matrix
haoyu-zc Oct 16, 2024
3c14345
[doc]: Add function documentation
haoyu-zc Oct 16, 2024
195e269
[test/kernel]: Update input matrix
haoyu-zc Oct 16, 2024
d13e946
[metrics/device]: Add device function string for get_pair_conf matrics
haoyu-zc Oct 16, 2024
806dfdc
[test/device]: Fix device memory type
haoyu-zc Oct 16, 2024
124ef0d
[test/device]: Compile kernel test successfully
haoyu-zc Oct 16, 2024
4e3cb6a
[test/device]: Get incorrect conf matrix results
haoyu-zc Oct 16, 2024
f666d4d
[metrics/device]: Fix get_conf_matrix and pass the first test
haoyu-zc Oct 17, 2024
662bfc6
[test/device]: Add parameterized tests
haoyu-zc Oct 17, 2024
c28cf9a
[metrics/device]: Update CUDA C++ code
haoyu-zc Oct 17, 2024
10e2e22
[test/ari]: Yield correct GPU ari
haoyu-zc Oct 18, 2024
5176540
[test]: Encounter illegal memory access in the ari kernel
haoyu-zc Oct 23, 2024
8640583
[test/kernel]: Fix test_art_parts_selection
haoyu-zc Oct 23, 2024
bf6d6a5
[test/kernel]: Pass ari tests
haoyu-zc Oct 23, 2024
9964cc7
[kernel]: Clean up debugging code
haoyu-zc Oct 24, 2024
33cfb1b
[kernel]: Add CUDA check macro
haoyu-zc Oct 24, 2024
7a3f7d4
[kernel]: Clean up cupy raw strings
haoyu-zc Oct 25, 2024
e1ab1a0
[bench]: Add simple benchmark for gpu_ari
haoyu-zc Oct 26, 2024
34cb479
[kernel]: Begin to do refactoring
haoyu-zc Oct 28, 2024
2232b66
[kernel]: Introduce the thurst library
haoyu-zc Oct 28, 2024
ba9d541
[build]: Add cmake file for the cuda kernel and tests
haoyu-zc Oct 28, 2024
3add53f
[test/kernel]: Build tests successfully
haoyu-zc Oct 28, 2024
78add78
[test/kernel]: Factor out tests for ari pair matching
haoyu-zc Oct 28, 2024
c0a0a98
[build]: Introduce gtest as a dependency
haoyu-zc Oct 28, 2024
94b5602
[test/ari]: Add simple cpp test cases for cudaAri
haoyu-zc Oct 28, 2024
a7237e7
[build]: Introduce dependency pybind11
haoyu-zc Oct 29, 2024
1b4aacf
[misc]: Move the cuda module to a separate folder
haoyu-zc Oct 30, 2024
6e29dc7
[build]: Test on an example of pybind11
haoyu-zc Oct 30, 2024
f55bf81
[build]: Compile cudaAri with pybind11 successfully
haoyu-zc Oct 31, 2024
89a6674
[binding]: Pass compilation using numpy array interface
haoyu-zc Nov 1, 2024
800ec2d
[binding]: Use py::array_t to accept np.ndarray
haoyu-zc Nov 1, 2024
a76de44
[test]: Pass tests for binded cuda code
haoyu-zc Nov 1, 2024
35ff703
[test/kernel]: Test ari with latest C++ API
haoyu-zc Nov 3, 2024
294f4b5
[build]: Name cuda extension as "ccc_cuda_ext"
haoyu-zc Nov 4, 2024
87603f5
[linker]: Separate template function declaration and implementation
haoyu-zc Nov 4, 2024
d06306c
[mist]: Rename ari function according to input type
haoyu-zc Nov 4, 2024
4723eb5
Clean up dead code
haoyu-zc Nov 4, 2024
4a0edd3
Clean up and rename tests
haoyu-zc Nov 4, 2024
d7f6768
[build]: Compile pybind11 with gtest successfully
haoyu-zc Nov 4, 2024
69255dc
[cuda]: Factor out ari function using vector for testing
haoyu-zc Nov 5, 2024
44ab7ee
[build]: Compile and pass c++ tests successfully
haoyu-zc Nov 5, 2024
c021a5c
[binding]: Embed python interpreter in c++ tests
haoyu-zc Nov 7, 2024
c47d129
[binding]: Call ccc python code in c++ successfully
haoyu-zc Nov 7, 2024
70c4f7d
[binding]: Replace raw python string evaluration
haoyu-zc Nov 7, 2024
969a79b
[test]: Migrate pytest to gtest
haoyu-zc Nov 8, 2024
d723383
[test]: Refactor ari tests by separating test code and data generatio…
haoyu-zc Nov 8, 2024
03a6aca
[doc]: Add test documentation
haoyu-zc Nov 8, 2024
452a074
[build]: Use scikit-build to repace setuptools
haoyu-zc Nov 8, 2024
a68efbc
Clean up code and add docs
haoyu-zc Nov 8, 2024
9c717a9
[bug]: Try to fix illegal mem access
haoyu-zc Nov 8, 2024
8855916
[cuda]: Add input and parameter validation
haoyu-zc Nov 8, 2024
1a02809
[bug]: Test fail on large number of objects
haoyu-zc Nov 8, 2024
9f52c55
[cuda]: Remove debugging function argument
haoyu-zc Nov 10, 2024
23340ce
[cuda]: Introduce CUB for block-wise data loading
haoyu-zc Nov 11, 2024
7c076f0
[doc]: Add notes for CUDA module compilation and testing
haoyu-zc Jan 6, 2025
360b7ba
[cuda]: Add example code of CUB
haoyu-zc Jan 6, 2025
98c389b
[kernel]: Use global memory to load partition pairs for now
haoyu-zc Jan 9, 2025
955fee9
[doc]: Update readme
haoyu-zc Jan 9, 2025
cebbb24
[doc]: Initialize sphnix and read the docs
haoyu-zc Jan 9, 2025
e848ae7
[test]: Fix deprecated api calls
haoyu-zc Jan 10, 2025
90ae8ec
[impl]: Revert to the previous binning logic
haoyu-zc Jan 11, 2025
ba74458
[build]: Update conda dependencies
haoyu-zc Jan 11, 2025
bc4b088
[impl]: Revert to the original CPU version of ccc main logic
haoyu-zc Jan 13, 2025
dca231c
Clean up test files
haoyu-zc Jan 15, 2025
72b07d1
[build]: Update conda env name and dev env setup
haoyu-zc Jan 15, 2025
71b23ae
[doc]: Update readme
haoyu-zc Jan 16, 2025
18f6c10
[build]: Updare environment setup
haoyu-zc Jan 16, 2025
17ee32d
[fmt]: Add pre-commit configuration
haoyu-zc Jan 17, 2025
3369adf
[test]: Add script to run both Python and C++ tests
haoyu-zc Jan 17, 2025
e087fa9
[test]: Exclude unneeded tests
haoyu-zc Jan 17, 2025
a6b3964
[test/cpp]: Skip time-consuming unit tests
haoyu-zc Jan 17, 2025
f4b7ccf
[test]: Create script to automate all test suites
haoyu-zc Jan 17, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -127,3 +127,12 @@ dmypy.json

# Pyre type checker
.pyre/

# Pycharm
.idea/*

# Development directory
__dev

# Binary files
libs/ccc/sklearn/metrics
20 changes: 20 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: check-toml
- id: check-added-large-files
- id: mixed-line-ending
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.8.5 # Ruff version.
hooks:
- id: ruff # Run the linter.
types_or: [python, pyi]
args: [--fix]
- id: ruff-format # Run the formatter.
types_or: [python, pyi]
35 changes: 35 additions & 0 deletions .readthedocs.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Read the Docs configuration file for Sphinx projects
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details

# Required
version: 2

# Set the OS, Python version and other tools you might need
build:
os: ubuntu-22.04
tools:
python: "3.12"
# You can also specify other tool versions:
# nodejs: "20"
# rust: "1.70"
# golang: "1.20"

# Build documentation in the "docs/" directory with Sphinx
sphinx:
configuration: docs/source/conf.py
# You can configure Sphinx to use a different builder, for instance use the dirhtml builder for simpler URLs
# builder: "dirhtml"
# Fail on all warnings to avoid broken references
# fail_on_warning: true

# Optionally build your docs in additional formats such as PDF and ePub
# formats:
# - pdf
# - epub

# Optional but recommended, declare the Python requirements required
# to build your documentation
# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
# python:
# install:
# - requirements: docs/requirements.txt
100 changes: 100 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
cmake_minimum_required(VERSION 3.15...3.26)
project(${SKBUILD_PROJECT_NAME} LANGUAGES CUDA CXX)

# Add this near the top of your file, after project()
# Define the include directories for the whole project
set(PROJECT_INCLUDE_DIR ${CMAKE_SOURCE_DIR}/libs)
# Set extention name and source directory
set(CUDA_EXT_MODULE_NAME ccc_cuda_ext)
set(CUDA_EXT_DIR ${CMAKE_CURRENT_SOURCE_DIR}/libs/${CUDA_EXT_MODULE_NAME})

# Set C++ standard
set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

# Set Python Standard
# Get Python version dynamically
execute_process(
COMMAND "${Python_EXECUTABLE}" -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}')"
OUTPUT_VARIABLE PYTHON_VERSION_FULL
OUTPUT_STRIP_TRAILING_WHITESPACE
)
# Set the paths using the detected version
set(Python_EXECUTABLE $ENV{CONDA_PREFIX}/bin/python)
set(PYTHON_INCLUDE_DIR $ENV{CONDA_PREFIX}/include/python${PYTHON_VERSION_FULL})
set(PYTHON_LIBRARY $ENV{CONDA_PREFIX}/lib/libpython${PYTHON_VERSION_FULL}.so)


# Set CUDA architecture and Pybind11
find_package(Python REQUIRED Development)
set(PYBIND11_NEWPYTHON ON)
find_package(pybind11 CONFIG REQUIRED)

# Download and configure Google Test
include(FetchContent)
FetchContent_Declare(
googletest
GIT_REPOSITORY https://github.com/google/googletest.git
GIT_TAG v1.15.2 # Adjust version as needed
)
FetchContent_MakeAvailable(googletest)

# Setup Gtest
enable_testing()
# Function to automatically add tests from a directory
function(add_tests_from_directory TEST_DIR)
# Find all test files in the directory
file(GLOB_RECURSE TEST_FILES
"${TEST_DIR}/*_test.cpp" # Files ending with _test.cpp
"${TEST_DIR}/*_tests.cpp" # Files ending with _tests.cpp
"${TEST_DIR}/test_*.cpp" # Files starting with test_
)

# Loop through each test file
foreach(TEST_FILE ${TEST_FILES})
# Get the filename without extension
get_filename_component(TEST_NAME ${TEST_FILE} NAME_WE)

# Create an executable for this test
add_executable(${TEST_NAME} ${TEST_FILE} ${CUDA_EXT_DIR}/metrics.cu)

# target_include_directories(${TEST_NAME} PRIVATE
# ${PROJECT_INCLUDE_DIR} # Add this line
# ${Python_INCLUDE_DIRS}
# )

# Link against gtest and your project libraries
target_link_libraries(${TEST_NAME} PRIVATE
GTest::gtest_main
GTest::gtest
pybind11::headers
pybind11::embed
Python::Python
# Add your other project libraries here
# project_lib
)

# Add the test to CTest
add_test(NAME ${TEST_NAME} COMMAND ${TEST_NAME})

# Set test properties (optional)
# Set test properties (optional)
set_tests_properties(${TEST_NAME} PROPERTIES
TIMEOUT 10 # Timeout in seconds
WORKING_DIRECTORY "${CMAKE_RUNTIME_OUTPUT_DIRECTORY}"
)
endforeach()
endfunction()

# Specify your test directory and call the function
add_tests_from_directory(${CMAKE_CURRENT_SOURCE_DIR}/tests)

# Optional: Set output directories
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin)
set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/lib)


# Set up binding then do compilation and installation
pybind11_add_module(${CUDA_EXT_MODULE_NAME} ${CUDA_EXT_DIR}/binder.cu ${CUDA_EXT_DIR}/metrics.cu)

install(TARGETS ${CUDA_EXT_MODULE_NAME} LIBRARY DESTINATION .)
Loading
Loading