Clustering Algorithms

Core/CMakeLists.txt

@@ -32,6 +32,11 @@ set(FilterList
   RemoveFlaggedFeaturesFilter
   SplitAttributeArrayFilter
   FeatureDataCSVWriterFilter
+
+  KMedoidsFilter
+  KMeansFilter
+  SilhouetteFilter
+  ImportDeformKeyFilev12Filter
 )
 
 set(STUB_FILTERS
@@ -60,7 +65,11 @@ create_complex_plugin(NAME ${PLUGIN_NAME}
 # can use the target_sources(...) cmake call
 target_sources(${PLUGIN_NAME}
   PUBLIC
-  "${${PLUGIN_NAME}_SOURCE_DIR}/src/${PLUGIN_NAME}/Utilities/CoreUtilities.hpp"
+    "${${PLUGIN_NAME}_SOURCE_DIR}/src/${PLUGIN_NAME}/Utilities/CoreUtilities.hpp"
+    "${${PLUGIN_NAME}_SOURCE_DIR}/src/${PLUGIN_NAME}/Utilities/DistanceTemplate.hpp"
+    "${${PLUGIN_NAME}_SOURCE_DIR}/src/${PLUGIN_NAME}/Utilities/ClusteringAlgorithms/DBSCANTemplate.hpp"
+    "${${PLUGIN_NAME}_SOURCE_DIR}/src/${PLUGIN_NAME}/Utilities/ClusteringAlgorithms/KMedoidsTemplate.hpp"
+    "${${PLUGIN_NAME}_SOURCE_DIR}/src/${PLUGIN_NAME}/Utilities/ClusteringAlgorithms/KMeansTemplate.hpp"
 )
 source_group(TREE "${${PLUGIN_NAME}_SOURCE_DIR}/src/${PLUGIN_NAME}"
   PREFIX ${PLUGIN_NAME}
@@ -105,6 +114,11 @@ set(filter_algorithms
   CalculateArrayHistogram
   RemoveFlaggedFeatures
   SplitAttributeArray
+
+  KMedoids
+  KMeans
+  Silhouette
+  ImportDeformKeyFilev12
 )
 
 

Core/docs/ImportDeformKeyFilev12.md

@@ -0,0 +1,42 @@
+# ImportDeformKeyFilev12 #
+
+
+## Group (Subgroup) ##
+
+SimulationIO (SimulationIO)
+
+## Description ##
+
+This **Filter** reads DEFORM v12 key files and saves the data in a newly created **Data Container**.
+
+It reads the quadrilateral mesh data (nodal coordinates and connectivity), and the value of variables such as stress, strain, ndtmp, etc at cells and nodes. 
+
+## Parameters ##
+
+| Name | Type | Description |
+|------|------|------|
+| Input File | Path | Name and address of the input DEFORM v12 key file |
+
+## Required Geometry ##
+
+Not Applicable
+
+## Required Objects ##
+
+## Created Objects ##
+| Kind | Default Name | Type | Component Dimensions | Description |
+|------|--------------|-------------|---------|-----|
+| **Data Container** | DataContainer | N/A | N/A | Created **Data Container** |
+| **Attribute Matrix** | VertexData | Vertex | N/A | Created **Vertex Attribute Matrix** name |
+| **Attribute Matrix** | CellData | Cell | N/A | Created **Cell Attribute Matrix** name |
+
+## Example Pipelines ##
+
+## License & Copyright ##
+
+Please see the description file distributed with this plugin.
+
+## DREAM3D Mailing Lists ##
+
+If you need more help with a filter, please consider asking your question on the DREAM3D Users mailing list:
+https://groups.google.com/forum/?hl=en#!forum/dream3d-users

Core/docs/KMeans.md

@@ -0,0 +1,84 @@
+# K Means #
+
+## Group (Subgroup) ##
+
+DREAM3D Review (Clustering)
+
+## Description ##
+
+This **Filter** applies the k means algorithm to an **Attribute Array**.  K means is a _clustering algorithm_ that assigns to each point of the **Attribute Array** a _cluster Id_.  The user must specify the number of clusters in which to partition the array.  Specifically, a k means partitioning is a _Voronoi tesselation_; an optimal solution to the k means problem is such that each point in the data set is associated with the cluster that has the closest mean.  This partitioning is the one that minimizes the within cluster variance (i.e., minimizes the within cluster sum of squares differences).  Thus, the "metric" used for k means is the 2-norm (the _Euclidean norm_; the squared Euclidean norm may also be used since this maintains the triangle inequality).
+
+Optimal solutions to the k means partitioning problem are computationally difficult; this **Filter** used _Lloyd's algorithm_ to approximate the solution.  Lloyd's algorithm is an iterative algorithm that proceeds as follows:
+
+1. Choose k points at random to serve as the initial cluster "means"
+2. Until convergence, repeat the following steps:
+  * Associate each point with the closest mean, where "closest" is the smallest 2-norm distance
+  * Recompute the means based on the new tesselation
+
+Convergence is defined as when the computed means change very little (precisely, when the differences are within machine epsilon).  Since Lloyd's algorithm is iterative, it only serves as an approximation, and may result in different classifications on each execution with the same input data.  The user may opt to use a mask to ignore certain points; where the mask is _false_, the points will be placed in cluster 0.
+
+A clustering algorithm can be considered a kind of segmentation; this implementation of k means does not rely on the **Geometry** on which the data lie, only the _topology_ of the space that the array itself forms.  Therefore, this **Filter** has the effect of creating either **Features** or **Ensembles** depending on the kind of array passed to it for clustering.  If an **Element** array (e.g., voxel-level **Cell** data) is passed to the **Filter**, then **Features** are created (in the previous example, a **Cell Feature Attribute Matrix** will be created).  If a **Feature** array is passed to the **Filter**, then an **Ensemble Attribute Matrix** is created.  The following table shows what type of **Attribute Matrix** is created based on what sort of array is used for clustering:
+
+| Attribute Matrix Source             | Attribute Matrix Created |
+|------------------|--------------------|
+| Generic | Generic |
+| Vertex | Vertex Feature | 
+| Edge | Edge Feature |
+| Face | Face Feature | 
+| Cell | Cell Feature| 
+| Vertex Feature | Vertex Ensemble |
+| Edge Feature | Edge Ensemble |
+| Face Feature | Face Ensemble |
+| Cell Feature | Cell Ensemble|
+| Vertex Ensemble | Vertex Ensemble |
+| Edge Ensemble | Edge Ensemble |
+| Face Ensemble | Face Ensemble |
+| Cell Ensemble | Cell Ensemble|
+
+This **Filter** will store the means for the final clusters within the created **Attribute Matrix**.
+
+## Parameters ##
+
+| Name | Type | Description |
+|------|------|-------------|
+| Number of Clusters | int32_t | The number of clusters in which to partition the array |
+| Distance Metric | Enumeration | The metric used to determine the distances between points; only 2-norm metrics (i.e., Euclidean or squared Euclidean) may be chosen |
+| Use Mask | bool | Whether to use a boolean mask array to ignore certain points flagged as _false_ from the algorithm |
+
+## Required Geometry ###
+
+None
+
+## Required Objects ##
+
+| Kind | Default Name | Type | Component Dimensions | Description |
+|------|--------------|------|----------------------|-------------|
+| Any **Attribute Array** | None | Any| Any | The **Attribute Array** to cluster |
+| **Attrubute Array** | Mask | bool | (1) | Specifies if the point is to be counted in the algorithm, if _Use Mask_ is checked |
+
+## Created Objects ##
+
+| Kind | Default Name | Type | Component Dimensions | Description |
+|------|--------------|------|----------------------|-------------|
+| **Attribute Matrix** | ClusterData | Feature/Ensemble | N/A | The **Attribute Matrix** in which to store information associated with the created clusters |
+| **Attribute Array** | ClusterIds | int32_t | (1) | Specifies to which cluster each point belongs |
+| **Attribute Array** | ClusterMeans | double | (1) | The means of the final clusters |
+
+
+## References ## 
+
+[1] Least squares quantization in PCM, S.P. Lloyd, IEEE Transactions on Information Theory, vol. 28 (2), pp. 129-137, 1982.
+
+## Example Pipelines ##
+
+
+
+## License & Copyright ##
+
+Please see the description file distributed with this plugin.
+
+## DREAM3D Mailing Lists ##
+
+If you need more help with a filter, please consider asking your question on the DREAM3D Users mailing list:
+https://groups.google.com/forum/?hl=en#!forum/dream3d-users
+

Core/docs/KMedoids.md

@@ -0,0 +1,84 @@
+# K Medoids #
+
+## Group (Subgroup) ##
+
+DREAM3D Review (Clustering)
+
+## Description ##
+
+This **Filter** applies the k medoids algorithm to an **Attribute Array**.  K medoids is a _clustering algorithm_ that assigns to each point of the **Attribute Array** a _cluster Id_.  The user must specify the number of clusters in which to partition the array.  Specifically, a k medoids partitioning is such that each point in the data set is associated with the cluster that minimizes the sum of the pair-wise distances between the data points and their associated cluster centers (medoids).  This approach is analogous to [k means](@ref kmeans), but uses actual data points (the medoids) as the cluster exemplars instead of the means.  Medoids in this context refer to the data point in each cluster that is most like all other data points, i.e., that data point whose average distance to all other data points in the cluster is smallest.  Unlike [k means](@ref kmeans), since pair-wise distances are minimized instead of variance, any arbirtary concept of "distance" may be used; this **Filter** allows for the selection of a variety of distance metrics.    
+
+This **Filter** uses the _Voronoi iteration_ algorithm to produce the clustering.  The algorithm is iterative and proceeds as follows:
+
+1. Choose k points at random to serve as the initial cluster medoids
+2. Associate each point to the closest medoid
+3. Until convergence, repeat the following steps:
+  * For each cluster, change the medoid to the point in that cluster that minimizes the sum of distances between that point and all other points in the cluster
+  * Reassign each point to the closest medoid
+
+Convergence is defined as when the medoids no longer change position.  Since the algorithm is iterative, it only serves as an approximation, and may result in different classifications on each execution with the same input data.  The user may opt to use a mask to ignore certain points; where the mask is _false_, the points will be placed in cluster 0.
+
+A clustering algorithm can be considered a kind of segmentation; this implementation of k medoids does not rely on the **Geometry** on which the data lie, only the _topology_ of the space that the array itself forms.  Therefore, this **Filter** has the effect of creating either **Features** or **Ensembles** depending on the kind of array passed to it for clustering.  If an **Element** array (e.g., voxel-level **Cell** data) is passed to the **Filter**, then **Features** are created (in the previous example, a **Cell Feature Attribute Matrix** will be created).  If a **Feature** array is passed to the **Filter**, then an **Ensemble Attribute Matrix** is created.  The following table shows what type of **Attribute Matrix** is created based on what sort of array is used for clustering:
+
+| Attribute Matrix Source             | Attribute Matrix Created |
+|------------------|--------------------|
+| Generic | Generic |
+| Vertex | Vertex Feature | 
+| Edge | Edge Feature |
+| Face | Face Feature | 
+| Cell | Cell Feature| 
+| Vertex Feature | Vertex Ensemble |
+| Edge Feature | Edge Ensemble |
+| Face Feature | Face Ensemble |
+| Cell Feature | Cell Ensemble|
+| Vertex Ensemble | Vertex Ensemble |
+| Edge Ensemble | Edge Ensemble |
+| Face Ensemble | Face Ensemble |
+| Cell Ensemble | Cell Ensemble|
+
+This **Filter** will store the medoids for the final clusters within the created **Attribute Matrix**.
+
+## Parameters ##
+
+| Name | Type | Description |
+|------|------|-------------|
+| Number of Clusters | int32_t | The number of clusters in which to partition the array |
+| Distance Metric | Enumeration | The metric used to determine the distances between points |
+| Use Mask | bool | Whether to use a boolean mask array to ignore certain points flagged as _false_ from the algorithm |
+
+## Required Geometry ###
+
+None
+
+## Required Objects ##
+
+| Kind | Default Name | Type | Component Dimensions | Description |
+|------|--------------|------|----------------------|-------------|
+| Any **Attribute Array** | None | Any| Any | The **Attribute Array** to cluster |
+| **Attrubute Array** | Mask | bool | (1) | Specifies if the point is to be counted in the algorithm, if _Use Mask_ is checked |
+
+## Created Objects ##
+
+| Kind | Default Name | Type | Component Dimensions | Description |
+|------|--------------|------|----------------------|-------------|
+| **Attribute Matrix** | ClusterData | Feature/Ensemble | N/A | The **Attribute Matrix** in which to store information associated with the created clusters |
+| **Attribute Array** | ClusterIds | int32_t | (1) | Specifies to which cluster each point belongs |
+| **Attribute Array** | ClusterMeans | double | (1) | The means of the final clusters |
+
+
+## References ## 
+
+[1] A simple and fast algorithm for K-medoids clustering, H.S. Park and C.H. Jun, Expert Systems with Applications, vol. 28 (2), pp. 3336-3341, 2009.
+
+## Example Pipelines ##
+
+
+
+## License & Copyright ##
+
+Please see the description file distributed with this plugin.
+
+## DREAM3D Mailing Lists ##
+
+If you need more help with a filter, please consider asking your question on the DREAM3D Users mailing list:
+https://groups.google.com/forum/?hl=en#!forum/dream3d-users

Core/docs/Silhouette.md

@@ -0,0 +1,54 @@
+# Silhouette #
+
+## Group (Subgroup) ##
+
+DREAM3D Review (Clustering)
+
+## Description ##
+
+This **Filter** computes the silhouette for a clustered **Attribute Array**.  The user must select both the original array that has been clustered and the array of cluster Ids.  The silhouette represents a measure for the quality of a clustering.  Specifically, the silhouette provides a measure for how strongly a given point belongs to its own cluster compared to all other clusters.  The silhouette is computed as follows:
+
+\f[ s_{i} = \frac{b_{i} - a_{i}}{\max\{a_{i},b_{i}\}} \f]
+
+where \f$ a \f$ is the average distance between point \f$ i \f$ and all other points in the cluster point \f$ i \f$ belongs to, \f$ b \f$ is the _next closest_ average distance among all other clusters, and \f$ s \f$ is the silhouette value.  Using this definition, \f$ s \f$ exists on the interval \f$ [-1, 1] \f$, where 1 indicates that the point strongly belongs to its current cluster and -1 indicates that the point does not belong well to its current cluster.  The user may select from a variety of options to use as the distance metric.  Additionally, the user may opt to use a mask array to ignore points in the silhouette; these points will contain a silhouette value of 0.
+
+The silhouette can be used to determine how well a particular clustering has performed, such as [k means](@ref kmeans) or [k medoids](@ref kmedoids). 
+
+## Parameters ##
+
+| Name | Type | Description |
+|------|------|-------------|
+| Distance Metric | Enumeration | The metric used to determine the distances between points |
+| Use Mask | bool | Whether to use a boolean mask array to ignore certain points flagged as _false_ from the algorithm |
+
+## Required Geometry ###
+
+None
+
+## Required Objects ##
+
+| Kind | Default Name | Type | Component Dimensions | Description |
+|------|--------------|------|----------------------|-------------|
+| Any **Attribute Array** | None | Any| Any | The **Attribute Array** to silhouette |
+| **Attribute Array** | ClusterIds | int32_t | (1) | Specifies to which cluster each point belongs |
+| **Attribute Array** | Mask | bool | (1) | Specifies if the point is to be counted in the algorithm, if _Use Mask_ is checked |
+
+## Created Objects ##
+
+| Kind | Default Name | Type | Component Dimensions | Description |
+|------|--------------|------|----------------------|-------------|
+| **Attribute Array** | Silhouette | double | (1) | Silhouette value for each point  |
+
+## Example Pipelines ##
+
+
+
+## License & Copyright ##
+
+Please see the description file distributed with this plugin.
+
+## DREAM3D Mailing Lists ##
+
+If you need more help with a filter, please consider asking your question on the DREAM3D Users mailing list:
+https://groups.google.com/forum/?hl=en#!forum/dream3d-users
+

Core/src/Core/Filters/Algorithms/ImportDeformKeyFilev12.cpp

@@ -0,0 +1,48 @@
+#include "ImportDeformKeyFilev12.hpp"
+
+#include "complex/DataStructure/DataArray.hpp"
+#include "complex/DataStructure/DataGroup.hpp"
+
+using namespace complex;
+
+// -----------------------------------------------------------------------------
+ImportDeformKeyFilev12::ImportDeformKeyFilev12(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ImportDeformKeyFilev12InputValues* inputValues)
+: m_DataStructure(dataStructure)
+, m_InputValues(inputValues)
+, m_ShouldCancel(shouldCancel)
+, m_MessageHandler(mesgHandler)
+{
+}
+
+// -----------------------------------------------------------------------------
+ImportDeformKeyFilev12::~ImportDeformKeyFilev12() noexcept = default;
+
+// -----------------------------------------------------------------------------
+const std::atomic_bool& ImportDeformKeyFilev12::getCancel()
+{
+  return m_ShouldCancel;
+}
+
+// -----------------------------------------------------------------------------
+Result<> ImportDeformKeyFilev12::operator()()
+{
+  /**
+  * This section of the code should contain the actual algorithmic codes that
+  * will accomplish the goal of the file.
+  *
+  * If you can parallelize the code there are a number of examples on how to do that.
+  *    GenerateIPFColors is one example
+  *
+  * If you need to determine what kind of array you have (Int32Array, Float32Array, etc)
+  * look to the ExecuteDataFunction() in complex/Utilities/FilterUtilities.hpp template 
+  * function to help with that code.
+  *   An Example algorithm class is `CombineAttributeArrays` and `RemoveFlaggedVertices`
+  * 
+  * There are other utility classes that can help alleviate the amount of code that needs
+  * to be written.
+  *
+  * REMOVE THIS COMMENT BLOCK WHEN YOU ARE FINISHED WITH THE FILTER_HUMAN_NAME
+  */
+
+  return {};
+}