Skip to content

Updatable inference lens #290

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
SleepyTeaDrinker wants to merge 20 commits into
base: master
Choose a base branch
from
Open

Updatable inference lens #290

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
ea9850c
Implemented updateable models and tests
Nov 27, 2017
08763a9
Fixing merge conflicts
Nov 28, 2017
a07ac9d
adding plugins
Nov 28, 2017
af0c793
fixing merge conflicts
Nov 28, 2017
bce1560
Type Inference Model merging for comparison purposes
Dec 19, 2017
31707e8
Merge branch 'master' of https://github.com/UBOdin/mimir into updatab…
Dec 19, 2017
08623ac
Merge branch 'master' of https://github.com/UBOdin/mimir
Dec 19, 2017
c78ed1a
Merge branch 'master' of https://github.com/UBOdin/mimir
Jan 11, 2018
fb13193
Merge branch 'master' into updatable_inference_lens
Jan 11, 2018
31d5f32
Merge branch 'updatable_inference_lens' of https://github.com/UBOdin/…
Jan 11, 2018
3061834
Fixing minor bugs
Jan 22, 2018
ce8b84e
Adding csv for accuracy error bounds
Jan 22, 2018
be97fa2
Developing error bound testing information
Jan 25, 2018
638056d
adding comparison tests between old/new methods
Jan 26, 2018
c10daa9
Keeping things updated, expanding test suite
Jan 31, 2018
55de607
Removing unnecessary lens
Jan 31, 2018
dad8974
Updating minor things to bugfix
Feb 7, 2018
c879aa4
Cleaning tests
Feb 22, 2018
e972a16
Merge branch 'master' of https://github.com/UBOdin/mimir
Feb 22, 2018
ca626d3
Merge branch 'master' into updatable_inference_lens
Feb 22, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 112 additions & 0 deletions .classpath
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
<?xml version="1.0" encoding="UTF-8"?>
<classpath>
<classpathentry kind="src" path="src/main/scala"/>
<classpathentry kind="src" path="src/main/java"/>
<classpathentry kind="src" path="src/main/resources"/>
<classpathentry kind="src" path="lib_extra"/>
<classpathentry kind="src" path="src/test/scala"/>
<classpathentry kind="src" path="src/test/resources"/>
<classpathentry kind="src" path="conf"/>
<classpathentry kind="lib" path="lib/gprom-jdbc.jar"/>
<classpathentry kind="lib" path="lib/javacc.jar"/>
<classpathentry kind="lib" path="lib/ojdbc7.jar"/>
<classpathentry kind="lib" path="lib/py4j0.10.4.jar"/>
<classpathentry kind="con" path="org.scala-ide.sdt.launching.SCALA_CONTAINER"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.rogach/scallop_2.11/jars/scallop_2.11-0.9.5.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.jline/jline/jars/jline-3.2.0.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.typesafe.scala-logging/scala-logging-slf4j_2.11/jars/scala-logging-slf4j_2.11-2.1.2.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.typesafe.scala-logging/scala-logging-api_2.11/jars/scala-logging-api_2.11-2.1.2.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.1.7.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/ch.qos.logback/logback-core/jars/logback-core-1.1.7.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.slf4j/slf4j-api/jars/slf4j-api-1.7.20.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.github.nscala-time/nscala-time_2.11/jars/nscala-time_2.11-1.2.0.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.apache.lucene/lucene-spellchecker/jars/lucene-spellchecker-3.6.2.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.apache.lucene/lucene-core/jars/lucene-core-3.6.2.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.apache.lucene/lucene-analyzers/jars/lucene-analyzers-3.6.2.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.apache.servicemix.bundles/org.apache.servicemix.bundles.collections-generic/bundles/org.apache.servicemix.bundles.collections-generic-4.01_1.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.scala-lang.modules/scala-parser-combinators_2.11/bundles/scala-parser-combinators_2.11-1.0.6.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.apache.commons/commons-csv/jars/commons-csv-1.4.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/commons-io/commons-io/jars/commons-io-2.5.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.github.wnameless/json-flattener/jars/json-flattener-0.2.2.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.eclipsesource.minimal-json/minimal-json/bundles/minimal-json-0.9.4.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.apache.commons/commons-lang3/jars/commons-lang3-3.4.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.typesafe.play/play-json_2.11/jars/play-json_2.11-2.4.11.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.typesafe.play/play-iteratees_2.11/jars/play-iteratees_2.11-2.4.11.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.scala-stm/scala-stm_2.11/jars/scala-stm_2.11-0.7.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.typesafe/config/bundles/config-1.3.0.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.typesafe.play/play-functional_2.11/jars/play-functional_2.11-2.4.11.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.typesafe.play/play-datacommons_2.11/jars/play-datacommons_2.11-2.4.11.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/joda-time/joda-time/jars/joda-time-2.8.1.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.joda/joda-convert/jars/joda-convert-1.7.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.fasterxml.jackson.core/jackson-core/bundles/jackson-core-2.5.4.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.fasterxml.jackson.core/jackson-annotations/bundles/jackson-annotations-2.5.4.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.fasterxml.jackson.core/jackson-databind/bundles/jackson-databind-2.5.4.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.fasterxml.jackson.datatype/jackson-datatype-jdk8/bundles/jackson-datatype-jdk8-2.5.4.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/bundles/jackson-datatype-jsr310-2.5.4.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/nz.ac.waikato.cms.weka/weka-stable/jars/weka-stable-3.8.1.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/nz.ac.waikato.cms.weka.thirdparty/java-cup-11b/jars/java-cup-11b-2015.03.26.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/nz.ac.waikato.cms.weka.thirdparty/bounce/jars/bounce-0.18.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.googlecode.matrix-toolkits-java/mtj/jars/mtj-1.0.4.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/net.sourceforge.f2j/arpack_combined_all/jars/arpack_combined_all-0.1-javadoc.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/net.sourceforge.f2j/arpack_combined_all/jars/arpack_combined_all-0.1.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.github.fommil.netlib/core/jars/core-1.1.2.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.github.fommil.netlib/netlib-native_ref-osx-x86_64/jars/netlib-native_ref-osx-x86_64-1.1-natives.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.github.fommil.netlib/native_ref-java/jars/native_ref-java-1.1.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.github.fommil/jniloader/jars/jniloader-1.1.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.github.fommil.netlib/netlib-native_ref-linux-x86_64/jars/netlib-native_ref-linux-x86_64-1.1-natives.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.github.fommil.netlib/netlib-native_ref-linux-i686/jars/netlib-native_ref-linux-i686-1.1-natives.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.github.fommil.netlib/netlib-native_ref-win-x86_64/jars/netlib-native_ref-win-x86_64-1.1-natives.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.github.fommil.netlib/netlib-native_ref-win-i686/jars/netlib-native_ref-win-i686-1.1-natives.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.github.fommil.netlib/netlib-native_ref-linux-armhf/jars/netlib-native_ref-linux-armhf-1.1-natives.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.github.fommil.netlib/netlib-native_system-osx-x86_64/jars/netlib-native_system-osx-x86_64-1.1-natives.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.github.fommil.netlib/native_system-java/jars/native_system-java-1.1.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.github.fommil.netlib/netlib-native_system-linux-x86_64/jars/netlib-native_system-linux-x86_64-1.1-natives.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.github.fommil.netlib/netlib-native_system-linux-i686/jars/netlib-native_system-linux-i686-1.1-natives.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.github.fommil.netlib/netlib-native_system-linux-armhf/jars/netlib-native_system-linux-armhf-1.1-natives.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.github.fommil.netlib/netlib-native_system-win-x86_64/jars/netlib-native_system-win-x86_64-1.1-natives.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.github.fommil.netlib/netlib-native_system-win-i686/jars/netlib-native_system-win-i686-1.1-natives.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.googlecode.netlib-java/netlib-java/jars/netlib-java-1.1.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/nz.ac.waikato.cms.moa/moa/jars/moa-2014.11.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.googlecode.sizeofag/sizeofag/jars/sizeofag-1.0.0.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/net.sf.jung/jung-graph-impl/jars/jung-graph-impl-2.0.1.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/net.sf.jung/jung-api/jars/jung-api-2.0.1.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/net.sourceforge.collections/collections-generic/jars/collections-generic-4.01.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/net.sf.jung/jung-algorithms/jars/jung-algorithms-2.0.1.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/colt/colt/jars/colt-1.2.0.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/concurrent/concurrent/jars/concurrent-1.3.4.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/net.sf.jung/jung-visualization/jars/jung-visualization-2.0.1.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/jgraph/jgraph/jars/jgraph-5.13.0.0.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/javax.media/jai_core/jars/jai_core-1.1.3.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.geotools/gt-referencing/jars/gt-referencing-16.2.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/com.googlecode.efficient-java-matrix-library/core/jars/core-0.26.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/commons-pool/commons-pool/jars/commons-pool-1.5.4.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.geotools/gt-metadata/jars/gt-metadata-16.2.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.geotools/gt-opengis/jars/gt-opengis-16.2.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/net.java.dev.jsr-275/jsr-275/jars/jsr-275-1.0-beta-2.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/jgridshift/jgridshift/jars/jgridshift-1.0.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/net.sf.geographiclib/GeographicLib-Java/jars/GeographicLib-Java-1.44.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.geotools/gt-epsg-hsql/jars/gt-epsg-hsql-16.2.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.hsqldb/hsqldb/jars/hsqldb-2.3.0.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.xerial/sqlite-jdbc/jars/sqlite-jdbc-3.16.1.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/net.java.dev.jna/jna/jars/jna-4.2.2.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/net.java.dev.jna/jna-platform/jars/jna-platform-4.2.2.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/log4j/log4j/bundles/log4j-1.2.17.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.specs2/specs2-core_2.11/jars/specs2-core_2.11-3.8.4.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.specs2/specs2-matcher_2.11/jars/specs2-matcher_2.11-3.8.4.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.specs2/specs2-common_2.11/jars/specs2-common_2.11-3.8.4.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.specs2/specs2-codata_2.11/jars/specs2-codata_2.11-3.8.4.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.scalaz/scalaz-core_2.11/bundles/scalaz-core_2.11-7.2.3.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.scalaz/scalaz-effect_2.11/bundles/scalaz-effect_2.11-7.2.3.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.scalaz/scalaz-concurrent_2.11/bundles/scalaz-concurrent_2.11-7.2.3.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.scala-lang.modules/scala-xml_2.11/bundles/scala-xml_2.11-1.0.5.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.specs2/specs2-matcher-extra_2.11/jars/specs2-matcher-extra_2.11-3.8.4.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.specs2/specs2-analysis_2.11/jars/specs2-analysis_2.11-3.8.4.jar"/>
<classpathentry kind="con" path="org.scala-ide.sdt.launching.SCALA_COMPILER_CONTAINER"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.specs2/classycle/jars/classycle-1.4.3.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.specs2/specs2-junit_2.11/jars/specs2-junit_2.11-3.8.4.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/junit/junit/jars/junit-4.12.jar"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/org.hamcrest/hamcrest-core/jars/hamcrest-core-1.3.jar"/>
<classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER"/>
<classpathentry kind="lib" path="/Users/Minos/.ivy2/cache/info.mimirdb/jsqlparser/jars/jsqlparser-1.0.3.jar"/>
<classpathentry kind="output" path="bin"/>
</classpath>
13 changes: 13 additions & 0 deletions .project
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
<projectDescription>
<name>Mimir-Core</name>
<buildSpec>
<buildCommand>
<name>org.scala-ide.sdt.core.scalabuilder</name>
</buildCommand>
</buildSpec>
<natures>
<nature>org.scala-ide.sdt.core.scalanature</nature>
<nature>org.eclipse.jdt.core.javanature</nature>
</natures>
<linkedResources> </linkedResources>
</projectDescription>
3 changes: 3 additions & 0 deletions .settings/org.eclipse.core.resources.prefs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#Generated by sbteclipse
#Tue Nov 28 13:23:48 EST 2017
encoding/<project>=UTF-8
47 changes: 47 additions & 0 deletions .settings/org.scala-ide.sdt.core.prefs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
//src/main/java=main
//src/main/resources=main
//src/main/scala=main
//src/test/resources=tests
//src/test/scala=tests
P=
Xcheckinit=false
Xdisable-assertions=false
Xelide-below=-2147483648
Xexperimental=false
Xfatal-warnings=false
Xfuture=false
Xlog-implicits=false
Xno-uescape=false
Xplugin=
Xplugin-disable=
Xplugin-require=
Xpluginsdir=misc/scala-devel/plugins
Ypresentation-debug=false
Ypresentation-delay=0
Ypresentation-log=
Ypresentation-replay=
Ypresentation-verbose=false
apiDiff=false
compileorder=Mixed
conf=macros
deprecation=false
eclipse.preferences.version=1
explaintypes=false
feature=false
g=vars
lib_extra=main
no-specialization=false
nowarn=false
optimise=false
recompileOnMacroDef=true
relationsDebug=false
scala.compiler.additionalParams=-feature -Xsource\:2.11 -Ymacro-expand\:none
scala.compiler.installation=2.11
scala.compiler.sourceLevel=2.11
scala.compiler.useProjectSettings=true
stopBuildOnError=true
target=jvm-1.8
unchecked=false
useScopesCompiler=true
verbose=false
withVersionClasspathValidator=true
8 changes: 0 additions & 8 deletions bin/mimir-profiler
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,4 +254,4 @@ pomExtra := <url>http://mimirdb.info</url>
/////// Publishing Options ////////
// use `sbt publish` to update the package in
// your own local ivy cache
publishTo := Some(Resolver.file("file", new File(Path.userHome.absolutePath+"/.m2/repository")))
publishTo := Some(Resolver.file("file", new File(Path.userHome.absolutePath+"/.m2/repository")))
1 change: 1 addition & 0 deletions project/plugins.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "5.2.2")
4 changes: 3 additions & 1 deletion src/main/scala/mimir/adaptive/TypeInference.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,9 @@ object TypeInference
new TypeInferenceModel(
s"MIMIR_TI_ATTR_${viewName}",
modelColumns,
stringDefaultScore
stringDefaultScore,
1000,
config.query
)

val columnIndexes =
Expand Down
19 changes: 16 additions & 3 deletions src/main/scala/mimir/lenses/LensManager.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
package mimir.lenses;

import java.sql._
import java.sql._;
import scala.collection.concurrent.TrieMap;
import scala.concurrent.Future;
import scala.concurrent.ExecutionContext.Implicits.global;

import mimir.Database
import mimir.algebra._
Expand All @@ -11,8 +14,7 @@ import mimir.util.JDBCUtils
import mimir.util.ExperimentalOptions

class LensManager(db: Database) {

val lensTypes = Map[String,((Database,String,Operator,Seq[Expression]) =>
val lensTypes = TrieMap[String,((Database,String,Operator,Seq[Expression]) =>
(Operator,TraversableOnce[Model]))](
"MISSING_VALUE" -> MissingValueLens.create _,
"DOMAIN" -> MissingValueLens.create _,
Expand Down Expand Up @@ -53,12 +55,23 @@ class LensManager(db: Database) {
for(model <- models){
db.models.persist(model, s"LENS:$saneName")
}

val updateModels = Future
{
this.updateModels()
}

}

def drop(name: String): Unit =
{
db.views.drop(name)
db.models.dropOwner(s"LENS:$name")
}

def updateModels() : Unit =
{

}

}
17 changes: 15 additions & 2 deletions src/main/scala/mimir/models/ModelBuildingBlocks.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
package mimir.models

import mimir.Database
import mimir.algebra._

trait NoArgModel
Expand All @@ -8,9 +8,22 @@ trait NoArgModel
def hintTypes(idx: Int) = Seq()
}

/**
* This trait is a flag for a concurrent operation that allows progressive
* updates to a given model as handled by the model manager. In practice,
* models are updated in the background
*/
trait ProgressiveUpdate
{
def progressiveTrain(db: Database, query:Operator): Unit
def getQuery(): Operator
def isCompleted(): Boolean
def getNextSample(): Int
}

trait ModelCache
{
val cache = scala.collection.mutable.Map[String,PrimitiveValue]()
val cache = scala.collection.mutable.Map[String,PrimitiveValue]()
def getCacheKey(idx: Int, args: Seq[PrimitiveValue], hints: Seq[PrimitiveValue] ) : String
def getCache(idx: Int, args: Seq[PrimitiveValue], hints: Seq[PrimitiveValue]) : Option[PrimitiveValue] = {
cache.get(getCacheKey(idx,args,hints))
Expand Down
12 changes: 12 additions & 0 deletions src/main/scala/mimir/models/ModelManager.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,13 +55,16 @@ class ModelManager(db:Database)
)
""")
}
//Begin the thread to progressively update
progressivelyUpdate()
}

/**
* Declare (and cache) a new Name -> Model association
*/
def persist(model:Model): Unit =
{
this.synchronized{
val (serialized,decoder) = model.serialize

db.backend.update(s"""
Expand All @@ -73,6 +76,7 @@ class ModelManager(db:Database)
StringPrimitive(decoder.toUpperCase)
))
cache.put(model.name, model)
}
}

/**
Expand Down Expand Up @@ -267,6 +271,14 @@ class ModelManager(db:Database)
}
return ret
}

/**
* Progressively update models that have the appropriate trait
*/
private def progressivelyUpdate() = {
val thread = new Thread(new mimir.util.ModelConcurrency(this,db))
thread.start()
}
}

trait NeedsReconnectToDatabase {
Expand Down
Loading