Skip to content

(🚧🚜 WIP 👷🚧 ) Split fabric-specific code into individual back-ends #1617

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cchepelov wants to merge 44 commits into
base: cascading3
Choose a base branch
from
Open

(🚧🚜 WIP 👷🚧 ) Split fabric-specific code into individual back-ends #1617

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
e9e09da
Split-fabrics: initial WIP commit
Oct 13, 2016
a11988d
Beginning of the great test-fixing exercise.
Oct 13, 2016
ecdfbfc
debug PartitionedDelimitedSourceTest
Oct 14, 2016
b5cdb0d
fixed breakage in Executions
Oct 14, 2016
5fa042a
fix Mode parse test: forgot --hdfs ;)
cchepelov Oct 14, 2016
d09112b
fix autoCluster mode
cchepelov Oct 14, 2016
644b9a3
not much, moving stuff around
cchepelov Oct 17, 2016
f3212cb
Rewrite away dependency to guava, using JDK 7 instead
Oct 17, 2016
826ad0b
Manage more precisely the version of guava included (guava [16.0,) is…
Oct 18, 2016
474dbed
carry over the configuration from the local cluster to upcoming test …
Oct 18, 2016
41133bc
More guava
Oct 18, 2016
54df9eb
Fix JsonLineTest
Oct 18, 2016
0aee566
PSA
Oct 18, 2016
23f16a8
fix repl: need to have cascading-hadoop around
Oct 19, 2016
fd6ee73
fix WeightedPageRank test
Oct 19, 2016
3342ce9
scalariform
Oct 19, 2016
0958edc
make the temporary file path (in HadoopTest mode) more explicit
Oct 19, 2016
33807aa
better debugging temp file name clean-up
Oct 19, 2016
c7efbf1
concentrate test-only code
Oct 19, 2016
4e4d107
(scalariform)
cchepelov Oct 20, 2016
1be785e
reworking a bit Hadoop dependencies (compile-time, test-time, run-time)
cchepelov Oct 20, 2016
30a9211
Move LegacyHadoopMode & friends into scalding-fabric-hadoop
Oct 20, 2016
26bbcb3
Implement modes for Hadoop2-mr1, Tez and Flink (not all work for now)
Oct 20, 2016
04368cf
PartitionedDelimitedSourceTest: pass test (adapt to changing file pat…
cchepelov Oct 29, 2016
492e2d2
fixing a List-to-String comparison (which should've failed much earlier)
cchepelov Oct 29, 2016
4559462
in generated setters, add the suppression against the PartialOption wart
cchepelov Nov 1, 2016
abd0d81
ensure test-mode doesn't accidentally return a LocalTest when autoClu…
cchepelov Nov 1, 2016
0874b91
WIP: splitting cluster-aware tests away from scalding-core
cchepelov Nov 1, 2016
ba4ba26
Fix unpossible comparison ("Step Number" ceased to be an Int)
cchepelov Nov 1, 2016
f16946a
bring in a fabric to test parquet
cchepelov Nov 2, 2016
0e45a2d
fix Reduce Operations tests; Tsv cannot be used for non-primitive types
Nov 2, 2016
4eec1a6
make Tez use the Hadoop reducers specification if needed
Nov 2, 2016
f0d28a1
OK, let's build Flink too. And run tests to ensure scriptValidator's …
cchepelov Nov 2, 2016
ce8e67b
dip a toe: try enabling Tez on the untyped REPL
cchepelov Nov 2, 2016
66a3b60
(scalariform)
cchepelov Nov 2, 2016
13c17a2
ruby typo
cchepelov Nov 3, 2016
4b55e31
more character escapes
cchepelov Nov 3, 2016
0f0542d
fixing PartitionedDelimitedSourceTest on Tez too
cchepelov Nov 3, 2016
9f05dbe
testing the tez fabric on scala 2.10.6 too (or the script validation …
cchepelov Nov 3, 2016
2e00633
attempt to build & test repl against all fabrics
Nov 3, 2016
1965385
Counters: port to non-Hadoop fabrics
Nov 3, 2016
40c792a
Fabric-switching logic & stats: fix & clean various unfinished bits
Nov 4, 2016
ddb9efa
early, very early baby steps of using Flink
Nov 4, 2016
5c5127e
minor Tez details + notes about non-working tezts
Nov 4, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 81 additions & 0 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,14 @@ matrix:
env: BUILD="base" TEST_TARGET="scalding-core scalding-jdbc scalding-json scalding-db"
script: "scripts/run_test.sh"

- scala: 2.10.6
env: BUILD="base" TEST_TARGET="scalding-core-fabric-tests scalding-fabric-hadoop scalding-fabric-hadoop2-mr1 scalding-fabric-tez scalding-fabric-flink"
script: "scripts/run_test.sh"

- scala: 2.11.7
env: BUILD="base" TEST_TARGET="scalding-core-fabric-tests scalding-fabric-hadoop scalding-fabric-hadoop2-mr1 scalding-fabric-tez scalding-fabric-flink"
script: "scripts/run_test.sh"

- scala: 2.10.6
env: BUILD="base" TEST_TARGET="scalding-hadoop-test"
script: "scripts/run_test.sh"
Expand Down Expand Up @@ -80,6 +88,78 @@ matrix:
- "scripts/build_assembly_no_test.sh scalding-assembly"
- "scripts/test_matrix_tutorials.sh"

- scala: 2.10.6
env: BUILD="test tutorials and matrix tutorials and repl" TEST_TARGET="scalding-repl" SCALD_FABRIC_SUFFIX="hadoop"
script:
- "scripts/run_test.sh"
- "scripts/build_assembly_no_test.sh scalding-assembly-${SCALD_FABRIC_SUFFIX}"
- "scripts/test_tutorials.sh"
- "scripts/build_assembly_no_test.sh scalding-assembly-${SCALD_FABRIC_SUFFIX}"
- "scripts/test_matrix_tutorials.sh"

- scala: 2.11.7
env: BUILD="test tutorials and matrix tutorials and repl" TEST_TARGET="scalding-repl" SCALD_FABRIC_SUFFIX="hadoop"
script:
- "scripts/run_test.sh"
- "scripts/build_assembly_no_test.sh scalding-assembly-${SCALD_FABRIC_SUFFIX}"
- "scripts/test_tutorials.sh"
- "scripts/build_assembly_no_test.sh scalding-assembly-${SCALD_FABRIC_SUFFIX}"
- "scripts/test_matrix_tutorials.sh"

- scala: 2.10.6
env: BUILD="test tutorials and matrix tutorials and repl" TEST_TARGET="scalding-repl" SCALD_FABRIC_SUFFIX="hadoop2-mr1"
script:
- "scripts/run_test.sh"
- "scripts/build_assembly_no_test.sh scalding-assembly-${SCALD_FABRIC_SUFFIX}"
- "scripts/test_tutorials.sh"
- "scripts/build_assembly_no_test.sh scalding-assembly-${SCALD_FABRIC_SUFFIX}"
- "scripts/test_matrix_tutorials.sh"

- scala: 2.11.7
env: BUILD="test tutorials and matrix tutorials and repl" TEST_TARGET="scalding-repl" SCALD_FABRIC_SUFFIX="hadoop2-mr1"
script:
- "scripts/run_test.sh"
- "scripts/build_assembly_no_test.sh scalding-assembly-${SCALD_FABRIC_SUFFIX}"
- "scripts/test_tutorials.sh"
- "scripts/build_assembly_no_test.sh scalding-assembly-${SCALD_FABRIC_SUFFIX}"
- "scripts/test_matrix_tutorials.sh"

- scala: 2.10.6
env: BUILD="test tutorials and matrix tutorials and repl" TEST_TARGET="scalding-repl" SCALD_FABRIC_SUFFIX="tez"
script:
- "scripts/run_test.sh"
- "scripts/build_assembly_no_test.sh scalding-assembly-${SCALD_FABRIC_SUFFIX}"
- "scripts/test_tutorials.sh"
- "scripts/build_assembly_no_test.sh scalding-assembly-${SCALD_FABRIC_SUFFIX}"
- "scripts/test_matrix_tutorials.sh"

- scala: 2.11.7
env: BUILD="test tutorials and matrix tutorials and repl" TEST_TARGET="scalding-repl" SCALD_FABRIC_SUFFIX="tez"
script:
- "scripts/run_test.sh"
- "scripts/build_assembly_no_test.sh scalding-assembly-${SCALD_FABRIC_SUFFIX}"
- "scripts/test_tutorials.sh"
- "scripts/build_assembly_no_test.sh scalding-assembly-${SCALD_FABRIC_SUFFIX}"
- "scripts/test_matrix_tutorials.sh"

- scala: 2.10.6
env: BUILD="test tutorials and matrix tutorials and repl" TEST_TARGET="scalding-repl" SCALD_FABRIC_SUFFIX="flink"
script:
- "scripts/run_test.sh"
- "scripts/build_assembly_no_test.sh scalding-assembly-${SCALD_FABRIC_SUFFIX}"
- "scripts/test_tutorials.sh"
- "scripts/build_assembly_no_test.sh scalding-assembly-${SCALD_FABRIC_SUFFIX}"
- "scripts/test_matrix_tutorials.sh"

- scala: 2.11.7
env: BUILD="test tutorials and matrix tutorials and repl" TEST_TARGET="scalding-repl" SCALD_FABRIC_SUFFIX="flink"
script:
- "scripts/run_test.sh"
- "scripts/build_assembly_no_test.sh scalding-assembly-${SCALD_FABRIC_SUFFIX}"
- "scripts/test_tutorials.sh"
- "scripts/build_assembly_no_test.sh scalding-assembly-${SCALD_FABRIC_SUFFIX}"
- "scripts/test_matrix_tutorials.sh"

- scala: 2.10.6
env: BUILD="test repl and typed tutorials"
script:
Expand All @@ -99,3 +179,4 @@ matrix:
- "scripts/test_typed_tutorials.sh"
- "scripts/build_assembly_no_test.sh execution-tutorial"
- "scripts/test_execution_tutorial.sh"

Loading