Skip to content

Commit 819074d

Browse files
Ishakaushik-cloudIshakaushik-cloud
committed
bq validation
1 parent b4850eb commit 819074d

10 files changed

Lines changed: 67 additions & 36 deletions

File tree

‎cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlRunTime.feature‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ Feature: CloudMySql Sink - Run time scenarios
5757
Then Open and capture logs
5858
Then Verify the pipeline status is "Succeeded"
5959
Then Close the pipeline logs
60-
Then Validate the values of records transferred to target CloudSQLMySQL table is equal to the values from source BigQuery table
60+
Then Validate the values of records transferred to target CloudSQLMySql table is equal to the values from source BigQuery table
6161

6262
@BQ_SOURCE_TEST @CLOUDMYSQL_TEST_TABLE @PLUGIN-1526
6363
Scenario: To verify data is getting transferred from BigQuery source to CloudMySql sink successfully when connection arguments are set
@@ -102,7 +102,8 @@ Feature: CloudMySql Sink - Run time scenarios
102102
Then Open and capture logs
103103
Then Verify the pipeline status is "Succeeded"
104104
Then Close the pipeline logs
105-
Then Validate the values of records transferred to target CloudSQLMySQL table is equal to the values from source BigQuery table
105+
Then Validate the values of records transferred to target CloudSQLMySql table is equal to the values from source BigQuery table
106+
106107

107108
@BQ_SOURCE_TEST @CLOUDMYSQL_TEST_TABLE @PLUGIN-1526
108109
Scenario: To verify data is getting transferred from BigQuery source to CloudSMySQL sink with Advanced property Connection timeout
@@ -147,6 +148,7 @@ Feature: CloudMySql Sink - Run time scenarios
147148
Then Open and capture logs
148149
Then Verify the pipeline status is "Succeeded"
149150
Then Close the pipeline logs
150-
Then Validate the values of records transferred to target CloudSQLMySQL table is equal to the values from source BigQuery table
151+
Then Validate the values of records transferred to target CloudSQLMySql table is equal to the values from source BigQuery table
152+
151153

152154

‎cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlRunTimeMacro.feature‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ Feature: CloudMySql Sink - Run time scenarios (macro)
6363
Then Open and capture logs
6464
Then Verify the pipeline status is "Succeeded"
6565
Then Close the pipeline logs
66-
Then Validate the values of records transferred to target CloudSQLMySQL table is equal to the values from source BigQuery table
66+
Then Validate the values of records transferred to target CloudSQLMySql table is equal to the values from source BigQuery table
6767

6868
@BQ_SOURCE_TEST @CLOUDMYSQL_TEST_TABLE @PLUGIN-1526
6969
Scenario: To verify data is getting transferred from BigQuery source to CloudMySql sink using macro arguments in basic section
@@ -108,7 +108,7 @@ Feature: CloudMySql Sink - Run time scenarios (macro)
108108
Then Open and capture logs
109109
Then Verify the pipeline status is "Succeeded"
110110
Then Close the pipeline logs
111-
Then Validate the values of records transferred to target CloudSQLMySQL table is equal to the values from source BigQuery table
111+
Then Validate the values of records transferred to target CloudSQLMySql table is equal to the values from source BigQuery table
112112

113113
@BQ_SOURCE_TEST @CLOUDMYSQL_TEST_TABLE
114114
Scenario: Verify pipeline failure message in logs when user provides invalid Table Name of plugin with Macros
@@ -232,4 +232,4 @@ Feature: CloudMySql Sink - Run time scenarios (macro)
232232
Then Open and capture logs
233233
Then Verify the pipeline status is "Succeeded"
234234
Then Close the pipeline logs
235-
Then Validate the values of records transferred to target CloudSQLMySQL table is equal to the values from source BigQuery table
235+
Then Validate the values of records transferred to target CloudSQLMySql table is equal to the values from source BigQuery table

‎cloudsql-mysql-plugin/src/e2e-test/features/source/CloudMySqlRunTime.feature‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -56,8 +56,7 @@ Feature: CloudMySql - Verify data transfer from CloudMySql source to BigQuery si
5656
Then Open and capture logs
5757
Then Verify the pipeline status is "Succeeded"
5858
Then Close the pipeline logs
59-
Then Validate the values of records transferred to target BigQuery table is equal to the values from source table
60-
59+
Then Validate the values of records transferred to target Big Query table is equal to the values from source table
6160
@CLOUDMYSQL_SOURCE_DATATYPES_TEST @BQ_SINK_TEST
6261
Scenario: To verify data is getting transferred from CloudMySql source to BigQuery sink successfully with all datatypes
6362
Given Open Datafusion Project to configure pipeline
@@ -99,7 +98,8 @@ Feature: CloudMySql - Verify data transfer from CloudMySql source to BigQuery si
9998
Then Open and capture logs
10099
Then Verify the pipeline status is "Succeeded"
101100
Then Close the pipeline logs
102-
Then Validate the values of records transferred to target BigQuery table is equal to the values from source table
101+
Then Validate the values of records transferred to target Big Query table is equal to the values from source table
102+
103103

104104
@CLOUDMYSQL_SOURCE_DATATYPES_TEST @CLOUDMYSQL_SINK_TEST @PLUGIN-20670
105105
Scenario: To verify data is getting transferred from CloudMySql source to CloudMySql sink successfully

‎cloudsql-mysql-plugin/src/e2e-test/features/source/CloudMySqlRunTimeMacro.feature‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -212,7 +212,7 @@ Feature: CloudMySql - Verify CloudMySql plugin data transfer with macro argument
212212
Then Open and capture logs
213213
Then Verify the pipeline status is "Succeeded"
214214
Then Close the pipeline logs
215-
Then Validate the values of records transferred to target BigQuery table is equal to the values from source table
215+
Then Validate the values of records transferred to target Big Query table is equal to the values from source table
216216

217217
@CLOUDMYSQL_SOURCE_TEST @BQ_SINK_TEST
218218
Scenario: To verify data is getting transferred from CloudMySql source to BigQuery sink successfully using macro arguments in basic section
@@ -256,7 +256,8 @@ Feature: CloudMySql - Verify CloudMySql plugin data transfer with macro argument
256256
Then Open and capture logs
257257
Then Verify the pipeline status is "Succeeded"
258258
Then Close the pipeline logs
259-
Then Validate the values of records transferred to target BigQuery table is equal to the values from source table
259+
Then Validate the values of records transferred to target Big Query table is equal to the values from source table
260+
260261

261262
@CLOUDMYSQL_SOURCE_TEST @BQ_SINK_TEST
262263
Scenario: To verify data is getting transferred from CloudMySql source to BigQuery sink successfully using macro arguments in advance section
@@ -304,4 +305,5 @@ Feature: CloudMySql - Verify CloudMySql plugin data transfer with macro argument
304305
Then Open and capture logs
305306
Then Verify the pipeline status is "Succeeded"
306307
Then Close the pipeline logs
307-
Then Validate the values of records transferred to target BigQuery table is equal to the values from source table
308+
Then Validate the values of records transferred to target Big Query table is equal to the values from source table
309+

‎cloudsql-mysql-plugin/src/e2e-test/java/io/cdap/plugin/CloudMySql/BQValidation.java‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ public class BQValidation {
3535
* @return true if the values in source and target side are equal
3636
*/
3737

38+
3839
public static boolean validateBQAndDBRecordValues(String sourceTable, String targetTable)
3940
throws SQLException, ClassNotFoundException, IOException, InterruptedException, ParseException {
4041
getBigQueryTableData(sourceTable, bigQueryRows);
@@ -198,12 +199,12 @@ public static boolean compareResultSetWithJsonData(ResultSet rsSource, List<Json
198199
break;
199200

200201
case Types.TIMESTAMP:
201-
// String sourceTS= String.valueOf(rsSource.getTimestamp(currentColumnCount));
202-
// String targetTS=bigQueryData.get(jsonObjectIdx).get(columnName).getAsString();
203-
// LocalDateTime timestamp = LocalDateTime.parse(targetTS, DateTimeFormatter.ISO_DATE_TIME);
204-
// DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss.S");
205-
// String formattedTimestamp = timestamp.format(formatter);
206-
// Assert.assertEquals(sourceTS, formattedTimestamp);
202+
String sourceTS= String.valueOf(rsSource.getTimestamp(currentColumnCount));
203+
String targetTS=bigQueryData.get(jsonObjectIdx).get(columnName).getAsString();
204+
LocalDateTime timestamp = LocalDateTime.parse(targetTS, DateTimeFormatter.ISO_DATE_TIME);
205+
DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss.S");
206+
String formattedTimestamp = timestamp.format(formatter);
207+
Assert.assertEquals(sourceTS, formattedTimestamp);
207208
break;
208209

209210
default:

‎cloudsql-mysql-plugin/src/e2e-test/java/io/cdap/plugin/CloudMySql/stepsdesign/CloudMysql.java‎

Lines changed: 37 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,23 @@
11
package io.cdap.plugin.CloudMySql.stepsdesign;
22

3+
import io.cdap.e2e.pages.actions.CdfPipelineRunAction;
4+
import io.cdap.e2e.utils.BigQueryClient;
35
import io.cdap.e2e.utils.CdfHelper;
46
import io.cdap.e2e.utils.PluginPropertyUtils;
57
import io.cdap.plugin.CloudMySqlClient;
68
import io.cucumber.java.en.Then;
79
import org.junit.Assert;
810
import stepsdesign.BeforeActions;
9-
import io.cdap.e2e.utils.PluginPropertyUtils;
11+
import io.cdap.plugin.CloudMySql.BQValidation;
1012

13+
import java.io.IOException;
1114
import java.sql.SQLException;
15+
import java.text.ParseException;
1216

1317
/**
1418
* CloudSqlMySql Plugin related step design.
1519
*/
1620
public class CloudMysql implements CdfHelper {
17-
1821
@Then("Validate the values of records transferred to target table is equal to the values from source table")
1922
public void validateTheValuesOfRecordsTransferredToTargetTableIsEqualToTheValuesFromSourceTable()
2023
throws SQLException, ClassNotFoundException {
@@ -27,5 +30,37 @@ public void validateTheValuesOfRecordsTransferredToTargetTableIsEqualToTheValues
2730
Assert.assertTrue("Value of records transferred to the target table should be equal to the value " +
2831
"of the records in the source table", recordsMatched);
2932
}
33+
34+
@Then("Validate the values of records transferred to target Big Query table is equal to the values from source table")
35+
public void validateTheValuesOfRecordsTransferredToTargetBigQueryTableIsEqualToTheValuesFromSourceTable()
36+
throws IOException, InterruptedException, IOException, SQLException, ClassNotFoundException, ParseException {
37+
int targetBQRecordsCount = BigQueryClient.countBqQuery(PluginPropertyUtils.pluginProp("bqTargetTable"));
38+
BeforeActions.scenario.write("No of Records Transferred to BigQuery:" + targetBQRecordsCount);
39+
Assert.assertEquals("Out records should match with target BigQuery table records count",
40+
CdfPipelineRunAction.getCountDisplayedOnSourcePluginAsRecordsOut(), targetBQRecordsCount);
41+
42+
boolean recordsMatched = BQValidation.validateDBAndBQRecordValues(
43+
PluginPropertyUtils.pluginProp("sourceTable"),
44+
PluginPropertyUtils.pluginProp("bqTargetTable"));
45+
Assert.assertTrue("Value of records transferred to the target table should be equal to the value " +
46+
"of the records in the source table", recordsMatched);
47+
}
48+
49+
@Then("Validate the values of records transferred to target CloudSQLMySql table is equal to the values from source " +
50+
"BigQuery table")
51+
public void validateTheValuesOfRecordsTransferredToTargetCloudSQLMySqlTableIsEqualToTheValuesFromSourceBigQueryTable()
52+
throws IOException, InterruptedException, IOException, SQLException, ClassNotFoundException, ParseException, ParseException {
53+
int sourceBQRecordsCount = BigQueryClient.countBqQuery(PluginPropertyUtils.pluginProp("bqSourceTable"));
54+
BeforeActions.scenario.write("No of Records from source BigQuery table:" + sourceBQRecordsCount);
55+
Assert.assertEquals("Out records should match with target PostgreSQL table records count",
56+
CdfPipelineRunAction.getCountDisplayedOnSourcePluginAsRecordsOut(), sourceBQRecordsCount);
57+
58+
boolean recordsMatched = BQValidation.validateBQAndDBRecordValues(
59+
PluginPropertyUtils.pluginProp("bqSourceTable"),
60+
PluginPropertyUtils.pluginProp("targetTable"));
61+
Assert.assertTrue("Value of records transferred to the target table should be equal to the value " +
62+
"of the records in the source table", recordsMatched);
63+
}
64+
3065
}
3166

‎cloudsql-mysql-plugin/src/e2e-test/java/io/cdap/plugin/CloudMySqlClient.java‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,9 @@ public static Connection getCloudSqlConnection() throws ClassNotFoundException,
2121
String password = System.getenv("CLOUDSQLMYSQL_PASSWORD");
2222

2323
String jdbcUrl = String.format(
24-
"jdbc:mysql:///%s?cloudSqlInstance=%s&socketFactory=com.google.cloud.sql.mysql.SocketFactory&user=%s&password=%s",
24+
PluginPropertyUtils.pluginProp("jdbcURL"),
2525
database, instanceConnectionName, username, password);
2626
Connection conn = DriverManager.getConnection(jdbcUrl);
27-
System.out.println("connected to database");
2827
return conn;
2928
}
3029

‎cloudsql-mysql-plugin/src/e2e-test/java/io/cdap/plugin/common/stepsdesign/TestSetupHooks.java‎

Lines changed: 1 addition & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -24,16 +24,6 @@
2424
*/
2525

2626
public class TestSetupHooks {
27-
public static void main(String[] args) throws SQLException, ClassNotFoundException, IOException, InterruptedException {
28-
// setTableName();
29-
// createCloudMysqlTestTable();
30-
// createTables();
31-
// createDatatypesTable();
32-
createTempSourceBQTable();
33-
//createCloudMysqlTestTable();
34-
// createTempSourceBQTable();
35-
// createCloudMysqlTestTable();
36-
}
3727
public static void setTableName() {
3828
String randomString = RandomStringUtils.randomAlphabetic(10);
3929
String sourceTableName = String.format("SourceTable_%s", randomString);
@@ -60,7 +50,7 @@ public static void initializeDBProperties() {
6050

6151
@Before(order = 2, value = "@CLOUDMYSQL_SOURCE_TEST")
6252
public static void createTables() throws SQLException, ClassNotFoundException {
63-
CloudMySqlClient.createSourceTable(PluginPropertyUtils.pluginProp("sourceTable"));
53+
CloudMySqlClient.createSourceTable(PluginPropertyUtils.pluginProp("CLOUDMYSQL_SOURCE_TEST"));
6454
CloudMySqlClient.createTargetTable(PluginPropertyUtils.pluginProp("targetTable"));
6555
}
6656

‎cloudsql-mysql-plugin/src/e2e-test/resources/BigQuery/BigQueryInsertDataQuery.txt‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
insert into `DATASET.TABLE_NAME` (COL1, COL2, COL3, COL4, COL6, COL8, COL9, COL10) values
2-
(b'01011011','priya','2021-01-28',1.110,'2019-03-10 04:50:01 UTC',false,92233720,'21:26:00'),
3-
(b'01011011','surya','2021-01-21',1.110000001,'2018-03-10 04:50:01 UTC',true,92233729,'20:26:00');
2+
(b'01011011','priya','2021-01-27',1.110,'2019-03-10 04:50:01 UTC',false,92233720,'21:26:00'),
3+
(b'01011011','surya','2021-01-27',1.110000001,'2018-03-10 04:50:01 UTC',true,92233729,'20:26:00');

‎cloudsql-mysql-plugin/src/e2e-test/resources/pluginParameters.properties‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,10 +15,11 @@ CloudMySqlImportQuery=select * from mytable
1515
fetchSize=1000
1616
NumSplits=1
1717
SplitBy=ID
18+
jdbcURL=jdbc:mysql:///%s?cloudSqlInstance=%s&socketFactory=com.google.cloud.sql.mysql.SocketFactory&user=%s&password=%s
1819
projectId=cdf-athena
1920
datasetprojectId=cdf-athena
2021
BQReferenceName=reference
21-
targetTable=mytable4
22+
targetTable=mytable5
2223
bqDatasetId=1234
2324
dataset=test_automation
2425
bqSourceTable=mysql
@@ -29,6 +30,7 @@ pass=PASS
2930
invalidUserName=testUser
3031
invalidPassword=testPassword
3132
invalidTable=data
33+
jdbcURL=jdbc:mysql:///%s?cloudSqlInstance=%s&socketFactory=com.google.cloud.sql.mysql.SocketFactory&user=%s&password=%s
3234
CloudMySqlDriverName=cloudsql-mysql
3335
bqTruncateTable=truncateTable
3436
bqUpdateTableSchema=updateSchema

0 commit comments

Comments
 (0)