Model monitor baseline processing job crashes on CSV file

[G-ecs]

Describe the bug
Model monitor baseline processing job crashes on CSV files.

To reproduce
From a SageMaker notebook following the SageMaker tutorial training XGBoost on mnist, and I adding the model monitor tutorial to the notebook.

Download mnist:

import pickle, gzip, urllib.request, json
import numpy as np

# Load the dataset
urllib.request.urlretrieve("http://deeplearning.net/data/mnist/mnist.pkl.gz", "mnist.pkl.gz")
with gzip.open('mnist.pkl.gz', 'rb') as f:
    train_set, valid_set, test_set = pickle.load(f, encoding='latin1')

create dataset:

import numpy as np
import boto3
from sagemaker import get_execution_role

role = get_execution_role()

region = boto3.Session().region_name

bucket='myBucket' # Replace with your s3 bucket name
prefix = 'sagemaker/xgboost-mnist' # Used as part of the path in the bucket where you store data

def convert_data():
    data_partitions = [('train', train_set), ('validation', valid_set), ('test', test_set)]
    for data_partition_name, data_partition in data_partitions:
        print('{}: {} {}'.format(data_partition_name, data_partition[0].shape, data_partition[1].shape))
        labels = [t.tolist() for t in data_partition[1]]
        features = [t.tolist() for t in data_partition[0]]
        
        if data_partition_name != 'test':
            examples = np.insert(features, 0, labels, axis=1)
        else:
            examples = features
        #print(examples[50000,:])
        
        
        np.savetxt('data.csv', examples, delimiter=',', format='%f')
        
        
        
        key = "{}/{}/examples.csv".format(prefix,data_partition_name)
        url = 's3://{}/{}'.format(bucket, key)
        boto3.Session().resource('s3').Bucket(bucket).Object(key).upload_file('data.csv')
        print('Done writing to {}'.format(url))
        
convert_data()

create baseline:

from sagemaker.model_monitor import DefaultModelMonitor
from sagemaker.model_monitor.dataset_format import DatasetFormat

my_default_monitor = DefaultModelMonitor(
    role=role,
    instance_count=1,
    instance_type='ml.m5.xlarge',
    volume_size_in_gb=20,
    max_runtime_in_seconds=3600,
)

my_default_monitor.suggest_baseline(
    baseline_dataset=baseline_data_uri+'examples.csv',            ## filename must have csv extension
    dataset_format=DatasetFormat.csv(header=False),            ## mnist has no header
    output_s3_uri=baseline_results_uri,
    wait=True
)

Expected behavior
Successful Processing job with float inferred type in Model Monitor.

Screenshots or logs

1. If format='%f' is not set processing job successful but inferred type is string because of the scientific notation. See results in screenshot:

2. If `format='%f'` is set the processing job fails:

Note that dumping the dataset to CSV using pandas.DataFrame.to_csv() produces the same error.

System information
A description of your system. Please provide:

• SageMaker Python SDK version: 1.55.3
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): Model Monitor processing job
• Framework version: 468650794304.dkr.ecr.eu-west-1.amazonaws.com/sagemaker-model-monitor-analyzer
• Python version:3.6.5
• CPU or GPU:CPU
• Custom Docker image (Y/N):N
• Running from SageMaker instance on a ml.m4.10xlarge

Additional context
I ran through the SageMaker tutorial training XGBoost on mnist, and I added the model monitor tutorial to the notebook.

create dataset:

1. had to add .csv extension to filename
2. had to add format='%f' to np.savetxt to avoid writing in scientific notation otherwise inferred type is string in model monitor.

[metrizable]

@G-ecs Thanks for filing the bug. Just for clarity, I am assuming you added fmt='%f' to your call to np.savetxt?

[metrizable]

I've been able to produce an exception that looks like the one you've posted using fmt='%f':

...
2020-07-02 17:12:41,240 - __main__ - ERROR - Exception performing analysis: Command 'bin/spark-submit --master yarn --deploy-mode client --conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider --conf spark.serializer=org.apache.spark.serializer.KryoSerializer /opt/amazon/sagemaker-data-analyzer-1.0-jar-with-dependencies.jar --analytics_input /tmp/spark_job_config.json' returned non-zero exit status 1.
...
...
UnexpectedStatusException: Error for Processing job baseline-suggestion-job-2020-07-02-17-02-52-208: Failed. Reason: AlgorithmError: See job logs for more information

Investigating.

[G-ecs]

@metrizable Yes, I added fmt='%f' to np.savetxt.
I first tried without and got 0.000000000000000000e+00. Due to this scientific notation, the inferred type in the processing job was string (as shown in the 1st screenshot).

With fmt="%f" I get 0.000000 - which should be easier type inference. But it caused the processing job to crash. I also tried writing the CSV file with pandas.to_csv with which I get 0.0. Same error

Note that the training jobs work just fine. I will try with the iris dataset and a few experiments in a notebook and post it here.

[G-ecs]

Here's a notebook with MNIST and iris datasets using 3 methods: npsavetxt as in tuto, np.savetxt with fmt="%f" and pandas to_csv. The error happens only on MNIST

https://gist.github.com/G-ecs/0676671003765aeaec016bc3b273db12