Error: [CONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT] Remote client cannot create a SparkContext. Create SparkSession instead.

[mooks623]

Hello Zingg Team,

I am very new to Databricks and Python and following the instructions on Open Source Identity Resolution on Databricks for Customer 360 and running into the following error in step 4:

[CONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT] Remote client cannot create a SparkContext. Create SparkSession instead.

I am not sure how to create a SparkSession. Here is the details code:

##please leave the following unchanged
MARKED_DIR = zinggDir + "/" + modelId + "/trainingData/marked/"
UNMARKED_DIR = zinggDir + "/" + modelId + "/trainingData/unmarked/"
MARKED_DIR_DBFS = "/dbfs" + MARKED_DIR
UNMARKED_DIR_DBFS = "/dbfs" + UNMARKED_DIR

import pandas as pd
import numpy as np
 
import time
import uuid
 
from tabulate import tabulate
from ipywidgets import widgets, interact, GridspecLayout
import base64
import pyspark.sql.functions as fn

##this code sets up the Zingg Python interface
from zingg.client import *
from zingg.pipes import *

def cleanModel():
    dbutils.fs.rm(MARKED_DIR, recurse=True)
    # drop unmarked data
    dbutils.fs.rm(UNMARKED_DIR, recurse=True)
    return

# assign label to candidate pair
def assign_label(candidate_pairs_pd, z_cluster, label):
  '''
  The purpose of this function is to assign a label to a candidate pair
  identified by its z_cluster value.  Valid labels include:
     0 - not matched
     1 - matched
     2 - uncertain
  '''
  
  # assign label
  candidate_pairs_pd.loc[ candidate_pairs_pd['z_cluster']==z_cluster, 'z_isMatch'] = label
  
  return
 
def count_labeled_pairs(marked_pd):
  '''
  The purpose of this function is to count the labeled pairs in the marked folder.
  '''
n_total = len(np.unique(marked_pd['z_cluster']))
n_positive = len(np.unique(marked_pd[marked_pd['z_isMatch']==1]['z_cluster']))
n_negative = len(np.unique(marked_pd[marked_pd['z_isMatch']==0]['z_cluster']))
  
return n_positive, n_negative, n_total
# setup widget 
available_labels = {
    'No Match':0,
    'Match':1,
    'Uncertain':2
    }
#dbutils.widgets.dropdown('label', 'Uncertain', available_labels.keys(), 'Is this pair a match?')

Thanks,
Mukesh

[sonalgoyal]

Hello @mooks623, which version of DBR are you using? Please try 16.4 ML LTS