Handling ConnectionError, HTTPError thrown during a query

[CorbinFoucart]

Versions

OS: Linux (Ubuntu 18.04)      
Python:  3.8.6
krakenex: 2.1.0

What are you trying to achieve?

I'm looking to record price data in a persistent session. I do this through a simple wrapper class

class KrakenSession(object):
    def __init__(self):
        self.k = krakenex.API()
        self.k.load_key('src/kraken.key')

    def get_current_asking_price(self):
        """queries kraken API for currency pair
        :note: don't query this too fast or you'll get request errors
            0.25 seconds per request or faster is too fast
        """
        pairstr='XXBTZUSD'
        query = self.k.query_public('Ticker', {'pair':pairstr})
        asking_price = query['result'][pairstr]['a'][0]

What happens instead?

The above snippet works for a few hours, then inevitably crashes

Traceback (most recent call last):
  File "src/kraken.py", line 96, in <module>
    krs.record()
  File "src/kraken.py", line 84, in record
    price = self.get_current_asking_price(pairstr='XXBTZUSD')
  File "src/kraken.py", line 31, in get_current_asking_price
    query = self.k.query_public('Ticker', {'pair':pairstr})
  File "/home/foucartc/anaconda3/envs/ctrade_env/lib/python3.8/site-packages/krakenex/api.py", line 162, in query_public
    return self._query(urlpath, data, timeout = timeout)
  File "/home/foucartc/anaconda3/envs/ctrade_env/lib/python3.8/site-packages/krakenex/api.py", line 134, in _query
    self.response = self.session.post(url, data = data, headers = headers,
  File "/home/foucartc/anaconda3/envs/ctrade_env/lib/python3.8/site-packages/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/home/foucartc/anaconda3/envs/ctrade_env/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/foucartc/anaconda3/envs/ctrade_env/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/foucartc/anaconda3/envs/ctrade_env/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

I presume this could be a problem on the Kraken end (or perhaps with my internet connection), so I wrap the body of get_current_asking_price in a try/except:

    def get_current_asking_price(self):
        """queries kraken API for currency pair
        :note: don't query this too fast or you'll get request errors
            0.25 seconds per request or faster is too fast
        """
        pairstr='XXBTZUSD'
        try:
            query = self.k.query_public('Ticker', {'pair':pairstr})
            asking_price = query['result']['XXBTZUSD']['a'][0]
        except ConnectionError:
            print('encountered a server error. retrying...')
            time.sleep(60)
            query = self.k.query_public('Ticker', {'pair':pairstr})
            asking_price = query['result'][pairstr]['a'][0]
        return float(asking_price)

which again runs for a few hours, but returns a different error (without printing 'encountered a server error. retrying...'):

Traceback (most recent call last):
  File "src/kraken.py", line 102, in <module>
    krs.record()
  File "src/kraken.py", line 90, in record
    price = self.get_current_asking_price(pairstr='XXBTZUSD')
  File "src/kraken.py", line 32, in get_current_asking_price
    query = self.k.query_public('Ticker', {'pair':pairstr})
  File "/home/foucartc/anaconda3/envs/ctrade_env/lib/python3.8/site-packages/krakenex/api.py", line 162, in query_public
    return self._query(urlpath, data, timeout = timeout)
  File "/home/foucartc/anaconda3/envs/ctrade_env/lib/python3.8/site-packages/krakenex/api.py", line 138, in _query
    self.response.raise_for_status()
  File "/home/foucartc/anaconda3/envs/ctrade_env/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 520 Server Error:  for url: https://api.kraken.com/0/public/Ticker

which seems like a deeper error in requests. Before I make my try/except handling more elaborate, I want to understand what's going on here.

• Is this a problem on the kraken side?
• Does it require resetting / opening a new connection through krakenex and closing the current session?
• Something else?

Any pointers would be appreciated!

[veox]

Is this a problem on the kraken side?

More often than not. You can avoid a few 'Connection reset by peer' by sending less queries. However, during a DDoS or heavy load Kraken may choose to drop/reset the connection regardless.

It can also happen due to a flaky connection.

Does it require resetting / opening a new connection through krakenex and closing the current session?

See the approaches in PRs #99 and #100, as well as the issues linked there, for a three-year old (:disappointed:) discussion on this.

I never got to merging either approach into the package, since both solutions have issues, and are hard to test properly these days.

Github says either of those can still be merged into the code as-is.

Something else?

Be careful with retries:

https://www.reddit.com/r/krakenex/comments/778uvh/psa_http_error_502_does_not_mean_the_query_wont/

This is old, and Kraken has had an iteration on their production code, so the PSA may not apply verbatim - just linking as an example of "unexpected things can happen".

[veox]

Marking as duplicate of issue #66, mainly because the solution seems to be the same. Ping back if I misunderstood.

[CorbinFoucart]

For those who encounter the same problem, I found the solutions in PRs #99 and #100 were insufficient to prevent the program from crashing. My approach was instead to wrap the price querying code in a shell script that restarts upon failure, which seems to work great although it doesn't solve the underlying problem at the krakenex level.

Closing. Thank you for the input!