Error running the example script

[jdebacker]

@jpycroft reports:

Then I thought I’d try to run the example “python run_oguk.py” in the “examples/” folder. This generated over 5000 lines of errors. Many errors reference matplotlib (I checked that my matplotlib is up-to-date, of course). I've also attached that output below.

See Issue #28 for the above referenced txt files.

[jdebacker]

The matplotlib lines are warnings, I believe. It maybe that the OG-Core style file needs updating, or it could be an issue with trying to import this file from a URL as is done in both OG-USA and OG-UK (see here).

The real issue, and what is breaking execution, is

Number of workers =  4
Using baseline tax parameters from  C:\Users\jonat\repos\OG-UK\examples\OG-UK-Example\OUTPUT_BASELINE\TxFuncEst_baseline.pkl
distributed.worker - WARNING -  Compute Failed
Function:  get_calculator_output
args:      (True, 2018, None, 'cps')
kwargs:    {}
Exception: KeyError("Unable to open object (object 'P_person_id' doesn't exist)")

Traceback (most recent call last):
  File "run_oguk.py", line 176, in <module>
    main()
  File "run_oguk.py", line 60, in main
    c = Calibration(p, estimate_tax_functions=True, client=client)
  File "c:\users\jonat\repos\og-uk\oguk\calibrate.py", line 47, in __init__
    tax_func_path=tax_func_path,
  File "c:\users\jonat\repos\og-uk\oguk\calibrate.py", line 126, in get_tax_function_parameters
    num_workers=num_workers,
  File "c:\users\jonat\repos\og-uk\oguk\get_micro_data.py", line 138, in get_data
    results = client.gather(futures)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\client.py", line 1992, in gather
    asynchronous=asynchronous,
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\client.py", line 833, in sync
    self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\utils.py", line 340, in sync
    raise exc.with_traceback(tb)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\utils.py", line 324, in f
    result[0] = yield future
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\tornado\gen.py", line 762, in run
    value = future.result()
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\distributed\client.py", line 1851, in _gather
    raise exception.with_traceback(traceback)
  File "c:\users\jonat\repos\og-uk\oguk\get_micro_data.py", line 44, in get_calculator_output
    sim = Microsimulation(year=year)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\tools\simulation.py", line 243, in __init__
    self.load_dataset(dataset, self.year)
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\openfisca_uk\tools\simulation.py", line 357, in load_dataset
    "person": np.array(data["P_person_id"][year]),
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\h5py\_hl\group.py", line 305, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py\h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (object 'P_person_id' doesn't exist)"
distributed.worker - WARNING - Heartbeat to scheduler failed

(oguk-dev) C:\Users\jonat\repos\OG-UK\examples>

I've seen this issue with object 'P_person_id' doesn't exist before, but I can't recall exactly how it was resolved. It was some issue with the data used for OpenFisca-UK, I believe. Let me see if I can recreate.

[jdebacker]

@jpycroft I was able to generate this same error. I had thought I ran this example before, but not sure.

PR #27 maybe helpful here (though it needs to be updated with the master).

Also, it would be good to learn more about OpenFisca-UK. I'm still not totally sure how to do many things with the model and don't fully understand how the data is loaded and hierarchical data structures handled. It might be good for all of us working with OG-UK to learn more about the microsimulation model.

[jpycroft]

@jdebacker @nikhilwoodruff @rickecon

Continuing to try to understand why I getting the litany of errors with run_oguk.py.

I realise now that I'm not even getting past the import commands at the beginning of file. If I run to here:

import multiprocessing
from distributed import Client
import json
import time
import os,sys
from openfisca_core.model_api import Reform
from oguk.calibrate import Calibration
print('stop after oguk.calibrate')
sys.exit()

I already get a load of errors: see attached "run_oguk_stop_after_importCalibrate_11oct2021.txt"

Going to the next import command, and commenting out the previous import command, i.e.:

import multiprocessing
from distributed import Client
import json
import time
import os,sys
from openfisca_core.model_api import Reform
# from oguk.calibrate import Calibration
from ogcore import output_plots as op
print('stop after ogcore - output plots; Calibration off')
sys.exit()

I again get a load of errors: see attached "run_oguk_stop_after_ogcore-outputPlots_CalibrationOff_11oct2021.txt"

The same happens for the other "from ogcore" imports.

run_oguk_stop_after_importCalibrate_11oct2021.txt
run_oguk_stop_after_ogcore-outputPlots_CalibrationOff_11oct2021.txt

[jdebacker]

@jpycroft All those were warnings. Your set of commands did execute (see the stop after ogcore - output plots; Calibration off printed at the end).

But those messages are annoying. I just merged a PR to OG-Core and a PR to OG-UK that should fix this (they completely remove any imports of matplotlib style files from scripts used in the two packages). Install new versions of each and try that above again. That warnings should disappear.

[jpycroft]

Thanks @jdebacker!

I updated my branch. It hard to tell if there are fewer warnings. There are certainly still many (attached).

In any case, if it's only warnings, then perhaps the only important part of the output is the last lines:

usage: run_oguk.py [-h] reform
run_oguk.py: error: the following arguments are required: reform

I'll have a look at this later on.

run_oguk_after-poss-removing-some-warnings_12oct2021.txt
.

[jdebacker]

@jpycroft Did you reinstall the ogcore package? Did you rebuild the oguk package after updating your branch from the upstream main branch?

[jpycroft]

@jdebacker Thanks again!

I hadn't updated my OG-Core. I updated that and tried to install its environment. This only partially worked, ending with the following error message (full output attached):

Pip subprocess error:
ERROR: Could not install packages due to an OSError: [WinError 5] Acceso denegado: 'C:\\Users\\jonat\\AppData\\Local\\Temp\\pip-uninstall-lahmt6kq\\sphinx-apidoc.exe'
Consider using the `--user` option or check the permissions.

failed

CondaEnvException: Pip failed

In any case, I updated OG-UK again and updated the environment (oguk-dev). Trying to run "run_oguk.py" still results in the litany of warnings as before, culminating in the same error as above (full output attached):

usage: run_oguk.py [-h] reform
run_oguk.py: error: the following arguments are required: reform

og-core_try_to_build_environment_13oct2021.txt

run_oguk_after-updating_13oct2021.txt

[nikhilwoodruff]

Great to see it (at least partially) working @jpycroft - this one I can answer: I while ago I wrote this format for run_oguk.py to take as argument a reform object in a Python file - it should work with python run_oguk.py small_ubi_reform.ubi_reform (pointing to the ubi_reform object in small_ubi_reform.py). I can take it out if it's causing inconvenience.

[jdebacker]

@jpycroft, Nikhil pointed out the issue with your attempted run of the example script.

I think the matplotllib warnings are still there because OG-Core was not updated because your env failed to updated.

I'd remove the environment with: conda remove --name oguk-dev --all

Then create it anew with conda env create

[jdebacker]

@rickecon because he may have seen the same sphinx install error that Pycroft is seeing recently. Suggests perhaps removing those dependencies that just help with documentation.

[jpycroft]

Thanks for the ideas @nikhilwoodruff @jdebacker.

Running the python run_oguk.py small_ubi_reform.ubi_reform (having reinstalled the environment) I get the same litany of errors, but now I'm ending on the person_id error:

KeyError: "Unable to open object (object 'person_id' doesn't exist)"

Output attached.
Best, Jon

run_oguk_small_ubi_reform_23oct2021.txt

[jpycroft]

@rickecon, @jdebacker, @nikhilwoodruff,

Exciting news: I’m getting different error messages :-) Seriously, that is exciting!

Having updated to the latest environment.yml and doing “conda update conda”, the litany of hundreds of error messages is gone. I’ve worked my way through a number of error messages and seem now to be stuck on accessing the synthetic data set for OpenFiscaUK.

I’ll follow this up below with two more posts on:

1. Adjustments I’ve already made to get past errors &
2. Notes on error messages regarding the default / synthetic data set

[nikhilwoodruff]

Amazing - admittedly it's been a while since I last used the synthetic dataset, happy to debug anything else you get. Or, I don't suppose you can get access directly to the FRS by any means?

[jpycroft]

Hi @nikhilwoodruff,
Yes, going for the full FRS might be the best solution.
I'll look into how to download. It seems that I'll need to agree to an End User Licence first ...

[jpycroft]

With reference to the thread above, #31 (comment)
I'm trying to run: python run_og_uk.py

Adjustments I’ve already made to get past errors:

1. change line in run_og_uk.py:
   from og_uk_calibrate.calibrate import Calibration
   --> from oguk.calibrate import Calibration

2. Copy two files into the OG-UK folder:
   a. ogusa
   b. taxcalc (pip install taxcalc did not work. seemingly many errors: "Preparing metadata (setup.py) ... error" ... "Command errored out with exit status 1"; then tries again with earlier version (e.g. 0.6.2.) and same message appears; also tried pip install taxcalc==0.4 / 0.4.1 / etc), Instead, fork & git clone Tax-Calculator. Then cut & paste "taxcalc" folder into OG-UK

3. Re-install a few packages, which were missing because I had reinstalled Anaconda:
   pip install numba
   pip install eurostat
   pip install xlsxwriter

4. Remove command to put OG-USA version in parameters to save for reference
   As ogusa is no longer part of the environment installation, the line in parameters.py:
   self.ogusa_version = pkg_resources.get_distribution("ogusa").version
   Throws an error:

  File "C:\Users\jonat\anaconda3\envs\oguk-dev\lib\site-packages\pkg_resources\__init__.py", line 777, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'ogusa' distribution was not found and is required by the application

As it appears to be non-essential, for now I simply edited it out.

5. The next error is a failure to find the default / synthetic data set; see next comment.

@jdebacker @rickecon @nikhilwoodruff