undefined method `fields' for nil:NilClass

[telezoic]

Hi folks,

I'm working through the production build as outlined at: https://digitalnz.github.io/supplejack/start/production-install.html

I think I've found a way (almost) to connect a production solr instance to the API. With solr 5 I have created a ‘production’ core, generated the sunspot solr files, copied those files to the production core, changed the sunspot.yml to point to production core, and created a symlink from the sunspot production folder to the solr production core.

I can start my solr instance and see my new production core [at localhost:8983/solr] by running rake sunspot:solr:start RAILS_ENV=production from the api folder.

From the api folder if I run bundle exec rails s -b 0.0.0.0 -p 3000, the api gives the following error:

{
  "errors": [
    "RSolr::Error::Http - 404 Not Found\nError:     Not Found\n\nURI: http://localhost:8982/solr/default/select?wt=json\nRequest Headers: {\"Content-Type\"=>\"application/x-www-form-urlencoded; charset=UTF-8\"}\nRequest Data: \"fq=type%3ASupplejackApi%5C%3A%5C%3ARecord&fq=record_type_i%3A0&&start=0&rows=20&q=*%3A*\"\n\nBacktrace: /home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/rsolr-2.2.1/lib/rsolr/client.rb:206:in `rescue in execute'\n/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/rsolr-2.2.1/lib/rsolr/client.rb:196:in `execute'\n/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/rsolr-2.2.1/lib/rsolr/client.rb:191:in `send_and_receive'\n/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sunspot_rails-2.2.8/lib/sunspot/rails/solr_instrumentation.rb:16:in `block in send_and_receive_with_as_instrumentation'\n/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activesupport-5.1.4/lib/active_support/notifications.rb:166:in `block in instrument'\n/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activesupport-5.1.4/lib/active_support/notifications/instrumenter.rb:21:in `instrument'\n/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activesupport-5.1.4/lib/active_support/notifications.rb:166:in `instrument'\n/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sunspot_rails-2.2.8/lib/sunspot/rails/solr_instrumentation.rb:15:in `send_and_receive_with_as_instrumentation'\n(eval):2:in `post'\n/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sunspot-2.2.8/lib/sunspot/search/abstract_search.rb:49:in `execute'\n/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/bundler/gems/supplejack_api-c262ce8a72d7/app/models/supplejack_api/search.rb:350:in `execute_solr_search_and_handle_errors'"
  ]
}

I'm thinking this is because I haven't specified the production environment (8983)

I added a line to the puma.rb file

environment ENV.fetch("RAILS_ENV") { "production" }

and then tried starting the api with bundle exec rails s -e production -b 0.0.0.0 -p 3000.

this throws the following:

/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/devise-4.3.0/app/controllers/devise/sessions_controller.rb:5: syntax error, unexpected '{', expecting keyword_end (SyntaxError)
...ion only: [:create, :destroy] { request.env["devise.skip_tim...
...                              ^
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/devise-4.3.0/app/controllers/devise/sessions_controller.rb:5: syntax error, unexpected '}', expecting keyword_end
..."devise.skip_timeout"] = true }

I see some discussion online of a bug with ruby 2.5.+ (I'm at ruby 2.5.3p105) and devise 4.3.0 around the session controller. Updating to devise 4.4.0 removes the error but throws a new one from the api gem:

/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/bundler/gems/supplejack_api-c262ce8a72d7/app/models/supplejack_api/set_item.rb:7:in `<class:SetItem>': undefined method `fields' for nil:NilClass (NoMethodError)

I hoping you might have a strategy/pointer to help me get around this :)

Thanks,

D.

[motizuki]

Hi @telezoic

Would you be able to share the following files?

app/supplejack_api/record_schema.rb
Gemfile
Gemfile.lock

Feel free to email me if you don't want to share your files here.

[telezoic]

Thanks @motizuki!
No trouble.

class RecordSchema
  include SupplejackApi::SupplejackSchema
  # Namespaces
  namespace :dc, url: 'http://purl.org/dc/elements/1.1/'
  # Fields
  string    :record_id, store: false
  string    :internal_identifier
  string    :title,  search_boost: 10,  search_as: [:filter, :fulltext], namespace: :dc
  string    :description,  search_boost: 2,  search_as: [:filter, :fulltext],  namespace:   :dc
  string    :display_collection,  search_as: [:filter, :fulltext],  namespace: :sj
  string    :display_content_partner, multi_value: true,  namespace: :dc
  string    :category,  multi_value: true,  search_as: [:filter]
  string    :source_url
  string    :landing_url
  string    :status
  datetime  :created_at, date_format: '%y/%d/%m'

  # Groups
  group :default do
    fields [
      :record_id,
      :internal_identifier,
      :title,
      :description,
      :category,
      :display_content_partner,
      :display_collection,
      :source_url,
      :landing_url
    ]
  end

  # Roles
  role :developer, default: true
  role :admin, admin: true
  role :harvester, harvester: true
end

git_source(:github) do |repo_name|
  repo_name = "#{repo_name}/#{repo_name}" unless repo_name.include?("/")
  "https://github.com/#{repo_name}.git"
end

gem 'supplejack_api', git: 'http://github.com/DigitalNZ/supplejack_api.git'
gem 'sunspot_rails', '~> 2.2.0'
gem 'active_model_serializers', '~> 0.10.7'
gem 'mongoid_auto_increment'
gem 'whenever', '~> 0.10.0'
# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'
gem 'rails', '~> 5.1.4'
# Use sqlite3 as the database for Active Record
gem 'sqlite3','1.3.13'
# Use Puma as the app server
gem 'puma', '~> 3.7'
# Use SCSS for stylesheets
gem 'sass-rails', '~> 5.0'
# Use Uglifier as compressor for JavaScript assets
gem 'uglifier', '>= 1.3.0'
# See https://github.com/rails/execjs#readme for more supported runtimes
# gem 'therubyracer', platforms: :ruby

# Use CoffeeScript for .coffee assets and views
gem 'coffee-rails', '~> 4.2'
# Turbolinks makes navigating your web application faster. Read more: https://github.com/turbolinks/turbolinks
gem 'turbolinks', '~> 5'
# Build JSON APIs with ease. Read more: https://github.com/rails/jbuilder
gem 'jbuilder', '~> 2.5'
# Use Redis adapter to run Action Cable in production
# gem 'redis', '~> 3.0'
# Use ActiveModel has_secure_password
# gem 'bcrypt', '~> 3.1.7'

# Use Capistrano for deployment
# gem 'capistrano-rails', group: :development

group :development, :test do
  # Call 'byebug' anywhere in the code to stop execution and get a debugger console
  gem 'byebug', platforms: [:mri, :mingw, :x64_mingw]
  # Adds support for Capybara system testing and selenium driver
  gem 'capybara', '~> 2.13'
  gem 'selenium-webdriver'
end

group :development do
  # Access an IRB console on exception pages or by using <%= console %> anywhere in the code.
  gem 'web-console', '>= 3.3.0'
  gem 'listen', '>= 3.0.5', '< 3.2'
  # Spring speeds up development by keeping your application running in the background. Read more: https://github.com/rails/spring
  gem 'spring'
  gem 'spring-watcher-listen', '~> 2.0.0'
end

# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw, :jruby]

GIT
  remote: http://github.com/DigitalNZ/supplejack_api.git
  revision: c262ce8a72d7d91bceaca09a2a9cb15e47fdf270
  specs:
    supplejack_api (1)
      active_model_serializers
      activemodel-serializers-xml
      activeresource
      coffee-rails
      compass-rails
      dalli
      devise (= 4.4.0)
      devise-token_authenticatable (<= 1.0.0)
      dimensions
      dry-struct
      dry-validation (= 0.13.0)
      figaro
      gabba
      jquery-rails
      json-ld
      kaminari
      kaminari-mongoid
      lazy_high_charts
      mimemagic
      mongoid (= 7.0.1)
      mongoid_auto_increment
      paperclip
      progressbar
      rails (= 5.1.4)
      responders
      rest-client
      rufus-scheduler
      sass-rails
      sidekiq
      state_machine
      sunspot_rails
      sunspot_solr
      thin
      uglifier
      unicode_utils

GEM
  remote: https://rubygems.org/
  specs:
    actioncable (5.1.4)
      actionpack (= 5.1.4)
      nio4r (~> 2.0)
      websocket-driver (~> 0.6.1)
    actionmailer (5.1.4)
      actionpack (= 5.1.4)
      actionview (= 5.1.4)
      activejob (= 5.1.4)
      mail (~> 2.5, >= 2.5.4)
      rails-dom-testing (~> 2.0)
    actionpack (5.1.4)
      actionview (= 5.1.4)
      activesupport (= 5.1.4)
      rack (~> 2.0)
      rack-test (>= 0.6.3)
      rails-dom-testing (~> 2.0)
      rails-html-sanitizer (~> 1.0, >= 1.0.2)
    actionview (5.1.4)
      activesupport (= 5.1.4)
      builder (~> 3.1)
      erubi (~> 1.4)
      rails-dom-testing (~> 2.0)
      rails-html-sanitizer (~> 1.0, >= 1.0.3)
    active_model_serializers (0.10.9)
      actionpack (>= 4.1, < 6)
      activemodel (>= 4.1, < 6)
      case_transform (>= 0.2)
      jsonapi-renderer (>= 0.1.1.beta1, < 0.3)
    activejob (5.1.4)
      activesupport (= 5.1.4)
      globalid (>= 0.3.6)
    activemodel (5.1.4)
      activesupport (= 5.1.4)
    activemodel-serializers-xml (1.0.2)
      activemodel (> 5.x)
      activesupport (> 5.x)
      builder (~> 3.1)
    activerecord (5.1.4)
      activemodel (= 5.1.4)
      activesupport (= 5.1.4)
      arel (~> 8.0)
    activeresource (5.1.0)
      activemodel (>= 5.0, < 7)
      activemodel-serializers-xml (~> 1.0)
      activesupport (>= 5.0, < 7)
    activesupport (5.1.4)
      concurrent-ruby (~> 1.0, >= 1.0.2)
      i18n (~> 0.7)
      minitest (~> 5.1)
      tzinfo (~> 1.1)
    addressable (2.6.0)
      public_suffix (>= 2.0.2, < 4.0)
    arel (8.0.0)
    bcrypt (3.1.13)
    bindex (0.7.0)
    bson (4.5.0)
    builder (3.2.3)
    byebug (11.0.1)
    capybara (2.18.0)
      addressable
      mini_mime (>= 0.1.3)
      nokogiri (>= 1.3.3)
      rack (>= 1.0.0)
      rack-test (>= 0.5.4)
      xpath (>= 2.0, < 4.0)
    case_transform (0.2)
      activesupport
    childprocess (1.0.1)
      rake (< 13.0)
    chronic (0.10.2)
    chunky_png (1.3.11)
    climate_control (0.2.0)
    coffee-rails (4.2.2)
      coffee-script (>= 2.2.0)
      railties (>= 4.0.0)
    coffee-script (2.4.1)
      coffee-script-source
      execjs
    coffee-script-source (1.12.2)
    compass (0.12.2)
      chunky_png (~> 1.2)
      fssm (>= 0.2.7)
      sass (~> 3.1)
    compass-rails (2.0.0)
      compass (>= 0.12.2)
    concurrent-ruby (1.1.5)
    connection_pool (2.2.2)
    crass (1.0.4)
    daemons (1.3.1)
    dalli (2.7.10)
    devise (4.4.0)
      bcrypt (~> 3.0)
      orm_adapter (~> 0.1)
      railties (>= 4.1.0, < 5.2)
      responders
      warden (~> 1.2.3)
    devise-token_authenticatable (1.0.0)
      devise (= 4.4.0)
    dimensions (1.3.0)
    domain_name (0.5.20180417)
      unf (>= 0.0.5, < 1.0.0)
    dry-configurable (0.8.2)
      concurrent-ruby (~> 1.0)
      dry-core (~> 0.4, >= 0.4.7)
    dry-container (0.7.0)
      concurrent-ruby (~> 1.0)
      dry-configurable (~> 0.1, >= 0.1.3)
    dry-core (0.4.7)
      concurrent-ruby (~> 1.0)
    dry-equalizer (0.2.2)
    dry-inflector (0.1.2)
    dry-logic (0.6.1)
      concurrent-ruby (~> 1.0)
      dry-core (~> 0.2)
      dry-equalizer (~> 0.2)
    dry-struct (0.7.0)
      dry-core (~> 0.4, >= 0.4.3)
      dry-equalizer (~> 0.2)
      dry-types (~> 0.15)
      ice_nine (~> 0.11)
    dry-types (0.15.0)
      concurrent-ruby (~> 1.0)
      dry-container (~> 0.3)
      dry-core (~> 0.4, >= 0.4.4)
      dry-equalizer (~> 0.2, >= 0.2.2)
      dry-inflector (~> 0.1, >= 0.1.2)
      dry-logic (~> 0.5, >= 0.5)
    dry-validation (0.13.0)
      concurrent-ruby (~> 1.0)
      dry-configurable (~> 0.1, >= 0.1.3)
      dry-core (~> 0.2, >= 0.2.1)
      dry-equalizer (~> 0.2)
      dry-logic (~> 0.5, >= 0.5.0)
      dry-types (~> 0.14, >= 0.14)
    erubi (1.8.0)
    et-orbi (1.2.1)
      tzinfo
    eventmachine (1.2.7)
    execjs (2.7.0)
    faraday (0.15.4)
      multipart-post (>= 1.2, < 3)
    ffi (1.11.1)
    figaro (1.1.1)
      thor (~> 0.14)
    fssm (0.2.10)
    fugit (1.2.1)
      et-orbi (~> 1.1, >= 1.1.8)
      raabro (~> 1.1)
    gabba (1.0.1)
    globalid (0.4.2)
      activesupport (>= 4.2.0)
    hamster (3.0.0)
      concurrent-ruby (~> 1.0)
    hash-deep-merge (0.1.1)
    http-cookie (1.0.3)
      domain_name (~> 0.5)
    i18n (0.9.5)
      concurrent-ruby (~> 1.0)
    ice_nine (0.11.2)
    jbuilder (2.9.1)
      activesupport (>= 4.2.0)
    jquery-rails (4.3.3)
      rails-dom-testing (>= 1, < 3)
      railties (>= 4.2.0)
      thor (>= 0.14, < 2.0)
    json-ld (3.0.2)
      multi_json (~> 1.12)
      rdf (>= 2.2.8, < 4.0)
    jsonapi-renderer (0.2.0)
    kaminari (1.1.1)
      activesupport (>= 4.1.0)
      kaminari-actionview (= 1.1.1)
      kaminari-activerecord (= 1.1.1)
      kaminari-core (= 1.1.1)
    kaminari-actionview (1.1.1)
      actionview
      kaminari-core (= 1.1.1)
    kaminari-activerecord (1.1.1)
      activerecord
      kaminari-core (= 1.1.1)
    kaminari-core (1.1.1)
    kaminari-mongoid (1.0.1)
      kaminari-core (~> 1.0)
      mongoid
    lazy_high_charts (1.5.8)
      hash-deep-merge
    link_header (0.0.8)
    listen (3.1.5)
      rb-fsevent (~> 0.9, >= 0.9.4)
      rb-inotify (~> 0.9, >= 0.9.7)
      ruby_dep (~> 1.2)
    loofah (2.2.3)
      crass (~> 1.0.2)
      nokogiri (>= 1.5.9)
    mail (2.7.1)
      mini_mime (>= 0.1.1)
    method_source (0.9.2)
    mime-types (3.2.2)
      mime-types-data (~> 3.2015)
    mime-types-data (3.2019.0331)
    mimemagic (0.3.3)
    mini_mime (1.0.1)
    mini_portile2 (2.4.0)
    minitest (5.11.3)
    mongo (2.8.0)
      bson (>= 4.4.2, < 5.0.0)
    mongoid (7.0.1)
      activemodel (>= 5.1, < 6.0.0)
      mongo (>= 2.5.1, < 3.0.0)
    mongoid_auto_increment (0.1.5)
      mongoid (>= 2.0)
    multi_json (1.13.1)
    multipart-post (2.1.1)
    netrc (0.11.0)
    nio4r (2.3.1)
    nokogiri (1.10.3)
      mini_portile2 (~> 2.4.0)
    orm_adapter (0.5.0)
    paperclip (6.1.0)
      activemodel (>= 4.2.0)
      activesupport (>= 4.2.0)
      mime-types
      mimemagic (~> 0.3.0)
      terrapin (~> 0.6.0)
    pr_geohash (1.0.0)
    progressbar (1.10.0)
    public_suffix (3.0.3)
    puma (3.12.1)
    raabro (1.1.6)
    rack (2.0.7)
    rack-protection (2.0.5)
      rack
    rack-test (1.1.0)
      rack (>= 1.0, < 3)
    rails (5.1.4)
      actioncable (= 5.1.4)
      actionmailer (= 5.1.4)
      actionpack (= 5.1.4)
      actionview (= 5.1.4)
      activejob (= 5.1.4)
      activemodel (= 5.1.4)
      activerecord (= 5.1.4)
      activesupport (= 5.1.4)
      bundler (>= 1.3.0)
      railties (= 5.1.4)
      sprockets-rails (>= 2.0.0)
    rails-dom-testing (2.0.3)
      activesupport (>= 4.2.0)
      nokogiri (>= 1.6)
    rails-html-sanitizer (1.0.4)
      loofah (~> 2.2, >= 2.2.2)
    railties (5.1.4)
      actionpack (= 5.1.4)
      activesupport (= 5.1.4)
      method_source
      rake (>= 0.8.7)
      thor (>= 0.18.1, < 2.0)
    rake (12.3.2)
    rb-fsevent (0.10.3)
    rb-inotify (0.10.0)
      ffi (~> 1.0)
    rdf (3.0.12)
      hamster (~> 3.0)
      link_header (~> 0.0, >= 0.0.8)
    redis (4.1.1)
    responders (2.4.1)
      actionpack (>= 4.2.0, < 6.0)
      railties (>= 4.2.0, < 6.0)
    rest-client (2.0.2)
      http-cookie (>= 1.0.2, < 2.0)
      mime-types (>= 1.16, < 4.0)
      netrc (~> 0.8)
    rsolr (2.2.1)
      builder (>= 2.1.2)
      faraday (>= 0.9.0)
    ruby_dep (1.5.0)
    rubyzip (1.2.3)
    rufus-scheduler (3.6.0)
      fugit (~> 1.1, >= 1.1.6)
    sass (3.7.4)
      sass-listen (~> 4.0.0)
    sass-listen (4.0.0)
      rb-fsevent (~> 0.9, >= 0.9.4)
      rb-inotify (~> 0.9, >= 0.9.7)
    sass-rails (5.0.7)
      railties (>= 4.0.0, < 6)
      sass (~> 3.1)
      sprockets (>= 2.8, < 4.0)
      sprockets-rails (>= 2.0, < 4.0)
      tilt (>= 1.1, < 3)
    selenium-webdriver (3.142.3)
      childprocess (>= 0.5, < 2.0)
      rubyzip (~> 1.2, >= 1.2.2)
    sidekiq (5.2.7)
      connection_pool (~> 2.2, >= 2.2.2)
      rack (>= 1.5.0)
      rack-protection (>= 1.5.0)
      redis (>= 3.3.5, < 5)
    spring (2.0.2)
      activesupport (>= 4.2)
    spring-watcher-listen (2.0.1)
      listen (>= 2.7, < 4.0)
      spring (>= 1.2, < 3.0)
    sprockets (3.7.2)
      concurrent-ruby (~> 1.0)
      rack (> 1, < 3)
    sprockets-rails (3.2.1)
      actionpack (>= 4.0)
      activesupport (>= 4.0)
      sprockets (>= 3.0.0)
    sqlite3 (1.3.13)
    state_machine (1.2.0)
    sunspot (2.2.8)
      pr_geohash (~> 1.0)
      rsolr (>= 1.1.1, < 3)
    sunspot_rails (2.2.8)
      rails (>= 3)
      sunspot (= 2.2.8)
    sunspot_solr (2.3.0)
    terrapin (0.6.0)
      climate_control (>= 0.0.3, < 1.0)
    thin (1.7.2)
      daemons (~> 1.0, >= 1.0.9)
      eventmachine (~> 1.0, >= 1.0.4)
      rack (>= 1, < 3)
    thor (0.20.3)
    thread_safe (0.3.6)
    tilt (2.0.9)
    turbolinks (5.2.0)
      turbolinks-source (~> 5.2)
    turbolinks-source (5.2.0)
    tzinfo (1.2.5)
      thread_safe (~> 0.1)
    uglifier (4.1.20)
      execjs (>= 0.3.0, < 3)
    unf (0.1.4)
      unf_ext
    unf_ext (0.0.7.6)
    unicode_utils (1.4.0)
    warden (1.2.8)
      rack (>= 2.0.6)
    web-console (3.7.0)
      actionview (>= 5.0)
      activemodel (>= 5.0)
      bindex (>= 0.4.0)
      railties (>= 5.0)
    websocket-driver (0.6.5)
      websocket-extensions (>= 0.1.0)
    websocket-extensions (0.1.3)
    whenever (0.10.0)
      chronic (>= 0.6.3)
    xpath (3.2.0)
      nokogiri (~> 1.8)

PLATFORMS
  ruby

DEPENDENCIES
  active_model_serializers (~> 0.10.7)
  byebug
  capybara (~> 2.13)
  coffee-rails (~> 4.2)
  jbuilder (~> 2.5)
  listen (>= 3.0.5, < 3.2)
  mongoid_auto_increment
  puma (~> 3.7)
  rails (~> 5.1.4)
  sass-rails (~> 5.0)
  selenium-webdriver
  spring
  spring-watcher-listen (~> 2.0.0)
  sqlite3 (= 1.3.13)
  sunspot_rails (~> 2.2.0)
  supplejack_api!
  turbolinks (~> 5)
  tzinfo-data
  uglifier (>= 1.3.0)
  web-console (>= 3.3.0)
  whenever (~> 0.10.0)

BUNDLED WITH
   2.0.1

[motizuki]

@telezoic

Can you try adding this code into your record schema?

  group :valid_set_fields do
    includes []
    fields []
  end

[motizuki]

Hi @telezoic, have you had a chance to try the suggested fix above?

[telezoic]

Hi @motizuki,

Sorry, I've been away for a few weeks, and have just come back at this. I modified the schema as requested, and then restarted the API with:

bundle exec rails s -e production -b 0.0.0.0 -p 3000

This gives me a new 'nilclass' error, this time at:

...
	 3: from /home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/bundler/gems/supplejack_api-c262ce8a72d7/app/serializers/supplejack_api/user_set_serializer.rb:3:in `<top (required)>'
	 2: from /home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/bundler/gems/supplejack_api-c262ce8a72d7/app/serializers/supplejack_api/user_set_serializer.rb:4:in `<module:SupplejackApi>'
	 1: from /home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/bundler/gems/supplejack_api-c262ce8a72d7/app/serializers/supplejack_api/user_set_serializer.rb:28:in `<class:UserSetSerializer>'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/bundler/gems/supplejack_api-c262ce8a72d7/app/serializers/supplejack_api/user_set_serializer.rb:29:in `<class:SetItemSerializer>': undefined method `fields' for nil:NilClass (NoMethodError)

Thanks for any advice you could share.

[telezoic]

Hi @motizuki,

I found a way out of this error [above]:

/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/bundler/gems/supplejack_api-c262ce8a72d7/app/serializers/supplejack_api/user_set_serializer.rb:29:in `<class:SetItemSerializer>': undefined method `fields' for nil:NilClass (NoMethodError)

Looking at your 'record_schema.rb` edit for the earlier error, I added another:

  group :sets do
    includes []
    fields []
  end

I have all the apps running production now . . . with one error in the manager that I'm attempting to troubleshoot.

Regardless of the target or parser, my harvests fail with the following message in the preview window:

ActiveResource::ResourceNotFound

Failed. Response code = 404. Response message = Not Found.

/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activeresource-5.0.0/lib/active_resource/connection.rb:154:in `handle_response'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activeresource-5.0.0/lib/active_resource/connection.rb:133:in `request'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activeresource-5.0.0/lib/active_resource/connection.rb:92:in `block in get'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activeresource-5.0.0/lib/active_resource/connection.rb:227:in `with_auth'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activeresource-5.0.0/lib/active_resource/connection.rb:92:in `get'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activeresource-5.0.0/lib/active_resource/base.rb:1051:in `find_single'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activeresource-5.0.0/lib/active_resource/base.rb:938:in `find'
/data/sites/supplejack_worker/app/models/abstract_job.rb:82:in `parser'
/data/sites/supplejack_worker/app/models/harvest_job.rb:56:in `records'
/data/sites/supplejack_worker/app/workers/preview_worker.rb:34:in `perform'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:185:in `execute_job'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:167:in `block (2 levels) in process'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:166:in `block in process'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/job_retry.rb:108:in `local'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/rails.rb:42:in `block in call'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activesupport-5.2.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activesupport-5.2.1/lib/active_support/reloader.rb:73:in `block in wrap'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activesupport-5.2.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/activesupport-5.2.1/lib/active_support/reloader.rb:72:in `wrap'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/rails.rb:41:in `call'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:243:in `stats'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/job_logger.rb:8:in `call'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/job_retry.rb:73:in `global'
/home/ArchMage/.rbenv/versions/2.5.3/lib/ruby/gems/2.5.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:125:in `block in dispatch'

I can curl my harvest targets from my server and see that they are indeed up and running.

The manager log doesn't help me much :). A sample below

method=POST path=/parsers/5d546d9b2a041875cb1e165c/previewers format=*/* controller=PreviewersController action=create status=200 duration=108.45 view=2.03 params={"parser"=>{"content"=>"class SJHorses < SupplejackCommon::Json::Base\r\n\r\n  \r\n    #a horse specific search against the digitalNZ API\r\n\r\n\tbase_url \"http://api.digitalnz.org/v3/records.json?api_key=[]&fields=title,display_content_partner,creator,landing_url,source_url,thumbnail_url,subject,rights,category,description,id,date,publisher,language,locations&geo_bbox=-41,174,-42,175&text=horse\"\r\n  \r\n\r\n  record_selector \"$.search.results\"\r\n\r\n  attribute :display_content_partner, default: \"South Pacific Surprise!\"\r\n  attribute :category, default: \"image\"\r\n  \r\n  attribute :publisher, path: \"$.publisher\"\r\n  attribute :subject, path: \"$.subject\"\r\n  attribute :language, path: \"$.language\"\r\n  attribute :title, path: \"$.title\"\r\n  attribute :description, path: \"$.description\", truncate: 170\r\n  attribute :source_url, path: \"$.source_url\"\r\n  attribute :landing_url, path: \"$.landing_url\"\r\n  attribute :thumbnail_url, path: \"$.thumbnail_url\"\r\n  attribute :internal_identifier do get(:landing_url).downcase\r\n\tend\r\n  attribute :display_date, path: \"$.date\", truncate: {length: 4, omission: \"\"}\r\n  attribute :locations, path: \"$.locations\"\r\n  \r\n  \r\nend"}, "environment"=>"staging", "index"=>"1", "parser_id"=>"5d546d9b2a041875cb1e165c"} time=2019-08-14 21:24:14 +0000
method=GET path=/previews/5d547bff2a04187fc31e917c.js format=js controller=PreviewsController action=show status=200 duration=51.16 view=1.50 params={} time=2019-08-14 21:24:17 +0000

Here's a snippet of the worker log where I'm viewing records in the manager preview window:

D, [2019-08-14T21:27:12.517745 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.find | SUCCEEDED | 0.011s
D, [2019-08-14T21:27:12.519066 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | STARTED | {"update"=>"abstract_jobs", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x47237766824280 type=uuid data=0xd08c32548bd741ee...>}, "updates"=>[{"q"=>{"_id"=>BSON::ObjectId('5d547cb02a04187fc31e917f')}, "u"=>{"$set"=>{"status"=>"active", "updated_at"=>...
D, [2019-08-14T21:27:12.519782 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | SUCCEEDED | 0.001s
D, [2019-08-14T21:27:12.521364 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | STARTED | {"update"=>"abstract_jobs", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x47237766824280 type=uuid data=0xd08c32548bd741ee...>}, "updates"=>[{"q"=>{"_id"=>BSON::ObjectId('5d547cb02a04187fc31e917f')}, "u"=>{"$set"=>{"start_time"=>2019-08-14 21:27:12 ...
D, [2019-08-14T21:27:12.522120 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | SUCCEEDED | 0.001s
D, [2019-08-14T21:27:12.774962 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | STARTED | {"update"=>"abstract_jobs", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x47237766824280 type=uuid data=0xd08c32548bd741ee...>}, "updates"=>[{"q"=>{"_id"=>BSON::ObjectId('5d547cb02a04187fc31e917f')}, "u"=>{"$set"=>{"harvest_failure"=>{"_id"=>BSON::O...
D, [2019-08-14T21:27:12.775753 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | SUCCEEDED | 0.001s
D, [2019-08-14T21:27:12.776854 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | STARTED | {"update"=>"abstract_jobs", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x47237766824280 type=uuid data=0xd08c32548bd741ee...>}, "updates"=>[{"q"=>{"_id"=>BSON::ObjectId('5d547cb02a04187fc31e917f')}, "u"=>{"$set"=>{"status_message"=>"Failed.  Respon...
D, [2019-08-14T21:27:12.779217 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | SUCCEEDED | 0.002s
D, [2019-08-14T21:27:12.780880 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | STARTED | {"update"=>"abstract_jobs", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x47237766824280 type=uuid data=0xd08c32548bd741ee...>}, "updates"=>[{"q"=>{"_id"=>BSON::ObjectId('5d547cb02a04187fc31e917f')}, "u"=>{"$set"=>{"status"=>"failed", "updated_at"=>...
D, [2019-08-14T21:27:12.790886 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | SUCCEEDED | 0.010s
D, [2019-08-14T21:27:12.792796 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | STARTED | {"update"=>"abstract_jobs", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x47237766824280 type=uuid data=0xd08c32548bd741ee...>}, "updates"=>[{"q"=>{"_id"=>BSON::ObjectId('5d547cb02a04187fc31e917f')}, "u"=>{"$set"=>{"end_time"=>2019-08-14 21:27:12 UT...
D, [2019-08-14T21:27:12.793608 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | SUCCEEDED | 0.001s
D, [2019-08-14T21:27:12.795137 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | STARTED | {"update"=>"abstract_jobs", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x47237766824280 type=uuid data=0xd08c32548bd741ee...>}, "updates"=>[{"q"=>{"_id"=>BSON::ObjectId('5d547cb02a04187fc31e917f')}, "u"=>{"$set"=>{"status"=>"finished", "updated_at"...
D, [2019-08-14T21:27:12.795858 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | SUCCEEDED | 0.001s
D, [2019-08-14T21:27:12.797444 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | STARTED | {"update"=>"abstract_jobs", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x47237766824280 type=uuid data=0xd08c32548bd741ee...>}, "updates"=>[{"q"=>{"_id"=>BSON::ObjectId('5d547cb02a04187fc31e917f')}, "u"=>{"$set"=>{"end_time"=>2019-08-14 21:27:12 UT...
D, [2019-08-14T21:27:12.797982 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | SUCCEEDED | 0.000s
D, [2019-08-14T21:27:12.799627 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | STARTED | {"update"=>"previews", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x47237766824280 type=uuid data=0xd08c32548bd741ee...>}, "updates"=>[{"q"=>{"_id"=>BSON::ObjectId('5d547cb02a04187fc31e9180')}, "u"=>{"$set"=>{"harvest_failure"=>"{\"_id\":\"5d547cb0...
D, [2019-08-14T21:27:12.800423 #32542] DEBUG -- : MONGODB | localhost:27017 | worker_production.update | SUCCEEDED | 0.001s
I, [2019-08-14T21:27:13.873749 #32707]  INFO -- : [8fcdd272-e4bd-469f-be1a-a52c5948d842] Started GET "/previews/5d547cb02a04187fc31e9180.json" for 127.0.0.1 at 2019-08-14 21:27:13 +0000
I, [2019-08-14T21:27:13.874323 #32707]  INFO -- : [8fcdd272-e4bd-469f-be1a-a52c5948d842] Processing by PreviewsController#show as JSON
I, [2019-08-14T21:27:13.874371 #32707]  INFO -- : [8fcdd272-e4bd-469f-be1a-a52c5948d842]   Parameters: {"id"=>"5d547cb02a04187fc31e9180"}
D, [2019-08-14T21:27:13.875347 #32707] DEBUG -- : [8fcdd272-e4bd-469f-be1a-a52c5948d842] MONGODB | localhost:27017 | worker_production.find | STARTED | {"find"=>"previews", "filter"=>{"_id"=>BSON::ObjectId('5d547cb02a04187fc31e9180')}, "$readPreference"=>{"mode"=>"primary"}, "lsid"=>{"id"=><BSON::Binary:0x47211718073360 type=uuid data=0xebb98896b03849b1...>}}
D, [2019-08-14T21:27:13.885652 #32707] DEBUG -- : [8fcdd272-e4bd-469f-be1a-a52c5948d842] MONGODB | localhost:27017 | worker_production.find | SUCCEEDED | 0.010s
I, [2019-08-14T21:27:13.886763 #32707]  INFO -- : [8fcdd272-e4bd-469f-be1a-a52c5948d842] [active_model_serializers] Rendered ActiveModel::Serializer::Null with Preview (0.57ms)
I, [2019-08-14T21:27:13.886910 #32707]  INFO -- : [8fcdd272-e4bd-469f-be1a-a52c5948d842] Completed 200 OK in 12ms (Views: 0.9ms)

Here's a parser I'm using:

[I've removed the api_key from the post]

class SouthPacificSurprise < SupplejackCommon::Json::Base

  #a horse specific search against the digital NZ API

	base_url "http://api.digitalnz.org/v3/records.json?api_key=[]&fields=title,display_content_partner,creator,landing_url,source_url,thumbnail_url,subject,rights,category,description,id,date,publisher,language,locations&geo_bbox=-41,174,-42,175&text=horse"
  

  record_selector "$.search.results"

  attribute :display_content_partner, default: "South Pacific Surprise!"
  attribute :category, default: "image"
  
  attribute :publisher, path: "$.publisher"
  attribute :subject, path: "$.subject"
  attribute :language, path: "$.language"
  attribute :title, path: "$.title"
  attribute :description, path: "$.description", truncate: 170
  attribute :source_url, path: "$.source_url"
  attribute :landing_url, path: "$.landing_url"
  attribute :thumbnail_url, path: "$.thumbnail_url"
  attribute :internal_identifier do get(:landing_url).downcase
	end
  attribute :display_date, path: "$.date", truncate: {length: 4, omission: ""}
  attribute :locations, path: "$.locations"
   
end

Can you recommend another log that might be helpful here, or any other advice that might set me on the right path? :)

Thanks,

D

[richardmatthewsdev]

Hey @telezoic,

Thanks for your message, I have been able to run the parser that you are trying locally.

Does your harvest work or is it just the preview that is broken? Also what configuration details do you have for the API, Manager, and Worker endpoints in your respective application.yml files?

Just a note that the preview uses the details for your staging worker.

Thanks,
Richard

[telezoic]

Thanks @reinkaos!

The harvest fails as well, but without errors in the 'harvest progress', but 'zeroes' in the 'number of records' line.

I've included my 'application.yml' files:

/manager/config/application.yml

production:
  WORKER_HOST: http://127.0.0.1:3002
  API_HOST: http://127.0.0.1:3000
  HARVESTER_API_KEY: LKisxbcTwynxxd3ssyvn
  API_MONGOID_HOSTS: localhost:27017
  AIRBRAKE_PROJECT_ID: AIRBRAKE_PROJECT_ID
  AIRBRAKE_PROJECT_API_KEY: AIRBRAKE_PROJECT_API_KEY
  WORKER_KEY: 7fdc382fc0bee62752f8cab38a0a04dbfb6c4535bdaf585a55e7eace560d64aeebf5f495edb628e35e739bedb0432580ddef23356e4bbf091acbb278b8a16111
    #36214098349d084e31e80c21b1e4a82657d80e0fd853982f0da4d7d783041fae42329cd41647cef5e94a6d3ebb2b0bb13e9413c44a7340d540a84ed65dbaae77

development:
  WORKER_HOST: http://127.0.0.1:3002
  API_HOST: http://127.0.0.1:3000
  HARVESTER_API_KEY: XxyAzmB6ummqn8YxAiCZ
  API_MONGOID_HOSTS: localhost:27017
  AIRBRAKE_PROJECT_ID: AIRBRAKE_PROJECT_ID
  AIRBRAKE_PROJECT_API_KEY: AIRBRAKE_PROJECT_API_KEY
  WORKER_KEY: W36214098349d084e31e80c21b1e4a82657d80e0fd853982f0da4d7d783041fae42329cd41647cef5e94a6d3ebb2b0bb13e9413c44a7340d540a84ed65dbaae77

test:
  WORKER_HOST: http://127.0.0.1:3002
  API_HOST: http://127.0.0.1:3000
  HARVESTER_API_KEY: XxyAzmB6ummqn8YxAiCZ
  API_MONGOID_HOSTS: localhost:27017
  AIRBRAKE_PROJECT_ID: AIRBRAKE_PROJECT_ID
  AIRBRAKE_PROJECT_API_KEY: AIRBRAKE_PROJECT_API_KEY
  WORKER_KEY: 36214098349d084e31e80c21b1e4a82657d80e0fd853982f0da4d7d783041fae42329cd41647cef5e94a6d3ebb2b0bb13e9413c44a7340d540a84ed65dbaae77

staging:
  WORKER_HOST: http://127.0.0.1:4002
  API_HOST: http://127.0.0.1:4000
  HARVESTER_API_KEY: sd-vbuMeyfywzXaVaFqu
  API_MONGOID_HOSTS: localhost:27017
  WORKER_KEY: ade6ffdb44133c1efe3fc1e73560fd1649b561eef4e620e25b26757f095f36f244a2f042640264fcd89f1853236842b8a3559e0870c1c1600437fc2d44766b62
  #WORKER_KEY: 36214098349d084e31e80c21b1e4a82657d80e0fd853982f0da4d7d783041fae42329cd41647cef5e94a6d3ebb2b0bb13e9413c44a7340d540a84ed65dbaae77

/worker/config/application.yml

development:
  API_HOST: "http://127.0.0.1:3000"
  HARVESTER_API_KEY: XxyAzmB6ummqn8YxAiCZ
  API_MONGOID_HOSTS: "localhost:27017"
  MANAGER_HOST: "http://127.0.0.1:3001"
  HARVESTER_CACHING_ENABLED: true
  AIRBRAKE_API_KEY: "anc123"
  LINK_CHECKING_ENABLED: "true"
  LINKCHECKER_RECIPIENTS: "test@test.com"
  WORKER_KEY: 36214098349d084e31e80c21b1e4a82657d80e0fd853982f0da4d7d783041fae42329cd41647cef5e94a6d3ebb2b0bb13e9413c44a7340d540a84ed65dbaae77

test:
  API_HOST: "http://127.0.0.1:3000"
  HARVESTER_API_KEY: XxyAzmB6ummqn8YxAiCZ
  API_MONGOID_HOSTS: "localhost:27017"
  MANAGER_HOST: "http://127.0.0.1:3001"
  HARVESTER_CACHING_ENABLED: false
  AIRBRAKE_API_KEY: "anc123"
  LINK_CHECKING_ENABLED: "true"
  LINKCHECKER_RECIPIENTS: "test@test.com"
  WORKER_KEY: YOUR_WORKER_KEY

staging:
  API_HOST: "http://127.0.0.1:4000"
  HARVESTER_API_KEY: sd-vbuMeyfywzXaVaFqu
  MANAGER_HOST: "http://manager.bcdltestbed.ca"
  HARVESTER_CACHING_ENABLED: true
  AIRBRAKE_API_KEY: "anc123"
  LINKCHECKER_EMAIL: "linkchecker@harvester.org"
  LINK_CHECKING_ENABLED: "true"
  HOST: 'worker.harvester.org'
  SECRET_TOKEN: 'some long hash value'
  DEVISE_MAILER: "info@harvester.org"
  LINKCHECKER_RECIPIENTS: "bill@harvester.org"
  API_MONGOID_HOSTS: "localhost:27017"
  WORKER_KEY: ade6ffdb44133c1efe3fc1e73560fd1649b561eef4e620e25b26757f095f36f244a2f042640264fcd89f1853236842b8a3559e0870c1c1600437fc2d44766b62
  #WORKER_KEY: 36214098349d084e31e80c21b1e4a82657d80e0fd853982f0da4d7d783041fae42329cd41647cef5e94a6d3ebb2b0bb13e9413c44a7340d540a84ed65dbaae77

production:
  API_HOST: "http://api.bcdltestbed.ca"
  MANAGER_HOST: "http://manager.bcdltestbed.ca"
  HARVESTER_CACHING_ENABLED: true
  AIRBRAKE_API_KEY: "anc123"
  LINKCHECKER_EMAIL: "linkchecker@harvester.org"
  LINK_CHECKING_ENABLED: "true"
  SMTP_ADDRESS: "your SMTP address"
  HOST: "worker.harvester.org"
  SECRET_TOKEN: 'some long hash value'
  DEVISE_MAILER: "info@harvester.org"
  LINKCHECKER_RECIPIENTS: "bill@harvester.org"
  API_MONGOID_HOSTS: "localhost:27017"
  HARVESTER_API_KEY: LKisxbcTwynxxd3ssyvn
  WORKER_KEY: 7fdc382fc0bee62752f8cab38a0a04dbfb6c4535bdaf585a55e7eace560d64aeebf5f495edb628e35e739bedb0432580ddef23356e4bbf091acbb278b8a16111 
    #36214098349d084e31e80c21b1e4a82657d80e0fd853982f0da4d7d783041fae42329cd41647cef5e94a6d3ebb2b0bb13e9413c44a7340d540a84ed65dbaae77

/api/config/application.yml

defaults: &defaults
  SOLR_PING: 'http://localhost:8982/solr/admin/ping'
  HTTP_HOST: 'http://localhost:3000'
  WORKER_API_URL: 'http://localhost:3002'
  WWW_DOMAIN: 'www.dev'
  DEVISE_MAILER: 'mailer@example.com'
  RESQUE_USER: 'admin'
  RESQUE_PASS: 'password'
  HARVESTER_IPS: '127.0.0.1'
  REQUEST_LIMIT_MAILER: 'develop@digitalnz.org'
  BLACKLIST_MAILER_TO: 'blacklist@mailer.com'
  BLACKLIST_MAILER_FROM: 'mailer@blacklist.org'
  HTTP_HOST: 'http://test.host'
  MONGOID_HOSTS: 'localhost:27017'
  SECRET_TOKEN: '4a843fb4303933c16c1aacb057f976546315b620'

development:
  <<: *defaults

test:
  <<: *defaults
  SOLR_PING: 'http://test.host'
  HTTP_HOST: 'http://test.host'
  WWW_DOMAIN: 'test.host'

staging:
  <<: *defaults

production:
  <<: *defaults

If the preview using the staging worker, I'm assuming I need to run a staging worker (a staging api too?) sorry . . .coming from your docker/dev builds this is new to me :)

Thanks again !

[richardmatthewsdev]

Hi @telezoic,

Sorry if that was confusing, the preview will use the worker of the RAILS_ENV that you are running the manager in. For instance if you run the manager in staging mode it will use the staging worker for previews, otherwise if you run it in production mode it will use the production worker for previews.

One other thing is that when a harvest runs the Worker will pull the parser script from the Manager over active resource. If you go into the rails console on the worker app, bundle exec rails c. Can you find the parser your are attempting to run? You should be able to do something like this Parser.find(<id-of-your-parser'). Where you can get the id of the parser from the manager on it's parser page.

Thanks,
Richard

[telezoic]

Thanks for your help with this @reinkaos!

I found the parser with bundle exec rails c production

I took a closer look at my application.yml files, made some adjustments, and restarted everything.
I can now successfully harvest and see the previews with a variety of parsers.

The one (or two) things I can't do however: are suppress a collection or suppress/delete a record in the relevant sections in the manager.

When I try and suppress a collection the collection status page remains blank below the 'suppressed collections" section. And the record update status page tells me my action is successful, but the record remains (even after a manual re-index).

I'm trying to trace how these events take place in the hopes it will help me troubleshoot this issue. Do you have any suggestions :) ?

Thanks,

Dan

[richardmatthewsdev]

Hi @telezoic,

I'm glad that your harvest is working now.

For the suppress collection the manager makes a request to the API on the route /harvester/sources/:source_id to update the source on the API side.
Sources then get excluded by Solr which you can see on the API if you do a request like so:

http://<your-api>/records.json?api_key=<your-api-key>&debug=true

At the bottom of the page you will see the Solr query and it should have something like this in it:

solr_request_params: {
fq: [
"type:Record",
"record_type_i:0",
"-source_id_s:<your-source-id>",
null
],

This does not require a commit from Solr.

For the specific record, the manager makes a request to /harvester/records/:record_id on the API and will update the status. Records then have a call back on them after they have been saved which will remove the record from the index if it the record is not active. Solr will then need a commit for the changes to take affect.

I would have a look at the API logs and see if there are any errors first. The API needs to have the same sources as the manager so if they do not exist on the API then that would be another place to look. If you resave the source on the Manager it will be posted back to the API.

Hope that helps,
Richard

[telezoic]

Hi Richard,

This is super helpful! Thanks again.

There were no sources on the api side.

from mongo:

switched to db supplejack_api_production
> show collections
partners
preview_records
preview_sequences
record_interactions
records
sequences
supplejack_api_request_metrics
users

And my manager gave me a updating sources page error

	"We're sorry something went wrong "

And the manager log, showed me these errors:

app/models/partner.rb:26:in `block in update_apis'
app/models/partner.rb:23:in `each'
app/models/partner.rb:23:in `update_apis'
app/models/source.rb:28:in `update_apis'
app/controllers/sources_controller.rb:44:in `update'

I swapped the API_HOST variable in app/models/partner.rb

from:
RestClient.post("#{env['API_HOST']}/harvester/partners", . . .

to:
RestClient.post("http://127.0.0.1:8190/harvester/partners", . . .

and I can now suppress collections without difficulty!

My individual record suppression/deletion is still problematic however.

My solr install is set to autocommit.

     <autoCommit> 
        <!--<maxTime>60000</maxTime> -->
       <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
       <openSearcher>false</openSearcher> 
     </autoCommit>

I can update the record status and have the page confirm the record has been updated, and from the manager log it looks fine

method=GET path=/production/collection_records format=html controller=CollectionRecordsController action=index status=200 duration=60.70 view=5.84 params={"utf8"=>"✓", "commit"=>"Search Record", "environment"=>"production"} time=2019-09-05 21:04:47 +0000
method=PUT path=/production/collection_records/5 format=html controller=CollectionRecordsController action=update status=302 duration=106.62 view=0.00 location=http://manager.bcdltestbed.ca/production/collection_records?id=5 params={"utf8"=>"✓", "_method"=>"put", "authenticity_token"=>"r3/Pabbek4bXUkHvwGDuwN+lzSRLPuXusg+4G0p31AbI7wDPeGNpzva6vKyLEGFd07Nf8/gM3/Je/1hGAvOX+w==", "status"=>"deleted", "environment"=>"production"} time=2019-09-05 21:04:52 +0000
method=GET path=/production/collection_records format=html controller=CollectionRecordsController action=index status=200 duration=70.61 view=6.99 params={"environment"=>"production"} time=2019-09-05 21:04:52 +0000

and in the api log:

D, [2019-09-05T21:04:52.583825 #25152] DEBUG -- : [b9acd6b5-9f3e-4a0c-8a00-02abc5ba9a70] MONGODB | 127.0.0.1:27017 | supplejack_api_production.update | STARTED | {"update"=>"records", "ordered"=>true, "lsid"=>{"id"=><BSON::Binary:0x47232225922860 type=uuid data=0x1e20b7f7f4f3462c...>}, "updates"=>[{"q"=>{"_id"=>BSON::ObjectId('5d5f813b2a041820f874fc80')}, "u"=>{"$set"=>{"status"=>"deleted", "updated_at"=>2019-...
D, [2019-09-05T21:04:52.594271 #25152] DEBUG -- : [b9acd6b5-9f3e-4a0c-8a00-02abc5ba9a70] MONGODB | 127.0.0.1:27017 | supplejack_api_production.update | SUCCEEDED | 0.010s
I, [2019-09-05T21:04:52.612972 #25152]  INFO -- : [b9acd6b5-9f3e-4a0c-8a00-02abc5ba9a70] Completed 204 No Content in 76ms

But here's the part that's really throwing me. In an open search, my deleted record is visible

http://api.bcdltestbed.ca/records.json?api_key=LKisxbcTwynxxd3ssyvn

...      {
        "internal_identifier": "https://www.inaturalist.org/observations/10181692",
        "created_at": "19/23/08",
        "updated_at": "2019-09-05T21:04:52.582Z",
        "record_id": 5,
        "title": "Horse Mussel",
        "description": null,
        "display_collection": null,
        "display_content_partner": [
          "South Pacific Surprise!"
        ],
        "category": [
          "image"

          ...

but if I target the individual record: http://api.bcdltestbed.ca/records/5.json?api_key=LKisxbcTwynxxd3ssyvn

I get

	{
  "errors": "Record with ID 5 was not found"
  	}

Naturally I'd like to see the record gone from the open search :)

I'm thinking I'm missing a step, but I'm not sure what it is . . . something with solr?

I'd be grateful for any wisdom you could share.

Thanks,

D.

[richardmatthewsdev]

Hey @telezoic,

My suspicion is that you aren't running the cron which reindexes the records, can you confirm that you are running this cronjob? We run the cron every minute.

bundle exec rails runner -e production 'SupplejackApi::IndexRemainingRecordsInQueue.perform_async'

Also instead of removing the API HOST environment variable, just change the value in the application.yaml :).

Thanks,
Richard