error 13: "not authorized for update on <database>.<collection>" after rs.stepDown()

[leemhenson]

I have seen this happen on two 2.4.9 replica sets, and a development 2.6 replica set. All of 3 nodes each with authentication enabled:

Moped::Errors::OperationFailure - The operation: #<Moped::Protocol::Command
  @length=104
  @request_id=21
  @response_to=0
  @op_code=2004
  @flags=[]
  @full_collection_name="<database>.$cmd"
  @skip=0
  @limit=-1
  @selector={:getlasterror=>1, "w"=>"majority"}
  @fields=nil>
failed with error 13: "not authorized for update on <database>.<collection>"

See https://github.com/mongodb/mongo/blob/master/docs/errors.md
for details about this error.:
/Users/leemhenson/src/forks/moped/lib/moped/operation/read.rb:50:in `block in execute'
  /Users/leemhenson/src/forks/moped/lib/moped/node.rb:594:in `block (2 levels) in flush'
  /Users/leemhenson/src/forks/moped/lib/moped/node.rb:593:in `block in flush'
  /Users/leemhenson/src/forks/moped/lib/moped/node.rb:617:in `block in logging'
  activesupport (4.1.0) lib/active_support/notifications.rb:159:in `block in instrument'
  activesupport (4.1.0) lib/active_support/notifications/instrumenter.rb:20:in `instrument'
  activesupport (4.1.0) lib/active_support/notifications.rb:159:in `instrument'
  /Users/leemhenson/src/forks/moped/lib/moped/instrumentable.rb:31:in `instrument'
  /Users/leemhenson/src/forks/moped/lib/moped/node.rb:616:in `logging'
  ... app stack ...
  /Users/leemhenson/src/forks/moped/lib/moped/node.rb:587:in `flush'
  /Users/leemhenson/src/forks/moped/lib/moped/node.rb:358:in `pipeline'
  /Users/leemhenson/src/forks/moped/lib/moped/operation/write.rb:47:in `execute'
  /Users/leemhenson/src/forks/moped/lib/moped/node.rb:665:in `write'
  /Users/leemhenson/src/forks/moped/lib/moped/node.rb:502:in `update'
  /Users/leemhenson/src/forks/moped/lib/moped/query.rb:427:in `block in update'
  /Users/leemhenson/src/forks/moped/lib/moped/cluster.rb:243:in `block in with_primary'
  /Users/leemhenson/src/forks/moped/lib/moped/node.rb:204:in `block in ensure_primary'
  /Users/leemhenson/src/forks/moped/lib/moped/executable.rb:25:in `execute'
  /Users/leemhenson/src/forks/moped/lib/moped/node.rb:203:in `ensure_primary'
  /Users/leemhenson/src/forks/moped/lib/moped/cluster.rb:242:in `with_primary'
  /Users/leemhenson/src/forks/moped/lib/moped/query.rb:426:in `update'
   () Users/leemhenson/.gem/ruby/2.1.1/bundler/gems/mongoid-dba1c17b1dba/lib/mongoid/query_cache.rb:117:in `update_with_clear_cache'
   () Users/leemhenson/.gem/ruby/2.1.1/bundler/gems/mongoid-dba1c17b1dba/lib/mongoid/persistable.rb:209:in `persist_atomic_operations'
   () Users/leemhenson/.gem/ruby/2.1.1/bundler/gems/mongoid-dba1c17b1dba/lib/mongoid/persistable.rb:192:in `persist_or_delay_atomic_operation'
   () Users/leemhenson/.gem/ruby/2.1.1/bundler/gems/mongoid-dba1c17b1dba/lib/mongoid/persistable.rb:146:in `prepare_atomic_operation'
   () Users/leemhenson/.gem/ruby/2.1.1/bundler/gems/mongoid-dba1c17b1dba/lib/mongoid/persistable/settable.rb:23:in `set'

To repro:

• boot a rails app
• click around the site a bit to warm up the connection pool
• rs.stepDown() the primary
• load a page

Mongoid: 4.0.0.beta1 ref dba1c17b1dba667e50347ab616aaf6f2e1bee094
Moped: 2.0.0.rc1 ref a44b4e1

Restarting the app fixes the problem, so presumably moped isn't re-auth-ing or is discarding credentials or something. I tried to figure out a way to simulate this in the spec suite but I struggled to get over the first hurdle of correctly setting the MONGOHQ envvars to run the full suite against my local replica set.

[arthurnn]

closing this as #324 was merged and it should solve this issue