Unmarshalling errors due to DesignDocumentViewsMapReduce.Map schema assumptions

[jchunkins]

Describe the bug

Calls to CloudantV1.GetDesignDocument and CloudantV1.GetDesignDocumentWithContext will fail because the model struct DesignDocumentViewsMapReduce does not take into account other possibilities that can be generated through valid cloudant index creation. To be clear, the struct structure that is causing the problem is DesignDocument.Views.map[<view name>]DesignDocumentViewsMapReduce.Map.

To illustrate the issue, here's a design document that was created as a byproduct of using the Query Indexes to create an index:

{
 "id": "_design/428a2f2506db4e77001bfe08fa0b79ea9eaf0279",
 "key": "_design/428a2f2506db4e77001bfe08fa0b79ea9eaf0279",
 "value": {
  "rev": "1-a3d9d6e8096d729c0b258dd96ab59fa8"
 },
 "doc": {
  "_id": "_design/428a2f2506db4e77001bfe08fa0b79ea9eaf0279",
  "_rev": "1-a3d9d6e8096d729c0b258dd96ab59fa8",
  "language": "query",
  "views": {
   "foo-json-index": {
    "map": {
     "fields": {
      "foo": "asc"
     },
     "partial_filter_selector": {}
    },
    "reduce": "_count",
    "options": {
     "def": {
      "fields": [
       "foo"
      ]
     }
    }
   }
  }
 }
}

Note that the doc.language is a query (as opposed to some other query server implementation like javascript) and the doc.views.foo-json-index.map value contains an object rather than a string as the DesignDocumentViewsMapReduce model expects. This results in the following unmarshalling error:

error unmarshalling cloudantv1.DesignDocument: error unmarshalling property 'views' as map[string]cloudantv1.DesignDocumentViewsMapReduce: error unmarshalling property 'map': json: cannot unmarshal object into Go value of type string

The code fails when the core.UnmarshalModel function is called:

core.UnmarshalModel(rawResponse, "", &result, UnmarshalDesignDocument)

To Reproduce

1. In the Cloudant web browser, click the Design Documents + button and then chose Query Indexes.
2. Enter the following JSON:

{
  "index": {
    "fields": [
      "foo"
    ]
  },
  "name": "foo-json-index",
  "type": "json"
}

3. Click Create Index
4. Look through the design documents and find the design document that was created as a result of creating the index
5. Use the CloudantV1.GetDesignDocument and CloudantV1.GetDesignDocumentWithContext against the resulting design document.

Expected behavior

I expected that the unmarshalling would succeed by taking into account the type of query server used. Presumably this would require custom unmarshalling logic to handle this scenario.

As a workaround, if there was set of functions to allow access to the raw data such as the following examples:

• A way to get the content without unmarshalling
  • GetDesignDocumentAsStream
  • GetDesignDocumentAsStreamWithContext
• A way to create design documents that don't necessarily conform to the DesignDocument schema. Maybe accept a map or just a raw string body?
  • PutDesignDocumentWithMap
  • PutDesignDocumentWithMapWithContext

Screenshots

Must gather (please complete the following information):

• SDK Version [e.g. 1.2.1]: v0.7.7
• Go Version [e.g. Go 1.21]: 1.22.1
• Name of service that you're trying to use (if applicable)
• Name of operation that you're trying to invoke (if applicable)

Additional context

I was not able to see another way to handle this aside from rolling my own functions to handle design documents of the type described.

[ricellis]

Thanks for reporting we'll look into it.

[ricellis]

A way to get the content without unmarshalling

It should be possible to use PostDesignDocs with IncludeDocs to get the design document content as a Document where the extra properties (like views) are not unmarshalled to a specific type. You can supply a Key if you want a single design document.

If Document is not suitable and you really wanted the raw content you could use PostAllDocsAsStream.

A way to create design documents that don't necessarily conform to the DesignDocument schema. Maybe accept a map or just a raw string body?

Similarly it is possible to write via PostBulkDocs.

All that said the expectation is that "language": "query" type design documents are written by the system using the PostIndex endpoint. It is unusual to be writing those design documents "by hand" and I wouldn't recommend it. Do you have a specific use case that requires this?

[jchunkins]

@ricellis Thanks for the followup.

My motivation here is to follow the recommended approaches for handling indexes (and since a design document "backs" the index, I was trying to follow best practices for design documents). According to https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-design-document-management we need to manage design documents carefully. One of the suggested approaches is “move and switch” approach which consists of:

1. Create a duplicate copy of the design document that you want to change, for example by adding _OLD to its name: _design/fetch_OLD.
2. Put the new or “incoming” design document into the database by using a name with the suffix _NEW: _design/fetch_NEW.
3. Query the fetch_NEW view to ensure that it starts to build.
4. Poll the _active_tasks endpoint and wait until the index is finished building.
5. Put a duplicate copy of the new design document into _design/fetch.
6. Delete design document _design/fetch_NEW.
7. Delete design document _design/fetch_OLD.

I think I can still follow the spirit of "move and switch" using the "higher level" index API instead, and make sure to reference the design doc ID for each call.

[ricellis]

For now #514 has documented a known issue for this: https://github.com/IBM/cloudant-go-sdk/blob/main/KNOWN_ISSUES.md#query-language-design-documents