Issue using blank file_type in :upload_file

[baserz]

Hi!
It seems that sepafm do not accept empty file_types for the :upload_file operation.
Since DanskeBank utilizes blank file_type as a valid format we have a need to get this working.

Example of their formats below:

UMTL Global Transfer (SWIFT MT101). – only used in Finland
Must be blank EDIFACT 96 A format: PAYMUL
Must be blank EDIFACT 96A format: DIRDEB
Must be blank Danske Bank commaseparated format: Payment files
CMBO,CMUO,CMGB,CMBODK,CMSI,CMNI,CMFI,CMDE,CMGBDC,CMII,
CMUOGB,CMIIDC,CMUOIE,CMUO10,CMUOSE,CMUOFI,CMUODE,CMUONO
Must be blank Danske Bank commaseparated format: Collection files
COLLECTION,COLDEBMOD,COLDELETE

The error i receive when setting file_type: "" is "File type needs to be present and under 35 characters"

I did a check into your source and confirmed that in your code the following check is done:

def check_file_type
      if file_type.present?
        valid = file_type.size < 35
      else
        return if bank == :op && %i(download_file download_file_list).include?(command)

Tried inserting both "" and " " as file_type but i guess that it requires some form of content as it is built now? Do you have a workaround (Like som special notation if empty type) to get this working in current version? Or is this an issue that needs to be fixed? Thanks a lot in advance for your assistance!

[baserz]

I tried overriding this check by always returning true and if i do that the request goes through but i receive schema validation failed in the response. My guess is that this is caused by the <file_type></file_type> is skipped if the parameter is empty (danske require that the tag exists but is empty), but i havent had the time to verify this yet. But i just wanted to lift that adjusting the check_file_type method is not enough in itself.

[jonikanerva]

Thanks for the report!

Haven't yet tried this out myself, but I think Nokogiri saves empty tags by default as <file_type/> instead of <file_type ></file_type>. This might cause problems in the Danske schema validation.

This behaviour can be changed with the SaveOptions. Could you try saving the xml with save_with: Nokogiri::XML::Node::SaveOptions::NO_EMPTY_TAGS? Maybe that could help.

[baserz]

Sounds like that could very well be the issue. Will see if i can try this workaround during the day. Just have to locate where to add this option (i guess should be set somewhere in soap_builder.rb or soap_danske.rb?).

[jonikanerva]

You can pass the save_with -option to the to_xml -method as a parameter. So maybe changing this to

@application_request.to_xml save_with: Nokogiri::XML::Node::SaveOptions::NO_EMPTY_TAGS

could work..

I'll probably have some time this evening to look into this as well.

I think we could remove the requirement that upload_file needs to have the file type set. Then it would be up to the user to make sure they are creating requests that are accepted by the banks.

[baserz]

Just wanted to mention that i tried the above suggestion and it seems like i wasnt enough to get it running. Still seems to get <ResponseCode>12</ResponseCode><ResponseText>Schema validation error.</ResponseText>. I also think i got failed, not error, before the nokogiri adjustment. Now response.valid? seems to return false.

To summarize, there still seems to be some issue with the request xml format.

[baserz]

@jonikanerva did you find the time to look at this?

[baserz]

I did some more debugging on this and it looks like that when applying NO_EMPTY_TAGS it still works as long as a file type is actually set. NO_EMPTY_TAGS does however make CanonicalizationMethod non self closing as well so it affects other parts as well.
When running this with an empty filetype the xml looks ok, seen to that the filetype is no longer self closing. But still get XML validation error. This makes me suspect the real villain in this issue is hiding elsewhere in the code (perhaps related to digest / signature)? Since everything looks ok when debugging the to_xml method but it still refuses to work.

By the way, i also tried just doing a .sub("<FileType/>", "<FileType></FileType>") as well after the to_xml but that didnt work either.

[jonikanerva]

Thank you for the debugging effort.

This can be a hard problem to solve without the help of Danske Bank. Solving this by trial and error could take a long time, as the soap request must be identical to the version Danske creates on their end. The soap request is canonicalized, which should in theory create identical files.

Did Danske provide you with an example of a working application request for uploading this file type? Or are you able to manually generate a valid request file? This would make debugging easier.

[baserz]

No i have not gotten a raw xml sample request from them, but as long as we enter some form of filetype (no matter what it is), it works in the the sense as we get a 00. And according to a rep for Danske that i talked to, it should work fine as long as we blank the file type for the targeted files. Which is the state in which we now reside.

I could talk to Danske again and ask for this and if he can see in more detail what goes wrong when sending blank type (not sure he can see that).

But the reason i was thinking about if it could be an issue with the digest / signature was the following:

If i set the parameter file_type: 'TEST' for example it works. BUT if i set parameter file_type: 'TESTABC' and then i edit the to_xml method as below (the only thing i actually add is the .sub) it will return the same schema validation error:

def to_xml
      set_common_nodes
      set_nodes_contents
      process_signature
      xmlReq = @application_request.to_xml save_with: Nokogiri::XML::Node::SaveOptions::NO_EMPTY_TAGS
      xmlReq = xmlReq.sub('TESTABC', 'TEST')
      xmlReq
    end

Havent checked this much deeper so far but as seen in the method above, process_signature is run befor turning the application_request to an XML string. This method both generates a digest of the request (INCLUDING the filetype tag) and then creates signature based on this. So my thought was that since it seems to digest the whole tag, maybe this causes the issue when assigning empty file_type value and using NO_EMPTY_TAG as well?
So that generating a non-self-closing file_type tag makes the signature / digest no longer match the payload -> causing a xml schema validation error? (this is just a wild guess based on what i see in the code, but i wanted to mention it).

[jonikanerva]

Yes, you are correct! That was not the correct place to change the XML options. It should be done in the initializer.

If you modify the request after you've calculated the signature (as you do above), you should get an error. That is what the signature is for. It guarantees that the file content is not modified after it's signed.

[baserz]

[Realized this post could be hard to grasp so i rewrote it]
Great then i got that part correctly.

Editing the file_type value in the to_xml method is of course just something i ran to demonstrate a potential problemsource, it is NOT something ive done during my other debugging. The below code shows the actual to_xml method i use when i debug atm:

def to_xml
      set_common_nodes
      set_nodes_contents
      process_signature
      @application_request.to_xml save_with: Nokogiri::XML::Node::SaveOptions::NO_EMPTY_TAGS
    end

But the reason wanted to point this out is that it seems that the digest is constructed of the whole tags, not just the values?

This makes me wonder if this isnt the cause of the error i get, something goes wrong here when i use a blank filetype which makes the digest differ from the actual payload. Could be a result of the change of @application_request.to_xml to @application_request.to_xml save_with: Nokogiri::XML::Node::SaveOptions::NO_EMPTY_TAGS or it would occur nevertheless (since, without this tag, the xml get self-closed its hard to tell).

For example if the file_type tag will be formatted differently in the digest compared to the actual payload.

As i said above, not sure about this, just speculating at this point (but it seems like the most likely cause for this since DanskeBank atleast claimed that the only thing that should need to change for us is removing the filetype value but keeping the tag itself) :). I hope this is clear enough described.

[jonikanerva]

I just wrote a small test to see how the canonilization affects the file. The canonilized version of the XML does include the empty tags. So the signature is calculated from a file that contains the closing tags. Ie:

<ApplicationRequest xmlns=\"http://bxd.fi/xmldata/\">
  <CustomerId>360817</CustomerId>
  <Command>UploadFile</Command>
  <Timestamp>2016-12-05T14:46:41Z</Timestamp>
  <Environment>TEST</Environment>
  <SoftwareId>Sepa Transfer Library version 1.1.8</SoftwareId>
  <FileType></FileType>
  <Content>YTJsemMyRT0K
</Content>

This should match what Danske is expecting in their end. So I don't think the problem is in the XML representation. It must be something else.

[baserz]

I noticed that too when i debugged more now. Seems like, as you say, the canonilization use correct type tag no matter the to_xml setting.

Anyway i plan to call my contact at danske tomorrow and ask him if he can see anything in their logs that can give more details on the issue than just the response code.

[baserz]

Sorry for the delay @jonikanerva , took a while to get an answer to this, but i have some information now. The information available was limited but the error message i got was as follows:
Schema validation error..http://1............/: cvc-simple-type 1: element {http://bxd.fi/xmldata/}FileType with anonymous type may not be empty..

If Danske runs the file on their side it works fine with empty file type. So it seems like a xml format issue. Might be a matter of explicitly setting a type for the filetype tag?

[baserz]

I think i perhaps found a solution to this (just maybe). After having failed to get it working following a lot of debugging i did a simple hack to remove the tag completely instead (contrary to the information i have received). And then i got an OK. So awaiting confirmation if this actually gave the correct type in the end. If so ill get back with what i did.