Building NT 11 app to listen for DocuSign events

[jking]

Hello all,

     I recently started using DocuSign.  I can create/upload PDF documents and use DocuSign to send them to recipients for signature, etc.  A feature of my account is called DocuSign Custom Connect.  It allows me to define forms/envelopes and wait for certain events on the DocuSign server side, like a recipient completing and signing the form.  DocuSign Connect then sends a XML file to a "listener app" via https.  This XML file has all the info about the form and sender/recipient.  It also contains the completed and signed PDF document (base 64 encoded) and any form fields with entered data.  I'm looking for some direction on how to create the "listener app" with NT 11.  Once I have this in place I'll work on parsing the XML for the data I need.  My initial thought is to use one of the example NT 11 apps and modify it.  Maybe there is one that someone can suggest?  Or maybe suggest a different approach?

Thank you

Jeff King

[DonRidley]

According to the DocuSign docs at:

https://developers.docusign.com/overview

"The DocuSign eSignature API lets you eSign documents, request signatures, automate your forms and data, and much more. You can integrate the eSignature REST and SOAP APIs into any app, website, or embedded system that can make https requests."

Don

[Bruce]

Hi Jeff,

>> DocuSign Connect then sends a XML file to a "listener app" via https.

This technique is sometimes called a "web hook". Basically they'll act as a _Client_ to your _web server_ and send you a "request" when something is available.

>> I'm looking for some direction on how to create the "listener app" with NT 11.

So, on the WebServer side you create a NetWebServiceMethod procedure. You can use the Wizard to create a base API app, and then just add the WebServiceMethod to that. Usually you can call it anything you like (they'll let you specify the URL on theor side.) They call this, you receive it and process it.
(You also send a reply, but oftentimes that's either trivial or blank.)

>> This XML file has all the info about the form and sender/recipient.  It also contains the completed and signed PDF document (base 64 encoded) and any form fields with entered data.

If the xml is "simple" then it may be possible to just add the various bits of it as parameters to the WebServiceMethod.If the XML is too complicated for that then you can access the "whole xml" as
p_web.GetValue('xml')
and just parse it out yourself.

Cheers
Bruce

[osquiabro]

"This technique is sometimes called a "web hook". Basically they'll act as a _Client_ to your _web server_ and send you a "request" when something is available."

My telegram example is a web hook and a good example for begin..

[jking]

Bruce,

     Thanks!  I have made some progress.  I started with NT Example app 77 as my web hook listener app.  I now get the XML that DocuSign sends.  See the attached image. 
     Now I would like to get the PDF document embedded in the XML, see near the end of the attached image.  My thought is I need to use GetValue to load the tag PDFBytes into a string theory object and then do a "decode", then finally write the decoded data to a PDF file.  Does this sound correct?  One problem I'm having is deciding if I need to use Start or Init before calling GetValue.
     In addition, I'll need to get a lot of tags (not seen in image) that represent form data that is also sent in the xml.  In this case I may create a queue or file to load this data into.  I'm going through the xfiles docs but wanted to see if you could provide some samples to get me going.

Thanks,

Jeff

[jking]

Bruce,

     I'm experimenting with some code, trying to get the PDF attachment from the XML:

          st.SetValue( p_web.GetValue('xml') )
          st.SaveFile( 'docusign.xml' )

          st.Between('<PDFBytes>', '</PDFBytes>')
          st.base64 = 1
          st.Base64Decode()
          st.SaveFile('clinhistTEST.pdf')

     The docusign.xml is created.  I next try to get the PDF data between <PDFBytes> and </PDFBytes>.  The Docusign docs indicate this data is base 64 encoded.  Assuming it needs to be decoded, I use Base64Decode then save the decoded string to a file.  Upon loading the PDF in Acrobat, I get an error message, saying file unsupported or damaged.  Any thoughts where I went wrong?

Thanks,

Jeff

[Jane]

Jeff,
Maybe look at your PDF with a text editor.  (Or hex editor).

I typed in the first little bit of your string into a couple of online base-64 decoders, and they do indicate it's a PDF.  But I didn't have enough to see more than the header.

I put

JVBERi0xLjUKJfv8/f4KJVdyaXRpbmcgb2JqZWN0cy4

into https://base64.guru/converter/decode
and got:
%PDF-1.5
%樺詰
%Writing objects.

This one: https://www.dcode.fr/base-64-encoding
returned
%PDF-1.5
%ûüýþ
%Writing objects.

Again, looking at what's actually in the file you saved would be instructive.
Then look at a working PDF in a text editor for comparison and see what clues you glean.

Jane

[jking]

Jane,

     Thanks for taking a look at this.  I uploaded the PDF data to the first link but it could not handle the size of the data I think.  The second link did decode all the data.  I next loaded the PDF data into a HEX editor along with an original version of the PDF.  There are some differences in the header and in data throughout, when comparing the original with that produced by DocuSign in the XML.  I really don't know what to look for so I have attached the XML file should you like to take a look.

note:  the Forum does not allow uploads of XML files, so I changed the extension to .txt.

Thanks,

Jeff

[Jane]

Jeff,

I took 3 minutes to build a little test app.

First, I extracted the <PDFBytes> from your attachment and saved it as jeff.txt

Then I did this in the app:

DECODE              routine
  st.loadfile('.\jeff.txt')
  st.Base64Decode()
!  message(st.getvalue())
  st.savefile('.\jeff.pdf')

The attached is what resulted.

So it appears to me that you're receiving correct data.  I'd suggest putting some debug statements into the code where you extract the PDFBytes part of the XML and process it to see where you're going wrong.

Cheers,

Jane

[jking]

Jane,

     Thanks!  That is the form I expected to see.  In my code I use st.between(,) to extract the PDF data.  I assume you did this manually and then created the jeff.txt file.  I'll test your app on my side and then look at my original code to see if I can determine what the problem is.

Jeff

[Jane]

Just a guess.

Maybe try

 st.Between('<<PDFBytes>', '<</PDFBytes>')

[jking]

Jane,

     I thought that was going to be the fix, but nope!  Turns out that when using st.Between(), a string is returned.  It does not "update" the current st object.  So, I added another object, st2.  Here is the working code:

    st.loadfile('.\docusign.xml')   
    st2.SetValue(st.Between('<PDFBytes>', '</PDFBytes>'))
    st2.Base64Decode()
    st2.SaveFile('.\clinhistTEST.pdf')

    Also works with FindBetween().  Anyway, it is working now.  Thanks for your help.

Jeff

[Jane]

Glad you got it sorted!

[jking]

Bruce,

     I have the Listener/Web Hook app working pretty well.  I have the following code in the Processed Code embed of my NetWebServiceMethod procedure:

 st.SetValue( p_web.GetValue('xml') )
 st.SaveFile( '.\docusign.xml' )

 !get form name
 st2.SetValue( st.Between( '<DocumentPDF><Name>', '</Name>') )
 FormName = st2.FileNameOnly( st2.GetValue(), 0 )

 !get and save PDF
 st2.SetValue( st.Between( '<PDFBytes>', '</PDFBytes>') )
 st2.Base64Decode()
 st2.SaveFile( '.\'&Clip(FormName)&'.pdf' )

 !get and save XFDF
 st2.SetValue( st.Between( '<Data>', '</Data>') )
 st2.Base64Decode()
 st2.SaveFile( '.\'&Clip(FormName)&'.xfdf' )

     I currently save the incoming XML file to disk as well as the PDF and XFDF files I extract from the XML.  In production, we could receive hundreds of XML files per day.  I don't see the need to save the XML, PDF and XFDF to disk if all this is fast enough.  The XFDF file will contain form field data that I will be sending to target TPS files.  So, I worry about how fast things will be.  I don't want subsequent XML file processing to interrupt any that may still be running.  I suspect that each incoming XML file is on a separate thread and if so this should not be a concern.  Can you verify that each will come in on it's own thread?

Thanks,

Jeff 

[Bruce]

>> Can you verify that each will come in on it's own thread?

yes.

>> So, I worry about how fast things will be.

Ultimately I suppose it depends on the TPS Write speed of your disk.
I'm guessing the whole thing probably happens in less than a tenth of a second.
So you can probably get 10 of these a second without the machine even raising a sweat.

cheers
Bruce