Show Posts
This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.
46
Web Server - Ask For Help / Re: Problem with Mandrill Webhooks and large POSTs
« on: July 10, 2020, 03:57:06 AM »
Hi Bruce,
I traced it through into the NetWebServerWorkerBase._UnEscape procedure.
Specifically the "Check for Hex" section.
Given in my example there are 213374 hex codes that need translation, its not unsurprising it takes some time.
I've tweaked the procedure to use StringTheory's Replace function and in my example its reduced the duration from 360 seconds to 50 seconds.
Frankly, I think that will get me out of trouble.
I also downloaded NT11.39 to ensure nothing has been changed in NT since I last looked, and that code is identical to NT10.
Regards
Bill
I traced it through into the NetWebServerWorkerBase._UnEscape procedure.
Specifically the "Check for Hex" section.
Given in my example there are 213374 hex codes that need translation, its not unsurprising it takes some time.
I've tweaked the procedure to use StringTheory's Replace function and in my example its reduced the duration from 360 seconds to 50 seconds.
Frankly, I think that will get me out of trouble.
I also downloaded NT11.39 to ensure nothing has been changed in NT since I last looked, and that code is identical to NT10.
Regards
Bill
47
Web Server - Ask For Help / Re: Problem with Mandrill Webhooks and large POSTs
« on: July 10, 2020, 01:48:36 AM »
Hi Bruce,
Thanks for taking a look.
I realised (after posting the example) I probably have access to your code to look for whats causing it.
Like all things once its reproducible its practically solved, and at least it is reproducible here!
I'll let you know how I go.
Regards
Bill
Thanks for taking a look.
I realised (after posting the example) I probably have access to your code to look for whats causing it.
Like all things once its reproducible its practically solved, and at least it is reproducible here!
I'll let you know how I go.
Regards
Bill
48
Web Server - Ask For Help / Problem with Mandrill Webhooks and large POSTs
« on: July 09, 2020, 04:54:37 AM »
Hi All (Probably Bruce),
I've got a tricky problem.
We use Mandrill to transport large volumes of emails roughly 200k emails per month.
Mandrill has a webhook where it POSTs back into our system some events in json (but URL encoded).
Nothing weird so far.
If the post data is large, say > 1Mb of json data (URL Encoded) the web server maxes out a CPU (and will continue to do so if more POSTs arrive). This does not happen if the POSTs are small (kilobytes, not megabytes).
I think if we wait long enough (hours) the thread may complete what its doing, but Mandrill has given up waiting for a response.
Important observations are:
1. Happens in NT10.40 & Clarion 10
2. Happens in NT11.30 & Clarion 11
3. Doesn't get to the NetWebPage code that is waiting to process the data (does get there if the payload is small).
4. If Ii simulate the POST via Postman it fails in the same manner as a POST direct from Mandrill.
5. If I decode the URL encoding and simulate the POST via Mandrill everything works as expected.
I suspect the built-in URL decoding system in NetTalk is not happy with the obscene amount of POST data that is requiring decoding > 1Mb.
I've attached a cut down APP that will hopefully compile (C11/NT11.30 - you might have to cut out some missing icons etc). But its just an empty web server with 1 Proc called MandrillHook (which doesn't even execute if the payload is large).
More importantly I've included a postman collection with the original post as it arrives from Mandrill that fails and another post that does the same thing but as i've already URL decoded the payload (and replaced & with + just to make sure it behaves the same as the first post) it succeeds.
Any help would be appreciated!
Regards
Bill
I've got a tricky problem.
We use Mandrill to transport large volumes of emails roughly 200k emails per month.
Mandrill has a webhook where it POSTs back into our system some events in json (but URL encoded).
Nothing weird so far.
If the post data is large, say > 1Mb of json data (URL Encoded) the web server maxes out a CPU (and will continue to do so if more POSTs arrive). This does not happen if the POSTs are small (kilobytes, not megabytes).
I think if we wait long enough (hours) the thread may complete what its doing, but Mandrill has given up waiting for a response.
Important observations are:
1. Happens in NT10.40 & Clarion 10
2. Happens in NT11.30 & Clarion 11
3. Doesn't get to the NetWebPage code that is waiting to process the data (does get there if the payload is small).
4. If Ii simulate the POST via Postman it fails in the same manner as a POST direct from Mandrill.
5. If I decode the URL encoding and simulate the POST via Mandrill everything works as expected.
I suspect the built-in URL decoding system in NetTalk is not happy with the obscene amount of POST data that is requiring decoding > 1Mb.
I've attached a cut down APP that will hopefully compile (C11/NT11.30 - you might have to cut out some missing icons etc). But its just an empty web server with 1 Proc called MandrillHook (which doesn't even execute if the payload is large).
More importantly I've included a postman collection with the original post as it arrives from Mandrill that fails and another post that does the same thing but as i've already URL decoded the payload (and replaced & with + just to make sure it behaves the same as the first post) it succeeds.
Any help would be appreciated!
Regards
Bill
49
Web Server - Ask For Help / Re: HL7 communication with Nettalk - possible?
« on: July 03, 2020, 11:01:48 PM »
Hi Wolfgang,
Given the HL7 standard is OSI-Level 7 (ie Application) its independent of transport. Just like xml and json describe data but are also independent of transport method.
HL7 v2 works like lots of very old systems that use a record identifier at the beginning of each line that in turn defined the fields that would follow on that line. Think back to mainframe systems in the 70s and 80s.
HL7 v3 is based on an xml style syntax.
Your ability to interopt with other systems via Nettalk is likely very true, NT can do pretty much anything.
But, just like sharing xml or json with other parties, where the data needs to be secure, some methodology to ensure security would be required. This doesn't appear to be part of the HL7 messaging standards. Not unsurprising given its pitched at the ISO-7 level.
Regards
Bill
Given the HL7 standard is OSI-Level 7 (ie Application) its independent of transport. Just like xml and json describe data but are also independent of transport method.
HL7 v2 works like lots of very old systems that use a record identifier at the beginning of each line that in turn defined the fields that would follow on that line. Think back to mainframe systems in the 70s and 80s.
HL7 v3 is based on an xml style syntax.
Your ability to interopt with other systems via Nettalk is likely very true, NT can do pretty much anything.
But, just like sharing xml or json with other parties, where the data needs to be secure, some methodology to ensure security would be required. This doesn't appear to be part of the HL7 messaging standards. Not unsurprising given its pitched at the ISO-7 level.
Regards
Bill
50
Web Server - Ask For Help / Re: Remote Desktop Connection not working for one EC2 Instance
« on: June 14, 2020, 10:57:35 PM »
cool 
51
Web Server - Ask For Help / Re: Remote Desktop Connection not working for one EC2 Instance
« on: June 14, 2020, 09:42:31 PM »
Hi Richard,
I've seen that happen, once in my AWS. I ended up killing the instance and restarting and it started behaving again.
Regards
Bill
I've seen that happen, once in my AWS. I ended up killing the instance and restarting and it started behaving again.
Regards
Bill
52
Web Server - Ask For Help / Re: Remote Desktop Connection not working for one EC2 Instance
« on: June 14, 2020, 08:56:27 PM »
Hi Richard,
Have you setup inbound rules in your security group(s) for RDP?
As RDP is not safe, you normally lock it down to your know client IP, but pretty sure you still need some inbound rule for port 3389.
Regards
Bill
Have you setup inbound rules in your security group(s) for RDP?
As RDP is not safe, you normally lock it down to your know client IP, but pretty sure you still need some inbound rule for port 3389.
Regards
Bill
53
Web Server - Ask For Help / Re: Facial Recognition template
« on: June 11, 2020, 03:56:23 AM »
Hi Jason,
I've used Karios (https://www.kairos.com/) for Facial Recognition. Its a hosted solution (they also have an on-premises solution). Its not free, but we found it reasonably priced.
API is very easy to use.
Regards
Bill
I've used Karios (https://www.kairos.com/) for Facial Recognition. Its a hosted solution (they also have an on-premises solution). Its not free, but we found it reasonably priced.
API is very easy to use.
Regards
Bill
54
Web Server - Ask For Help / Re: Bootstrap and NT?
« on: June 09, 2020, 03:48:08 AM »
Sort of, but why? Bootstrap can do forms but compared with NT its very basic. It just does the layout and client side validation. So no ajax. NT is all about ajax, you get to write server side code in Clarion. If you want to do server side validation using "plain" Bootstrap you need to write client side javascript and then some server side endpoints to help out. Its a total pain (I know as I've had to do it).
If you like the Bootstrap visual layout, you can combine this (mostly) into NT, and frankly that is easier than bootstrap forms. They are ok where no (or very very little server side stuff is required), think of a contact form or similar.
If your motivation for bootstrap forms is visual (including responsive layout) and your forms are complicated (server side validation), i'd recommend NT with bootstrap CSS.
If your forms are simple, merge your server side data into Bootstrap HTML and have it post back to a NetTalk Page that knows how to process it (pretty easy).
But, I don't think you can get the best of both, without some pain.
If you like the Bootstrap visual layout, you can combine this (mostly) into NT, and frankly that is easier than bootstrap forms. They are ok where no (or very very little server side stuff is required), think of a contact form or similar.
If your motivation for bootstrap forms is visual (including responsive layout) and your forms are complicated (server side validation), i'd recommend NT with bootstrap CSS.
If your forms are simple, merge your server side data into Bootstrap HTML and have it post back to a NetTalk Page that knows how to process it (pretty easy).
But, I don't think you can get the best of both, without some pain.
55
Web Server - Ask For Help / Re: Twilio Webhook for Receiving SMS?
« on: May 24, 2020, 03:03:23 PM »
Hi Jeff,
SMS providers are a dim a dozen. We support about 6 (mostly domestic Australian providers) and let our clients choose which one they want to use. Twilio is good for international SMS, in situations where the bulk of your SMS could end up in any country. As they claim a higher percent of delivery for some of the more difficult countries compared with local providers (this probably matter more Outside the US).
Regards
Bill
SMS providers are a dim a dozen. We support about 6 (mostly domestic Australian providers) and let our clients choose which one they want to use. Twilio is good for international SMS, in situations where the bulk of your SMS could end up in any country. As they claim a higher percent of delivery for some of the more difficult countries compared with local providers (this probably matter more Outside the US).
Regards
Bill
56
Web Server - Ask For Help / Re: Twilio Webhook for Receiving SMS?
« on: May 22, 2020, 11:02:17 PM »
Hi,
I've done Twilio Verify and Twilio Chat, but not SMS. We use other suppliers for SMS.
I usually read their code examples to try and distil it down to what is required from a RAW API call point of view, many of the examples for other languages assume a library they supply that does much of the work, and we don't get that.
Regards
Bill
I've done Twilio Verify and Twilio Chat, but not SMS. We use other suppliers for SMS.
I usually read their code examples to try and distil it down to what is required from a RAW API call point of view, many of the examples for other languages assume a library they supply that does much of the work, and we don't get that.
Regards
Bill
57
Web Server - Ask For Help / Re: Working with CSS and Server Restart
« on: May 01, 2020, 02:45:02 PM »
Hi,
I turn off "Enable File Cache" while i'm actively developing. This means I dont have to restart the server to clear the file cache.
You also either have to turn off "Combine Common Files" to remove the need to re-combine and compress resources (styles & scripts) each time you make a change OR you can add your own .CSS and .JS files into NT, just remember it is ok for us in development to have the press Ctrl-F5 but this is not satisfactory when in production (you can't tell hundreds or thousands of customers to do this). In this case a common technique is to place a version or random number as part of the css or js filename.
In my example move.js?v=2.29 as shown in the attachment.
As long as I change the version all clients browsers will reload the file.
Regards
Bill
I turn off "Enable File Cache" while i'm actively developing. This means I dont have to restart the server to clear the file cache.
You also either have to turn off "Combine Common Files" to remove the need to re-combine and compress resources (styles & scripts) each time you make a change OR you can add your own .CSS and .JS files into NT, just remember it is ok for us in development to have the press Ctrl-F5 but this is not satisfactory when in production (you can't tell hundreds or thousands of customers to do this). In this case a common technique is to place a version or random number as part of the css or js filename.
In my example move.js?v=2.29 as shown in the attachment.
As long as I change the version all clients browsers will reload the file.
Regards
Bill
58
Web Server - Ask For Help / Re: Third party certificates - PEM encoded options
« on: February 13, 2020, 04:43:00 AM »
Hi Casey,
Use https://www.ssllabs.com/ssltest to verify you have it correctly.
Nettalk usually requires the certificate plus chain but not root (some domain registry's may require the root).
But ssllabs will tell you exactly, so no guess work.
Regards
Bill
Use https://www.ssllabs.com/ssltest to verify you have it correctly.
Nettalk usually requires the certificate plus chain but not root (some domain registry's may require the root).
But ssllabs will tell you exactly, so no guess work.
Regards
Bill
59
Web Server - Ask For Help / Re: Address autocomplete service
« on: January 18, 2020, 07:57:34 PM »
Hi Johan,
HERE also have one.
https://developer.here.com/documentation/geocoder-autocomplete/dev_guide/topics/example-location-id.html
I've used both. They are simple. From memory HERE is cheaper in scale.
Depending upon exactly what you want, there are other slightly different end points like /places etc.
Regards
Bill
PS. I have examples in Postman. My source code depends on other tools i've written so I cannot just pull out the code.
HERE also have one.
https://developer.here.com/documentation/geocoder-autocomplete/dev_guide/topics/example-location-id.html
I've used both. They are simple. From memory HERE is cheaper in scale.
Depending upon exactly what you want, there are other slightly different end points like /places etc.
Regards
Bill
PS. I have examples in Postman. My source code depends on other tools i've written so I cannot just pull out the code.
60
Web Server - Ask For Help / Re: Running two different Secure apps from the one server
« on: January 02, 2020, 05:35:02 PM »
Hi Richard,
You can have multiple external IP address on AWS if you use an Elastic IP as well as the original external IP. I haven't tried it, but i expect you could use multiple elastic IPs and point them all to the same box.
Also, you could use ELB (Load Balancer) It will allows you to have various external IPs (elastic IPs are best) and you can then map them to one EC2 instance on various ports. The downside is you'll need to use AWS to handle your certificates (as NT will only need to do http). This is called off-loading of SSL and allows NT to run faster by getting someone else to do the cpu intensive SSL encryption. But, frankly its a bit fiddly and you'll be paying AWS to do something you could probably do yourself.
Back to the first option, you should just be able to run two services on your EC2 instance and bind them to the appropriate IP. I've not personally tried to bind a server to an Elastic IP, but it should work. You cannot have two internal IPs on AWS (to my knowledge).
Regards
Bill
You can have multiple external IP address on AWS if you use an Elastic IP as well as the original external IP. I haven't tried it, but i expect you could use multiple elastic IPs and point them all to the same box.
Also, you could use ELB (Load Balancer) It will allows you to have various external IPs (elastic IPs are best) and you can then map them to one EC2 instance on various ports. The downside is you'll need to use AWS to handle your certificates (as NT will only need to do http). This is called off-loading of SSL and allows NT to run faster by getting someone else to do the cpu intensive SSL encryption. But, frankly its a bit fiddly and you'll be paying AWS to do something you could probably do yourself.
Back to the first option, you should just be able to run two services on your EC2 instance and bind them to the appropriate IP. I've not personally tried to bind a server to an Elastic IP, but it should work. You cannot have two internal IPs on AWS (to my knowledge).
Regards
Bill