NetWebClient .Start .Post -- how to safely post rapidly to API server (mailgun)

[RichCPT]

I need to process many records from a work file and turn them into email messages, using NetWebClient, and POST them to mailgun.com.  The number of emails (POSTS) I am talking about could be up to 2,000 at a time.

So, that I have complete control over how rapidly the POST's will occur, I have coded a Generic window with my own looping and timer settings. But I need to know:
   1. How quickly (or slowly) should I be POSTing to the mailgun server?
   2. Do I need to wait for a netTalk event before moving onto sending the next email?  Which event?
   3. Do I need to pause after sending a certain amount of emails and wait for that amount of responses to come back or timeout?
   4. Do I use the same NetWebClient object for every POST and do I call NetWebClient.start() before initializing the fields for each post?
   5. If #4 is "NO" then do I use a limited queue of NetWebClient objects, where each POST will be using an entry from the queue and that queue entry won't be re-used until a response is received or it times out?

[Bruce]

Hi Rich,

>> So, that I have complete control over how rapidly the POST's will occur, I have coded a Generic window with my own looping and timer settings. But I need to know:

Firstly - if you've used a timer you're doing it wrong
Noting about the NetWebclient requires a timer event, and if you are feeling you need one then there's something missing in your implementation. (Actually that's not 100% true - you would need a timer in the case where you needed to _slow down_ the sending because of rate limits on the other end.)

>>    1. How quickly (or slowly) should I be POSTing to the mailgun server?

In pure time - NetTalk doesn't care. You can send one message after the other as fast as you like - as long as you wait for one to be completed before starting the next.

>>    2. Do I need to wait for a netTalk event before moving onto sending the next email?  Which event?

You would send the next mail out the PAGERECEIVED method. In other words;
a) you POST
b) the post completes by calling either ERRORTRAP or PAGERECEIVED
c) in both (?) places you can move onto sending the next POST.

>>    3. Do I need to pause after sending a certain amount of emails and wait for that amount of responses to come back or timeout?

yes. You pause after 1. In other words send one - that one completes - send the next one.
(but see Note 1 below)

>>   4. Do I use the same NetWebClient object for every POST

Yes (but see Note 1)

and do I call NetWebClient.start() before initializing the fields for each post?

you can. No harm in doing that (although since you are sending the same thing to the same server it's probably not necessary in this case.)

>>    5. If #4 is "NO" then do I use a limited queue of NetWebClient objects, where each POST will be using an entry from the queue and that queue entry won't be re-used until a response is received or it times out?


Note 1:
Ok, bear in mind that these POSTS will likely go fast - probably very fast. I would expect at least 2 per second - so about 120 per minute - so even 1000 would be done inside 10 minutes. But that may be too slow.

In that case you can get a bit fancier. An example of this is the way the NetMaps (netMaps.Inc and NetMaps.clw) are coded.
Basically, you're making one request at a time "per object". In NetMaps there are multiple objects, and so multiple requests and multiple responses all happening at the same time.

So by declaring say 10 objects in the procedure, all 10 can be "working" at the same time - start with 10 POSTS, then as each completed in turn it grabs the next item in the queue, and so on. That way you'd see a 10 fold increase in time - so you could do thousands per second.

HOWEVER some services rate-limit. Meaning that _they_ have rules as to how fast, and how many, you can send. So it's good to research those limits first before going too far down this road.

cheers
Bruce

[RichCPT]

Thanks.

The reason for the timer, while perhaps contradictory to this post's subject title, is to allow the process of sending emails to run on a background window/thread within my application.  I want the user to be able to continue doing other things in my Clarion application.  Eventually, I will probably pull this code out of the main application and let it run at the server and I will want it to be a well behaved server process and not dominate the CPU usage when it sends email.

[Bruce]

Hi Rich,

I don't think you'll need to slow it down if it's running on another thread.
Threads in the program run on different cores, so it's unlikely the CPU usage will even overlap.
And even if it does the CPU usage is tiny compared to the time delays created by the networking.

cheers
Bruce