Server unresponsive after massive requests

[jlavera]

A rest server with NT 11.15.
If the server receives a lot of requests at once, it becomes unresponsive. It keeps receiving the packets, but seems like it cannot deliver the answers any more. It is not hanged, it does not gpf, but I don´t know what is it doing. There is no cpu usage. Al debug is active and no error is reported, but... it dies.
Server just have a bunch of methods to read records and deliver them with json. When I do that manually from a browser, all of them work perfect.
By the way, no memory files, no global queues, no semaphores, logout ot anything that can be causing a deadly embrace.
Any suggestion on what to change or look at?

[peterH]

Are all of the requests valid?
IOW, could it be that you receive a request that the server don’t know how to handle?

Peter

[Jane]

Do you know how high the Threads count is  on the Performance tab when it hangs?

Not helpful to your finding the cause, but as an aside...

I have several NT webserver/API service apps running as services.  They also contain netclient procedures running on separate threads that connect to an outside system.

Occasionally, one of my services will hang. 

In two of those services I've included a timer window (running on its own thread) that writes an "I'm alive" heartbeat timestamp to an INI file every minute or so.

I have another service running on that machine that checks the INI files from the services it's monitoring.
If there's no heartbeat for a set amount of time (4 minutes or whatever), then it uses OddJob to kill the stalled process then stop and restart the service. 

This has been solid for me for the past 8 or 9 months.  The watchdog service writes a text log to disk and I can see what it's done.  My NetTalk services also write a text log to disk showing startup timestamps. 
Between the two services it monitors, the watchdog log shows that it has done 7 restarts since the beginning of March. 

May not be the best solution, but has saved me some headaches.

Jane

[jlavera]

Peter:

Apparently yes, they are valid requests.  Anyway, the server shouldn't hang when faced with a lot of garbage, shouldn't it? Isn't there any way to just reject an invalid request with a FU answer, block the IP for a while, or something alike?

[jlavera]

Jane: thank you for your suggestion, it is a good solution that I used in the past. I'm in a need to try to avoid the hanging, this time, as it happens too often. I had the hope to find a way to prevent the hanging, instead of acting after it. Thanks!

[jlavera]

Taking about banning, I just saw the banning documentation section. It is actually not what I want to do (I shoulnd't ban a legitic client), but I see this mentioned: "Banning is not the same as Rate Limiting, which simply applies a standard of "fair use", Rate Limiting will be discussed elsewhere. "
Howhever, i couldn't stll find where is it that "elsewhere".

[Bruce]

Hi Jorge,

>> If the server receives a lot of requests at once, it becomes unresponsive.

We're going to need to talk numbers here.
a) If you look on the "performance" tab, what sort of numbers are we looking at? Especially number of threads, number of connections, and so on.

b) what backend are you using? (I've had reports of problems with ODBC drivers to MySQL and so on where it's the backend that dies causing the frontend to die.

>> It keeps receiving the packets, but seems like it cannot deliver the answers any more. It is not hanged, it does not gpf, but I don´t know what is it doing. There is no cpu usage. Al debug is active and no error is reported, but... it dies.

but you can still change tabs and so on? or has the UI died as well?

>> Are all of the requests valid?

Should make no difference. "Completely invalid" requests are very fast to deal with and typically won't be an issue. The only ones that would make a difference would be valid API calls which allow the user to collect vast amounts of data (like generate a sales analysis report for the last 10 years etc.)

>> They also contain netclient procedures running on separate threads that connect to an outside system.
>> Occasionally, one of my services will hang. 

I found an issue (probably fixed in 11.16) that would affect web clients - so it's worth using that.

>> I'm in a need to try to avoid the hanging, this time, as it happens too often.

Often is "good" in the sense that it's easier (and faster) to get to the root of the problem and apply a fix. Problems that happen once a month are wicked to work on.

>> I had the hope to find a way to prevent the hanging, instead of acting after it.

yep, it's always better to attack the root of the problem, and not just "patch" it.

>> Rate Limiting will be discussed elsewhere. However, i couldn't still find where is it that "elsewhere".

I don't think I've written up an article on that yet.

cheers
Bruce

[jlavera]

Hi Bruce,

>> We're going to need to talk numbers here.
a) If you look on the "performance" tab, what sort of numbers are we looking at? Especially number of threads, number of connections, and so on.

It was a service, so I run it as an exe and found a couple of methods that were not closing connections immediately. I fixed that.

>> b) what backend are you using? (I've had reports of problems with ODBC drivers to MySQL and so on where it's the backend that dies causing the frontend to die.

Topspeed files.

but you can still change tabs and so on? or has the UI died as well?

Can't tell... it was as a service.

>> They also contain netclient procedures running on separate threads that connect to an outside system.
>> Occasionally, one of my services will hang. 

>> I found an issue (probably fixed in 11.16) that would affect web clients - so it's worth using that.

Now I recompiled with 11.16 and the hangs went away; but I actually changed so many things, that I'm not sure if this was the cause.

>>> I had the hope to find a way to prevent the hanging, instead of acting after it.
>> yep, it's always better to attack the root of the problem, and not just "patch" it.

>>> Rate Limiting will be discussed elsewhere. However, i couldn't still find where is it that "elsewhere".
>>I don't think I've written up an article on that yet.

Well, something to look forward to.
Thanks!
Kind regards,