NetTalk Central
NetTalk Web Server => Web Server - Ask For Help => Topic started by: Richard I on March 31, 2018, 06:22:01 PM
-
Hi ,
I have 6 netTalk apps (v 8.71) using Topspeed drivers only, running on a Windows 10 PC
They have been running continuously since Jan 2017 without any "leakage issues "until I loaded Self Service V3.49, this year and ran them as Services.
The problem was first discovered with slowness and the task manager reporting that the Utilization reached 80% and the RAM used was 7.2 from available 7.9 G, after about a month of continuous use
Since then the RAM is slowly creeping up from 3.7 and is now at 4.2 from 7.9 available since the last shut down about a fortnight ago. Utilization is not climbing and is steady about 5%
I have checked the performance tab on the app - nothing extraordinary.
Nothing has changed in the apps except the addition of Self Service
Changes to the PC.....
The Client has had installed and running as services the following :
StorageCraft Shadow Protect SPX
Team Viewer
Sophos Antivirus
Webroot Endpoint protection
Tight VNC Server
Windows 10 automatic Updates have been disabled.
This PC is a dedicated PC to my Software and is not part of any LAN
IT are saying that there is a potential bug in the Windows 10 build that could be causing Leakage.
Is there any Utility I can run that will report the source of the Leakage?
or any help/ suggestions as to how to sort ?
Turning my apps off as a test is not possible until the end of the University year.
thanks
Richard c10
-
Richard, i am having the same issues a few months back and i am unable to pinpoint the problem yet. Maybe Bruce can help us identify the problem.
-
Well thats good and bad news.
Good that someone else has the same problem but bad in that it might be NT ( which I doubt)
I neglected to mention in my earlier thread that IT have also installed Webroot Endpoint Security as well as Sophos Endpoint
Also need to include that savservice .exe which is part of Sophos appears to have a healthy appetite for resources....174652 K in fact.
check this link
SavService.exe is using high amount of CPU on servers - Sophos Endpoint Software - Endpoint Security and Control - Sophos Community
Cheers
Richard
-
Hi Richard,
FWIW, we run NetTalk apps 7.40 and 10.15 as services (selfservice 3.49) on Windows Server 2003, 2008 and 2012 without a memory leak.
Our EXEs RAM usage (as reported by TaskMgr) normally doesn't exceed 2-3GB under heavy load (per EXE, anymore is dangerous).
We us ODBC database, no TPS, so thats a point of difference.
We run no anti-virus on our servers (intentionally).
I know this doesn't help, but i'd check infrastructure before I checked code. Unless you are up to some creative memory allocation within your apps.
Regards
Bill
-
Are this flag is on in your application?
-
Hi Oggy, yes its ticked ON ....?
Personally, I think Sophos is sucking it dry with 174652 k being used and running.....
Thanks
Richard
-
In my limited experience I thought having multiple antivirus software suites running caused conflict between them?
-
Hi forgot to mention that following clients comments that the Web browser apps were timing out after periods of inactivity, I wound the Timeout time up from the default 15 minutes to 2 hours
This has stopped the timeout, but am I choking with multiple sessions?
Richard
-
Hi Richard,
Potentially. I have also seen over the years misbehaving anti-virus and anti-threat software.
Over the years we have refused to sell/install our software to people using Microsoft Security Essentials and a particular version of Nortons, due to their faults.
If you image how it works, they have to intercept disk I/O and network I/O and inspect the contents. Which causes the obvious performance degradation (why I don't use it) but also if it has any faults you can get weird problems.
If its practical, i'd disable both of them for a time and check the memory consumption. This could be difficult if it takes weeks to manifest or if your infrastructure is dictated by someone else.
Regards
Bill
-
Hi Richard,
It will depend upon the number of users and the size of your session data.
My guess would be no. We run up to thousands of concurrent users and the session data is not a problem (and we are a bit session data heavy, plus 2hr timeouts).
We do have a problems with threaded data, as each thread, while it is being processed, creates a complete copy of all your threaded variables (which includes the whole database record buffers).
This is only really a problem with large databases and high volume 50+ pages served per second. If the database is small it will do many hundreds of pages a second.
Regards
Bill
-
Hi Bill,
Thanks,
yes, I have encountered issues with Microsoft security Essentials too (AVAST as well)and have in fact code to prevent my software from running if MSE is found
The IT company administer the PC so its difficult for me to dictate terms, but, I just might sneak in there and disable for a 24 hour period and see what happens !
the web server has an option to change session ID on logout so its not that.
Cheers
Richard
-
Hi Richard,
It sounded like the OS RAM usage was climbing. I am wondering if your EXEs memory usage is also climbing in step.
TaskMgr is notoriously misleading when it comes to memory usage, but it is indicative.
Your EXE really cannot show more than about 3GB (in our experience in TaskMgr) before Windows kills it (even with the 4GB ticked in NT).
If you could identify what app belongs to the memory usage, you'd get a better idea. I often use SysInternals ProcExp to drill into my apps where i can see threads and memory usage in more detail.
Regards
Bill
-
Thanks Bill
I have two University clients
1.Auckland, running on a provided Windows10 PC and being managed by the companies IT contractor.
This is the PC that is showing a build up of Physical memory over time, peaking before shut down at 7.2 from 7.9 G RAM. and 80% utilization
This PC has the installed programs Webroot Endpoint, Sophos Endpoint, Shadow Protect, TightVNC TeamViewer and my program, with 8 NT exes running
The NT exes handle about 7000- 8000 insert records a day over 3 two hour meal slots.
2.Wellington University is being hosted by be on a XP PRO PC in my office running the same Net Talk version but with only one antivirus, 360 total protect.
The number of daily NT records is about the same.
On the XP machine, I am also running SysInternals Process explorer and the System information window is displaying that the memory usage is constant, at 1.2 GB over days, and the CPU usage while working, oscillates between 0 and 14% always returning to 0
From the comparison of the two machines and given that the NT traffic and the NT versions are the same, can I then suggest to Auckland IT that my NT program are not the cause of the Leakage and that perhaps the the security exes are?
Regards
Richard
-
Hi Richard,
It does look indicative, however, we'd need to know:
1. Who is using all the RAM (i rebooted a Win10 box here and its base RAM usage was a bit over 3GB, so if your app chews up 3+GB the memory consumption is plausible)
2. What processes are using 80% of the CPU?
It is possible for a single NT Thread to use up a complete CPU (core not socket).
It is also possible for this to occur more than once.
If this happens less and less power is available for other requests, you process the requests more slowly. There are more concurrent requests being handled. Memory usage spirals out of control and the system will eventually crash - i've done it :(
So, I'd check, what processes are using the CPU (the CPU is definately a solid clue)
If it your APP, check via ProcExp what threads are running and if any threads have been running for a very long time (other than the base APP thread). This can be done in a production environment.
If security software is misbehaving, it will normally be the one wasting CPU or memory, not your app.
Regards
Bill
-
Thanks Bill,
Ill bite the bullet and have the antivirus exes turned off
reboot and see what happens.
As I have said , in 2017 no problems , no doubling of antiV software or remote connection apps but , No Self Service either !!!
Thanks for your time
Cheers
Richard
-
Bill-
Guess what !!
"Sophos has recently identified an issue where, in certain environments, a memory leak can occur in the HTTP proxy on the UTM. A number of causes have been identified and these have been fixed in the v9.005 release, which will improve the stability of the HTTP proxy for many affected customers. However, we have not yet identified and fixed all possible causes, but investigating and fixing these is a high priority."
Cheers
R
-
Great news!
-
Richard,
You are reporting that the overall usage of memory on the machine is climbing.
but Task Manager shows you which processes in the machine are using what memory.
So from there it should be simple to understand which process is consuming the memory.
If it's the web server, then obviously one has to look at that - if it's another process then one loos at that and so on.
cheers
Bruce
-
Hi Bruce:
In my case, when this happens (and lately is very often), happens to the web server process. Lots of CPU and RAM consumed and even sits the server.
In my case (after reading the posts here), i did remove the option of use more than 4gb of ram and resource usage is better, but i don't know if this is gonna impact my clients at the long run.
Even one of my test was in a server without any Antivirus software installed (new server deployed for the server), and before the 4gm checkmark removed it doesn't even let the server work. OS was Windows Server 2016
All the test was using NetTalk ver. 10.19
-
hi Ura,
There are no known leaks in the NetTalk code, but that of course does not prove they are not there. There have been a spate of memory leak reports lately, but so far none have turned out to be NetTalk code - usually it's something in the app itself.
There is a rather serious leak in the Clarion ODBC driver, introduced in the C10 build 12799.
It's been reported to SV (PTSS 42658) but despite a long thread on the newsgroups SV have been very silent on the issue. Feel free to lobby them about that if you like... <g>.
I think I need to describe it as a "memory leak in H5 apps" before they'll get around to fixing it <g>.
cheers
Bruce
-
Cool. Thank God i did not use ODBC nor H5 :D
Bruce, any suggestions to track or pinpoint possible resource usage problems?
-
>> any suggestions to track or pinpoint possible resource usage problems?
Finding memory leaks is tricky. Programs can "use" memory and that's ok. A "leak" are cases where it used memory, but ended up with stuff stored in memory that it won't use again.
It's like you mom coming to your room as a teenager and deciding which stuff you have is valuable to you, and which you'll never use again. To her it's all just "stuff".
TaskManager can help to keep an eye on memory usage. You can't be "exact" about it, but you can keep an eye on it to spot trends.
The WebStrain client app is also useful for being able to repeat the same calls over and over thousands of times. This helps isolate which calls are leaking memory. (Just be sure to think about session cookies - a web stress tester that doesn't preserve the session will use memory on each call - which will only come back when the session is deleted.)
Ideally you want to find some call which when called a zillion times causes the me usage in the app to grow. Maybe it's a page with some embed code, or a report, or a web service method. If you can narrow it down to 1 call then that's a good start.
Then you start commenting out embed code in that procedure and so on.
It's a long and tedious process, and ultimately helps if you have a bit of luck. It took Chuck weeks to narrow the cause down to the ODBC driver.
Cheers
Bruce