Are my netTalk Self Service Apps causing memory leak

[Richard I]

Hi ,
I have 6  netTalk apps (v 8.71) using Topspeed drivers only, running  on a Windows 10 PC
They have been running continuously since Jan 2017 without any "leakage issues "until I  loaded Self Service V3.49, this year and ran them as Services.
The problem was first discovered with slowness and the task manager reporting that the Utilization reached 80% and the RAM used was 7.2 from available 7.9 G, after about a month of continuous use

Since then  the RAM is slowly creeping up from 3.7 and is now at 4.2  from 7.9 available since the last shut down about a fortnight ago. Utilization is not climbing and is steady about 5%
I have  checked the performance tab on the app - nothing extraordinary.

Nothing has changed in the apps except the addition of Self Service

Changes to  the PC.....
The Client has had installed and  running as services the following  :
StorageCraft Shadow Protect SPX
Team Viewer
Sophos Antivirus
Webroot Endpoint protection
Tight VNC Server

Windows 10 automatic Updates  have been disabled.
This PC is a dedicated PC to my Software and is not part of any LAN
 IT are saying that there is a potential bug in the Windows 10 build that could be causing Leakage.


Is there any Utility I can run that will report the source of the Leakage?
or any help/ suggestions as to how to  sort ?
Turning my apps off as a test is not possible until the end of the University year.
thanks
Richard c10

[urayoan]

Richard, i am having the same issues a few months back and i am unable to pinpoint the problem yet. Maybe Bruce can help us identify the problem.

[Richard I]

Well thats good and bad news.
Good that someone else has the same  problem but bad in that it might be NT ( which I doubt)
I neglected to mention in my earlier thread that IT have also installed Webroot Endpoint Security as well as Sophos Endpoint

Also need to include  that  savservice .exe which is part of Sophos appears to have a healthy appetite for  resources....174652 K in fact.
check this link
SavService.exe is using high amount of CPU on servers - Sophos Endpoint Software - Endpoint Security and Control - Sophos Community

Cheers
Richard

[bshields]

Hi Richard,

FWIW, we run NetTalk apps 7.40 and 10.15 as services (selfservice 3.49) on Windows Server 2003, 2008 and 2012 without a memory leak.

Our EXEs RAM usage (as reported by TaskMgr) normally doesn't exceed 2-3GB under heavy load (per EXE, anymore is dangerous).

We us ODBC database, no TPS, so thats a point of difference.

We run no anti-virus on our servers (intentionally).

I know this doesn't help, but i'd check infrastructure before I checked code. Unless you are up to some creative memory allocation within your apps.

Regards
Bill

[oggy]

Are this flag is on in your application?

[Richard I]

Hi Oggy, yes its ticked ON ....?

Personally, I think Sophos is sucking it dry with 174652 k being used and running.....
Thanks
Richard

[Richard I]

In my limited experience I thought having multiple antivirus software suites running caused conflict between them?

[Richard I]

Hi forgot to mention that following clients comments that the Web browser apps were timing out after periods of inactivity, I wound the Timeout time up from the default 15 minutes to 2 hours
This has stopped the timeout, but am I choking with multiple sessions?

Richard

[bshields]

Hi Richard,

Potentially. I have also seen over the years misbehaving anti-virus and anti-threat software.

Over the years we have refused to sell/install our software to people using Microsoft Security Essentials and a particular version of Nortons, due to their faults.

If you image how it works, they have to intercept disk I/O and network I/O and inspect the contents. Which causes the obvious performance degradation (why I don't use it) but also if it has any faults you can get weird problems.

If its practical, i'd disable both of them for a time and check the memory consumption. This could be difficult if it takes weeks to manifest or if your infrastructure is dictated by someone else.

Regards
Bill

[bshields]

Hi Richard,

It will depend upon the number of users and the size of your session data.

My guess would be no. We run up to thousands of concurrent users and the session data is not a problem (and we are a bit session data heavy, plus 2hr timeouts).

We do have a problems with threaded data, as each thread, while it is being processed, creates a complete copy of all your threaded variables (which includes the whole database record buffers).

This is only really a problem with large databases and high volume 50+ pages served per second. If the database is small it will do many hundreds of pages a second.

Regards
Bill

[Richard I]

Hi Bill,
Thanks,
yes, I have encountered  issues with Microsoft security Essentials too (AVAST as well)and have in fact code to prevent my software from running if MSE is found
The IT company administer the PC so its difficult for me to  dictate terms, but, I just might sneak in there and disable for a 24 hour period and see what happens !
the web server has an option to change session ID on logout so its not that.

Cheers
Richard

[bshields]

Hi Richard,

It sounded like the OS RAM usage was climbing. I am wondering if your EXEs memory usage is also climbing in step.

TaskMgr is notoriously misleading when it comes to memory usage, but it is indicative.

Your EXE really cannot show more than about 3GB (in our experience in TaskMgr) before Windows kills it (even with the 4GB ticked in NT).

If you could identify what app belongs to the memory usage, you'd get a better idea. I often use SysInternals ProcExp to drill into my apps where i can see threads and memory usage in more detail.

Regards
Bill

[Richard I]

Thanks Bill
I have two University clients
1.Auckland, running on a provided Windows10 PC and being managed by the companies IT contractor.
This is the PC that is showing a build up of Physical memory over time, peaking before shut down at 7.2  from 7.9 G RAM. and 80% utilization
This PC has the installed programs  Webroot Endpoint, Sophos Endpoint, Shadow Protect, TightVNC TeamViewer and my program, with 8 NT exes running
The NT exes handle about 7000- 8000 insert records a day over 3 two hour meal slots.

2.Wellington University is being hosted by be on a XP PRO PC in my office running the same Net Talk version but with only one antivirus, 360 total protect.
The number of daily NT records is about the same.
On the XP machine, I am also running SysInternals Process explorer and the System information window is displaying that the memory usage is constant, at 1.2 GB over days, and the CPU usage while working, oscillates between  0 and 14% always returning to 0

From the comparison of the two machines and given that the NT traffic and the NT versions are the same,  can I then suggest to Auckland IT that my NT program are not the cause of the Leakage and that perhaps the the security exes are?

Regards
Richard

[bshields]

Hi Richard,

It does look indicative, however, we'd need to know:

1. Who is using all the RAM (i rebooted a Win10 box here and its base RAM usage was a bit over 3GB, so if your app chews up 3+GB the memory consumption is plausible)

2. What processes are using 80% of the CPU?

It is possible for a single NT Thread to use up a complete CPU (core not socket).

It is also possible for this to occur more than once.

If this happens less and less power is available for other requests, you process the requests more slowly. There are more concurrent requests being handled. Memory usage spirals out of control and the system will eventually crash - i've done it

So, I'd check, what processes are using the CPU (the CPU is definately a solid clue)

If it your APP, check via ProcExp what threads are running and if any threads have been running for a very long time (other than the base APP thread). This can be done in a production environment.

If security software is misbehaving, it will normally be the one wasting CPU or memory, not your app.

Regards
Bill

[Richard I]

Thanks Bill,
Ill bite the bullet and have the antivirus exes turned off
reboot and see what happens.
As I have said , in 2017 no problems , no doubling of antiV software or remote connection apps but , No Self Service either !!!
Thanks for your time
Cheers
Richard