NetTalk Central
NetTalk Web Server => Web Server - Ask For Help => Topic started by: lanmicro on March 03, 2013, 10:48:27 AM
-
Hi Guys,
Sometimes afte sitting for a while (usually overnight) the Webserver stops responding. The program is running (this happens both when running as an exe and as a service), but the browser won't connect.
Attached is a tps file with a log of traffic showing the gets and posts. Has this happened to anyone else?
The stuff dated 3/03 11:23 is where no response to browser occurred.
The stuff dated 3/03 11:31 is after the program was stopped and restarted.
NETTALK 6.51 Clarion 6.3
[attachment deleted by admin]
-
Hi Greg,
yeah I've seen this happen a couple times over the years and it's generally a pig to debug,
there are a couple of general ideas in terms of what is happening, but the specifics can vary a lot from app to app.
a) The WebServer procedure is event driven of course. So you mustn't "run" any procedure from this procedure. You can START new threads, but if you have any kind of background routines - like session clear-ups or backups, or file maintainence or whatever, they shouldn't "run" directly on the WebServer thread - rather START them onto their own thread.
b) Any kind of MESSAGE or STOP procedure of course breaks the server. If you're running as a service you can't see these (but they exist). I recommend MessageBox to make sure these can't happen (or use prop:messageHook and prop:StopHook to make sure they can't appear.)
Running the program in Exe mode for a bit to see if anything visual does appear is a good thing to do as well.
c) Keep an eye on Ram and CPU usage. It's possible something is accumulating over time and ultimately overloads the machine.
d) make sure there's no power-saving on the machine, or specifically the network card. Nothing like the machine shutting down the hard disk, or network card to create some fun.
cheers
Bruce
-
Hi Bruce,
No background threads. No messages. I am using messagebox so they would have shown up in the log I sent. When I looked at the task manager I didn't see any memory use jump out at me.
All power saving settings are never.
More info.
Problem signature:
Problem Event Name: APPCRASH
Application Name: MySQLWeb.exe
Application Version: 0.0.0.0
Application Timestamp: 51325203
Fault Module Name: c60runx.dll
Fault Module Version: 0.6.300.9054
Fault Module Timestamp: 44ff2dea
Exception Code: c0000005
Exception Offset: 0006e863
OS Version: 6.1.7601.2.1.0.272.7
Locale ID: 1033
Additional Information 1: cb3d
Additional Information 2: cb3df5922acec54bba6f355078e26882
Additional Information 3: 6cc3
Additional Information 4: 6cc32171d2e6387f3738fb1ed024b320
Read our privacy statement online:
http://go.microsoft.com/fwlink/?linkid=104288&clcid=0x0409
If the online privacy statement is not available, please read our privacy statement offline:
C:\Windows\system32\en-US\erofflps.txt
-
if you're getting an actual GPF then perhaps make use of GPF Reporter, or even the Clarion 8 debug runtime DLL may give you more info.
cheers
Bruce
-
Hi Bruce,
I am using GPF Reporter. I got that message after starting the program with the /ISS switch. Now GPF reporter doesn't restart the program because it tries to restart it with the /ISS switch and the program gets stuck with an operating system error.
Part of that error is the error mesage from my prior posting.
-
The /ISS switch is just used (once) to install the program as a service.
So you don't need to run it with that switch again.
So remove the switch, and report back on the GPF Report.
cheers
Bruce
-
HI Bruce,
If I understand correctly, you run program /iss once. After that you just run program.
This is what I did and when the program attempts to restart it gives the error that the program is already running as a service after the message that I sent above is displayed by the OS.
I have (as a test) now run program /rs, and just program to see if GPF reporter restarts the program in these problem instances. I know GPF reporter works outside of this problem instance because I have a button on my window that creates a GPF and the program restarts just fine.
-
>> If I understand correctly, you run program /iss once.
yes.
>> After that you just run program.
no it's already running.
/ISS stands for Install and Start Service
/IS would be Install Service
If you want to run the program as an Exe you first have to Stop the servcice (using the windows service manager services.msc)
>> This is what I did and when the program attempts to restart it gives the error that the program is already running as a service
as I predicted. <g>
>> I have (as a test) now run program /rs,
don't play with the /RS parameter. Removing services (which is not the same as stopping them) often requires a reboot.
>> and just program to see if GPF reporter restarts the program in these problem instances.
you want to be able to trigger the GPF "at some time" so tie it into some outside thing the program detects. like say a file with a very specific name. (only then remove the file before GPF'ing <g>)
running the program again can't trigger the test.
Or perhaps you're going down a road you don't need to go down. Running the program to crash the service will likely not work.
cheers
Bruce
-
Hi Bruce,
In the services dialog I cannot find my program. In TaskManager I cannot find my program.
However when the error (whatever it is occurs) I get a message saying my program cannot restart because it is already running.
When I do restart my program manually it starts with no problem and I am able to access the website. Until it sits overnight and then the whole thing starts again.
Having looked at the things you've identified before, is there anywhere else to poke at?
-
does the app use SQL?
is the SQL Server on the same machine?
-
Hi Bruce,
The app does use SQL. But not MSSQL, MySQL.
The tests are not done on the same machine as the MySQL server. One was connected over the internet. The other colocated in the same GoDaddy server farm.
-
Hi Greg,
it occurs to me that if the SQL is on another machine, then there may be a "connect" issue between the program and the data. ie Clarion programs typically don't like being "disconnected" from the DB, so maybe there's something going on around that?
One possible test (if you can) is to run the program as an Exe for a period of time and see if
a) that's more stable and
b) if there are any visual messages generated which might give you some clues.
I've started a thread for collected knowledge here;
http://www.nettalkcentral.com/index.php?option=com_smf&Itemid=36&topic=4317.0
When converting to C8 I found a possible hang - in ABC - which I've documented there as well. Not sure if it's the one you have, but maybe.
cheers
Bruce
PS - this thread is long, and this forum software doesn't like me and long threads, so if you find out any more please start a new thread...
-
Hi Bruce,
To make sure that there was no disconnect with the other server with the MySQL database on it I put a heartbeat in a timer on the WebServer window that just accessed the MySQL server every 5 minutes.
I wrote an message out to CSMessage when it happened so that I could see when it failed.
It doesn't fail. Not until I try and access the login the next day. Then the program issues a GPF and restarts but the restarted program does not respond to any messages.
I have to manuall stop the program and restart it before the website responds to any messages. Then I'm good until tomorrow.
Tonight I am running the server program with the /netall parameter and capturing the output in debug. You'll get that tomorrow.
-
It doesn't fail. Not until I try and access the login the next day. Then the program issues a GPF
> I would add trace statements to see where in your login routine it GPF's also paying attention to memory and cpu levels on your server and mysql server
and restarts but the restarted program does not respond to any messages.
> It could be the original program still has a hold on the port when the program restarts and hence the new instance can't listen on the port. In one of your previous post you identified Problem Event Name: APPCRASH Application Name: MySQLWeb.exe - do you think your login is crashing MySQL somehow. Not sure what you are performing on your heartbeat but maybe it needs to replicate the login sequence?
-
The most useful information on a GPF is provided by GPF Reporter. That tells you the line of code that GPF'd and the code leading up to that code. In my experience it's the most effective way to find a GPF.
Cheers
Bruce
-
Hi Bruce,
Here is the GPF report.
Program : H:\6\MySQL\MySQLWeb.exe
Version :
At : 12:53:18 on 2013/03/19
Workstation: : bailey64xp32
User Name: : Gregory Bailey
Reported error : EXCEPTION_ACCESS_VIOLATION - Error reading data at : 0000000Ch
Windows : Win XP - 5.1.2600 Service Pack 3
Clarion : 6.3
Thread : 3 Field : 0 Event : 0 Keycode : 0
Error at address : 0070E715h no debug info, Module=H:\6\MySQL\c60odbx.dll 6.300.9054
Stack Trace
??? 004DD808h Line ?=147 no proc Src=SelfService.Clw
??? 0040726Ch Line=495 Proc=FUNCTIONCALLED_@_FILECALLBACKINTERFACE_@_FILEMANAGER@F21FILECALLBACKINTERFACEl6PARAMSRscRsc Src=ABFILE.CLW
??? 010DE7A0h Line ?=3023 no proc Src=wsl.cpp Module=H:\6\MySQL\c60runx.dll 6.300.9054
??? 004DD808h Line ?=147 no proc Src=SelfService.Clw
??? 004DD808h Line ?=147 no proc Src=SelfService.Clw
??? 00407FFDh Line=943 Proc=TRYFETCH@F11FILEMANAGERBk Src=ABFILE.CLW
??? 004DD808h Line ?=147 no proc Src=SelfService.Clw
[01] 00456F53h Line=1361 Proc=R$VALIDATEUPDATE Src=MySQL017.clw
??? 004D7614h Line ?=147 no proc Src=SelfService.Clw
??? 004D9484h Line ?=147 no proc Src=SelfService.Clw
??? 00457974h Line=350 Proc=R$RESTOREMEM Src=MySQL017.clw
[02] 0046060Ch Line=199 Proc=LOGINFORM@F18NETWEBSERVERWORKERl Src=MySQL017.clw
??? 004D3F70h Line ?=147 no proc Src=SelfService.Clw
[03] 0044F390h Line=53 Proc=CALLFORM@F5P_WEBBfl Src=MySQL013.clw
??? 004D7614h Line ?=147 no proc Src=SelfService.Clw
??? 004D7614h Line ?=147 no proc Src=SelfService.Clw
[04] 0048E165h Line=7774 Proc=VALIDATEFORM@F18NETWEBSERVERWORKERBfBkl Src=NetWeb.Clw
??? 004D7614h Line ?=147 no proc Src=SelfService.Clw
??? 004D7614h Line ?=147 no proc Src=SelfService.Clw
[05] 0048E957h Line=7677 Proc=SAVEFORM@F18NETWEBSERVERWORKERBfBkl Src=NetWeb.Clw
??? 004D7614h Line ?=147 no proc Src=SelfService.Clw
[06] 004AA602h Line=2699 Proc=PROCESSLINK@F18NETWEBSERVERWORKEROsb Src=NetWeb.Clw
??? 004D7614h Line ?=147 no proc Src=SelfService.Clw
??? 004D7614h Line ?=147 no proc Src=SelfService.Clw
[07] 0044F2FEh Line=77 Proc=PROCESSLINK@F5P_WEBOsb Src=MySQL013.clw
[08] 004A5CC0h Line=5879 Proc=_HANDLEPOST@F18NETWEBSERVERWORKER Src=NetWeb.Clw
??? 004D7614h Line ?=147 no proc Src=SelfService.Clw
??? 004E7D30h Line ?=147 no proc Src=SelfService.Clw
[09] 004AC3CAh Line=2503 Proc=PROCESSREQUEST@F18NETWEBSERVERWORKERsb Src=NetWeb.Clw
??? 004D7614h Line ?=147 no proc Src=SelfService.Clw
[10] 0044F50Ah Line=41 Proc=WEBHANDLER@Fsb Src=MySQL013.clw
[11] 01091AB1h no line number no proc Module=H:\6\MySQL\c60runx.dll 6.300.9054
-
Hi Greg,
cool - thanks.
The key starting point in this report is the line prefixed with [1]. The numbers with actual line numbers, [1], [2] and so on are the "definites" - the others are "maybes". so I always start on the more recent definite and work forwards and backwards from there;
[01] 00456F53h Line=1361 Proc=R$VALIDATEUPDATE Src=MySQL017.clw
so first thing to do is look at this line of code - ie right click on the procedure in module 17 and choose "module" (_not_ source).
then go to line 1361. It'll likely be a call to do ValidateUpdate,or possibly it'll be in the ValidateUpdate routine.
That's useful because it tells us where and what the machine was doing when it died. You can start by looking at the code in that routine _especially_ any hand-code you may have added to that routine.
Note also this is the LoginForm procedure - so someone was trying to log in at the time - and it's just possible there was something funny going on. Not terribly likely, but possible. But I'd look first at the embed code in there and see what it says.
Incidentally note that it's likely doing a database interaction at that point - a TRYFETCH - which may bring us back to some sort of file access error. It may be worth collecting a few of these reports to see if they're always in the same place, or if the happen at different places, but always when accessing the database. That gives us quite a few clues.
I also see you're using C6 9054 - you might have more joy with the Clarion runtimes from a later build of C6, I know they did a lot of SQL stuff in the later builds, and that may help - hard to say for sure though.
Cheers
Bruce
-
Hi Bruce,
I added some code there to validate the entries in the logon process.
In an effort to prevent this GPF I have surrounded that code with code that opens and closes the file that is used for validation.
Let you know tomorrow if that is successful.
-
Hi Bruce,
Not successful. Here is my Validate Routine in the LoginForm. This line only GPFs if the program sits overnight. I just let it sit from 5:45Am to 11:45 AM and there was no problem. But last night I let it sit from 9:45PM to 5:45AM and GPF.
ValidateUpdate Routine
do CompleteForm
do ValidateRecord
if p_web.GetValue('loc:hash') = p_web.GetSessionValue('loc:hash')
access:MyStores.Open()
access:MyStores.UseFile()
mysto:NbStore = loc:Login
if access:MyStores.TryFetch( mysto:K_NbStore ) !!!!!!!!!!!!This is the line that is GPFing.
loc:invalid = 'loc:Login'
p_web.SetValue( 'retry', p_web.site.LoginPage )
loc:Alert = 'Login Failed. Try Again.'
p_web.DeleteCookie( 'loc__login' )
p_web.DeleteCookie( 'loc__password' )
else
if upper( loc:password ) = upper( mysto:Password )
p_web.ValidateLogin()
p_web.SSV( 'loc:hash', 0 )
p_web.SetSessionLevel( 1 )
p_web.SSV( 'CurrentStore', mysto:NbStore )
p_web.SSV( 'CurrentStoreSysId', mysto:SysId )
p_web.SSV( 'CurrentStoreName', mysto:Name )
p_web.SSV( 'DateLastUpdated', format( mysto:DtLastUpdated, @d02 ) )
if loc:remember = 1
p_web.SetCookie( 'loc__login', loc:login, today() + 30 )
p_web.SetCookie( 'loc__password', loc:password, today() + 30 )
else
p_web.DeleteCookie('loc__login')
p_web.DeleteCookie('loc__password')
end!if
else
loc:invalid = 'Loc:Login'
p_web.SetValue('retry',p_web.site.LoginPage)
loc:Alert = 'Login Failed. Try Again.'
p_web.DeleteCookie('loc__login')
p_web.DeleteCookie('loc__password')
end!if
end!If
access:MyStores.Close()
Else
p_web.DeleteCookie('loc__login')
p_web.DeleteCookie('loc__password')
End
! delete the session values, so if the user comes back to the form, the items are not set.
p_web.deletesessionvalue('loc:login')
p_web.deletesessionvalue('loc:password')
p_web.deletesessionvalue('loc:remember')
p_web.deletevalue('loc:login')
p_web.deletevalue('loc:password')
p_web.deletevalue('loc:remember')
-
Hi Greg,
that's a normal ABC file access line, and I'm assuming it's a MySQL table, so I'm not sure how much help I'll be. I'm not the SQL expert by any means, and I've never used MySql myself.
Perhaps other can step in and offer suggestions - but it seems like perhaps some sort of SQL error.
You might have some joy using the RTL DLL's from a later build of C6 - I know there were a lot of SQL updates in the builds after 9054.
Cheers
Bruce
-
If the updating of clarion does not work, I would try a a couple of things...
Add the line that GPF's to your heartbeat test
try to rw-write that block of code as Prop:SQL
-
Hi Greg,
In Clarion code using MySQL were it opens the file(s) always GPF'ed for me! I had to use the OMIT() around the the file open code and put it after OPEN(Window) was called. For whatever reason MySql needed a window handle when using odbc according to the techs at MySQL. All I know is that it worked.
BTW my code is almost identical to yours and I cannot get it to validate user name and password, using a tps file for mine. Go figure...mine was generated using the application wizard and does not work. All I done is put the file access code in.
Ashley
-
Hi Kevin,
That code is in my heartbeat code with no problems. The problem only occurs if the application has been up overnight with the first attempt at logging in in the morning.
I Think that I will place code in the program to automatically GPF at midnight (GPF Reporter will auto restart) and see it that resolves the problem.
-
Hi Ashley,
The code does work the first and subsequent times I try and access the website. It only fails if the website has been idle overnight. (which makes it really hard to test).
-
Hi Gregory,
You mention your site being idle some time at night.
Is there a possibility that the mySql server times out in some sort of way?
Rene Simons
-
Hi Rene,
That was my first thought, but the heartbeat code accesses the MySQL database.
Since I am only able to run a single test per day I made 3 changes to my program.
1. Updated it to a later build of c6.
2. Changed my access code to loop next (no ABC) through file to find the correct password (like the heartbeat code )
3. After midnight cause the program to GPF and restart using GPF reporter.
In the morning the website works! Don't know which of the above was the correct solution.
I also had to changed the server 2008 r2 to not display the dialog box "Windows is searching for a solution" so that the old instance of the program would just fail and the new instance would run with no interferance.
-
Hi Greg,
>> In the morning the website works! Don't know which of the above was the correct solution.
I'm sure your client really wants you to make it unstable again, but it might be worth undoing the 3 changes, then trying them one at a time to see which one is helping. It'll be useful information to you, but also to everyone else I think.
The one I'd test first, and the one I think is the most likely is the Clarion 6 upgrade. If they can tolerate a few more nights of pain then narrowing that down for sure will be most interesting.
Cheers
Bruce
-
Hi Bruce,
I spoke too soon. This morning was the first test without the stupid Micorsoft "Searching for a solution" dialog and the program just sat there. Could not connect from my browser. Causing a GPF by the code
poke( 0, x# )
caused the program to restart and start accepting messages from my browser.
So none of the things I tried was successful.
Clarion 9059
NetTalk6.51
-
Some new information.
I am getting an error 08S01 when attempting to access the MySQL server through ODBC.
05D0H(3) 08:48:00.437 Allocating Statement 0EDEF08H on Connection 0EC7050H Time Taken:0.00 secs
05D0H(3) 08:48:00.437 OPEN(Stores:0691B6CH) Time Taken:0.00 secs
05D0H(3) 08:48:00.437 GET_PROPERTY(Stores:0691B6CH) Time Taken:0.00 secs
05D0H(3) 08:48:00.437 SETNULLS(Stores:0691B6CH) Error: Invalid Field Type Descriptor Time Taken:0.00 secs
05D0H(3) 08:48:00.437 SETkeykey(Stores:0691B6CH,mysto_K_NbStore[2]) Time Taken:0.00 secs
05D0H(3) 08:48:00.437 MYSTO:NBSTORE: 720
05D0H(3) 08:48:00.437
05D0H(3) 08:48:00.437 Allocating Statement 0EDDB58H on Connection 0EC7050H Time Taken:0.00 secs
05D0H(3) 08:48:00.437 Setting Concurrency to Read Only for Statement 0EDDB58H Time Taken:0.00 secs
05D0H(3) 08:48:00.437 Setting Cursor Type to Forward Only for Statement 0EDDB58H Time Taken:0.00 secs
05D0H(3) 08:48:00.437 Resetting Parameters Statement 0EDDB58H Time Taken:0.00 secs
05D0H(3) 08:48:00.437 Preparing Statement 0EDDB58H : SELECT SYSID,NAME,NBSTORE,PASSWORD,DTLASTUPDATED FROM Stores WHERE (NBSTORE >= ? ) ORDER BY NBSTORE
05D0H(3) 08:48:00.453 Error Occurred: 08S01 [MySQL][ODBC 5.2(a) Driver][mysqld-5.1.61]MySQL server has gone away
05D0H(3) 08:48:00.453 Time Taken:0.01 secs
05D0H(3) 08:48:00.453 NEXT(Stores:0691B6CH) MySQL server has gone away Time Taken:0.01 secs [font=Verdana]!>> This is strange not 2 seconds earlier the heartbeat code found it[/font]
05D0H(3) 08:48:00.453 MYSTO:SYSID : 0
05D0H(3) 08:48:00.453 MYSTO:NAME : ''
05D0H(3) 08:48:00.453 MYSTO:NBSTORE: 720
05D0H(3) 08:48:00.453 MYSTO:PASSWORD : ''
05D0H(3) 08:48:00.453 MYSTO:DTLASTUPDATED : Invalid date (0)
I have written code to explicitly reopen the file just before attempting to access the record, but that doesn't appear to solve the problem. First access in the morning after midnight the program GPFs on an access to the file restarts, hangs and must be restarted manually.
Any suggestions?
-
Greg,
I understand that this is not your own machine.
Does the client of yours run a BackUp or anything during the night?
Is that server publicly abailable, does it have a static IP-Address or a dynamic one? Thats a wild guess, but maybe the router disconnects during the night to prevent an automatic disconnect by the provider during the office hours. Mine does, albeit I have a static address.
Getting a new IP-Address during the night is less painful (well, in most cases) because at 3am you can expect that not many users are connected. If webserver and DB-server are permanently conncted over a LAN then this will likely have no influence.
just two thoughts....
Wolfgang
-
I might not have anything to add from the MySQL side, but I've noticed with a MSSQL NT Webserver of mine, that it used to appear to not respond after a GPF.
I use GPFReporter to restart the service, but it appeared to restart almost too quickly, and couldn't properly bind to the port that I wanted, because it was still in use.
I spoke to Bruce to see if there was a way to test if the init had worked correctly, but he couldn't think of a way of doing it, so I placed a 10 second pause before the init of the webserver, and I haven't had a hang since.
Regards,
Neil.
-
Hi Neil,
That's interesting. I am using GPFReporter to restart the application too, and it always hangs on the restart.
That's why I am using task scheduler to restart the program. If the task scheduler continues to work I will try the 10 second pause before the init of the webserver.
-
Wolfgang,
The Website is reachable at 50.62.74.129:88. This takes you to a logon screen where the problem manifests itself.
Problem is that there is a heartbeat in the WebServer procedure that reads the MySQL file every two minutes. It never gives an error.
Opening the file just before accessing a record doen't give an error. It is only upon accessing a record in the logon screen that the error comes back that MySQL server has gone away.
I believe there is a backup run every night but shouldn't that affect the heartbeat code that runs every two minutes too?
The IP address is static.
Thanks for the thoughts.
-
My suspicion is that the Backup locks some files, hence the unresponsive db.
-
Gregory, do you have all of the hooks hooked in your sever so you can everything to a file? Message, Assert, Stop, Halt, etc...
Larry Sand
-
Hi Wolfgang,
That's an idea, but I don't think that its the problem. The file that has the problem is the same file that I do a heartbeat on every 2 minutes. (heartbeat = open and read first record in file)
My code for accessing the file before the GPF is essentially.
open( file )
clear( record )
fil:field = username
set( key, key )
next( file ) !This is where I get the message MySQL has gone away. Why didn't I get the message on the file open?
-
Gregory, do you have all of the hooks hooked in your sever so you can everything to a file? Message, Assert, Stop, Halt, etc...
Larry Sand
Not sure what your asking here. After starting the program during the day I have no problems accessing anything in the file. After the site sits overnight the first access to the password file causes a GPF.
-
I would add the exact code to your heartbeat that you use in your login but hardcode the username for starters.
Add a trace to your dbowner value in the login before file access.
Then try to add a super user to your login that will bypass the login file access and see if the rest of the program operates ok to isolate if it is just your login.
If it is just the login try using prop:sql for file access.
-
Hi Kevin,
I have tried prop:SQL and get the same error on the attempt to read the logon.
I have run the program with the ODBC trace turned on. Everything looks fine (even the open of the file) until the attempt to read the logon. When I get the MySQL has gone away message.
I have not tried the super user. But since I only get one test per day it is difficult to try everything.
BTW, so far for 3 days in a row. Closing the app at midnight and using the scheduler to restart at 12:05AM has worked.
-
do you think it is the time you restart or the hours since last accessed is the key? If you restart at 11:55pm or 8:00pm or 6:00am does that still work in the morning?
I would even run a 2nd web app on another port to try and isolate if it is the app or mySQL.
It is odd that the heatbeat does not throw an error (although you didn't mention if you changed the code to be the same as the login) which suggests that it is not MySQL. It's almost like it looses the connection string.
-
Hi Kevin,
Don't know if it is the time. I have tried loging in at 11pm and it works. The next morning it fails.
The code in the heartbeat is the same as the code in the login.
Really the only difference would be that the login would be on a different thread.
But shouldn't the MySQL has gone away message happen on the open and not the row access?
And when testing the PROP:SQL method of accessing the password file I used a dummy table to return the data into so its not something wrong with the Clarion buffer. It's the access of the row.
BTW, 4 days in a row working with the shutting down of the app at midnight and using task manager to restart at 12:05AM.