Show Posts
This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.
61
Web Server - Ask For Help / Re: POST and parameters
« on: December 21, 2015, 05:40:16 AM »
Hi Bruce,
you're right it works in the examples.
And now the strange thing:
I've dived into the netweb.clw a little more and added/uncommented some self._trace calls. What I found out really troubles me: in my app the calls to the _ParseSettings method where the 5/6th parameter is omitted, the OMITTED(p_sep) still is False (and that's why the whole _postdata_ string isn't being split) !!!
For each post to the server the _Parsettings is called (at least) three times and the last one should be splitting the parameters. In my app the following code is called (fragment from the _HandlePost method), both by my app as well as for example the web77 app:
elsif self.RequestBoundary <> ''
self._ParseSettings(self.RequestData.DataString,x+4,self.RequestData.DataStringLen,0,self.RequestBoundary,net:Multipart)
else
self._trace('!!! this call to _ParseSettings is used !!!')
self._ParseSettings(self.RequestData.DataString,x+4,self.RequestData.DataStringLen)
end
end
self._trace('recs localdataq : ' & records(self._LocalDataQueue))
loop qindex# = 1 to records(self._LocalDataQueue)
get(self._LocalDataQueue, qindex#)
self._trace('element : ' & qindex# & ' param : ' & self._LocalDataQueue.name & ' val : ' & self._LocalDataQueue.value)
end
So in both apps the parameters in the calling statement are the same, but somehow, the implementation of the _ParseSettings method fails because of the omitted(p_sep) is not working ...
So it's more that omitted is not returning the right value somehow ...
(already removed obj files etc and recompiled stuff a dozen of time ..., but I am really clueless to what is going on here ...)
Regards,
Ton
you're right it works in the examples.
And now the strange thing:
I've dived into the netweb.clw a little more and added/uncommented some self._trace calls. What I found out really troubles me: in my app the calls to the _ParseSettings method where the 5/6th parameter is omitted, the OMITTED(p_sep) still is False (and that's why the whole _postdata_ string isn't being split) !!!
For each post to the server the _Parsettings is called (at least) three times and the last one should be splitting the parameters. In my app the following code is called (fragment from the _HandlePost method), both by my app as well as for example the web77 app:
elsif self.RequestBoundary <> ''
self._ParseSettings(self.RequestData.DataString,x+4,self.RequestData.DataStringLen,0,self.RequestBoundary,net:Multipart)
else
self._trace('!!! this call to _ParseSettings is used !!!')
self._ParseSettings(self.RequestData.DataString,x+4,self.RequestData.DataStringLen)
end
end
self._trace('recs localdataq : ' & records(self._LocalDataQueue))
loop qindex# = 1 to records(self._LocalDataQueue)
get(self._LocalDataQueue, qindex#)
self._trace('element : ' & qindex# & ' param : ' & self._LocalDataQueue.name & ' val : ' & self._LocalDataQueue.value)
end
So in both apps the parameters in the calling statement are the same, but somehow, the implementation of the _ParseSettings method fails because of the omitted(p_sep) is not working ...
So it's more that omitted is not returning the right value somehow ...
(already removed obj files etc and recompiled stuff a dozen of time ..., but I am really clueless to what is going on here ...)
Regards,
Ton
62
Web Server - Ask For Help / POST and parameters
« on: December 19, 2015, 03:16:30 AM »
Hi all,
I can't get parameters working with a webservice via a POST. I'm posting three parameters (see below the request that arrives serverside): "tabbed=1&traditional=0&readable=1"
In the primefields routine I expect the parameters to be accesible. In this routine the _postdata_ is filled with this string, but I can only get the FIRST parameter with the p_Web.GetValue('') command ....
I've dived into the netweb.clw file a little and it looks like something goes wrong in the NetwebserverWorker._ParseSettings method: I expect the line to be split on the "&" characters but it only sees one parameter (the part after the first "=", goes into the value of the first parameter). But I'm clueless to what is causing this
Can somebody tell me what I'm doing wrong here or what to check for?
TIA,
Ton
This is the request (incoming at the server)
POST /wsSupply/setLines HTTP/1.0
Accept: application/json
Accept-Language: en
Accept-Encoding: gzip
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16 ( .NET CLR 3.5.30729; .NET4.0C)
Host: localhost:8088
Content-Length: 33
Connection: Keep-Alive
tabbed=1&traditional=0&readable=1
I can't get parameters working with a webservice via a POST. I'm posting three parameters (see below the request that arrives serverside): "tabbed=1&traditional=0&readable=1"
In the primefields routine I expect the parameters to be accesible. In this routine the _postdata_ is filled with this string, but I can only get the FIRST parameter with the p_Web.GetValue('') command ....
I've dived into the netweb.clw file a little and it looks like something goes wrong in the NetwebserverWorker._ParseSettings method: I expect the line to be split on the "&" characters but it only sees one parameter (the part after the first "=", goes into the value of the first parameter). But I'm clueless to what is causing this
Can somebody tell me what I'm doing wrong here or what to check for?
TIA,
Ton
This is the request (incoming at the server)
POST /wsSupply/setLines HTTP/1.0
Accept: application/json
Accept-Language: en
Accept-Encoding: gzip
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16 ( .NET CLR 3.5.30729; .NET4.0C)
Host: localhost:8088
Content-Length: 33
Connection: Keep-Alive
tabbed=1&traditional=0&readable=1
63
Web Server - Ask For Help / Re: Webservice and (threading?) performance issues with json output - Part 2
« on: December 17, 2015, 02:54:52 PM »
Hi Bruce,
I've implemented Dave's memory management model and allthough it hughly improves the speed for the jFiles class, I'm not convinced on other parts of my app yet. I see some minor improvements here and there, but also speed degradation for some procedures. Also I'm not sure about any side effects (not initializing object properties ...) it might have. In short:for me (!) it's not worth the risk ...
so I decided to write my little "json" class that mimics the output I'm after (just a bunch of queues in a collection mainly ...), but than as fast as possible. That resulted in the producing of the return packet from taking 1.39 seconds to 0.18/0.19 seconds for each run (so the the total server side time take dropped from roughly 3.3 to 2.1 seconds for a single run) and as this code only creates a single (stringtheory) object it serves multiple request fast too!
I whish we could tie this memory improvement only to the creation of objects, then we all could have best of both worlds. But that's probably only up to softvelocity ...
Regards,
Ton
I've implemented Dave's memory management model and allthough it hughly improves the speed for the jFiles class, I'm not convinced on other parts of my app yet. I see some minor improvements here and there, but also speed degradation for some procedures. Also I'm not sure about any side effects (not initializing object properties ...) it might have. In short:for me (!) it's not worth the risk ...
so I decided to write my little "json" class that mimics the output I'm after (just a bunch of queues in a collection mainly ...), but than as fast as possible. That resulted in the producing of the return packet from taking 1.39 seconds to 0.18/0.19 seconds for each run (so the the total server side time take dropped from roughly 3.3 to 2.1 seconds for a single run) and as this code only creates a single (stringtheory) object it serves multiple request fast too!
I whish we could tie this memory improvement only to the creation of objects, then we all could have best of both worlds. But that's probably only up to softvelocity ...
Regards,
Ton
64
Web Server - Ask For Help / Re: Webservice and (threading?) performance issues with json output - Part 2
« on: December 07, 2015, 01:57:56 AM »
Hi Bruce,
that's good news, if you need a tester for this, let me know!
thanks so far
regards,
Ton
that's good news, if you need a tester for this, let me know!
thanks so far
regards,
Ton
65
Web Server - Ask For Help / Re: Webservice and (threading?) performance issues with json output - Part 2
« on: November 27, 2015, 09:59:44 AM »
Hi Bruce,
actually I agree with you (that it's best solved in the json class ...), although that doesn't help much of course <g>.
If you need any additional info/testing etc. just let me know
thanks so far
and regards
Ton
actually I agree with you (that it's best solved in the json class ...), although that doesn't help much of course <g>.
If you need any additional info/testing etc. just let me know
thanks so far
and regards
Ton
66
Web Server - Ask For Help / Re: Webservice and (threading?) performance issues with json output
« on: November 26, 2015, 02:37:28 AM »
Hi Bruce,
I've looked at the jFiles class a little trying to improve performance and can't really tell what's making it slower (but the mechanism is not entirely clear (yet), so that might be a problem too <g>).
In itself the json is roughly a factor 1.2 to 1.5 slower than xml (for the tests I'v run) that's fine/acceptable. but when the requests come in randomly serving json becomes really troublesome. My production server will have to server even larger result sets and I can't hope that there will be no "collisions"
So I'm still looking for a way to produce and serve json fast
Maybe a really stupid question: but is there a chance of producing JSON with the xFiles class (in the end it goes through the same structures and I can see "some" resemblense between xml and json <g>)?
Or what would be another simple method of producing "flat" json output easily (my result sets are always "simply" a set of queues and some of them will have child/grandchild result sets, that it <g>)?
Regards,
Ton
I've looked at the jFiles class a little trying to improve performance and can't really tell what's making it slower (but the mechanism is not entirely clear (yet), so that might be a problem too <g>).
In itself the json is roughly a factor 1.2 to 1.5 slower than xml (for the tests I'v run) that's fine/acceptable. but when the requests come in randomly serving json becomes really troublesome. My production server will have to server even larger result sets and I can't hope that there will be no "collisions"
So I'm still looking for a way to produce and serve json fast
Maybe a really stupid question: but is there a chance of producing JSON with the xFiles class (in the end it goes through the same structures and I can see "some" resemblense between xml and json <g>)?
Or what would be another simple method of producing "flat" json output easily (my result sets are always "simply" a set of queues and some of them will have child/grandchild result sets, that it <g>)?
Regards,
Ton
67
Web Server - Ask For Help / Re: Webservice and (threading?) performance issues with json output
« on: November 24, 2015, 04:33:28 AM »
Hi Bruce,
yep, that was it to produce json.
And letting webstrain put 4 request in simultaniously shows almost identical result as you have indicated But my example still choked (by pressing a button with a second or so pause between presses). Then (for fun) I started two session of the webstrain app and pressed the "play" button in each app with a second or two pause between them. Now I see the performance degredation too !
In this case I asked for 2 x 4 = a total of 8 sessions. Two finish in normal/expected time (3.19 s each), but the other 6 take 141 seconds in total (so for 24s each request)
When I asked for xml the 8 request get served in 22 seconds (so 2.69s each). What I did notice that the server never hit 8 thread running (actualy the maximum was 2 threads ....), don't know wether this has something to do with it ...
Again if you any additional info I'll do my best
Regards,
Ton
yep, that was it to produce json.
And letting webstrain put 4 request in simultaniously shows almost identical result as you have indicated But my example still choked (by pressing a button with a second or so pause between presses). Then (for fun) I started two session of the webstrain app and pressed the "play" button in each app with a second or two pause between them. Now I see the performance degredation too !
In this case I asked for 2 x 4 = a total of 8 sessions. Two finish in normal/expected time (3.19 s each), but the other 6 take 141 seconds in total (so for 24s each request)
When I asked for xml the 8 request get served in 22 seconds (so 2.69s each). What I did notice that the server never hit 8 thread running (actualy the maximum was 2 threads ....), don't know wether this has something to do with it ...
Again if you any additional info I'll do my best
Regards,
Ton
68
Web Server - Ask For Help / Re: Webservice and (threading?) performance issues with json output
« on: November 23, 2015, 07:52:33 AM »
now I'm 99% sure that webstrain was asking for xml in my case ..., just tried with Fiddler and it shows an xml result ... still figuring out how to ask for json with webstrain ...
69
Web Server - Ask For Help / Re: Webservice and (threading?) performance issues with json output
« on: November 23, 2015, 07:31:33 AM »
Hi Bruce,
first of all : thanks very much for ruining your weekend for this!
But, I still have performance issues ...
The single json request runs in 3.22 (that's faster!). But the 5 request still take roughly 445 seconds ...
I tried the Strain tester too and it does finish in 8 seconds (for four requests), but can it be that this is only serving xml? Can I ask for json with this application somehow? I
b.t.w. I upgraded jFiles and Stringtheory, both Nettalk and xFiles already were on the latest
I removed all obj/rsc etc. and did a full rebuild (I think ... still not sure about that in c10 ...), but I can't get it any faster
sorry for ruining your day
If you need any additional info or want me to test something let me know!
Regards
Ton
first of all : thanks very much for ruining your weekend for this!
But, I still have performance issues ...
The single json request runs in 3.22 (that's faster!). But the 5 request still take roughly 445 seconds ...
I tried the Strain tester too and it does finish in 8 seconds (for four requests), but can it be that this is only serving xml? Can I ask for json with this application somehow? I
b.t.w. I upgraded jFiles and Stringtheory, both Nettalk and xFiles already were on the latest
I removed all obj/rsc etc. and did a full rebuild (I think ... still not sure about that in c10 ...), but I can't get it any faster
sorry for ruining your day
If you need any additional info or want me to test something let me know!
Regards
Ton
70
Web Server - Ask For Help / Re: Webservice and (threading?) performance issues with json output
« on: November 20, 2015, 01:24:41 AM »
Hi Bruce,
just sent you a pm with steps to (hopefully) reproduce the issue
Instructions;
Run the server and from a client "get" the localhost:88//Test/TestLargeResult method just once. Depends on hardware but over here it finished in 2.00 seconds (as xml), now request this several times in a row, let's say 5 times), now it takes 35 seconds but finishes normally
Now ask for json ("Accept: application/json") once and the request finishes in 3.78 seconds over here. Now ask for json 5 times in a row and it will take the server 790 seconds (to proces these 5 request)
The "first" request finished in 60 seconds than several minutes the cpu settles at around 25 % (4 cores over here) and the last couple of seconds the cpu on the web77 proces goes up to 50 % and it "serves" the other 4 pending requests (gathering from the performance tab on the webserver)
Also note:
after serving 1 xml set, the web77.exe uses 7.072 kB of memory
after serving 1 json set the web77.exe uses 269.044 kB of memory !
I'm running latest NT/JFiles/xFiles as of this morning on an i5 , 4 cores with over 2 gB available when starting web77. After serving 5 json sets (concurrently) the exe uses 1.8 gB ...
If you need any more info/details, just let me know
just sent you a pm with steps to (hopefully) reproduce the issue
Instructions;
Run the server and from a client "get" the localhost:88//Test/TestLargeResult method just once. Depends on hardware but over here it finished in 2.00 seconds (as xml), now request this several times in a row, let's say 5 times), now it takes 35 seconds but finishes normally
Now ask for json ("Accept: application/json") once and the request finishes in 3.78 seconds over here. Now ask for json 5 times in a row and it will take the server 790 seconds (to proces these 5 request)
The "first" request finished in 60 seconds than several minutes the cpu settles at around 25 % (4 cores over here) and the last couple of seconds the cpu on the web77 proces goes up to 50 % and it "serves" the other 4 pending requests (gathering from the performance tab on the webserver)
Also note:
after serving 1 xml set, the web77.exe uses 7.072 kB of memory
after serving 1 json set the web77.exe uses 269.044 kB of memory !
I'm running latest NT/JFiles/xFiles as of this morning on an i5 , 4 cores with over 2 gB available when starting web77. After serving 5 json sets (concurrently) the exe uses 1.8 gB ...
If you need any more info/details, just let me know
71
Web Server - Ask For Help / Webservice and (threading?) performance issues with json output
« on: November 19, 2015, 09:52:50 AM »
Hi (Bruce),
I think there is an issue with a webmethods that proceduce (large sets of) json output on several threads concurrently.
I have a method that produces a rather large resultset (10,000+ items in a Queue). When I serve this data as JSON and activate this method multiple times (more or less concurrently) the server starts launching new threads (as it should), but then the server starts to crawl.
I've reproduced this by adding a new "method" to the 77 example app that simply loads data from an xml file to a queue and than simply serves that queue as JSON. Serving a single set takes roughly 20 seconds, asking for more sets while the first is still being processed, gets the webserver on its knees (I think memory consumption is also too high ... and sometimes it crashes (and gpf reporter doesn't kick in) ...
Simply letting the method serve xml instead of JSON (very same Queue with the same data, but now only served as xml) and the problem doesn't appear!!! Asking for three sets "concurrently" takes a little over a minute and nicely ends in all threads being closed and server ready for other requests.
Now if can have that for json too, I'll be more than happy <g>
using NT 8.65 and jFiles 1.13/1.14
BTW, just starting with webservices, but really love the nettalk product very much, there's been some great work done in there!
I think there is an issue with a webmethods that proceduce (large sets of) json output on several threads concurrently.
I have a method that produces a rather large resultset (10,000+ items in a Queue). When I serve this data as JSON and activate this method multiple times (more or less concurrently) the server starts launching new threads (as it should), but then the server starts to crawl.
I've reproduced this by adding a new "method" to the 77 example app that simply loads data from an xml file to a queue and than simply serves that queue as JSON. Serving a single set takes roughly 20 seconds, asking for more sets while the first is still being processed, gets the webserver on its knees (I think memory consumption is also too high ... and sometimes it crashes (and gpf reporter doesn't kick in) ...
Simply letting the method serve xml instead of JSON (very same Queue with the same data, but now only served as xml) and the problem doesn't appear!!! Asking for three sets "concurrently" takes a little over a minute and nicely ends in all threads being closed and server ready for other requests.
Now if can have that for json too, I'll be more than happy <g>
using NT 8.65 and jFiles 1.13/1.14
BTW, just starting with webservices, but really love the nettalk product very much, there's been some great work done in there!
72
Web Server - Ask For Help / Re: webservice : field case ignored for JSON in service documentation?
« on: November 17, 2015, 05:21:37 AM »
I haven't gotten to that bit yet, so for the moment I'm fine <g>
(but keeping an eye on it)
Thanks so far
(but keeping an eye on it)
Thanks so far
73
Web Server - Ask For Help / Re: webservice : field case ignored for JSON in service documentation?
« on: November 16, 2015, 05:30:08 AM »
Hmmm, casing seems to revert to the default (uppercase) on several places (with JSON):
- data : caseAsIs is ignored
- when there are errors in p_web.ServiceErrorQueue these are returned in uppercase (while the help, is "hardcoded" to mixed case, see "GenereateHelp Routine"
I've added the following to the BuildResultFields Routine
! Start of "Start of BuildResultFields Routine"
! [Priority 5000]
json.TagCase = jF:CaseAsIs
jsonResults.TagCase = jF:CaseAsIs
That seems to "fix" things.
- data : caseAsIs is ignored
- when there are errors in p_web.ServiceErrorQueue these are returned in uppercase (while the help, is "hardcoded" to mixed case, see "GenereateHelp Routine"
I've added the following to the BuildResultFields Routine
! Start of "Start of BuildResultFields Routine"
! [Priority 5000]
json.TagCase = jF:CaseAsIs
jsonResults.TagCase = jF:CaseAsIs
That seems to "fix" things.
74
Web Server - Ask For Help / webservice : field case ignored for JSON in service documentation?
« on: November 12, 2015, 04:12:45 AM »
Hi,
I think the field case settings for (at least) the webservice documentation is ignored. I'm using Queues as return type and have external names on the fields and set the "Fieldname Case" to "AsIs", but the documentation for the JSON output shows up as uppercase. (XML output shows mixed case, so that's fine). Using Nettalk 8.65 and jFiles 1.13
first a snippet for the generated XML (correct):
<category>
<id>value</id>
<label>value</label>
</category>
and this is for the JSON:
"category" : [
{
"ID" : value,
"LABEL" : "value"
}
]
Don't know what the actual output is like (need to test this ...)
I think the field case settings for (at least) the webservice documentation is ignored. I'm using Queues as return type and have external names on the fields and set the "Fieldname Case" to "AsIs", but the documentation for the JSON output shows up as uppercase. (XML output shows mixed case, so that's fine). Using Nettalk 8.65 and jFiles 1.13
first a snippet for the generated XML (correct):
<category>
<id>value</id>
<label>value</label>
</category>
and this is for the JSON:
"category" : [
{
"ID" : value,
"LABEL" : "value"
}
]
Don't know what the actual output is like (need to test this ...)