webset - feature or fault?

[rossw]

Thanks guys, received my latest board on Friday (but was interstate and didn't get to play with it until now) v03.02.17a
 
(Stupid server won't take my post, trying to edit it in a little at a time)
 
I've started with webset (since it's one I have immediate need of). The documentation is somewhat unclear, so I've been trying to track down exactly what it's doing. I have a very simple CGI that does nothing other than return some numbers. Currently returning 10-digit unique number of the form   YYYYhhmmss 
 
My cgi is this:
 

  #! /bin/sh

  echo "content-type: text/plain"
  echo ""
  date +"SET_WC=%Y%H%M%Sabcdefghij"
 
The webcontrol code is quite simple too:
 

START    
    WEBSET URL1 VAR1  
    set var2 0
lp:
    set var3 wsrply
    delay 1000
    inc var2
    tstlt var2 20
    goto lp
    inc var1
    END    
 
I see, every 20 seconds, a new value arrive for wsrply which is loaded into var3. Now it's 2013155355
Which is exactly what I expect. 2013 (year), 15 53 55  (it was 3:53pm and 55 seconds).
 
The interesting thing is, this ONLY WORKS if there are enough characters AFTER the "webset=xxxx" for the webcontrol to store the number!
 
 
If my CGI has this line:   date +"SET_WC=%Y%H%M%Sabcd"
then the wc shows only:  "2013".   ("abcd" is 4 chars and that's all I get).
 
If my cgi has: date +"SET_WC=%Y%H%M%Sab"
then the wc shows "20"
 
If my cgi has: date +"SET_WC=%Y%H%M%Sabcdefg"
then the wc shows "2013155"
 
The full measure of data is definately sent from the apache webserver, so it's something within the WC.
 
Simply sending padding information after the data is a temporary work-around for me - but it'd be nice if the docs indicated the need for it, or if the code was fixed to not require it

 

[CAI_Support]

Ross,
 
Can you provide a TCPdump from your server, so that we can see what your server actually sending out over the network?
 
Thanks.

 

[rossw]

Ross,
 
Can you provide a TCPdump from your server, so that we can see what your server actually sending out over the network?
 
Thanks.

 
The example code is no longer running, but here's some from a subsequent piece. Basically, the WC board hits one of my servers and gets the suns current elevation and azimuth returned to it in a 6-digit packed word.
 
I've removed a couple of specific identifying bits, but they're immaterial for your purposes:
 
Request from webcontrol to server:
 
TCP:   0549 0050 | 349f 4be7 | 62ff 66aa | 5018 00c8 | abe8 0000 | Src port: 1353  Dst port: 80 Flags: Psh Ack
Data:  4745 5420 | 2f** **** | **** **2f | 7472 6163 | 6b65 722e | 6367 693f | 626c 7772 | 673d 3020 |
       4854 5450 | 2f31 2e30 | 0d0a 486f | 7374 3a20 | **** **** | **** 2e61 | 6c62 7572 | 792e 6e65 |
       742e 6175 | 0d0a 4175 | 7468 6f72 | 697a 6174 | 696f 6e3a | 2042 6173 | 6963 200d | 0a0d 0a55 |
       7365 722d | 4167 656e | 743a 20
17:18:37  *.*.*.*  *.*.*.*    GET /******/tracker.cgi?blwrg=0 HTTP/1.0..Host: ******.albury.net.au..Authorization: Basic ....User-Agent:
 
TCP:   0549 0050 | 349f 4c52 | 62ff 66aa | 5018 00c8 | dbdf 0000 | Src port: 1353  Dst port: 80 Flags: Psh Ack
Data:  5745 4243 | 4f4e 5452 | 4f4c 2020 | 2020 2020 | 3030 3a32 | 323a 3132 | 3a30 323a | 3032 3a30 |
       350d 0a43 | 6f6e 6e65 | 6374 696f | 6e3a 2063 | 6c6f 7365 | 0d0a 0d0a |
17:18:37  *.*.*.*  *.*.*.*    WEBCONTROL      00:22:12:02:02:05..Connection: close....
 
 
Reply to webcontrol:
 
TCP:   0050 0555 | 4459 c5a8 | 7771 3854 | 5018 ffff | aa0e 0000 | Src port: 80  Dst port: 1365 Flags: Psh Ack
Data:  4854 5450 | 2f31 2e31 | 2032 3030 | 204f 4b0d | 0a44 6174 | 653a 204d | 6f6e 2c20 | 3235 204d |
       6172 2032 | 3031 3320 | 3036 3a32 | 383a 3034 | 2047 4d54 | 0d0a 5365 | 7276 6572 | 3a20 4170 |
       6163 6865 | 2f32 2e32 | 2e31 3520 | 2846 7265 | 6542 5344 | 2920 6d6f | 645f 7373 | 6c2f 322e |
       322e 3135 | 204f 7065 | 6e53 534c | 2f30 2e39 | 2e38 6e20 | 4441 562f | 320d 0a43 | 6f6e 6e65 |
       6374 696f | 6e3a 2063 | 6c6f 7365 | 0d0a 436f | 6e74 656e | 742d 5479 | 7065 3a20 | 7465 7874 |
       2f70 6c61 | 696e 0d0a | 0d0a 5345 | 545f 5743 | 3d30 3230 | 3136 310a | 7878 7878 | 7878 7878 |
       7878
17:18:37  203.15.244.1    203.15.244.151  HTTP/1.1 200 OK..Date: Mon, 25 Mar 2013 06:28:04 GMT..Server: Apache/2.2.15 (FreeBSD) mod_ssl/2.2.15 OpenSSL/0.9.8n DAV/2..Connection: close..Content-Type: text/plain....SET_WC=020161.xxxxxxxxxx
 
The next packet was just the FIN/ACK, no further data.

 

[CAI_Support]

Hi Ross,
 
Could you do it using this command:
 

tcpdump -s 0 -elnnxX -i eth0 host 192.168.1.15
 
where eth0 is the server interface, 192.168.1.15 is the IP address of WebControl board.  You can change them into your information.
 
It is best to save the output to a file, so that we can load that into Wireshark to look.

 

[rossw]

Hi Ross,
 
Could you do it using this command:

 
I could, or I could just send you a working program so you can reproduce the problem. (Link to the actual code sent by PM)
 
In the code I am sending you, the 3rd line lets you set how much padding is sent after the data.
 


START
SET WSRPLY 0
WEBSET URL1 10

 
Change the "10" to 4, 5, 6 etc to see the problem.
 
The rest of my code is largely irrelevant to your purposes, however if you enable webpolling and open the system status window, watch VARs 4, 5 and 6 particularly.
VAR4 shows the whole wsrply value as received, which should be 8 or 9 digits, -EEEEAAAA (E=Elevation, A=Azimuth)
VAR5 shows the "Elevation above horizon" component
VAR6 shows the "Azimuth west-of-east" component
 
The server code already sets a minimum azimuth of 0 (due east, so if the sun is south-of-east it is still set to 0), and 180 (due west, limited to 180 even if the sun is further south), and 90 degrees for due north.
The elevation can (and will) go negative when the sun is below the horizon.
The webcontrol code positions the tracker horizontal (Parked, or 90 degrees, North) when the sun is below the horizon.
 
You can use wireshark/tcpdump/whatever on your board to observe what's happening, and work on it, including changing the padding after the data.

 

[CAI_Support]

No, Ross, I want to see first the reply from your server side.  I want to see the TCPdump on the server side to see what was your server actually sending out.  Only the data bytes sending over the wire actually being received and processed.  If we can see the data sending correctly, WebControl did not process that correctly, then we can figure out where is the problem.

 

[rossw]

No, Ross, I want to see first the reply from your server side.  I want to see the TCPdump on the server side to see what was your server actually sending out.  Only the data bytes sending over the wire actually being received and processed.  If we can see the data sending correctly, WebControl did not process that correctly, then we can figure out where is the problem.

 
I've only got one WC board with the webset command, and at the moment I'm using it to develop a different chunk of code.
Can you try loading the code I sent you onto one of your boards? It should produce exactly the same results I see here, that is, if you set the pad count to say, 4 (instead of the 10 in my supplied code) it should reproduce the fault perfectly.
 
You can capture the packets between your WC board and my server - they will be identical to what I saw here.

 

[CAI_Support]

Hi Ross,
 
WebControl WEBSET reading back feature depends on server reply.  We did reassemble the packets, so that if server sending back the reply in two packets, we can match them up. However, if server insert anything, like space or any other character, white character, it will be a different story.  We want to see how your server actually sending it back, that helping us to identify where is the problem. Maybe your server was sending in a different way that our logic did not handled correctly.
 
We had tested with many scenarios, however, we only tested on Apache server with CGI code published in the latest user guide.
 
PLC code could make some difference, because from calling WEBSET to WEBSET finish, it may take 100mS or longer, during that time, PLC logic already runs many lines.  However, in your PLC code, you already checked that not zero and delay waiting for WEBSET executed, so the PLC code is probably not the problem.  You could shorten the delay time and loop more times to check the return value.  If delayed too much time, the server may already finished sending reply however WEBSET had not chance to process it.
 
Please note WebControl when talking to server, it tells the server it wanted to reopen connection each time communicate, due to its internal limited RAM space. Server will close connection once it finish sending the reply.  If WEBSET call did not have chance to capture all the reply, then part of the data packet could be lost.  If PLC logic expects non-zero reply from server, it is better just busy looping check the return value, not using delay.

 

[rossw]

Which email address do you want the log sent to?
The entire reason I sent you the code, and provided access to the cgi, was so you could repeat the test with different parameters to actually nail this, rather than sending dumps around the world and not capturing quite what you wanted.
 
The reply packet is *NOT* split.
 
I've reconfigured everything to put it on the 192.168.1.x subnet you asked for. Here's a small extract from your specified tcpdump
 
07:28:53.472419 00:22:12:02:02:05 > 00:30:18:a6:90:ab, ethertype IPv4 (0x0800), length 167: 192.168.1.15.1197 > 192.168.1.200.80
: Flags [P.], ack 1, win 200, length 113
        0x0000:  4500 0099 ffb7 0000 6406 d27f c0a8 010f  E.......d.......
        0x0010:  c0a8 01c8 04ad 0050 5ea7 36cf 7cac 4427  .......P^.6.|.D'
        0x0020:  5018 00c8 b555 0000 4745 5420 2f7e 726f  P....U..GET./~ro
        0x0030:  7373 772f 7472 6163 6b65 722e 6367 693f  ssw/tracker.cgi?
        0x0040:  7061 643d 3520 4854 5450 2f31 2e30 0d0a  pad=5.HTTP/1.0..
        0x0050:  486f 7374 3a20 3139 322e 3136 382e 312e  Host:.192.168.1.
        0x0060:  3230 300d 0a41 7574 686f 7269 7a61 7469  200..Authorizati
        0x0070:  6f6e 3a20 4261 7369 6320 0d0a 0d0a 5573  on:.Basic.....Us
        0x0080:  6572 2d41 6765 6e74 3a20 5745 4243 4f4e  er-Agent:.WEBCON
        0x0090:  5452 4f4c 2020 2020 20                   TROL.....
 
 
And the reply, in ONE PACKET is:
 
07:28:53.501348 00:30:18:a6:90:ab > 00:22:12:02:02:05, ethertype IPv4 (0x0800), length 245: 192.168.1.200.80 > 192.168.1.15.1197
: Flags [P.], ack 155, win 65535, length 191
        0x0000:  4500 00e7 f0fa 4000 4006 c4ee c0a8 01c8  E.....@.@.......
        0x0010:  c0a8 010f 0050 04ad 7cac 4427 5ea7 3769  .....P..|.D'^.7i
        0x0020:  5018 ffff bc72 0000 4854 5450 2f31 2e31  P....r..HTTP/1.1
        0x0030:  2032 3030 204f 4b0d 0a44 6174 653a 2054  .200.OK..Date:.T
        0x0040:  7565 2c20 3236 204d 6172 2032 3031 3320  ue,.26.Mar.2013.
        0x0050:  3230 3a32 383a 3533 2047 4d54 0d0a 5365  20:28:53.GMT..Se
        0x0060:  7276 6572 3a20 4170 6163 6865 2f32 2e32  rver:.Apache/2.2
        0x0070:  2e31 3520 2846 7265 6542 5344 2920 6d6f  .15.(FreeBSD).mo
        0x0080:  645f 7373 6c2f 322e 322e 3135 204f 7065  d_ssl/2.2.15.Ope
        0x0090:  6e53 534c 2f30 2e39 2e38 6e20 4441 562f  nSSL/0.9.8n.DAV/
        0x00a0:  320d 0a43 6f6e 6e65 6374 696f 6e3a 2063  2..Connection:.c
        0x00b0:  6c6f 7365 0d0a 436f 6e74 656e 742d 5479  lose..Content-Ty
        0x00c0:  7065 3a20 7465 7874 2f70 6c61 696e 0d0a  pe:.text/plain..
        0x00d0:  0d0a 5345 545f 5743 3d30 3031 3230 3031  ..SET_WC=0012001
        0x00e0:  320a 7878 7878 78                        2.xxxxx
 
 
There are only 5 'x' characters after the SET_WC= value and the webcontrol has NOT read the reply correctly.

 

[CAI_Support]

Hi Ross,
 
Thanks for sharing the tcpdump output. The last two lines from the server reply was
 
0x00d0:  0d0a 5345 545f 5743 3d30 3031 3230 3031  ..SET_WC=0012001
0x00e0:  320a 7878 7878 78                        2.xxxxx
 
At the last line, there is a 0x0A, which is a new line character. WebControl reads to that and stop reading further.
Please note WebControl only read 0-9 numbers and '-' sign if there is one leading.  Any byte not in range 0x30 and 0x39 will be considered end of the reply number.
 
We do not recommend to format the number with leading zeros, since if there is a negative sign in front of that will confuse the number conversion.
 
You PLC code is good to check the value, and reset that value to zero after reading.  We do recommend change the delay to busy looping at zero value, that will give better chance for WEBSET to respond to server. (you may set a max loop number, so that it will not deadlock PLC code in case server down or network down).

You are always welcome to write to us at [email protected]

 

[rossw]

Hi Ross,
 
Thanks for sharing the tcpdump output. The last two lines from the server reply was
 
0x00d0:  0d0a 5345 545f 5743 3d30 3031 3230 3031  ..SET_WC=0012001
0x00e0:  320a 7878 7878 78                        2.xxxxx
 
At the last line, there is a 0x0A, which is a new line character. WebControl reads to that and stop reading the numbers.
Please note WebControl only read 0-9 numbers and '-' sign if there is one leading.
 
We do not recommend to format the number with leading zeros, since if there is a negative sign in front of that will confuse the number conversion.
 
You are always welcome to write to us at [email protected]

 
Yes, without the extra newline character, the WC was refusing to process the line at all. I've now removed the newline.
 
I deliberately padded leading zeros to reduce confusion as far as length of reply is concerned, but thats easily fixed too. (done now)
The problem however remains, if I don't pad enough after the data, the WC sets the value of wsrply incorrectly (missing digits off the right hand end)

 

[CAI_Support]

HI Ross,
 
Could you please try to use looping when WSRPLY is zero instead of delay? 
 
Another thing is like what you did send more bytes.
 
I don't think there is a bug in the firmware, rather there is no handshaking between the server and WebControl.  Because server understand WebControl WEBSET is a HTTP 1.0 client, it will send data then close connection.  It then at mercy of server TCP stack to send out immediately or hold it for short period of time, or immediately before close connection.
 
WEBSET has only one chance to process all the replies into very limited TCP buffer. If that is not processed in time, TCP stack will see the disconnect notice from the server side then free up the TCP buffer.   That is why I say using looping in PLC is better than blocking DELAY.  That will give WEBSET better chance to read from TCP buffer.  If server side after sending the SET_WC=00120012 line sending additional junks over, that actually helps WEBSET to have more time to process the server reply. because server will break the reply to more packets and TCP close notice will be send two packages later.
 
Even TCP protocol is supposedly reliable protocol, WebControl's TCP stack is trying hard to be efficient by discarding things when it seeing disconnect from the server end.  WEBSET will only be able to process the reply on WebControl TCP buffer before WebControl TCP stack received disconnection notice from other side.

 

[rossw]

HI Ross,
 
Could you please try to use looping when WSRPLY is zero instead of delay? 
 
Another thing is like what you did send more bytes.
 
I don't think there is a bug in the firmware, rather there is no handshaking between the server and WebControl.  Because server understand WebControl WEBSET is a HTTP 1.0 client, it will send data then close connection.  It then at mercy of server TCP stack to send out immediately or hold it for short period of time, or immediately before close connection.
 
WEBSET has only one chance to process all the replies into very limited TCP buffer. If that is not processed in time, TCP stack will see the disconnect notice from the server side then free up the TCP buffer.   That is why I say using looping in PLC is better than blocking DELAY.  That will give WEBSET better chance to read from TCP buffer.  If server side after sending the SET_WC=00120012 line sending additional junks over, that actually helps WEBSET to have more time to process the server reply. because server will break the reply to more packets and TCP close notice will be send two packages later.
 
Even TCP protocol is supposedly reliable protocol, WebControl's TCP stack is trying hard to be efficient by discarding things when it seeing disconnect from the server end.  WEBSET will only be able to process the reply on WebControl TCP buffer before WebControl TCP stack received disconnection notice from other side.

 
 
I *REALLY* wish you'd run this code on one of *YOUR* boards so you can make these changes and observe in REAL TIME and save everyone hours or days of work!
 
Section of code, comments added for clarity
 

START    
    SET WSRPLY 0  
    WEBSET URL1 var3   # set var3 externally to tell server how many pad characters to add after the data.
    SET VAR2 0  # counter for timeout, otherwise we will never re-request after a failure
LP:
    set var4 wsrply   # we can monitor status to see what was returned
    TSTNE var4 0   # test if it is still zero
    GOTO ELEV    # if not, go do stuff
    INC VAR2      # otherwise, bump timeout counter
    TSTLT VAR2 5000  # is it around 5 seconds?
    GOTO LP     # go around again if not expired
    GOTO START   # if timeout, start the whole thing again
ELEV:
    SET VAR8 500  # start of serial output routine, not relevant
    SET OP1 0  
    SET OP2 0  
    SET OP8 1  
    SET RAM1 32  
    CALLSUB SEND   
    DIV var4 10000 VAR5
    MUL VAR5 10000 VAR6
    SUB var4 VAR6 VAR6
    TSTLT VAR5 2  
    SET VAR6 90  
    SET RAM1 VAR6  
    CALLSUB SEND 

 
 
With the code modified as shown, it fails to get ANY data if var3<=5, if var3=6, wsrply shows only "4" although the data sent is "400044", with var3=9, wsrply shows "4000", with var3=10, wsrply="40004".

 

[CAI_Support]

Ross,
 
We can run the PLC code, but we need to also know the server, so that we can have WebControl to talk to the server and get reply.  If you don't want to let others to see how to connect to your server, please do send us email with your server information, like URL to reach, IP address, so that we can configure WEBSET to get to the server.  We also like to be able to see how the TCPdump respond when WEBSET is called.
 
To get server reply is like two persons talking, we have to examine both sides to see where is the problem.
 
Thanks,

 

[rossw]

Ross,
 
We can run the PLC code, but we need to also know the server, so that we can have WebControl to talk to the server and get reply.  If you don't want to let others to see how to connect to your server, please do send us email with your server information, like URL to reach, IP address, so that we can configure WEBSET to get to the server.  We also like to be able to see how the TCPdump respond when WEBSET is called.
 
To get server reply is like two persons talking, we have to examine both sides to see where is the problem.
 
Thanks,

 
 
I already did that in my original mail days ago!!!