[Verse-dev] New optimized verse

[Eskil Steenberg]

Hi

This is copy of a post i made on www.blender.org

I received an E-mail form averse user who was having some issues with an
application he is writing using verse. He sent me a small test
application that attempted to upload an image to verse but failed.

As it turns out there where some bugs in the in the re-send mechanism.
there where quite easily fixable, but is I tried to send larger images i
found that verse is just very slow. sending a 256 * 256 8 bit image (4
channels) took about 20+ seconds! i removed some waits, but it still
dint help.

So i have spent the last 5 nights trying to fix this, and as it turns
out is wasn't that easy. One of the problems is that im trying to send
10.000+ packets over the need and trying to figure out why they don't
run very good is hard with that much data to look threw.

First of all, i found that verse out going queue is just way too slow.
when a new command is placed in the out going queue it has to be
compared to every existing command in the que. with 16000 commands this
turns out to be slow even if its just a check sum compare. so I split
the queue in to 2192 queues, making it 2192 * 2192 times fewer compares.
this made things a lot faster.

But still there are some problems. when you upload an image to verse you
usually call the "set tile" command _a lot_. And you don't ever inside
that loop call verse_callback_update (the commands that looks on the net
for any incoming data). This means that you wont get any confirmation
that any thing got threw, and you cant cut down the history, and
basically the network chocks. This could be considered as just poorly
written code by the app and not at all a verse problem, but in my
opinion application developers should not have to care about this.

So I tried to fix this by making verse check the network for in coming
data. pick out the acks and naks and store the command until the next
time the user calls. verse_callback_update. this turned ot to be a good
thing but only marginal.

So next i found that when verse tried to send 16000 packets at once
almost none got there so i implemented a "sliding window" to allow the
stream to bee a little bit more controlled. How ever this turned out to
not wok at all! The packets just never got there at all,and both sides
where waiting for the other side to send something. later i found that
when both sides where receiving data and tried to confirm this, the
confirmation commands (acks) got last in the queue, and since it dint
send any data until it got any acks nothing happened.

So i made a separate "speed lane" and made all acks and nacks(negative
confirmations) a priority. And all of a sudden the network started
running like hell!

So i think the major issues are fixed now but there may still be
problems, so if you have any please mail me.

Another issue since im using windows is that windows switches tasks so
slow, and so seldom. Having two apps on two machines comunicate over a
network may even be faster then having two apps comunicate on the same
machine. I hope to get some feadback to see if this is better in linux. 

the new version is up at: http://www.blender.org/modules/verse

E