Ants P2P Roadmap per IRC Conversation

[Hornet]

Making Queries Scalable, Ants Supernodes and IP Filtering

G_G 20:10:57: tsafa?
G_G 20:13:14: how is it going?
G_G 20:13:49: everything fine?
G_G 20:15:17: yes queries are now controlled the way Jason suggested
G_G 20:15:21: to prevent flooding
G_G 20:15:30: but now there is an issue
G_G 20:15:43: filetransfers are scalable
G_G 20:15:50: but queries are not yet
G_G 20:16:13: so I'm working on a supernode infrastructure
G_G 20:16:17: this will take some time
G_G 20:16:47: supernode will collect list of file shared
G_G 20:16:52: and will manage queries
G_G 20:16:59: sure gnutella supernodes
G_G 20:17:05: same thing
G_G 20:17:24: the peers block list is already implemented
G_G 20:17:35: you can use in ants the ipfilter.dat of emule
G_G 20:17:53: eheh it's not easy
G_G 20:17:58: I'm still brainstorming
G_G 20:18:09: I've set up a discovery procedure
G_G 20:18:28: cuz it not so plain to state if a node have to be a supernode
G_G 20:18:39: and I want to keep the net self orgenized
G_G 20:18:50: so one cannot decide to be a supernode or note
G_G 20:19:01: it will be supernode if needed
G_G 20:20:01: well no
G_G 20:20:09: I want to keep a local broadcast
G_G 20:20:27: everytime you do a query
G_G 20:20:34: you actually do two queries
G_G 20:20:39: one is a local broadcast
G_G 20:20:45: to find near peers
G_G 20:20:53: without the aid of supernode
G_G 20:21:02: and the other is not a broadcast
G_G 20:21:10: but a message to one of your supernode list
G_G 20:21:19: item
G_G 20:21:32: this message once received by the supernode
G_G 20:21:36: is processed
G_G 20:21:44: and you get some result
G_G 20:21:45: s
G_G 20:22:01: them the supernode forward the message to other supernodes in its list
G_G 20:22:07: like in gnutella
G_G 20:22:20: eheh dunnow
G_G 20:22:45: anyway this will slightly affect the current behaviour
G_G 20:22:55: it will be very usefult with many more nodes
G_G 20:23:18: actually I've stated that a broadcast covers about half our net
G_G 20:23:38: so with two searches you get half of the total amount of file shared now with 30 node
G_G 20:23:39: s
G_G 20:24:14: the very good side will be that searches will become more precise and less "random"
G_G 20:24:47: you will get results every time you do a search... instead now sometimes you don't get any even if the file is shared
G_G 20:25:08: ok this is one point


Direct IP Connections in ANts

G_G 20:25:11: another
G_G 20:25:16: but still not developed
G_G 20:25:25: is the thing that let ants become popular
G_G 20:25:32: :)
G_G 20:25:41: I want to put a checkbox
G_G 20:25:50: to enable direct ip connections
G_G 20:25:58: zitto paes
G_G 20:26:13: sorry tsafa?
G_G 20:26:22: where?
G_G 20:26:32: ghgh
G_G 20:26:38: in the spernode list
G_G 20:26:48: or in the second thing i was mentioning
G_G 20:27:03: lascialo girare che adesso parte
G_G 20:27:26: tsafa whic is precisely your doubt?
G_G 20:29:29: no wait...
G_G 20:29:40: the supernode keeps only a list of file---ID
G_G 20:29:42: couples
G_G 20:30:04: just like you are doing now with your remote shared file list
G_G 20:30:12: instead the other idea
G_G 20:30:34: it to create "two kind" of ants clients running together in the same net
G_G 20:30:46: pure anonymous clients
G_G 20:31:07: why not scent based?
G_G 20:31:18: just mutual aid
G_G 20:31:41: node A knows that node B knows where is node C
G_G 20:32:00: so A asks B to pass the message to node C
G_G 20:32:12: if you get remote results from node B
G_G 20:32:20: you suppose it knows where node C is
G_G 20:32:37: or actually it can now the location of another node who had some more information
G_G 20:32:42: it's a chain process
G_G 20:32:55: yeah it's a "second level"
G_G 20:32:58: scent system
G_G 20:33:12: that works on the base scent system that also mute uses
G_G 20:33:28: got it?
G_G 20:33:46: eheh
G_G 20:33:48: instead
G_G 20:34:00: I was talking about the blockburter...
G_G 20:34:22: I'planning to build two ants node kinds
G_G 20:34:32: pure anonymous kind
G_G 20:34:39: the one you are running now
G_G 20:34:54: and half-anonymous clients
G_G 20:35:05: we talk much about the threats of this approach
G_G 20:35:12: I found a possible solution
G_G 20:35:20: to grant anonymity
G_G 20:35:32: we stated that hybrid net is not possible
G_G 20:35:39: that is not actually true
G_G 20:35:54: a hybrid net where a node is pure anonymous
G_G 20:36:04: and another is half anonimous
G_G 20:36:12: is actually NOT anonymous at all
G_G 20:36:33: but if two half anonymous nodes communicate
G_G 20:36:36: each other
G_G 20:36:47: they can decide to use direct ip connections
G_G 20:36:54: istead of using proxies
G_G 20:37:17: this doesn't harm the overal net anonymity
G_G 20:37:45: infact direct ip onnections will be used only if both the node have decided they don't want to be anonymous
G_G 20:37:51: setting a proper checkbox
G_G 20:38:37: this don't compromise your anonymity cuz you'll use not anonymous mode only if you explicitly set that parameter
G_G 20:38:57: and your peer have to do that tto to have a direct connection
G_G 20:39:12: every other configuration of couples of nodes will use anonymous way through proxies
G_G 20:39:36: actually I'm sure that ants way of spreading files
G_G 20:39:44: is more efficient than bittorrent
G_G 20:40:01: this way of hybridizing ants to pure p2p
G_G 20:40:20: will let use become popular also as 2nd gen p2p net
G_G 20:40:31: and will let us have some possibility to conenct to overnet
G_G 20:41:12: well you have already renounced to your anonymity if you set the checkbox
G_G 20:41:35: so anonymity is no more an issue if you enable half-anonymity in your client
G_G 20:42:34: yeah but this is only an idea
G_G 20:42:38: infact
G_G 20:42:39: :)
G_G 20:42:51: but now I just want to set up a good supernode structure
G_G 20:43:37: actually the half-anoninimity thing is not a great change in ants
G_G 20:43:50: I mean all the structure remain the same
G_G 20:44:06: I only have to add a proper field in queries answer
G_G 20:44:18: where you put your ip
G_G 20:44:29: if you want to remain anonimous you don't put the ip
G_G 20:44:40: ok what are they saing?
G_G 20:45:35: well we are still few but the supernode thing should grant us even more scalability
G_G 20:46:00: so what's the matter?
G_G 20:46:51: well you have to see the ants net as a wiretapped thing
G_G 20:47:07: that wiretap is divided in 32k blocks
G_G 20:47:20: you always have a constant flux
G_G 20:47:40: but some part of the blocks are not for you
G_G 20:48:03: so the transfers bursted actually make very sense
G_G 20:48:35: no if I put the direct connection once you choosed to expose your ip you loose anonymity



Start Stop Downloads in ANts

G_G 20:49:13: no that is impossible
G_G 20:49:46: was not talking to you tsafa
G_G 20:49:56: anyway I cannot control what people do :)
G_G 20:50:12: I just try to set up a net structure that is strong against such behaviours
G_G 20:50:26: I don't think that a guy doing manual queries is an issue
G_G 20:50:31: you know I separated fluxes
G_G 20:50:44: now queries have a dedicated bandwith limit
G_G 20:51:00: so they never influence data transfer
G_G 20:52:02: tsafa queries are not deterministic
G_G 20:52:15: untill the supernode system is not set up
G_G 20:52:27: so you can't reason this way
G_G 20:52:33: yeah it has
G_G 20:52:36: you are right
G_G 20:52:44: every time you do a search
G_G 20:52:51: you make the scent more strong
G_G 20:52:57: to your destination
G_G 20:53:05: so it's easiser to find the destination
G_G 20:53:12: I'm happy that this is so evident
G_G 20:53:18: the algorithm works good ;)
G_G 20:53:24: but this is not the issue
G_G 20:53:34: obiusly you can't do queries all the time
G_G 20:53:44: so I put a limit to grant a "good beahviour" to all
G_G 20:54:14: without blocking the net
G_G 20:54:26: also there are many other stuffs
G_G 20:54:33: free slots for instance is an issue
G_G 20:54:48: your requests are rejected when no free slots are availbable
G_G 20:54:57: and your filetransfer is interrupted
G_G 20:55:09: untill the next query
G_G 20:55:16: yeah it's interesting
G_G 20:55:50: and I'm trying to give you a rough image of the reason for wht you are seeing
G_G 20:56:09: exact
G_G 20:56:41: well the supernode stuff will make things a bit better
G_G 20:56:54: but don't expect great improvements
G_G 20:57:30: the burst thing cannot be overcome
G_G 20:57:50: you are routing stuffs for other
G_G 20:58:08: yeash
G_G 20:58:24: great speed (burst) then 0 speeds
G_G 20:58:34: that is necessary
G_G 20:58:43: if you route somethhing for someone else
G_G 20:58:46: you have 0 speed
G_G 20:58:56: untill you get the next packet
G_G 20:59:00: it makes sente to me
G_G 20:59:04: sense
G_G 20:59:13: you have 1 connection to the internet
G_G 20:59:17: so if it's occupied
G_G 20:59:23: you can't have stuff for you
G_G 20:59:39: well if you fill the net with your stuff
G_G 20:59:47: other cannot tranfers
G_G 21:00:04: actually the fact of transferring just 320k block is critical
G_G 21:00:10: to prevent queues in the system
G_G 21:01:01: well I hope that my explainations are good
G_G 21:01:08: for ppl doubts
G_G 21:01:30: I mean I could make stuffs appear more smooth...
G_G 21:01:44: I can just change the way stats are computed
G_G 21:01:56: but i don't think this is a significant improvement
G_G 21:02:07: eheh
G_G 21:02:13: a chunk is 320k
G_G 21:02:23: and it is made up of packets of 32k
G_G 21:02:40: the chunk is sent all together
G_G 21:02:57: yeah in ANts you have the ACTUAL speed
G_G 21:03:05: that is why you see also 90k transfer
G_G 21:03:08: or 100k
G_G 21:03:13: I got also 150k
G_G 21:03:27: but that is anyway the real speed
G_G 21:03:54: no holes in the transfer... when you see the speed you are getting infos
G_G 21:04:09: could be usefult to put two meters
G_G 21:04:15: one for data flow
G_G 21:04:21: and one for own data flow
G_G 21:04:48: because I didn't find the time to fixx it
G_G 21:04:53: the same way
G_G 21:05:11: upload is an average on a 20 second period
G_G 21:05:18: so you see that is more smoothed
G_G 21:05:32: but actually works exactly the same way as download
G_G 21:05:49: so you can compare them
G_G 21:06:14: well wait
G_G 21:06:20: it's not a so easy computation..
G_G 21:06:24: for upload
G_G 21:06:27: I mean...
G_G 21:06:31: in download you get data
G_G 21:06:41: you can know the actual speed you are getting data
G_G 21:06:53: in upload you upload data to your neighbour
G_G 21:07:15: but before computing the data you must wait that the real receiver (some hops away) gets the data
G_G 21:07:25: this is why speed is lower
G_G 21:07:33: you actually upload fast
G_G 21:07:42: but you have to wait for the receiver response
G_G 21:07:50: and this takes time
G_G 21:08:04: so the average appears low
G_G 21:08:20: cuz you are waiting the acknowledge that the data have been received
G_G 21:08:34: well you can never know if the data reach their destination
G_G 21:08:46: broken routes
G_G 21:08:50: node failure
G_G 21:08:52: stall
G_G 21:08:56: many issues
G_G 21:09:05: so you are waiting for the acknowledge
G_G 21:09:20: 3 times
G_G 21:09:34: in a span of 9 minutes
G_G 21:09:44: so a transmission every 3 minutes
G_G 21:10:02: thats what i do
G_G 21:10:15: I send 10 partial together
G_G 21:10:32: for a total amount of 320k
G_G 21:10:58: obiously I cannot send 1Mb together... so 320k in chunks of 32k is a good trade off
G_G 21:11:54: no
G_G 21:16:06: phone..
G_G 23:17:03: lol ants is going to look as the middle-earth
G_G 23:18:04: ??
G_G 23:18:30: no no they simply connect as the other ants nodes
G_G 23:18:43: there's nothing special to be a supernode
G_G 23:20:25: noooo
G_G 23:25:53: you are making only a big fuss...
G_G 23:27:15: I was talking to tsafa and I explained some "possible" future extensions... you put all these together but you didn't understand any of them
G_G 23:28:01: noone knows... it was just an idea I was talking with tsafa about...


Difference Between Ants Supernodes and Direct Connections

G_G 23:28:17: there are not direct connection yet
G_G 23:28:33: if there will be any form of direct connection i'll post details
G_G 23:29:26: no if I'll wimplemnet it that will be safe
G_G 23:30:19: vers... you know I know my job :)
G_G 23:30:29: chill out
G_G 23:30:47: obius
G_G 23:31:15: because exist also the "publish/suscribe" paradigm in this world
G_G 23:31:34: so I could be one node that "publishes" its intension to be not anonymous
G_G 23:32:12: and another node could suscribe in a later time its intension to extabilish a not anonymous channel with me
G_G 23:32:24: infact that will be possible
G_G 23:32:34: half node will be able to tunnel the net to kademlia
G_G 23:32:41: but this is another issue
G_G 23:32:56: and i will not code this part
G_G 23:33:41: the supernode discussion is not realed in any way to this discussion
G_G 23:33:52: this is why I say you didn't understand
G_G 23:34:03: I'm coding the supernode infrastructure right now
G_G 23:34:17: yes we move to a push paradigm
G_G 23:34:30: as you described it
G_G 23:34:46: well there is not so much dfference in my opinion
G_G 23:34:51: push or pull I mean
G_G 23:35:04: you just have messages roaming through the net
G_G 23:35:51: how?
G_G 23:36:08: I mean... how does he know you ip/id relation
G_G 23:36:27: you are just routing a file list push at the eye of an external node
G_G 23:36:36: ahhhhhhh
G_G 23:36:40: ASK
G_G 23:36:43: BEFORE
G_G 23:36:45: TAKLING
G_G 23:36:59: Have I ever talked about DIRECT IP CONNECTION?????
G_G 23:37:05: WHERE?
G_G 23:37:07: NOT HER?
G_G 23:37:25: I said that the directi ip issue is not related to supernodes
G_G 23:37:31: please read what i wrtie
G_G 23:38:26: ok ok you want a more detailed description...
G_G 23:38:33: it is possible
G_G 23:38:38: push is a paradigm
G_G 23:38:45: not an implementation issue
G_G 23:38:53: first version of windows
G_G 23:38:57: could emulate multithreading
G_G 23:39:05: tohugh they were single threaded
G_G 23:39:09: here it's the same..
G_G 23:39:17: we are EMULATING push
G_G 23:39:20: that is
G_G 23:39:27: A does a search for a supernode
G_G 23:39:35: the supernode answers...
G_G 23:39:52: and also gives to A all the information to set up an encrypted channel with the supernode itself
G_G 23:40:14: so the filelist passing process is an end to end encrypted process
G_G 23:40:24: sure over proxies with different possible paths
G_G 23:40:37: exactly like filetransfers
G_G 23:40:44: actually I always use the same protocol
G_G 23:40:55: the protocol doesn't change
G_G 23:41:18: I'm just shuffling our cards to have a better behaviour
G_G 23:41:43: I haven't coded the supernode connection protocol
G_G 23:41:54: but it will lay on the ants protocol too
G_G 23:42:14: because we must remain on this protocol is we want to stay anonymous
G_G 23:42:40: it should
G_G 23:42:50: and searches should be better and more deterministic
G_G 23:43:24: supernode cannot be distinguished by other nodes
G_G 23:43:28: they can share or node
G_G 23:43:29: not
G_G 23:44:13: it will be merged
G_G 23:44:20: by some kind of special messages
G_G 23:44:26: carrying IP and ID of a node
G_G 23:44:40: if this will stae it doesnt want to be anonyomus
G_G 23:44:54: so if another node doesn't want to be anonymus too
G_G 23:45:06: they will be able to extabilish a direct channel
G_G 23:45:53: actually the search process remain anonymous
G_G 23:48:46: i know... but a proper architecture is needed... and actually it is not a priority
G_G 23:49:05: exact
G_G 23:49:13: bu tin any case
G_G 23:49:25: direct connections will involve only filetransfers
G_G 23:49:30: not query forwarding
G_G 23:50:02: if you will be an half node you will connect directly to another halfnode only to share a file
G_G 23:50:25: well there is already an ip filtering system

Private Networks on ANts

G_G 23:50:56: I can set up an option that allow only connections with jeti buddies
G_G 23:51:11: no there is no way to do that...
G_G 23:51:24: but you will be able to have a rough extimation of the total sharesize
G_G 23:51:35: when the supernode system will be implemented
G_G 23:52:56: I'll see
G_G 23:53:15: you know these are not my issues
G_G 23:53:20: I'm not building a private netowkr
G_G 23:53:31: yes I'm just testing this for some days
G_G 23:53:39: to see if it is stable
G_G 23:53:48: as it looks
G_G 23:54:06: really dunnow how to test the new architecture
G_G 23:54:23: we should set up a small community
G_G 23:54:31: tsafa could be the supernode
G_G 23:55:22: don't you use it?
G_G 23:55:48: it's integrated in ants
G_G 23:56:53: mmm tell eric
G_G 23:57:02: that's a lot easy to implement
G_G 23:57:09: there is the bouncycastle library
G_G 23:57:17: full pgp support
G_G 23:57:42: go away from here!
G_G 20:10:51: hi

(taken from ANts IRC channel last night, only Gwrens words shown for brevity)

My message to Gwren:

Gwren why not use the private anonymous chat system to make direct connections as this would mean only the two parties making the direct connection would know IP addresses.

If you use queries then everyone knows but connection is not any better. Using anonymous messages would be more compatible with ANts as it would be PRIVATE NOT PUBLIC.

Also ANY IP address could be given as ants does not use IP addresses to route!

I know there is a moral question about using an IP address that is not your own.

Using anonymous IP messaging for direct connections also would not stop you from using half nodes.

Users that do not want some deniability and want to connect to hybrid network(emule/ANts) would use half nodes.

Others that want deniability would use anonymous IP messaging only. Other that want anonymity would use plain ANts.

Anyway just a thought :)


Hornet :bk :bk :bk

[lacitpo]

could someone past the full text somewhere?

[tsafa1]

Ants now has super nodes. Expect a lot of debugging. This is a big move.

[Abyss00]

I can't find the setting to tell Ants if you want to run in super node or not?

What is the difference between super node and normal user?

[tsafa1]

I can't find the setting to tell Ants if you want to run in super node or not?

What is the difference between super node and normal user?

When this is working right, all the peer ndeswill update a listof files they are sharing to the supernode. so now when search is done anywhere in the network, only the supernodes need be searched. The cuts down the workload on slower 56k users and puts it on fast connections.

[tsafa1]

re: 9.9c supernode version

This is what Gwren said:

This is just a middle-term test version... the supernode system is only partially implemented while the old system based on random queries is
deactivated, so actually now it's harder for nodes to find each others. The greatest problem is that now a supernode traces only the last seen source for a file, and obiously this is a huge limitation... I'm going to let supernodes track all the sources for a file so that a single query can give you all the current sources... (this release is actually helpful only to me to see if everything works...) also I've seen that there is a routing problem (some message cannot reach their destination), this is surely due to the latest changes cuz in 0.9.7.10 it was ok, so I have to find this problem.

[tsafa1]

ok, everyone on 9.9d

[tsafa1]

9.9e get it while its hot! :fire

all your files will be rehashed to new format for supernodes :green3