Bandwidth low? Ideas.

[SomebodyElse]

Most connection types used by home users have greater download speed than upload speed (eg. V90 modem, ADSL).

In current P2P networks, the total amount of upload data is equal to the total amount of download data, so, overall, we are limited to the _upload_ speed of users in general.

Here are a couple of ideas to get better usage out of the upload speed of user nodes.

1) Dialup users.

Typical modems compress data. The compression used works when the data has repetitions. The idea here is to have the P2P app. have fine control over the data flow (the probably means using UDP and writing our own protocol over UDP rather than using TCP) so we can contrive to upload a packet to user A, then the _same_ packet to user B, user C etc.

As long as the data packets are small enough to fit in the modem's compression dictionary, we probably get the transmission to users B, C etc. almost for free... A user on dialup would still only upload 3k/second or so, but would be uploading 3k/second to each of several other users.

Obviously, this idea can only work if we can find more than 1 node wanting to download the same chunk of a given file, but I doubt this will be a problem ;-)

2) Email.

Typically, when sending an email to several destinations, one copy of the email gets sent from the user's machine to the ISP's mail server. The ISP's server then sends the email on to all the recipients. If we create a P2P file sharing app. that works over email, we can have a node that has a wanted chunk of a file send it to almost any number of other peers at the same time.

In the simple case, imagine I have a 5 megabyte file that 20 people want. I simply email it to all 20 at once. I have to upload the file _once_ and it is sent to everybody who wants it. The data is actually larger because of the way email encodes binary data, so I upload 6.7 megs, and 20 people all download 6.7 megs, for a payload data transfer of 100megs.

Read that again: I uploaded 6.7 megs and achieved 100megs of payload data transfer - my effective upload speed is 15 times my connection speed!



I do not have spare time to work on programming to turn these ideas into useful applications - but, given the way that ports are being blocked, internet traffic is being monitored etc. I would like to suggest that a hybrid system which used traditional P2P methods to search and locate files and used email for bulk data transfer might be a useful tool.


The idea based on the compression built into modems might be earier to 'bolt on' to an existing app - worth thinking about if you are a developer.

[Krell]

Email servers were never never never never never never intended to send 5Mb attachments ! Just because some allow you to, doesnt mean it's not a bobo idea.

So exactly WHOS email servers do you think is going to handle this? I would shut you out and lock you down in a heartbeat. This is a classic example of the greed and lack of concern for the infrastructure.

I view that as a greedy and irresponsible approach.

SPAMMERS - Stay off our email servers and redirect your idea at multicasting other ways.


my 2 cents

[endersgame21]

Actually, I think it is a pretty interesting idea. He isn't a spammer and maybe they could make a email server just for this and is built to handle it. If one comes out I think it would be pretty useful.

[SomebodyElse]

I would like to point out:

1) Emailing somebody a file they have asked for is not usually regarded as spam!

2) It is true that some email relays would not like a 5M attachment. I chose this size as an easy illustration. A better chunksize for a real implementation might be 64k as this size has been used by ftp-over-email servers for years.

However, I do take the point that this idea would place quite a bit of load on email servers. Whether this would be excessive or not is not known to me - maybe someone who knows would like to comment on this.

There may be other methods of multicasting a file - I would be interested to hear of any - I started think about this issue when I noticed that I had downloaded more data in the last week than I had uploaded, and I don't particularly like leeching.

[woppi]

There is a java based file sharing client called CoolShare which uses email. It can be found at http://www.coolderry.com/ . It is for sending large files to one or more people, but it segments the files and can send those segments after a delay. So there is no major pressure put on your SMTP server, or on the inbox of the recipients. So it's usefull if either party have low bandwidth, or firewall restrictions on file sharing. The person(s) you send the email to must have CoolShare to reassemble the segments into the original file. One drawback is that it doesn't have a 'pull' facility so that others can request a file and get it sent automatically.

woppi

[Theinfamousone]

Ok, ever since I started using Kazaa almost 2 years ago I have been trying to email the programmers to impliment a multicast sort of system. Obviously Kazaa is the LAST P2P that will ever take the time to come up with it.

When I download files on eMule, I can be downloading a file, and as I get it, I am sending it to someone else, it's like Bittorrent and what have you. (Great idea BTW except for you have about a 50/50 chance of not even having the ability to finish files because once the person that has the complete file is gone, it's gone for good, someone please correct me if there is another way. From my experience, the speeds and quality of files are much better than most P2P programs, but about half the time the files just cut out and will never start if I let it sit there for the rest of my life because no one is left with the complete file.) Anyways, the email example is a good demonstration, but even though you are uploading the file once, your email provider still that has to send the 6.7MB 20 times doesn't it? I mean, someone has to send the data right?

If ever the day comes where a P2P app support TRUE multicasting (if it's even possible), P2P will be infinitely faster.

Combining multicasting with multisourcing, and mauhahaha, "I am INVINCIBLE!"

[isus]

but wouldnt the email server have a copy on the server? i mean, the riaa could look for it, and find you... and then you're screwed.

but i dont know about mail servers, so... if its not true that a copy of the email is left on the server from the sender then ignore me.

[Nothingface5384]

if i dont think they do..theires too many ueser for the server to keep a copy of a sent file..it would like overload or something lol
but then again..who knows....

[metale]

I think is a good idea.... but nothing else.

I mean... we'll have to pay to teh email server, or something like that... and....the p2p idea is to share... get things for free...
i don't know....

[endersgame21]

Originally posted by metale
I think is a good idea.... but nothing else.

I mean... we'll have to pay to teh email server, or something like that... and....the p2p idea is to share... get things for free...
i don't know....
Why would you have to pay for anything. Do you pay for your e-mail right now. Yahoo Mail is free and so is Hotmail and a few others. I know they would probably st up a new email service but they could make money off of advertisments and allowing people to spam.

[shellreef]

Originally posted by SomebodyElse
Typical modems compress data. The compression used works when the data has repetitions. The idea here is to have the P2P app. have fine control over the data flow (the probably means using UDP and writing our own protocol over UDP rather than using TCP) so we can contrive to upload a packet to user A, then the _same_ packet to user B, user C etc.
[...]
Obviously, this idea can only work if we can find more than 1 node wanting to download the same chunk of a given file, but I doubt this will be a problem ;-)

Nice idea - you should definitely suggest this to the BitTorrent developers (http://groups.yahoo.com/group/BitTorrent/), as their system seems to have a high probability of "more than 1 node wanting to download the same chunk of a given file". Implementing this would be easier over UDP but I believe it'll work with TCP just as well. Disable Nagle's algorithm so the data won't be buffered, and send data in chunk with sizes equal to the MSS, one to each peer receiving the file. Immediately flush the sockets of all peers receiving the same file, and the packets can be sent at roughly the same time. If the MSS is too great and the modem doesn't compress identical payload packets, the segment size could be lowered within the program.

2) Email.
What you are suggesting just places the burden of bandwidth on SMTP servers, instead of individual users. True, SMTP servers often have high bandwidth and if your ISP doesn't provide an SMTP server or you can't use it, there are plenty of open relays (as used but spammers, but as you pointed out, spam is only unsoliciated email) that can be used.

Krell, I run my own SMTP server and its pretty cool having someone email you and the message is already on your disk; there is no need to download from a third party. My email server can handle large messages, since I run it is under my control and what I do with it does not show greed or lack of concern for the Internet infrastructure, and its not "our" server. Its mine. Undoubtedly, abusing someone else's SMTP server to send 700MB to 1000 users will have a noticable impact on that server; many mail servers limit how many addresses you can Cc for this reason. But if everyone runs their own server it wouldn't be a problem.

Email is interesting for file sharing to me, because its a widely available and standardized "push" technology, not unlike IRC DCC. More common is "pull", where everyone that wants the file must ask for it. On Audiogalaxy, one could join groups and have the latest files sent to them as soon as they were released -- not only did the timely transfer please people that wanted the file(s), but it improved swarming immensely because of the increased amount of potential uploaders. To combat spam and malicious users sending viruses to everyone, groups could be created with limited sending abilities (i.e., only operators could send).

isus & NothingFace: nope, the servers don't store messages they relay.

The data is actually larger because of the way email encodes binary data, so I upload 6.7 megs
You don't have to; the base64 (or UUE, or XXE - just different alphabets) encoding encodes 3 arbitrary bytes to 4 ASCII characters, for a file size increase of 75%. Its annoying, but not intractable - yEnc (http://www.yenc.org/) only replaces unsafe characters, for a 1-2% (on average) bloat, and RFC1652 (http://www.nada.kth.se/sunet-mime/ref/rfc1652.txt) specifies an 8-bit transport extension for SMTP but it isn't widely implemented.

In regards to multicasting - its very possible, and very efficient, and supported by standard IP protocols. The technology is known as "multicasting", there is also unicast (sending to one host, like most of the Internet uses) and broadcast (sending to a group of hosts, like your LAN), as well as anycast (supported by IPv6). Basically, multicasting works using the Internet Group Management Protocol (http://www.cis.ohio-state.edu/cgi-bin/rfc/rfc2236.html), hosts with a membership in a multicast group. Certain IP addresses (http://www.iana.org/assignments/multicast-addresses), beginning with 224-239, are used for multicasting group management (the next generation IP also has its own reserved IP addresses (http://www.iana.org/assignments/ipv6-multicast-addresses)). This occurs on the MBone (http://www-itg.lbl.gov/mbone/) (How to connect to the MBone (http://www.live.com/mbone/)) which is a subset of the Internet, and is mostly used for video conferencing. It works, and people use it.

So why isn't it available to you? I don't know of any residential ISP that supports multicast. Your ISP needs special routers. Even home routers will need to be upgraded. The technology for multicasting is there, but non-commercial ISPs don't seem to jump at the chance of offering it. We're not using the Internet to its full potential because of this, but what if multicasting was widespread, and P2P programs were written to use it? Files could be sent over the Internet to large numbers of hosts efficiently. Bandwidth would be redefined as not how much data you send, but how much unique data you send. The Internet would completely change, I predict.

These ISPs support multicasting (http://www.multicast-isp-list.com/). Some even provide dialup and DSL with multicast support. There have been some efforts to make a multicast P2P, such as the Emcast Multicast Library (http://www.junglemonkey.net/emcast/). Hopefully, new P2P programs will support but not require multicast and take advantage of it if possible. If you want multicast, ask your current ISP, or find one on that list.

Hope this helps,
-shellreef
P.S. Sorry for the length of this reply.

Update: Here's a very good multicasting FAQ (http://www.multicasttech.com/multicast_faq.html), with links to sites that provide multicasted content. On The I (http://www.on-the-i.com/) has multicast streaming audio; it seems ideal. But read above if you can't play it.

[BigMole]

Typical modems compress data. The compression used works when the data has repetitions. The idea here is to have the P2P app. have fine control over the data flow (the probably means using UDP and writing our own protocol over UDP rather than using TCP) so we can contrive to upload a packet to user A, then the _same_ packet to user B, user C etc.

As long as the data packets are small enough to fit in the modem's compression dictionary, we probably get the transmission to users B, C etc. almost for free... A user on dialup would still only upload 3k/second or so, but would be uploading 3k/second to each of several other users.

That is a really, really smart idea! The idea in itself is totally correct. But from what I remember about how the compression works in today's modems it won't work. But with future modems with a longer "compression memory" it should work perfectly!

I have been working as a researcher in computer communications and computer security for many years and the last few years I have spent full time on p2p research. About ten years ago I read about the compression used in today's modems. Here's what I remember: The error correction standard used is called v42. And when they added compression they called it v42" (pronounced v42bis). V42bis packs 40 bytes of data into one block and ads error correction (probably 16 bits one's complement checksum, same as used in TCP/IP) and does LZW compression on the data. Unfortunately in v42bis the compression only "remembers" the data in the current block being compressed. That is 40 bytes. Since IP-packets has a 40 bytes header we can't send packets small enough to use the compression effect when sending to several recipients.

But I hope I remember wrong and the compression used in modems has a longer memory than 40 bytes. It should be easy to test, just send a bunch of small UDP/IP packets with the same data to several different recipients and se how much data you get through. If it works really well you should be able to get through almost four times as much data as the modem normally can send. (According to tests some of my friends have done the modems can compress data down to 25% the size.) I just realised a simpler way to test the modem's compression memory length: Make a big file with random data but split the data into "faked" IP blocks. Each block should be at least 80 bytes since the IP-header is 40 bytes. Then repeat each block in the file several times. The file can then be sent over an FTP upload or download to test the modem's compression. I just might do this test one day...

But then of-course you need to have several downloaders wanting the same file from the same user at the same time. Considering that users on average share 130 files that will only work for files that are big and rare, like a new "release" of a movie on a p2p network...

[Rickio]

I've sent music files many times over email. You simply must find one that accepts fairly large attachments.

2 that do accept up to 10mb file attachments are http://lycos.co.uk and http://hotbox.ru , when signing up for hotbox look for the english translation link, it was in upper right corner last time I looked.


I am sure there are a few more free email services that allow such large file attachments but I have not had to look since I use these from time to time.


peace

[reg]

Originally posted by Rickio
I've sent music files many times over email. You simply must find one that accepts fairly large attachments.

2 that do accept up to 10mb file attachments are http://lycos.co.uk and http://hotbox.ru , when signing up for hotbox look for the english translation link, it was in upper right corner last time I looked. . . . peace

...& just a fyi~if anyone does decide to receive large files via one of these emails (yup, i have received music files many times over email :;) ) ... i've found (with my rinky-dinky modem) that hotbox worked better than lycos ... hotbox downloaded faster; lycos, on a few files, wouldn't deliver the whole file ...
:upside

[noxiousneo]

or you could just get Broadband.

[shellreef]

Originally posted by noxiousneo
or you could just get Broadband.
Even if you got broadband, many types of broadband (residential DSL) are very limited in upload capacity. Email servers usually have much more upload bandwidth than a broadband line.

[metale]

Originally posted by shellreef
Even if you got broadband, many types of broadband (residential DSL) are very limited in upload capacity. Email servers usually have much more upload bandwidth than a broadband line.

Yeah but noxiousneo is right... you can just get broadband and stop thinking difficult ideas!

[shellreef]

Originally posted by metale
Yeah but noxiousneo is right... you can just get broadband and stop thinking difficult ideas!
Broadband does not solve the problem. I have broadband, and can only upload at 128kbps, while I can download at 1500Kbps. Directly transferring files between two DSL users of with the same speeds is not optimal -- you can achieve transfer rates about only 3 times that of a dial-up modem, in the best case. Email servers act as a sort of proxy, which the sender first uploads to (slowly) and the receiver downloads from (quickly). This wastes much bandwidth because the file has to be transferred twice.

I don't think throwing more bandwidth at the problem is effective. Sure it helps, but even today, new specifications are being developed to increase transfer rates. For example, selective acknowledgements are a relatively new TCP feature that Windows implements. SACKs can improve transfer rates significantly, but not all stacks yet implement them (FreeBSD comes to mind). It goes back to the old adage, "work smarter, not harder".

Anyone know if DSL compresses data during transmit to the CO? If so similar technology could applied to broadband.

[Krell]

DSL is by its nature already compressed, so it is applied to broadband. Your last post is correct and is about as close as it comes to streamlining transport.

Also Metale isnt versed in high speed communications, he just likes to remark on anything, so you dont have to respond.

http://www.vicomsoft.com/knowledge/reference/pppoe.html
http://www.adsl.com
http://www.westell.com
http://www.telechoice.com
http://www.adsl.com/adsl_forum.html
http://www.flashcom.com
http://www.americasnetwork.com
http://www.netspeed.com
http://www.adsl.com/menupub.html
http://www.centillium.com
http://www.analog.com
http://www.dslcenter.com
http://www.visi.com
http://www.sparnex.com
http://www.alcatel.com
http://www.acad.humberc.on.ca
http://think.atr.net/adsl.html
http://www.iols.net/dsl/whatisdsl.htm