From pb at bieringer.de Fri Jun 6 15:47:25 2003 From: pb at bieringer.de (Dr. Peter Bieringer) Date: Fri Jun 6 14:47:31 2003 Subject: [ds6-devel] privoxy, how to prevent much unusable AAAA DNS queries Message-ID: <18128126.1054910845@[10.3.62.6]> Hi, first thanks for the IPv6 patch for privoxy, working here very well now (will publish an adjusted spec file soon). But I have a question: How can I prevent privoxy from doing so much unusable AAAA queries? Preface: Host running privoxy has an /etc/resolv.conf with 3 search domains specified. On each new host request, privoxy tries to resolv (example here): AAAA www.domain.tld. -> NXDOMAIN AAAA www.domain.tld.searchdom1. -> NXDOMAIN AAAA www.domain.tld.searchdom2. -> NXDOMAIN AAAA www.domain.tld.searchdom3. -> NXDOMAIN A www.domain.tld. -> IPv4 address same happen on using telnet, but only on telnet www.domain.tld 80 not on telnet www.domain.tld. 80 It's sure a glibc resolver issue (here used: 2.2.5 / RHL 7.3), but would it be possible to add a config switch to privoxy for "only resolve fqdn". But I have too less knowledge how to implement this in a proper manner. related function is imho in "jbsockets.c" addr_list *resolve_hostname_to_ip(const char *host, const char *port, int pf) { ... result = getaddrinfo(host, port, &hints, &res0); Does it help to check, whether "host" is not an IPv4 or an IPv6 address (how) and append a "." at the end (by copy into a new char* - don't forget to check max size - 1....). Peter -- Dr. Peter Bieringer http://www.bieringer.de/pb/ GPG/PGP Key 0x958F422D mailto: pb at bieringer dot de Deep Space 6 Co-Founder and Core Member http://www.deepspace6.net/ From pb at bieringer.de Fri Jun 6 17:43:25 2003 From: pb at bieringer.de (Dr. Peter Bieringer) Date: Fri Jun 6 17:33:10 2003 Subject: [ds6-devel] privoxy: feature request: mangle title line in HTML depending on IPv4 or IPv6 connectivity Message-ID: <25088465.1054917805@[10.3.62.6]> Hi, since using IPv6-enabled privoxy now for some hours it would be interesting to know (without digging through log), which page was retrieved via IPv4 or IPv6. The easiest way imho is modifying the title line (by config option...) Original: Title of page IPv4: IPv4 - Title of page IPv6: IPv6 - Title of page Or something similar. Comments? Peter -- Dr. Peter Bieringer http://www.bieringer.de/pb/ GPG/PGP Key 0x958F422D mailto: pb at bieringer dot de Deep Space 6 Co-Founder and Core Member http://www.deepspace6.net/ From pb at bieringer.de Fri Jun 6 17:11:32 2003 From: pb at bieringer.de (Dr. Peter Bieringer) Date: Fri Jun 6 17:33:11 2003 Subject: [ds6-devel] privoxy, how to prevent much unusable AAAA DNS queries - solved In-Reply-To: <18128126.1054910845@[10.3.62.6]> References: <18128126.1054910845@[10.3.62.6]> Message-ID: <23175264.1054915892@[10.3.62.6]> --On Freitag, 6. Juni 2003 14:47 +0200 "Dr. Peter Bieringer" wrote: > But I have a question: > > How can I prevent privoxy from doing so much unusable AAAA queries? > > Preface: > Host running privoxy has an /etc/resolv.conf with 3 search domains > specified. On each new host request, privoxy tries to resolv (example > here): > > AAAA www.domain.tld. -> NXDOMAIN > AAAA www.domain.tld.searchdom1. -> NXDOMAIN > AAAA www.domain.tld.searchdom2. -> NXDOMAIN > AAAA www.domain.tld.searchdom3. -> NXDOMAIN > A www.domain.tld. -> IPv4 address > > same happen on using telnet, but only on > telnet www.domain.tld 80 > not on > telnet www.domain.tld. 80 my colleague Harald Geiger found a poor-mans-solution ;-): Adjust initscript like below: start () { # start daemon echo -n $"Starting $PRIVOXY_PRG: " if [ -f $PRIVOXY_PID ]; then killproc $PRIVOXY_PRG && rm -f $PRIVOXY_LOCK $PRIVOXY_PID RETVAL=$? [ $RETVAL != 0 ] && return $RETVAL fi + export LOCALDOMAIN="." daemon $PRIVOXY RETVAL=$? echo [ $RETVAL = 0 ] && touch $PRIVOXY_LOCK return $RETVAL Peter -- Dr. Peter Bieringer http://www.bieringer.de/pb/ GPG/PGP Key 0x958F422D mailto: pb at bieringer dot de Deep Space 6 Co-Founder and Core Member http://www.deepspace6.net/ From lionel at mamane.lu Fri Jun 6 17:45:01 2003 From: lionel at mamane.lu (Lionel Elie Mamane) Date: Fri Jun 6 17:33:13 2003 Subject: [ds6-devel] privoxy, how to prevent much unusable AAAA DNS queries In-Reply-To: <18128126.1054910845@[10.3.62.6]> References: <18128126.1054910845@[10.3.62.6]> Message-ID: <20030606144501.GA21314@tofu.home.mamane.lu> On Fri, Jun 06, 2003 at 02:47:25PM +0200, Dr. Peter Bieringer wrote: > first thanks for the IPv6 patch for privoxy, working here very well > now You are welcome. > On each new host request, privoxy tries to resolv (example here): > AAAA www.domain.tld. -> NXDOMAIN > AAAA www.domain.tld.searchdom1. -> NXDOMAIN > AAAA www.domain.tld.searchdom2. -> NXDOMAIN > AAAA www.domain.tld.searchdom3. -> NXDOMAIN > A www.domain.tld. -> IPv4 address > same happen on using telnet, but only on > telnet www.domain.tld 80 > not on > telnet www.domain.tld. 80 This is true for privoxy, too: master@tofu:~/src/privoxy/privoxy/src$ telnet localhost 8118 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. GET http://foo.bar.com./ HTTP/1.0 and a tcpdump says: 16:31:49.710383 tofu.mamane.lu.35492 > quorn.mamane.lu.domain: [udp sum ok] 49133+ AAAA? foo.bar.com. (29) (len 37, hlim 64) 16:31:49.711657 tofu.mamane.lu.35492 > quorn.mamane.lu.domain: [udp sum ok] 49134+ A? foo.bar.com. (29) (len 37, hlim 64) > It's sure a glibc resolver issue (here used: 2.2.5 / RHL 7.3), Exactly. > but would it be possible to add a config switch to privoxy for "only > resolve fqdn". I don't think the getaddrinfo() interface permits to express this wish. So it is not possible, unless you delve into the DNS protocol (possibly with a library) and do the requests yourself. I don't think it is worth it. > related function is imho in "jbsockets.c" > addr_list *resolve_hostname_to_ip(const char *host, const char *port, int > pf) Yes, indeed. > Does it help to check, whether "host" is not an IPv4 or an IPv6 > address (how) and append a "." at the end Pff... Try to parse it (evil), or feed it to getaddrinfo with the AI_NUMERICHOST set. If that fails, add the "." and try again without the flag. But I don't think it is worth it. I don't see why these extra requests are a problem in the first place, and implementing work-arounds in every and each program is the wrong way to "fix" it. Fix getaddrinfo. -- Lionel -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 240 bytes Desc: not available Url : http://lists.deepspace6.net/pipermail/ds6-devel/attachments/20030606/072fbd8e/attachment.bin From lionel at mamane.lu Sat Jun 7 08:06:45 2003 From: lionel at mamane.lu (Lionel Elie Mamane) Date: Mon Jun 9 10:33:04 2003 Subject: [ds6-devel] privoxy: feature request: mangle title line in HTML depending on IPv4 or IPv6 connectivity In-Reply-To: <25088465.1054917805@[10.3.62.6]> References: <25088465.1054917805@[10.3.62.6]> Message-ID: <20030607050645.GA23385@tofu.home.mamane.lu> On Fri, Jun 06, 2003 at 04:43:25PM +0200, Dr. Peter Bieringer wrote: > since using IPv6-enabled privoxy now for some hours it would be > interesting to know (without digging through log), which page was > retrieved via IPv4 or IPv6. > The easiest way imho is modifying the title line (by config > option...) Interesting idea. I haven't dug into the filtering / modification code at all though, so I'd have to first understand how it works. But there is an "activity log" or "activity console" that got committed to CVS, but alas after I did my IPv6 porting job. I never set to IPv-6ising it, mainly because it is Java-based, and I don't have Java (non-free, blah, blah), so I can't test my changes. I guess the "best" solution would be adding the info on this activity console. Or maybe in the logs? That would be simpler for me, because it is in the part of the code I already know. Instead of: Connect: to www.foo.bar have: Connect (IPv6): to www.foo.bar or Connect: to www.foo.bar (IP address) Would this fulfill your wish? Oh no, you said "without digging through logs". Oh well. -- Lionel From pb at bieringer.de Sat Jun 7 11:16:29 2003 From: pb at bieringer.de (Peter Bieringer) Date: Mon Jun 9 10:33:06 2003 Subject: [ds6-devel] privoxy: feature request: mangle title line in HTML depending on IPv4 or IPv6 connectivity In-Reply-To: <20030607050645.GA23385@tofu.home.mamane.lu> References: <25088465.1054917805@[10.3.62.6]> <20030607050645.GA23385@tofu.home.mamane.lu> Message-ID: <24790000.1054973789@worker.muc.bieringer.de> --On Saturday, June 07, 2003 07:06:45 AM +0200 Lionel Elie Mamane wrote: > Interesting idea. I haven't dug into the filtering / modification code > at all though, so I'd have to first understand how it works. > > But there is an "activity log" or "activity console" that got committed > to CVS, but alas after I did my IPv6 porting job. I never set to > IPv-6ising it, mainly because it is Java-based, and I don't have Java > (non-free, blah, blah), so I can't test my changes. I guess the "best" > solution would be adding the info on this activity console. > > Or maybe in the logs? That would be simpler for me, because it is in > the part of the code I already know. Instead of: > > Connect: to www.foo.bar > > have: > > Connect (IPv6): to www.foo.bar > > or > Connect: to www.foo.bar (IP address) > > > Would this fulfill your wish? Oh no, you said "without digging through > logs". Oh well. A related eintry in the log would be always nice...but it's not so transparent to normal users. I discussed with my colleague what on the filtered HTML can be changed easily without causing too much trouble on layout. We found - manipulation of title in header - add a footer Second is not so nice, because you always have to scroll down, so I would wish a manipulation of the title Peter -- Dr. Peter Bieringer http://www.bieringer.de/pb/ GPG/PGP Key 0x958F422D mailto: pb at bieringer dot de Deep Space 6 Co-Founder and Core Member http://www.deepspace6.net/ From qesmith at lacademy.edu Mon Jun 23 12:01:54 2003 From: qesmith at lacademy.edu (Quentin Smith) Date: Mon Jun 23 17:05:28 2003 Subject: [ds6-devel] Re: privoxy ipv6 patch - PRIVOXY-FORCE doesn't work? In-Reply-To: <20030623104237.GB3043@tofu.home.mamane.lu> References: <20030623104237.GB3043@tofu.home.mamane.lu> Message-ID: Lionel Elie Mamane writes: >Hi, > >On Sat, Jun 21, 2003 at 07:29:25PM -0400, Quentin Smith wrote: > >> I've been using your IPv6 patch for Privoxy for a few months. I've >noticed >> a couple of things: > >> - Some web sites just time out. Packet sniffing shows that Privoxy >> successfully gets the page, but then returns no data to the web >> browser. > >Hmm. Do you have any specific example, so that I can test? I.e., >please give me a few URL's that exhibit this problem. Hi- http://www.hb-rights.org/ does it for me on Internet Explorer 6 connected through Privoxy on my linux server One things to note: www.hb-rights.org is hosted on the same machine as Privoxy is running on. I'm not sure if this would affect the problem. Internet Explorer 6 throws up the "Cannot find server" generic error. Some other web browsers just hang. Mozilla shows "Document contains no data". If I add an exception to the proxy usage for hb-rights.org, it works just fine. Turning off all filtering for hb-rights.org in Privoxy does /not/ solve the problem. I've also seen a similar problem with http://www.aclu.org/ but I think that might be a bug in Internet Explorer, because non-IE clients can load it just fine. This also occurs with other sundry websites. I can't think of any off the top of my head, but it was annoying enough that I moved to a less than ideal host-based blocking mechanism on my regular web browser. --Quentin Quentin Smith - http://www.comclub.org/~quentins/about ComClub International - http://www.comclub.org/ quentins@comclub.org qesmith@lacademy.edu Two wrongs don't make a right, but they make a good excuse. -- Thomas Szasz From lionel at deepspace6.net Mon Jun 23 20:19:28 2003 From: lionel at deepspace6.net (Lionel Elie Mamane) Date: Mon Jun 23 19:19:43 2003 Subject: [ds6-devel] Re: privoxy ipv6 patch - PRIVOXY-FORCE doesn't work? In-Reply-To: References: <20030623104237.GB3043@tofu.home.mamane.lu> Message-ID: <20030623171928.GB11974@tofu.home.mamane.lu> On Mon, Jun 23, 2003 at 11:01:54AM -0400, Quentin Smith wrote: > Lionel Elie Mamane writes: >>> - Some web sites just time out. Packet sniffing shows that Privoxy >>> successfully gets the page, but then returns no data to the web >>> browser. >> Hmm. Do you have any specific example, so that I can test? I.e., >> please give me a few URL's that exhibit this problem. > http://www.hb-rights.org/ does it for me on Internet Explorer 6 > connected through Privoxy on my linux server Packet sniffing for me shows that Privoxy drops the connection early: Not the whole document goes through. Please confirm if in your case privoxy gets the *whole* page, or not. Anyway, it is an upstream bug: It shows up even if my patch is not applied. You might want to consider to report it upstream. (I fear the answer you'll get will be along the lines of "no time, we are concentrating on improving the 3.0.x line", though) > One things to note: www.hb-rights.org is hosted on the same machine > as Privoxy is running on. I'm not sure if this would affect the > problem. I don't think this changes anything. This site is served by Zope. Do you know any other web site served by Zope? Does it work? Let's try www.zope.org. That one works. OK. Are there any other web sites hosted on the same machine as www.hb-rights.org? Do they work? -- Lionel From lionel at deepspace6.net Tue Jun 24 06:32:16 2003 From: lionel at deepspace6.net (Lionel Elie Mamane) Date: Tue Jun 24 05:32:22 2003 Subject: [ds6-devel] Re: privoxy ipv6 patch - PRIVOXY-FORCE doesn't work? In-Reply-To: References: <20030623104237.GB3043@tofu.home.mamane.lu> <20030623171928.GB11974@tofu.home.mamane.lu> Message-ID: <20030624033216.GA26083@tofu.home.mamane.lu> On Mon, Jun 23, 2003 at 07:41:33PM -0400, Quentin Smith wrote: > Lionel Elie Mamane writes: >> On Mon, Jun 23, 2003 at 11:01:54AM -0400, Quentin Smith wrote: >>> Lionel Elie Mamane writes: >>> http://www.hb-rights.org/ does it for me on Internet Explorer 6 >>> connected through Privoxy on my linux server >> Packet sniffing for me shows that Privoxy drops the connection >> early: Not the whole document goes through. Please confirm if in >> your case privoxy gets the *whole* page, or not. > Nope, I do get the whole page. Here's a packet capture: > http://www.comclub.org/~root/hbrights.pcap. This might just be > because of the huge MTU on the loopback interface. Hmm... Maybe, maybe. But I see one strange thing in your packet capture: The second line sent by the HTTP server contains a few dozens of NULL characters in a row. This looks like an HTTP server bug, and I can quite understand that this confuses privoxy, because written in C, and strings in C are NULL-terminated. But there is still another problem left, because I can't get hb-rights.org through privoxy, and I don't get any NULL's from the server. >> Anyway, it is an upstream bug > Who should I report this to? I have no experience contacting the > Privoxy developers :) http://www.privoxy.org/user-manual/contact.html#CONTACT-BUGS > I should mention that I had to patch your patch to get it to compile > on my (relatively) old glibc - find and replace ss_family with > __ss_family. It's possible this affects the problem, though I doubt > it. It isn't related to my patch at all: The problem is present with an unpatched upstream CVS, too. -- Lionel -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 240 bytes Desc: not available Url : http://lists.deepspace6.net/pipermail/ds6-devel/attachments/20030624/2390b0df/attachment.bin From qesmith at lacademy.edu Mon Jun 23 20:41:33 2003 From: qesmith at lacademy.edu (Quentin Smith) Date: Tue Jun 24 23:14:45 2003 Subject: [ds6-devel] Re: privoxy ipv6 patch - PRIVOXY-FORCE doesn't work? In-Reply-To: <20030623171928.GB11974@tofu.home.mamane.lu> References: <20030623104237.GB3043@tofu.home.mamane.lu> <20030623171928.GB11974@tofu.home.mamane.lu> Message-ID: Lionel Elie Mamane writes: >On Mon, Jun 23, 2003 at 11:01:54AM -0400, Quentin Smith wrote: >> Lionel Elie Mamane writes: > >>>> - Some web sites just time out. Packet sniffing shows that Privoxy >>>> successfully gets the page, but then returns no data to the web >>>> browser. > >>> Hmm. Do you have any specific example, so that I can test? I.e., >>> please give me a few URL's that exhibit this problem. > >> http://www.hb-rights.org/ does it for me on Internet Explorer 6 >> connected through Privoxy on my linux server > >Packet sniffing for me shows that Privoxy drops the connection early: >Not the whole document goes through. Please confirm if in your case >privoxy gets the *whole* page, or not. > >Anyway, it is an upstream bug: It shows up even if my patch is not >applied. You might want to consider to report it upstream. (I fear the >answer you'll get will be along the lines of "no time, we are >concentrating on improving the 3.0.x line", though) Hi- Nope, I do get the whole page. Here's a packet capture: http://www.comclub.org/~root/hbrights.pcap. This might just be because of the huge MTU on the loopback interface. Who should I report this to? I have no experience contacting the Privoxy developers :) > > >> One things to note: www.hb-rights.org is hosted on the same machine >> as Privoxy is running on. I'm not sure if this would affect the >> problem. > >I don't think this changes anything. > >This site is served by Zope. Do you know any other web site served by >Zope? Does it work? Let's try www.zope.org. That one works. The site is served by the Plone framework running on Zope. www.plone.org loads just fine and consists of identical HTML code to my site (layout code; the contents is obviously different). > > >OK. Are there any other web sites hosted on the same machine as >www.hb-rights.org? Do they work? Yes. My personal web domain, www.comclub.org, is also hosted on the same machine. I've seen this problem with a few pages in the domain, but not everything in the domain. That's why this is so frustrating: it seems to depend on both the server *and* the HTML code. I should mention that I had to patch your patch to get it to compile on my (relatively) old glibc - find and replace ss_family with __ss_family. It's possible this affects the problem, though I doubt it. Thanks for all your help, --Quentin Quentin Smith - http://www.comclub.org/~quentins/about ComClub International - http://www.comclub.org/ quentins@comclub.org qesmith@lacademy.edu Two wrongs don't make a right, but they make a good excuse. -- Thomas Szasz From qesmith at lacademy.edu Tue Jun 24 01:17:59 2003 From: qesmith at lacademy.edu (Quentin Smith) Date: Tue Jun 24 23:14:45 2003 Subject: [ds6-devel] Re: privoxy ipv6 patch - PRIVOXY-FORCE doesn't work? In-Reply-To: <20030624033216.GA26083@tofu.home.mamane.lu> References: <20030623104237.GB3043@tofu.home.mamane.lu> <20030623171928.GB11974@tofu.home.mamane.lu> <20030624033216.GA26083@tofu.home.mamane.lu> Message-ID: Lionel Elie Mamane writes: >On Mon, Jun 23, 2003 at 07:41:33PM -0400, Quentin Smith wrote: >> Lionel Elie Mamane writes: >>> On Mon, Jun 23, 2003 at 11:01:54AM -0400, Quentin Smith wrote: >>>> Lionel Elie Mamane writes: > >>>> http://www.hb-rights.org/ does it for me on Internet Explorer 6 >>>> connected through Privoxy on my linux server > >>> Packet sniffing for me shows that Privoxy drops the connection >>> early: Not the whole document goes through. Please confirm if in >>> your case privoxy gets the *whole* page, or not. > >> Nope, I do get the whole page. Here's a packet capture: >> http://www.comclub.org/~root/hbrights.pcap. This might just be >> because of the huge MTU on the loopback interface. > >Hmm... Maybe, maybe. But I see one strange thing in your packet >capture: The second line sent by the HTTP server contains a few dozens >of NULL characters in a row. This looks like an HTTP server bug, and I >can quite understand that this confuses privoxy, because written in C, >and strings in C are NULL-terminated. Hi- Mozilla went crazy, continually reloading the page. I think that might have had something to do with the odd capture. In any case, this seems to be a known issue with Privoxy HEAD: http://sourceforge.net/tracker/index.php?func=detail&aid=639914&group_id=11118&atid=111118 > > >But there is still another problem left, because I can't get >hb-rights.org through privoxy, and I don't get any NULL's from the >server. > Thanks a lot, --Quentin Quentin Smith - http://www.comclub.org/~quentins/about ComClub International - http://www.comclub.org/ quentins@comclub.org qesmith@lacademy.edu Two wrongs don't make a right, but they make a good excuse. -- Thomas Szasz