[Web10g-user] Maximum polling frequency with web10g

Mon Mar 17 11:59:23 EDT 2014

Good afternoon,

Sorry for the late answer. I did not install web10G myself but the 
person who did it gave me the following informations:

" I have used Ubuntu 13.04 which comes with Kernel 3.8.
1) this is the link for the kernel
https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.8.tar.gz

2) this is the link for the appropriate version of web10g
http://web10g.org/index.php?option=com_remository&Itemid=65&func=fileinfo&id=63 

Please read the help file attached to install the kernel patch and then 
follow the install instruction in web10g package to install libmnl and 
eventually WEb10g. "

I do not know if this should help finding the errors I have

My script is using listcnxs cmd and poll it every 100 ms. A sequence of 
bash cmds is then launched to format the result of this cmd and check 
the state of the cnxs. When the connection is detected as finished (one 
of the final state of TCP) the formatted logs are uploaded to a remote 
database.

Please keep me updates if you have some news !

Thank you very much

Hadrien

Le 12/03/2014 02:27, Chris Rapier a écrit :
> No need to apologize. This is the sort of information we need. I'll be 
> discussing this withthe team this week and running some of our own 
> tests. I'll be back in touch late tomorrow to get more information.
>
>
> Sent with AquaMail for Android
> http://www.aqua-mail.com
>
>
> On March 11, 2014 8:06:37 PM Sebastian Zander <szander at swin.edu.au> 
> wrote:
>
>> Hi Chris,
>>
>> Sorry for crashing into this thread, but we just stumbled over what 
>> looks like exactly the same problem.
>>
>> We run a series of experiments where in each experiment we generate 
>> TCP traffic for a few minutes, each experiment has about 10 flows or 
>> so and in addition we have SSH control traffic before and after. We 
>> use openSuSE with a vanilla Linux 3.9.8 kernel with the 0.7 kernel 
>> patch and 2.0.7 userland code. We poll web10g often, time between 
>> poll is 10-20ms. Our machines (with 4GB RAM and 1GB swap) crash after 
>> a number of experiments, in the order of 50--100 experiments.
>>
>> Different error messages, but at least some point to memory issues:
>> "Kernel panic - not syncing: Out of memory and no killable processes..."
>> "Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0"
>> I have screen dumps of two crashes, can send if you want them.
>>
>> Quick and not very thorough look at memory consumption seemed to 
>> indicate we're loosing memory somewhere, albeit very slowly. With 
>> same kernel but without polling web10g we seem to have no issues. 
>> Since it takes a while to crash, as a workaround we reboot more often 
>> now...
>>
>> Unfortunately, I'm a bit limited on accessing the actual testbed at 
>> the moment due to experiments running, but let me know if you need 
>> more information.
>>
>> Cheers,
>>
>> Sebastian
>>
>> On 12/03/2014 3:25 AM, rapier wrote:
>> > I'd like to apologize for the delay on this. Since you aren't 
>> subscribed
>> > to the list this ended up being held by our mailing list software. 
>> I've
>> > been swamped with grant writing work and getting web10g finalized.
>> >
>> > Could you tell me a bit more about the parameters of your experiment?
>> > How long is the tcp flow running? At what point do you run out of
>> > memory? Are there any error messages that you are seeing on crash? 
>> Which
>> > version of the kernel patch set and userland are you using?
>> >
>> > Let me know and we'll look into this. If you have specific code 
>> that you
>> > are using to run this experiment it would you mind sending it to me 
>> so I
>> > can try to recreate the problems?
>> >
>> > Chris Rapier
>> >
>> >
>> > On 2/24/14, 6:10 AM, Hadrien Hours wrote:
>> >> Hi,
>> >>
>> >> I am not sure whether this is the correct mailing list to ask this
>> >> question or not but that's the best I have found.
>> >>
>> >> I am currently using the latest version of web10g on Ubuntu 13.04
>> >>
>> >> I am trying to track the evolution of TCP state by polling tcp stack
>> >> parameters every 100 ms. By doing so I have the machine crashing 
>> (memory
>> >> shortage while my conf on the machine has 4GB)
>> >>
>> >> Reducing to 1s the machine ends up crashing also but after a longer
>> >> running time (up to several hours so far).
>> >>
>> >> Does someone already experiment the same problem ? And had found a
>> >> solution ?
>> >>
>> >> Thank you very much !
>> >>
>> >> Hadrien
>> >> _______________________________________________
>> >> Web10g-user mailing list
>> >> Web10g-user at web10g.org
>> >> https://lists.psc.edu/mailman/listinfo/web10g-user
>> >>
>> >> To UNSUBSCRIBE visit 
>> https://lists.psc.edu/mailman/unsubscribe/web10g-user
>> >>
>> > _______________________________________________
>> > Web10g-user mailing list
>> > Web10g-user at web10g.org
>> > https://lists.psc.edu/mailman/listinfo/web10g-user
>> >
>> > To UNSUBSCRIBE visit 
>> https://lists.psc.edu/mailman/unsubscribe/web10g-user
>> >
>> >
>>
>>
>> -- 
>> Sebastian Zander
>> http://caia.swin.edu.au/cv/szander/
>
>

-- 
/Hadrien Hours
PhD Student at Eurecom
Networking and Security Dept./
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.psc.edu/pipermail/web10g-user/attachments/20140317/78e85808/attachment.html 
-------------- next part --------------

Building a Web10G Kernel 
                    =========================

This document covers general instructions and advice on how to:

    - apply the web10G patches to a linux kernel
    - build the kernel, and
    - install the newly built kernel on the system

BEFORE YOU PROCEED
------------------

Before you proceed, it may be a good idea to take the following steps
so that you can backtrack if you run into problems:

   - Make a copy of the current kernel
   - Make a copy of the .config file (usually in /usr/src/linux)
   - Tar up the files in the /lib/modules directory corresponding
     to the current working kernel

GETTING THE KERNEL SOURCES
------------------------------

If you do not have it, download the sources for kernel version 2.6.16
ftp.kernel.org or one of its many mirrors (http://www.kernel.org/mirrors/).
Extract the sources:

# tar -C /usr/src -xvf linux-3.8.xxxx.tar.gz.

Make a symbolic link:

# rm -rf /usr/src/linux
# ln -s /usr/src/linux-3.8 /usr/src/linux

APPLYING WEB100 PATCH
---------------------

# cd /usr/src/linux
# patch -p1 < [path-to-web10g-dir]/web10g-[version-name].patch

---------------------------------------------------------------------

The patch command should complete successfully without any errors.
There is, however, one file for which the patch may fail. This situation is
explained in the "EXTRAVERSION" section below. 

EXTRAVERSION
------------

The top level kernel Makefile defines "EXTRAVERSION" (typically on line 4).
EXTRAVERSION determines the suffix of the kernel version name.  It is used for
a number of things, including determining the directory name used by loadable
kernel modules. 

    - If EXTRAVERSION is not updated by web10g (ie. patch fails), then
      you should edit the Makefile by hand and append "-web10g" to the
      EXTRAVERSION string (no spaces between the exisiting string the
      added string).

CONFIGURING AND BUILDING THE KERNEL 
----------------------

We have found it to be safer to force a full rebuild:

# cd /usr/src/linux
# cp .config config.save
# make mrproper
# cp config.save .config

# make menuconfig
(you may need to install ncurses libraries before this by "apt-get install libncurses5-dev" )

To enable web10g features select the following options: 

    The pertinent configure options are:
		Networking support --->
		  Networking options --->
		    [*] TCP: Extended TCP statistics (TCP ESTATS) MIB

	This will enable the loadable kernel module:
		Networking support --->
		  Networking options --->
		    [*] TCP: Extended TCP statistics (TCP ESTATS) MIB
		    <M>   TCP: ESTATS netlink module

	Previous versions encouraged setting the default congestion control
	to RENO:

		Networking support --->
		  Networking options --->
		    [*] TCP: advanced congestion control --->
		        Default TCP congestion control (Reno)

	However accommodations for CUBIC, BIC, and H-TCP are now implemented.

	The loadable kernel module produced is named "tcp_estats_nl".

Save menuconfig options.

Now build the kernel and modules:

# make

Start compiling to kernel modules:
# make modules 

Install kernel modules (become a root user)
# make modules_install

So far we have compiled kernel and installed kernel modules. Now let's install the kernel itsself.
# make install