panic: ath0=ap + ath1=sta ==> soggy cornflakes

Local copy of wifi-up script. See ticket description.

Local copy of crashme script. See ticket description.

Possibly related: #160.

stack traces from madwifi crashes

1a) I copied down the stack-trace from a couple of different crashes. See newly attached file crash.log Also, I attached upgraded versions of the crashme script and the wifi-up script. In all cases, the latest and greatest versions of these items can be found at http://www.av8n.com/computer/wifi-bugs/

1b) I had to hand-type the stack trace. Setting up a serial console was a complete waste of time; no crash info appeared on the serial console.

1c) Note that in the two crashes in the log there is a block of 6 lines (beginning and ending with scan_next) that is identical in both cases. That's probably a big hint as to the nature of the problem.

2) I observe it to crash the same with *or* without WEP turned on. MRenzmann's previous comment mentioned bug #160, which mentions WEP, but evidence suggests the WEP issue is a red herring.

3) I observe it to crash the same with same *or* different essids given to the ap and sta.

4) I have reproduced the crash on another machine. One is a 300 MHz desktop with a Proxim 8482-WD on the PCI bus, while the other is a 1800 MHz laptop with built-in 802.11 a/b/g.

6) Question: has anybody tried to reproduce this bug? It is 100% reproducible chez moi. Am I the only person who has ever tried to instantiate an ap and a sta on the same card? This is a documented feature according to the wlanconfig manpage; indeed it is a prominently-featured feature. I was hoping a few gazillion people would contact me, either with sympathy or with success stories and workaround info.

7) If I may make a process suggestion: On a project of this size, it helps to have a suite of regression tests. The tests get run on the "current version" each night. This keeps down the number of new bugs, and is particularly effective at keeping old bugs from creeping back in. The examples given on the manpage would be a good place to start. They should either work, or be documented to not work.

same bug, slightly different call-pattern

I use ath1 to be AP and ath0 to be sta.
here is my script:

#!/bin/sh
wlanconfig ath1 create wlandev wifi0 wlanmode ap
sleep 3s
wlanconfig ath0 create wlandev wifi0 wlanmode sta nosbeacon
sleep 3s
iwconfig ath0 essid someap
iwconfig ath1 essid someotherap
ifconfig ath0 up
sleep 2s
/etc/init.d/killerwall stop
dhcpcd ath0
ifconfig ath1 192.168.1.1 up
dhcpd
firestarter

which worked fine is the signal was ok. If the signal was too low, it would crash as soon as dhcpcd would terminate. (because it calls ifconfig ath0 down?)

The following also crashed the computer, but everytime (I had made an error when editing the script, I replaced the wrong command):

#!/bin/sh
wlanconfig ath1 create wlandev wifi0 wlanmode ap
sleep 3s
wlanconfig ath0 create wlandev wifi0 wlanmode sta nosbeacon
sleep 3s
iwconfig ath0 essid someap
iwconfig ath1 essid someotherap
ifconfig ath0 up
sleep 2s
/etc/init.d/killerwall stop
dhcpcd ath0
ifconfig ath1 192.168.1.1 up
pump -i ath0
firestarter

If I brought down ath0 (sta) or destroyed it before ath1 (ap), it'd crash.

Also, too many connection drops caused it to crash as well.

I can reproduce the kernel panic using WEP. While using WPA seems to be stable.

This bug is still with us as of today: Sat 25-Feb-2006 madwifi-ng svn 1456 kernel 2.6.15.4

I commited a patch for #318 today, can you verify if r1482 fixes also this case?

I have good news and bad news concerning r1482.

1) I ran "crashme 0 1" and the system stayed up. That's a first.

2) It does not, however, function correctly. After bringing up ath0, ath0 appeared to exist as expected, but then after bringing up ath1, ath0 had vanished.

The objective has always been to have ath0 and ath1 working at the same time, as AP and STA respectively.

What do you mean wish 'vanished'? Was ath0 (running in AP mode, right?) invissible for the clients? This can happen because the AP device has to follow the STA VAP channel.

"ath0 apears to exist" means that I can see it using ifconfig ath0. It has an IP address as expected, as assigned by the crashme script.

"ath0 vanished" means ifconfig ath0 reports an error: ath0: error fetching interface information: Device not found

Clear enough?

hmm it is I guess the same bug concerning me. I shouldn't have created another ticket. Anyway, It does not seem to work and hangs the computer, I tested it on two dell inspiron laptops with FC4, linux 2.6.12.

ahhh I foung the bug. That is you have to set the channel of both interfaces, before bring them up with ifconfig. try iwconfig ath0 channel 1 iwconfig ath1 channel 1

and than ifconfig ath0 up etc. seems to work, this way.

1) Thanks for looking into this.

2) I added

iwconfig athN channel 1

in the obvious places in the crashme script.

Behavior is the same as before: setting up ath1 causes ath0 to vanish, i.e. ifconfig reports "device not found". This is observed with both r1482 and r1484. Changing the channel didn't affect things.

I was refused permission to "attach" the new crashme script, but you can download the latest & greatest version from: [ http://www.av8n.com/computer/wifi-bugs/]

Just a fyi: to get it to work, I need to set the channel again after ifconfig athX x.x.x.x Here is my little script

#!/bin/bash

function station() {
        echo "setting $1 up as a station"
        iwconfig $1 channel 6
        iwconfig $1 nick "none"
        iwconfig $1 essid "default"
	ifconfig $1 192.168.0.90

        #### without this it doesn't work, the status of ath1
        #### in /proc/net/wireless never changes from 0
        iwconfig $1 channel 6

        echo "setting route"
	route add default gw 192.168.0.1

        echo "enabling forwading"
        echo "1" > /proc/sys/net/ipv4/ip_forward
        echo "1" > /proc/sys/net/ipv4/ip_dynaddr
}

function stop_station() {
        echo "disabeling forwading"
        echo "0" > /proc/sys/net/ipv4/ip_forward
        echo "0" > /proc/sys/net/ipv4/ip_dynaddr

        echo "bring down station on $1"
        ifconfig $1 down
}

case "$1" in
  station)
        # set up a station only
        modprobe ath_pci autocreate=none rfkill=0
        wlanconfig ath0 create wlandev wifi0 wlanmode sta

        station ath0
        ;;
  stop_station)
        # destroy the station
        stop_station ath0

        wlanconfig ath0 destroy
	
	modprobe -r ath_pci
        ;;
  ap)
        # create a station and an access point
        modprobe ath_pci autocreate=none rfkill=0
        wlanconfig ath0 create wlandev wifi0 wlanmode ap
        wlanconfig ath1 create wlandev wifi0 wlanmode sta nosbeacon

	station ath1
        
        echo "setting up access point on ath0"
        iwconfig ath0 channel 6 mode master essid tobsnet nick none
        ifconfig ath0 192.168.2.1

        ;;
  stop_ap)
        stop_station ath0

        ifconfig ath0 down
        wlanconfig ath1 destroy
        wlanconfig ath0 destroy

	modprobe -r ath_pci
        ;;
  *)
        echo $"Usage: $0 {station|stop_station|ap|stop_ap}"
        ;;
esac

With the latest svn version, are any of these issues still happening ?

To answer the question raised by dyquith in the previous comment: No, we are not out of the woods yet.

BUG: soft lockup detected on CPU#0!

Pid: 0 comm:  swapper
EIP is at .... [ath_hal]
.....
ath_calcrxfilter+0x20/0x90      [ath_pci]
ath_scan_end+0x3a/0xc0         [ath_pci]
scan_next+0xbb/0x4b0            [wlan]
hrtimer_run_queues+0x5d/0x140
run_timer_softirq+0x9b/0xb0
__do_softirq+0x9b/0xb0
do_softirq+0x2b/0x30
irq_exit+0x37/0x40
do_IRQ
common_interrupt
default_idle
cpu_idle
start_kernel

This is 100% reproducible via:

crashme stop     ## unload driver modules
crashme 1 0      ## not 0 1

As soon as the crashme script completes, the keyboard is already locked up. The softlock detector BUGs the kernel shortly thereafter.

The latest version of the crashme script is at http://www.av8n.com/computer/wifi-bugs/

#define SVNVERSION "1533"

Linux 2.6.16.11

Can you try the lateset revision (r1538+) ?

Your script didn't crash on my system, either I'm doing something wrong, or its solved by r1534

Here's what I get running "crashme 1 0":

./crashme stop
./crashme: line 27: /etc/init.d/networking: No such file or directory
ERROR: Module wlan_scan_sta does not exist in /proc/modules
[root@nc6k00 ~]# ./crashme 1 0
... start of IFACE=ath1 wifi-up
wifi-up: MacNaughton sta nosbeacon 99111111111111111111111111 2
wlanconfig: ioctl: No such device
SIOCSIFADDR: No such device
ath1: unknown interface: No such device
ath1: error fetching interface information: Device not found
SIOCADDRT: Network is unreachable
... start of IFACE=ath0 wifi-up
wifi-up: snark ap 99000000000000000000000000 3
ath0
... end of wifi-up
ath0      Link encap:Ethernet  HWaddr 00:0F:20:95:0A:4F
          inet addr:192.168.2.1  Bcast:192.168.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

Quick note to let you know that things work for me now as expected. (Except for a few lokups when I remove ath_pci too often, no pattern yet). Thanks! Tobias

Great, sounds good to me,

If you have time to post up the kernel panics, I'll try to take time to fix them.

I think its best if we can try to make madwifi-ng code at least 99% stable (of course, we'll have to push it up a few 9's after a few releases)

thanks.

I'm going to close this ticket now,

please feel free to open this ticket again if you have the same problem.

For the other kernel panics, please either check with existing tickets or open a new one.

thanks.

Comment by John Denker on devel mailing list

Why close it?

The crash is still 100% reproducible.  All I need to do on
my system (Debian) is run my script "crashme 1 0".

Actually, it is slightly worse now than at last report; a
few moments ago I tested it again and got a hard (not soft)
lockup, i.e. the machine locked up and the softlock detector
was not able to print a BUG message.

This is with svn 1574 and Linux 2.6.16.11.

Okay, if this is still a bug, can you do a clean install of the madwifi-ng source, and read this: http://madwifi.org/wiki/UserDocs/Distro/Debian

Then, see if it crashes again. (I can't reproduce the crash).

And try to get the kernel panic/oops messages posted so I know where to start looking in the code.

What we will be helpful will be the dmesg log (when the driver loads).

>  Okay, if this is still a bug, can you do a clean install of the madwifi-ng
>  source,

I obtained a fresh copy of the source via svn.
It compiled cleanly AFAICT.
It installed cleanly AFAICT.
I can send you "script" logs of the compile and install upon request.

I checked that there is only one copy of:
  athstats 80211stats athkey athchans athctrl athdebug 80211debug wlanconfig
installed on my system.

I checked that every entry in:
  /lib/modules/2.6.16.11/net
has the expected recent timestamp.

If that's not what you mean by a "clean" install, please explain in
more detail what you want.

>  and read this:  http://madwifi.org/wiki/UserDocs/Distro/Debian

I read it.  I'm not sure what I'm supposed to do with the information.

>  Then, see if it crashes again. (I can't reproduce the crash).

I cannot imagine why it doesn't crash for you.
It remains 100% reproducible chez moi.

 >  What we will be helpful will be the dmesg log (when the driver loads).

Here you go:  From dmesg:

ath_pci: driver unloaded
ath_rate_sample: unloaded
ath_hal: driver unloaded
wlan: driver unloaded

xxxxxxxxxxxxxxxxx

ath_hal: 0.9.16.16 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
wlan: 0.8.4.2 (svn 1574)
ath_rate_sample: 1.2 (svn 1574)
ath_pci: 0.9.4.5 (svn 1574)
PCI: Found IRQ 11 for device 0000:02:02.0
PCI: Sharing IRQ 11 with 0000:00:1d.2
PCI: Sharing IRQ 11 with 0000:00:1f.1
wifi0: 11a rates: 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
wifi0: 11b rates: 1Mbps 2Mbps 5.5Mbps 11Mbps
wifi0: 11g rates: 1Mbps 2Mbps 5.5Mbps 11Mbps 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
wifi0: H/W encryption support: WEP AES AES_CCM TKIP
wifi0: mac 5.6 phy 4.1 5ghz radio 1.7 2ghz radio 2.3
wifi0: Use hw queue 1 for WME_AC_BE traffic
wifi0: Use hw queue 0 for WME_AC_BK traffic
wifi0: Use hw queue 2 for WME_AC_VI traffic
wifi0: Use hw queue 3 for WME_AC_VO traffic
wifi0: Use hw queue 8 for CAB traffic
wifi0: Use hw queue 9 for beacons
wifi0: Atheros 5212: mem=0xc0210000, irq=11
dummy0: no IPv6 routers present
ath1: no IPv6 routers present

>  And try to get the kernel panic/oops messages posted so I know where to
>  start looking in the code.

That is much harder.  There is no serial port on the machine I'm using (never
thought I'd see the day) ... so I cannot easily log the panic/oops/BUG messages.

Let me work on it.

Can we keep all the info in this ticket (ticket:182), so there's an easy way to refer to it.

thanks.

Will wait for the crash messages, I see that you're using a preemptible kernel, that may be a reason why I can't reproduce the crashes. when I get some time, I'll go ahead and build a preemptible kernel and try this. (i would imagine it would be the same with SMP systems, but haven't heard from anyone so far)

It's still broken as of 23 May 2006 * svn r1591 ("code freeze") * Linux 2.6.16.11

Same symptoms as before: Hard lockup, i.e. does not respond to soft-lockup detector, but does respond to watchdog pulling on the NMI line.

The bug is 100% reproducible chez moi using the command

crashme bracket 1 0

Here's the traceback:

Pid: 0 comm:    swapper
...
ath_calcrxfilter+0x20/0x90 [ath_pci]
ath_scan_start+0x41/0xb0 [ath_pci]
scan_next+0x183/0x4b0 [wlan]
hrtime_run_queues+0xd5/0x140
scan_next+0x0/0x4b0 [wlan]
run_timer_softirq+0xeb/0x210
__do_softirq+0x9b/0xb0
do_softirq+0x2d/0x30
irq_exit+0x37/0x40
do_IRQ+0x28/0x40
common_interrupt+0x1a/0x20
default_idle+0x41/0x70
apm_cpu_idle+0x97+0x160 [apm]
cpu_idle+0x60/0x80
start_kernwl+0x18d/0x1d0
unknown_bootoption+0x0/0x1d0

As always, the latest & greatest version of the test/example script can be found here: http://www.av8n.com/computer/wifi-bugs/ in particular http://www.av8n.com/computer/wifi-bugs/crashme

Unassign this ticket from svens, since he recently retired from the project.

@John: Is this issue still existing in a recent snapshot (with HAL v0.9.18.0)? Was dyqith right with his assumption about the use of a preemptible kernel?

At present, this bug is still 100% reproducible chez moi.

-- Wednesday 22 Nov 2006 -- Thinkpad T42p -- built-in Atheros Communications, Inc. AR5212 802.11abg -- Linux 2.6.18.2 PREEMPT i686 GNU/Linux -- madwifi-r1816 -- Invocation:

crashme unload crashme 1 0

-- Result: immediate crash with interrupts off; caps lock light does

not respond to the caps lock key.

I still get the following:

---

Feb 12 07:15:23 infinitynet kernel: BUG: soft lockup detected on CPU#0!
Feb 12 07:15:23 infinitynet kernel: [<c04051db>] dump_trace+0x69/0x1af
Feb 12 07:15:23 infinitynet kernel: [<c0405339>] show_trace_log_lvl+0x18/0x2c
Feb 12 07:15:23 infinitynet kernel: [<c04058ed>] show_trace+0xf/0x11
Feb 12 07:15:23 infinitynet kernel: [<c04059ea>] dump_stack+0x15/0x17
Feb 12 07:15:23 infinitynet kernel: [<c044d9b5>] softlockup_tick+0xad/0xc4
Feb 12 07:15:23 infinitynet kernel: [<c042e596>] update_process_times+0x39/0x5c
Feb 12 07:15:23 infinitynet kernel: [<c0418914>] smp_apic_timer_interrupt+0x5c/0x64
Feb 12 07:15:23 infinitynet kernel: [<c0404ad3>] apic_timer_interrupt+0x1f/0x24
Feb 12 07:15:23 infinitynet kernel: DWARF2 unwinder stuck at
apic_timer_interrupt+0x1f/0x24
Feb 12 07:15:23 infinitynet kernel: Leftover inexact backtrace:
Feb 12 07:15:23 infinitynet kernel: [<c04e935f>] _raw_spin_lock+0x6f/0xdc
Feb 12 07:15:23 infinitynet kernel: [<c041e77b>] activate_task+0x1c/0x29
Feb 12 07:15:23 infinitynet kernel: [<c041f069>] try_to_wake_up+0x371/0x37b
Feb 12 07:15:23 infinitynet kernel: [<f89fccac>] ath_tx_processq+0xd6/0x5e3
[ath_pci] Feb 12 07:15:23 infinitynet kernel: [<f89ff8bb>] ath_tx_tasklet+0x5c/0xe3 [ath_pci]
...

---

I use:
FC6 (2.6.18-1.2798.fc6)
Hostapd 0.5.9
Madwifi 0.9.3.3

I use the following script to bring up my AP:

---

ath=ath0
wifi=wifi0
essid=pk.wlan
channel=1
eth=eth0
br=br0
ip=192.168.0.94
gw=192.168.0.4
sleep_secs=5

wlanconfig $ath destroy
wlanconfig $ath create wlandev $wifi wlanmode ap

sleep $sleep_secs

iwconfig $ath essid $essid channel $channel

brctl addbr $br
brctl addif $br $eth
brctl addif $br $ath
brctl setfd $br 0

ifconfig $eth 0.0.0.0 up
ifconfig $ath 0.0.0.0 up
ifconfig $br $ip up

route add default gw $gw

hostapd -B /etc/hostapd.conf