[ale] SATA drive not recognized by BIOS
Steve Litt
slitt at troubleshooters.com
Thu Oct 26 16:40:07 EDT 2023
Here's my diagnostic advice Steve,
My first step would be to temporarily unplug your RAID and plug in each
disk as a single disk (presumably sata) and see if 1), the BIOS
recognizes it and 2) if it has errors.
If it fails as a single disk, it's probably the disk's fault: Return
it. If it succeeds, that points an accusing finger at your RAID, so
troubleshoot that.
HTH,
SteveT
Steve Litt
Autumn 2023 featured book: Rapid Learning for the 21st Century
http://www.troubleshooters.com/rl21
Steve Tynor via Ale said on Thu, 26 Oct 2023 20:03:47 +0000
>Steve -
>
>I backup nightly. But would definitely look deeper if I knew what to
>look for. Ideas? Here are the relevant entries out of dmesg showing
>the disk initialization: (the two new disks are ata3 and ata4). The
>ACPI warnings worry me, but I don't know what to make of them.
>
>Thanks for any advice!
>
>Steve
>
>...
>
>[ 2.132317] ACPI Warning:
>[ 2.274882] usb 1-1.4: New USB device found, idVendor=1a2c,
>idProduct=2124, bcdDevice= 1.10
>[ 2.290169] SystemIO range 0x0000000000001C30-0x0000000000001C3F
>conflicts with OpRegion 0x0000000000001C00-0x0000000000001C3F (\GPRL)
>(20210730/utaddress-204)
>[ 2.316695] usb 1-1.4: New USB device strings: Mfr=1, Product=2,
>SerialNumber=0
>[ 2.316696] usb 1-1.4: Product: USB Keyboard
>[ 2.343626] ACPI Warning:
>[ 2.370278] usb 1-1.4: Manufacturer: SEM
>[ 2.397059] SystemIO range 0x0000000000001C30-0x0000000000001C3F
>conflicts with OpRegion 0x0000000000001C00-0x0000000000001FFF (\GPR)
>[ 2.658691] e1000e: Intel(R) PRO/1000 Network Driver
>[ 2.685215] (20210730/utaddress-204)
>[ 2.739682] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
>[ 2.739742] hid: raw HID events driver (C) Jiri Kosina
>[ 2.768035] ACPI: OSL: Resource conflict; ACPI support missing from
>driver?
>[ 2.768425] i801_smbus 0000:00:1f.3: SPD Write Disable is set
>[ 2.768447] i801_smbus 0000:00:1f.3: SMBus using PCI interrupt
>[ 2.771923] i2c i2c-0: 4/4 memory slots populated (from DMI)
>[ 2.772216] i2c i2c-0: Successfully instantiated SPD at 0x50
>[ 2.772495] i2c i2c-0: Successfully instantiated SPD at 0x51
>[ 2.772787] i2c i2c-0: Successfully instantiated SPD at 0x52
>[ 2.773064] i2c i2c-0: Successfully instantiated SPD at 0x53
>[ 2.796245] e1000e 0000:00:19.0: Interrupt Throttling Rate
>(ints/sec) set to dynamic conservative mode
>[ 2.823893] ACPI Warning:
>[ 2.928035] e1000e 0000:00:19.0 0000:00:19.0 (uninitialized):
>registered PHC clock
>[ 2.935646] SystemIO range 0x0000000000001C00-0x0000000000001C2F
>conflicts with OpRegion 0x0000000000001C00-0x0000000000001C3F (\GPRL)
>[ 2.964764] ahci 0000:00:1f.2: version 3.0
>[ 2.993537] (20210730/utaddress-204)
>[ 2.993539] ACPI Warning: SystemIO range
>0x0000000000001C00-0x0000000000001C2F conflicts with OpRegion
>0x0000000000001C00-0x0000000000001FFF (\GPR) (20210730/utaddress-204)
>[ 2.993542] ACPI: OSL: Resource conflict; ACPI support missing from
>driver?
>[ 2.993543] lpc_ich: Resource conflict(s) found affecting gpio_ich
>[ 3.023221] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6
>Gbps 0xf impl SATA mode
>[ 3.082964] e1000e 0000:00:19.0 eth0: (PCI Express:2.5GT/s:Width
>x1) 98:90:96:b5:b7:20
>[ 3.109700] ahci 0000:00:1f.2: flags: 64bit ncq pm led clo pio slum
>part ems apst
>[ 3.137850] e1000e 0000:00:19.0 eth0: Intel(R) PRO/1000 Network
>Connection
>[ 3.144572] usbcore: registered new interface driver usbhid
>[ 3.144573] usbhid: USB HID core driver
>[ 3.205236] scsi host0: ahci
>[ 3.222318] e1000e 0000:00:19.0 eth0: MAC: 11, PHY: 12, PBA No:
>FFFFFF-0FF
>[ 3.250623] scsi host1: ahci
>[ 3.428072] e1000e 0000:00:19.0 eno1: renamed from eth0
>[ 3.457834] scsi host2: ahci
>[ 3.488978] AVX2 version of gcm_enc/dec engaged.
>[ 3.520282] scsi host3: ahci
>[ 3.825901] AES CTR mode by8 optimization enabled
>[ 3.854425] scsi host4: ahci
>[ 3.882362] hid-generic 0003:051D:0002.0001: hiddev0,hidraw0: USB
>HID v1.00 Device [American Power Conversion Back-UPS NS 650M1
>FW:929.a8 .D USB FW:a8 ] on usb-0000:00:1a.0-1.2/input0
>[ 3.909374] scsi host5: ahci
>[ 3.936308] input: SEM USB Keyboard as
>/devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.4/1-1.4:1.0/0003:1A2C:2124.0002/input/input2
>[ 3.991986] ata1: SATA max UDMA/133 abar m2048 at 0xf7c3a000 port
>0xf7c3a100 irq 31
>[ 4.076231] ata2: SATA max UDMA/133 abar m2048 at 0xf7c3a000 port
>0xf7c3a180 irq 31
>[ 4.076628] hid-generic 0003:1A2C:2124.0002: input,hidraw1: USB HID
>v1.10 Keyboard [SEM USB Keyboard] on usb-0000:00:1a.0-1.4/input0
>[ 4.105275] ata3: SATA max UDMA/133 abar m2048 at 0xf7c3a000 port
>0xf7c3a200 irq 31
>[ 4.134962] input: SEM USB Keyboard Consumer Control as
>/devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.4/1-1.4:1.1/0003:1A2C:2124.0003/input/input3
>[ 4.164273] ata4: SATA max UDMA/133 abar m2048 at 0xf7c3a000 port
>0xf7c3a280 irq 31
>[ 4.226932] ata5: DUMMY
>[ 4.256594] input: SEM USB Keyboard System Control as
>/devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.4/1-1.4:1.1/0003:1A2C:2124.0003/input/input4
>[ 4.258139] ata6: DUMMY
>[ 4.290536] hid-generic 0003:1A2C:2124.0003: input,hidraw2: USB HID
>v1.10 Device [SEM USB Keyboard] on usb-0000:00:1a.0-1.4/input1
>[ 4.635750] ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
>[ 4.669083] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
>[ 4.702111] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
>[ 4.735319] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
>[ 4.767812] ata4.00: ATAPI: PLDS DVD+/-RW DH-16AES, 3D11, max
>UDMA/100 [ 4.800890] ata3.00: ATA-10: WDC WD60EFPX-68C5ZN0,
>81.00A81, max UDMA/133 [ 4.834108] ata2.00: ATA-10: WDC
>WD60EFPX-68C5ZN0, 81.00A81, max UDMA/133 [ 4.865922] ata1.00:
>ATA-7: Hitachi HDS721075KLA330, GK8OAB0A, max UDMA/133
>[ 4.898374] ata2.00: 11721045168 sectors, multi 16: LBA48 NCQ
>(depth 32), AA
>[ 4.929252] ata2.00: Features: NCQ-prio
>[ 4.960353] ata3.00: 11721045168 sectors, multi 16: LBA48 NCQ
>(depth 32), AA
>[ 4.991976] ata3.00: Features: NCQ-prio
>[ 5.023484] ata1.00: ATA Identify Device Log not supported
>[ 5.054211] ata1.00: 1465149168 sectors, multi 16: LBA48 NCQ (depth
>32), AA
>[ 5.084691] ata4.00: configured for UDMA/100
>[ 5.117030] ata2.00: configured for UDMA/133
>[ 5.146743] ata3.00: configured for UDMA/133
>[ 5.176185] ata1.00: ATA Identify Device Log not supported
>[ 5.204540] ata1.00: configured for UDMA/133
>[ 5.232800] scsi 0:0:0:0: Direct-Access ATA Hitachi
>HDS72107 AB0A PQ: 0 ANSI: 5
>[ 5.261873] sd 0:0:0:0: Attached scsi generic sg0 type 0
>[ 5.262102] sd 0:0:0:0: [sda] 1465149168 512-byte logical blocks:
>(750 GB/699 GiB)
>[ 5.290979] scsi 1:0:0:0: Direct-Access ATA WDC
>WD60EFPX-68C 0A81 PQ: 0 ANSI: 5
>[ 5.319705] sd 0:0:0:0: [sda] Write Protect is off
>[ 5.349259] sd 1:0:0:0: Attached scsi generic sg1 type 0
>[ 5.349302] sd 1:0:0:0: [sdb] 11721045168 512-byte logical blocks:
>(6.00 TB/5.46 TiB)
>[ 5.349305] sd 1:0:0:0: [sdb] 4096-byte physical blocks
>[ 5.349311] sd 1:0:0:0: [sdb] Write Protect is off
>[ 5.349312] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
>[ 5.349323] sd 1:0:0:0: [sdb] Write cache: enabled, read cache:
>enabled, doesn't support DPO or FUA
>[ 5.371303] sd 1:0:0:0: [sdb] Attached SCSI disk
>[ 5.378885] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
>[ 5.378909] sd 0:0:0:0: [sda] Write cache: enabled, read cache:
>enabled, doesn't support DPO or FUA
>[ 5.408998] scsi 2:0:0:0: Direct-Access ATA WDC
>WD60EFPX-68C 0A81 PQ: 0 ANSI: 5
>[ 5.481357] sda: sda1 sda2 sda3
>[ 5.497373] sd 2:0:0:0: Attached scsi generic sg2 type 0
>[ 5.497401] sd 2:0:0:0: [sdc] 11721045168 512-byte logical blocks:
>(6.00 TB/5.46 TiB)
>[ 5.497403] sd 2:0:0:0: [sdc] 4096-byte physical blocks
>[ 5.497407] sd 2:0:0:0: [sdc] Write Protect is off
>[ 5.497408] sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00
>[ 5.497413] sd 2:0:0:0: [sdc] Write cache: enabled, read cache:
>enabled, doesn't support DPO or FUA
>[ 5.511525] sd 2:0:0:0: [sdc] Attached SCSI disk
>[ 5.526727] sd 0:0:0:0: [sda] Attached SCSI disk
>[ 5.558172] scsi 3:0:0:0: CD-ROM PLDS DVD+-RW
>DH-16AES 3D11 PQ: 0 ANSI: 5
>[ 5.929723] sr 3:0:0:0: [sr0] scsi3-mmc drive: 48x/48x writer
>dvd-ram cd/rw xa/form2 cdda tray
>[ 5.956577] cdrom: Uniform CD-ROM driver Revision: 3.20
>[ 6.015681] sr 3:0:0:0: Attached scsi CD-ROM sr0
>[ 6.015729] sr 3:0:0:0: Attached scsi generic sg3 type 5
>[ 6.092297] Console: switching to colour dummy device 80x25
>[ 6.092337] i915 0000:00:02.0: vgaarb: deactivate vga console
>[ 6.093630] i915 0000:00:02.0: vgaarb: changed VGA decodes:
>olddecodes=io+mem,decodes=io+mem:owns=io+mem
>[ 6.110969] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0
>on minor 0
>
>
>On 2023-10-26 3:17 PM, Steve Litt via Ale wrote:
>> If your system is this picky about individual disks, I think you've
>> got a much worse problem than you think, and your problems is
>> anything but solved. I'd advise to diagnose this before everything
>> goes to hell in a handbasket, and for gosh sakes back up very often.
>>
>> SteveT
>>
>>
>> Steve Tynor via Ale said on Wed, 25 Oct 2023 15:55:46 +0000
>>
>>> To follow up on how i managed to solve this... The original failed
>>> drive was a WD 4T. I tried two different Toshiba drives (8T and 4T)
>>> and neither were recognized by the Dell BIOS. So I tried WD.
>>> Their 8T was recognized by the BIOS, but its size was misidentified
>>> as 0T and the kernel refused to recognize it as a valid disk. WD
>>> 6T, however, worked fine. So I've build a new array with 6T WD.
>>> Why the WD works and Toshiba didn't will remain a mystery. I don't
>>> see anything significant in their spec sheets.
>>>
>>> In any case, Hurray for RAID1 - I didn't lose any data and didn't
>>> have to resort to backups.
>>>
>>> Steve
>>>
>>>
>>> On 2023-10-15 8:50 PM, Steve Tynor via Ale wrote:
>>>> Thanks Jim. It's a software RAID via mdadm. The array was
>>>> originally built with Ubuntu 18 - have kept it going through
>>>> various system upgrades - now on Ubuntu 22. And you are right - I
>>>> misspoke when calling it LVM RAID - if my notes are right from
>>>> back then, I created it via the "full disk" partitions:
>>>>
>>>> mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2
>>>> /dev/sdb1 /dev/sdc1
>>>>
>>>> When these errors start happening, it brings the machine to its
>>>> knees. Anything I can do to "repair" the array to avoid those
>>>> segments? Even when the errors are happening, mdstat looks
>>>> healthy:
>>>>
>>>> ricotta:~> cat /proc/mdstat
>>>> Personalities : [raid1] [linear] [multipath] [raid0] [raid6]
>>>> [raid5] [raid4] [raid10]
>>>> md0 : active raid1 sdc1[0] sdb1[1]
>>>> 3906884608 blocks super 1.2 [2/2] [UU]
>>>> bitmap: 10/30 pages [40KB], 65536KB chunk
>>>>
>>>> unused devices: <none>
>>>>
>>>> For now I've just removed the errant disk from the array again
>>>> pending inspiration...
>>>>
>>>> Steve
>>>>
>>>>
>>>> On 2023-10-15 5:47 PM, Jim Kinney wrote:
>>>>> The Errors look like the raid recognized a block/segment failure
>>>>> on sdb and handled it by redirects to a new location and copied
>>>>> in the data from the mirror.
>>>>>
>>>>> Is the the drive bad? Not yet. Sectors do fail in time.
>>>>>
>>>>> But raid system specified sdb, not sdb1, so the raid subsystem is
>>>>> either using the entire drive for sdb or there's a raid
>>>>> controlling hardware or software that is doing the low level
>>>>> hardware management and that layer needs to be worked on to
>>>>> recognize the new drive. It really sounds like there's a raid
>>>>> controller in there somewhere.
>>>>>
>>>>> On Sun, Oct 15, 2023, 2:40 PM Steve Tynor via Ale <ale at ale.org>
>>>>> wrote:
>>>>>
>>>>> And I guess one more question... I was not able to learn
>>>>> much about these errors from the interwebs, but perhaps they don't
>>>>> necessarily suggest an actual drive failure but some sort of
>>>>> soft problem with the array - perhaps repartition the bad drive
>>>>> and add it back to the array and hope the errors don't come back?
>>>>> The errors look scary to me, but most people on this list
>>>>> have a lot more sysadm chops than I do...
>>>>>
>>>>> Oct 13 21:11:48 ricotta kernel: [714090.323895] md/raid1:md0:
>>>>> read error correct
>>>>> ed (8 sectors at 7339070832 on sdb1)
>>>>> Oct 13 21:12:01 ricotta kernel: [714104.108320] md/raid1:md0:
>>>>> redirecting sector
>>>>> 7338806424 to other mirror: sdb1
>>>>> Oct 13 21:12:09 ricotta kernel: [714111.347389] ata2.00:
>>>>> exception Emask 0x0 SAc
>>>>> t 0x140000 SErr 0x0 action 0x0
>>>>> Oct 13 21:12:09 ricotta kernel: [714111.347501] ata2.00:
>>>>> irq_stat 0x40000008
>>>>> Oct 13 21:12:09 ricotta kernel: [714111.347558] ata2.00:
>>>>> failed command: READ FP
>>>>> DMA QUEUED
>>>>> Oct 13 21:12:09 ricotta kernel: [714111.347621] ata2.00: cmd
>>>>> 60/78:90:98:5c:71/0
>>>>> 1:00:b5:01:00/40 tag 18 ncq dma 192512 in
>>>>> Oct 13 21:12:09 ricotta kernel: [714111.347621] res
>>>>> 41/40:00:88:5d:71/0
>>>>> 0:00:b5:01:00/00 Emask 0x409 (media error) <F>
>>>>> Oct 13 21:12:09 ricotta kernel: [714111.347809] ata2.00:
>>>>> status: { DRDY ERR }
>>>>> Oct 13 21:12:09 ricotta kernel: [714111.347860] ata2.00:
>>>>> error: { UNC }
>>>>> Oct 13 21:12:09 ricotta kernel: [714111.350759] ata2.00:
>>>>> configured for UDMA/133
>>>>> Oct 13 21:12:09 ricotta kernel: [714111.350801] sd 1:0:0:0:
>>>>> [sdb] tag#18 FAILED
>>>>> Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=7s
>>>>> Oct 13 21:12:09 ricotta kernel: [714111.350811] sd 1:0:0:0:
>>>>> [sdb] tag#18 Sense K
>>>>> ey : Medium Error [current]
>>>>> Oct 13 21:12:09 ricotta kernel: [714111.350817] sd 1:0:0:0:
>>>>> [sdb] tag#18 Add. Se
>>>>> nse: Unrecovered read error - auto reallocate failed
>>>>> Oct 13 21:12:09 ricotta kernel: [714111.350824] sd 1:0:0:0:
>>>>> [sdb] tag#18 CDB: Re
>>>>> ad(16) 88 00 00 00 00 01 b5 71 5c 98 00 00 01 78 00 00
>>>>> Oct 13 21:12:09 ricotta kernel: [714111.350827]
>>>>> blk_update_request: I/O error, d
>>>>> ev sdb, sector 7339072664 op 0x0:(READ) flags 0x0 phys_seg 47
>>>>> prio class 0
>>>>> Oct 13 21:12:09 ricotta kernel: [714111.351018] md/raid1:md0:
>>>>> sdb1: rescheduling
>>>>> sector 7338806424
>>>>> Oct 13 21:12:09 ricotta kernel: [714111.351111] ata2: EH
>>>>> complete Oct 13 21:12:20 ricotta kernel: [714122.174015]
>>>>> md/raid1:md0: redirecting sector
>>>>> 7338806424 to other mirror: sdb1
>>>>> Oct 13 21:12:24 ricotta kernel: [714126.594981] ata2.00:
>>>>> exception Emask 0x0 SAc
>>>>> t 0x1000 SErr 0x0 action 0x0
>>>>> Oct 13 21:12:24 ricotta kernel: [714126.595089] ata2.00:
>>>>> irq_stat 0x40000008
>>>>> Oct 13 21:12:24 ricotta kernel: [714126.595146] ata2.00:
>>>>> failed command: READ FP
>>>>> DMA QUEUED
>>>>> Oct 13 21:12:24 ricotta kernel: [714126.595209] ata2.00: cmd
>>>>> 60/78:60:98:5c:71/0
>>>>> 1:00:b5:01:00/40 tag 12 ncq dma 192512 in
>>>>> Oct 13 21:12:24 ricotta kernel: [714126.595209] res
>>>>> 41/40:00:78:5d:71/0
>>>>> 0:00:b5:01:00/00 Emask 0x409 (media error) <F>
>>>>> Oct 13 21:12:24 ricotta kernel: [714126.603898] ata2.00:
>>>>> status: { DRDY ERR }
>>>>> Oct 13 21:12:24 ricotta kernel: [714126.608263] ata2.00:
>>>>> error: { UNC }
>>>>> Oct 13 21:12:24 ricotta kernel: [714126.615103] ata2.00:
>>>>> configured for UDMA/133
>>>>> Oct 13 21:12:24 ricotta kernel: [714126.615136] sd 1:0:0:0:
>>>>> [sdb] tag#12 FAILED
>>>>> Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=4s
>>>>> Oct 13 21:12:24 ricotta kernel: [714126.615145] sd 1:0:0:0:
>>>>> [sdb] tag#12 Sense K
>>>>> ey : Medium Error [current]
>>>>> Oct 13 21:12:24 ricotta kernel: [714126.615151] sd 1:0:0:0:
>>>>> [sdb] tag#12 Add. Se
>>>>> nse: Unrecovered read error - auto reallocate failed
>>>>> Oct 13 21:12:24 ricotta kernel: [714126.615158] sd 1:0:0:0:
>>>>> [sdb] tag#12 CDB: Re
>>>>> ad(16) 88 00 00 00 00 01 b5 71 5c 98 00 00 01 78 00 00
>>>>> Oct 13 21:12:24 ricotta kernel: [714126.615162]
>>>>> blk_update_request: I/O error, d
>>>>> ev sdb, sector 7339072664 op 0x0:(READ) flags 0x0 phys_seg 47
>>>>> prio class 0
>>>>> Oct 13 21:12:24 ricotta kernel: [714126.619470] md/raid1:md0:
>>>>> sdb1: rescheduling
>>>>> sector 7338806424
>>>>> Oct 13 21:12:24 ricotta kernel: [714126.623671] ata2: EH
>>>>> complete
>>>>>
>>>>> On 2023-10-15 3:01 PM, Steve Tynor via Ale wrote:
>>>>>> Forgot to include: the original WD drives are also "AF", so
>>>>>> I think that rules out sector size incompatibility.
>>>>>>
>>>>>> Steve
>>>>>>
>>>>>>
>>>>>> On 2023-10-15 2:16 PM, Steve Tynor via Ale wrote:
>>>>>>> I'm not certain how to be sure. A quick duckduckgo search
>>>>>>> suggests that 4k native sector also being marketed as
>>>>>>> "Advanced Format" - is that right? Looking at the Toshiba spec
>>>>>>> page, I see AF listed as "Yes" for the 8T (the first drive I
>>>>>>> tried) and "No" for the 4T (which also didn't work).
>>>>>>>
>>>>>>> https://storage.toshiba.com/docs/support-docs/toshiba_n300_salessheet_english_07-27-21.pdf?Status=Master
>>>>>>>
>>>>>>> Steve
>>>>>>>
>>>>>>> On 2023-10-15 2:08 PM, Robert Tweedy via Ale wrote:
>>>>>>>> Hi Steve,
>>>>>>>>
>>>>>>>> Out of curiosity, are the new drives "4K native" sector
>>>>>>>> drives, and is this the same as the old drives you're
>>>>>>>> replacing? If the system's old enough its BIOS might not
>>>>>>>> support that & is expecting to have "512n" (512-native) or
>>>>>>>> "512e" (512-emulated) sector-size drives.
>>>>>>>>
>>>>>>>> Other than that, I'm not sure what it could be beyond
>>>>>>>> being Toshiba brand (which I've personally never had good
>>>>>>>> experience with, but that's only anecdotal).
>>>>>>>>
>>>>>>>> -Robert
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Ale mailing list
>>>>>>> Ale at ale.org
>>>>>>> https://mail.ale.org/mailman/listinfo/ale
>>>>>>> See JOBS, ANNOUNCE and SCHOOLS lists at
>>>>>>> http://mail.ale.org/mailman/listinfo
>>>>>> _______________________________________________
>>>>>> Ale mailing list
>>>>>> Ale at ale.org
>>>>>> https://mail.ale.org/mailman/listinfo/ale
>>>>>> See JOBS, ANNOUNCE and SCHOOLS lists at
>>>>>> http://mail.ale.org/mailman/listinfo
>>>>> _______________________________________________
>>>>> Ale mailing list
>>>>> Ale at ale.org
>>>>> https://mail.ale.org/mailman/listinfo/ale
>>>>> See JOBS, ANNOUNCE and SCHOOLS lists at
>>>>> http://mail.ale.org/mailman/listinfo
>>>>>
>>>> _______________________________________________
>>>> Ale mailing list
>>>> Ale at ale.org
>>>> https://mail.ale.org/mailman/listinfo/ale
>>>> See JOBS, ANNOUNCE and SCHOOLS lists at
>>>> http://mail.ale.org/mailman/listinfo
>>
>> SteveT
>>
>> Steve Litt
>>
>> Autumn 2023 featured book: Rapid Learning for the 21st Century
>> http://www.troubleshooters.com/rl21
>> _______________________________________________
>> Ale mailing list
>> Ale at ale.org
>> https://mail.ale.org/mailman/listinfo/ale
>> See JOBS, ANNOUNCE and SCHOOLS lists at
>> http://mail.ale.org/mailman/listinfo
>_______________________________________________
>Ale mailing list
>Ale at ale.org
>https://mail.ale.org/mailman/listinfo/ale
>See JOBS, ANNOUNCE and SCHOOLS lists at
>http://mail.ale.org/mailman/listinfo
More information about the Ale
mailing list