Proxmox: enp0s31f6: Detected Hardware Unit Hang

Jarek Telco and IT 2 Comments

Since a few weeks, my Proxmox lab has started having issues with on-board network. The adapter enters in “hang” state, the logs are full of recurring errors like below:

[89276.274556] e1000e 0000:00:1f.6 enp0s31f6: Reset adapter unexpectedly
[89276.306147] vmbr0: port 1(enp0s31f6) entered disabled state
[89280.269563] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[89280.269626] vmbr0: port 1(enp0s31f6) entered blocking state
[89280.269631] vmbr0: port 1(enp0s31f6) entered forwarding state
[89282.226613] e1000e 0000:00:1f.6 enp0s31f6: Detected Hardware Unit Hang:
                 TDH                  <0>
                 TDT                  <1>
                 next_to_use          <1>
                 next_to_clean        <0>
               buffer_info[next_to_clean]:
                 time_stamp           <10153702e>
                 next_to_watch        <0>
                 jiffies              <101537150>
                 next_to_watch.status <0>
               MAC Status             <80083>
               PHY Status             <c8db>
               PHY 1000BASE-T Status  <a39b>
               PHY Extended Status    <ffff>
               PCI Status             <10>

Initially I was suspecting a hardware issue but after replacing the motherboard, the problem still persisted.

Next, I’ve found this thread in Proxmox forum. Workaround which is suggested there seems to be working, below how I’ve implemented it:

One time fix

root@wieloryb-pve:/etc/rc.d/init.d# /sbin/ethtool -K enp0s31f6 tx off rx off
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
Actual changes:
rx-checksumming: off
tx-checksumming: off
    tx-checksum-ip-generic: off
tcp-segmentation-offload: off
    tx-tcp-segmentation: off [requested on]
    tx-tcp6-segmentation: off [requested on]

Preserve the change across reboots

root@wieloryb-pve:~# cat /etc/network/if-up.d/ethtool2
#!/bin/sh

/sbin/ethtool -K enp0s31f6 tx off rx off

root@wieloryb-pve:~# chmod 755 /etc/network/if-up.d/ethtool2

Reboot and verify

root@wieloryb-pve:/etc#  shutdown -r now

root@wieloryb-pve:/etc/rc.d/init.d# /sbin/ethtool -k enp0s31f6
Features for enp0s31f6:
Cannot get device udp-fragmentation-offload settings: Operation not supported
rx-checksumming: off                   <--------- SHOULD BE OFF, HERE AND A FEW OTHER PLACES
tx-checksumming: off

 

Read more

PlantronicsHub – major memory leak?

Jarek Gadgets & hobbies No Comments

I’ve noticed that process SpokesUpdateService  is promoted to the top of my Memory list. Here an example which I captured today (just 24hrs after my last reboot) but on the other day I’ve seen 2.4 GB wasted:

It turned out to be part of PlantronicsHub software. I will disable to for a while an see how it will behave.

bash-4.4$ sudo launchctl remove com.PlantronicsUpdateService
bash-4.4$

Note: need to find out how to prevent starting this after every reboot.

References to other guys reporting very same:

  • http://forums.macresource.com/read.php?1,2266926
  • https://arstechnica.com/civis/viewtopic.php?f=19&t=1430655
  • I’ve opened Plantronics Case 06390520: ref:_00D507IyW._50038xj5MQ:ref

 

Update 2018-08-21

This message I got from Plantronics Support:

Dear Jaroslaw,

 

Please update the hub to 3.11.2 this should solve the issue you were experiencing.

Please let me know how it goes.

The case number for your reference is 06390520. If you need any further assistance, please let us know.

Kind regards

Angela

Technical Support

I’ve upgraded to 3.11.2, will see 😀

Read more

New home virtualisation lab

Jarek Gadgets & hobbies No Comments

Intro

It’s been a while since I’ve built my current home lab. It’s based on Lenovo M910t. In general, I’m quite happy but it’s lacking expansion flexibility.

So I’m looking for a replacement. Below my notes from research and calculations of total cost.

New system components

Cost estimation.

ComponentDescriptionPrice [PLN]
Total [PLN] / [USD]7307 [PLN]
1964 [USD]
Case - Fractal Design Define R5 👍
- be quiet! Silent Base 800
- Phanteks Enthoo Pro		439
CPUAMD Ryzen Threadripper 1950X3365
MotherboardMSI X399 SLI PLUS1290
RAM - 48GB from old PC
- 16GB - Crucial 2400

Total: 64 GB (with option for expansion up to 128 GB)		630
PSUSeaSonic Focus Plus Platinum 650W (SSR-650PX)478
HDDPC
- 2x 2TB from the old (ZFS root, RAID1)
- 1x 4TB from the old + 2x 4TB new (ZFS storage1, RAID5)

Total: 2+8 = 10 GB HDD

On-demand backups
- 1x 2TB from the old (backup, via old USB dock)		880
SSD - 256GB from the old (zfs storage2-ssd)0
M.2- 256GB from the old (ZFS cache)0
CPU Cooler- Noctua NH-U12S TR4-SP3
- Arcetic Freezer 33 TR 👍		170
Video cardUsed one from ebay or Allegro35
Delivery20

Considerations

  • Do I really need 16-cores/32-threads AMD 1950X? Perhaps 12-cores/24-threads 1920X for 2699 PLN (1k less!) would be sufficient?
  • CPU cooling fans may not have enough clearance to populate all 8 RAM sockets? Answer here.
  • Is it possible to use 6 RAM DIMMs? User guide says that allowed configurations are 1, 2, 4 & 8. This forum thread advises that any even number of DIMMs is perfectly supported.
  • Cases
    • 8x 3.5” HDD case or 6x 3.5” HDD + 3x 5.25” bays for my old Icy Docks bay?
    • What’s the min width to fit 155mm-high CPU cooling? Phanteks Enthoo Pro can fit max 193 mm cooling -> OK

References

Does the CPU cooler has enough clearance for all RAM DIMMs?

Tech It Out was kind enough to provide this response and the photo on the MoBo I’m interested (MSI X399):

And how does it look on the picture:

 

Sample quotations

From Morele.net

From x-kom.pl

Summary

After deeper deliberations (it’s hell lot of money! And I’m completely self-funded… Also Threadripper 2 is about to be released on August) I decided:

  • Immediately buy 2x 4 TB WD Blue disks and replace my Storage1 pool 3x 2 TB → 3x 4 TB which should give me enough disk space
  • Wait till August and see how (and if) release of new CPUs and MB chipset X399 change the situation.

Regardless of all, this was quite interesting exercise!

Read more

My current Home Virtualisation Lab “Wieloryb”

Jarek Gadgets & hobbies No Comments

Wieloryb as a home lab

My current home server which I use many for virtualisation, serving also some net services is reaching memory and disk-space limits. I named this server Wieloryb which in Polish means The Whale.

Hardware

So far I have:

  • Lenovo M910t server
    • 64 GB RAM
    • 1x 1 TB SATA HDD (3.5”) – configured root ZFS Pool (RAID1)
    • 3x 2 TB SATA HDD (3.5”) – storage1 ZFS Pool (RAID5)
    • 1x 254 GB SATA SDD (2.5”) – storage2 ZFS Pool
    • 1x 254 GB (M.2) – cache for ZFS pools (Cache and L2ARC). I know, L2ARC should be mirrored but I decided that I will take the risk
    • PCI-e 4x SATA board to provide 4 + 4 SATA connectors (M’board + PCI-e board)
  • The three 2 TB HDD disks and 254 GB SSD disks are stored in Icy Dock 4-disks cage. The disk cage allows to connect all 4 disks through 4 SATA connectors
  • The disk cage is powered by an external PSU (as the PC’s PSU is not strong enough)
  • On-demand backups are made to a 4 TB SATA HDD using i-tec USB HDD dock
  • Remote management is possible thanks to Belkin Net KVM
  • Sending critical alarms via SMSes as well as forwarding of SMSes from my old mobile SIM are done thanks to Nokia USB stick and Gammu software
  • And finally – a small UPS

Picture of the lab

 

Software

For greater flexibility and separation between different tasks I decided to use Proxmox as virtualisation manager. No issues with that: it does what it should, small issues can be justified by a very attractive price (0 USD…).

Read more