Telco and IT

Everything which is related to Telecommunications, IT and my work

Wireshark MATE for Diameter Charging-Control

That’s surprising that for all these years I didn’t know about Wireshark MATE:

MATE: Meta Analysis and Tracing Engine

What is MATE? Well, to keep it very short, with MATE you can create user configurable extension(s) of the display filter engine.

MATE creates a filterable tree based on information contained in frames that share some relationship with information obtained from other frames. The way this relationships are made is described in a configuration file. The configuration file tells MATE what makes a PDU and how to relate it to other PDUs.

I found it very useful when analysing Diameter logs. It helps to solve eternal problem: how to get whole diameter session when you know only MSISDN (or Result-Code or an other AVP which is not present in both CCR and CCA and the only way to bind CCR and CCA is Session-Id).

The answer is MATE. 

If I want to search for my MSISDN, I can use this filter:

mate.diam_transaction.msisdn == "46702904828"

In result, Wireshark will filter all CCR and CCAs where this subscriber is involved:

Mate in action

How to configure Wireshark for Diameter with MATE?

Lucky, MATE is built-in into current version of Wireshark. All we need to do is to create configuration and set it up in Wireshark preferences

MATE cfg file for Diameter

As I wanted to filter using MSISDN (actually Subscription-Id) and Result-Code, both needs to be extracted.

We will be binding messages using Session-Id, App Id and End-to-end Identifier

(credits needs to go to an anonymous poster to Wireshark forum; I’ve modified this file slightly adding MSISDN and Stop conditions)

// Create a "diam_pdu" that contains various pieces of the processed Diameter
// message.
Pdu diam_pdu Proto diameter Transport ip {
Extract command_code From diameter.cmd.code;
Extract app_id From diameter.applicationId;
Extract session_id From diameter.Session-Id;
Extract e2eid From diameter.endtoendid;
Extract resultcode From diameter.Result-Code;
Extract msisdn From diameter.Subscription-Id-Data;
Extract flag From diameter.flags.request;
Extract request_type From diameter.CC-Request-Type;
};

// Then create a GOP (Group Of Pdus) where the each GOP contains all the PDUs
// (msgs) that whose command_code, app_id, session_id, and e2eid match.
Gop diam_transaction On diam_pdu Match (command_code, app_id, session_id, e2eid) {
Start();
Stop(flag=0,request_type=3);

// Store the result code in the GOP
Extra(resultcode, msisdn);
};

Done;

Save the file somewhere on the disk.

Configure Wireshark

Wireshark protocols configuration set this file under MATE:

Now a new protocol tree MATE should appear in each Diameter Packet details. You can use these pseudo-AVPs for filtering.

How to enable R environment in Golden Cheetah on Mac?

Somehow I couldn’t find a complete guide how to install R and configure it properly in Golden Cheetah (I’m using v3.5).

There is a good guide how to work with R in GC, but actually – no description how to enable it.

1. Download R for MacOS

I got it from https://cran.r-project.org/bin/macosx/ page

(I’ve also installed R Studio but it seems not really needed)

2. Check where R home is located

In terminal, start R and check the home:

It’s /Library/Frameworks/R.framework/Resources in my environment.

3. In GC open Preferences and set the same in R Installation Directory

Wireshark LUA plugin

This is an example of a LUA plugin for Wireshark which goes through the packets and generates a statistic.

In my case, I was looking for Diameter Charging-Control stats around Reporting-Reason Valdity-Time.

The project is hosted in GitHub.

Pre-requisites

  • Wireshark
  • LUA environment

I’ve tested it on MacOS, same should work in Linux without modifications. I’m not sure about Windows – I will appreciate you comments.

How to start

The plugin can be started in two ways:

Command-line

Update start.sh to provide:

  • path to your Wireshark application
  • name to your pcap / snoop file
TSHARK='/Users/jhartman/Tools/Internet/Wireshark/Wireshark.app/Contents/MacOS/tshark'
INPUT='/Users/jhartman/Documents/Documents/Oracle/Telia/!Local/Logs and config/Diameter/!Production/spikes - 2020-02-03/dgw-spikes-tr001prdgw11.snoop'

Then you can invoke start.sh and see the result:

MBP:wiresharkLUA jhartman$ ./start.sh
Starting in command-line mode
Registering Listener
QUOTA_EXHAUSTED (3)            - 359 		(7.701 %)
FINAL (2)                      - 402 		(8.623 %)
VALIDITY_TIME (4)              - 3634 		(77.949 %)
QHT (1)                        - 267 		(5.727 %)

Total                          - 4662 		(100 %)

USU (octets) when VALIDITY_TIME (4)
Median  : 292458 octets, 285.60 kB
Average : 19140604 octets, 18692.00 kB
Min     : 112 octets
Max     : 310552054 octets, 303273.49 kB

Histogram of USU (Validity-Time)
 1:         0 MB -        11 MB :   2691 (74.05 %)
 2:        11 MB -        23 MB :    306 (8.42 %)
 3:        23 MB -        35 MB :    131 (3.60 %)
 4:        35 MB -        47 MB :     84 (2.31 %)
 5:        47 MB -        59 MB :     61 (1.68 %)
 6:        59 MB -        71 MB :     60 (1.65 %)
 7:        71 MB -        82 MB :     43 (1.18 %)
 8:        82 MB -        94 MB :     41 (1.13 %)
 9:        94 MB -       106 MB :     26 (0.72 %)
10:       106 MB -       118 MB :     24 (0.66 %)
11:       118 MB -       130 MB :     21 (0.58 %)
12:       130 MB -       142 MB :     15 (0.41 %)
13:       142 MB -       154 MB :     13 (0.36 %)
14:       154 MB -       165 MB :     16 (0.44 %)
15:       165 MB -       177 MB :     12 (0.33 %)
16:       177 MB -       189 MB :     11 (0.30 %)
17:       189 MB -       201 MB :     15 (0.41 %)
18:       201 MB -       213 MB :     10 (0.28 %)
19:       213 MB -       225 MB :      7 (0.19 %)
20:       225 MB -       236 MB :     10 (0.28 %)
21:       236 MB -       248 MB :      6 (0.17 %)
22:       248 MB -       260 MB :      9 (0.25 %)
23:       260 MB -       272 MB :      9 (0.25 %)
24:       272 MB -       284 MB :      8 (0.22 %)
25:       284 MB -       296 MB :      5 (0.14 %)
QUOTA_EXHAUSTED (3)            - 359 		(7.701 %)
FINAL (2)                      - 402 		(8.623 %)
VALIDITY_TIME (4)              - 3634 		(77.949 %)
QHT (1)                        - 267 		(5.727 %)

Total                          - 4662 		(100 %)

Note If you can see two invocations of LUA plugin, see the note in Wireshark GUI section:

Starting in command-line mode
Registering Listener
Starting in command-line mode
Registering Listener

Wireshark GUI

In order to invoke the plugin from the GUI, you need to install it (i.e. upload) into relevant folder which is ‘ (see Wireshark Plugin folders).

As I’m working with GitHub, I prefer to make a sym-link:

MBP:wiresharkLUA jhartman$ cd ../wiresharkLUA
MBP:wiresharkLUA jhartman$ ln -s $(pwd)/reportingReason-gui.lua ~/.local/lib/wireshark/plugins/reportingReason-gui.lua

After making the link, you need to reload the plugins by selecting Analyse -> Reload LUA plugins

Reload

Then you can open a Diameter Charging-Control log (i.e. one defined as per RFC 4006 or a relevant 3GPP spec) and start the analyze:

Reload

After a while, a window with statistics results will pop up: Reload

Wireshark LUA script starts twice when invoked from commandline

You may see that the script is invoked twice when starting from commandline:

MBP:wiresharkLUA jhartman$ ./start.sh
Starting in command-line mode
Registering Listener
Starting in command-line mode
Registering Listener
...
...

Reason for this the way how tshark loads the plugins:

  • first time it loads all user plugins from ~/.local/lib/wireshark/plugins/
  • and then it loads and executes all plugins defined through -X, i.e. -X lua_script:reportingReason-gui.lua

Solution: remove the plugin from the user plugins:

MBP:wiresharkLUA jhartman$ rm ~/.local/lib/wireshark/plugins/reportingReason-gui.lua

OUI issue NGINST-64002: Error occurred in validation of: “Host Name”

When installing an Oracle application using OUI (Oracle Universal Installer) sometimes it verifies if a given host is reachable. 

Unfortunately, in my case, it was always failing:

All this despite the host was perfectly reachable using Ping:

So what was happening? The OUI is using under the hood Java’s InetAddress.isReachable(). The algorithm is:

  1.  Try to send ICMP “ping”, if that fails
  2.  Try to use ICMP Echo service
 
Unfortunately, check (1) can only be attempted if you run the app with an elevated permissions (root or sudo), which typically does not happen.
 
So it goes to (2) and makes the following attempt:
 
[rms@tr005buwls11 ~]$ sudo tcpdump -nn -v -i eth0 host tr005buece12 and port not 22
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:40:57.566469 IP (tos 0x0, ttl 64, id 64840, offset 0, flags [DF], proto TCP (6), length 60)
    131.116.167.11.55766 > 131.116.167.13.7: Flags [S], cksum 0xaf64 (correct), seq 562514241, win 14600, options [mss 1460,sackOK,TS val 300069897 ecr 0,nop,wscale 7], length 0
14:40:57.566522 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    131.116.167.13.7 > 131.116.167.11.55766: Flags [R.], cksum 0x1628 (correct), seq 0, ack 562514242, win 0, length 0

So it probes Echo service over TCP port 7. 

Solution

Here you are! It was a missing firewall rule. Just open TCP/7 port on the target (checked) system and the test is passing like a charm.

Details for nerds

A small tool to run the test. Just run it as shown below:

[rms@tr005buece12 tmp]$ ./test.sh tr005buwls11.ddc.teliasonera.net
Testing tr005buwls11.ddc.teliasonera.net
OK

[rms@tr005buwls11 ~]$ sudo tcpdump -nn -v -i eth0 host tr005buece12 and port not 22
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:40:57.566469 IP (tos 0x0, ttl 64, id 64840, offset 0, flags [DF], proto TCP (6), length 60)
    131.116.167.11.55766 > 131.116.167.13.7: Flags [S], cksum 0xaf64 (correct), seq 562514241, win 14600, options [mss 1460,sackOK,TS val 300069897 ecr 0,nop,wscale 7], length 0
14:40:57.566522 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    131.116.167.13.7 > 131.116.167.11.55766: Flags [R.], cksum 0x1628 (correct), seq 0, ack 562514242, win 0, length 0

Compare with same test ran using sudo. This time the test is done using ICMP, so no TCP/7 port needed. But as mentioned earlier, this is highly impractical and even not recommended:

[rms@tr005buece12 tmp]$ sudo ./test.sh tr005buwls11.ddc.teliasonera.net
Testing tr005buwls11.ddc.teliasonera.net
OK

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:40:26.430473 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 72)
    131.116.167.11 > 131.116.167.13: ICMP echo request, id 12072, seq 1, length 52
14:40:26.430508 IP (tos 0x0, ttl 64, id 56716, offset 0, flags [none], proto ICMP (1), length 72)
    131.116.167.13 > 131.116.167.11: ICMP echo reply, id 12072, seq 1, length 52

Prometheus Node Exporter /lib/init/init-d-script: /usr/bin/daemon: not found

After Ubuntu upgrade 18.04 -> 18.10, Prometheus Node Exporter started to complain:

root@grafana:~# /etc/init.d/prometheus-node-exporter 
startStarting Prometheus exporter for machine metrics prometheus-node-exporter                                                         
/etc/init.d/prometheus-node-exporter: 45: /lib/init/init-d-script: /usr/bin/daemon: not found

Fix for this is quite easy – just install missing dependency:

root@grafana:~# apt install daemon
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  daemon
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 99.5 kB of archives.
After this operation, 288 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu cosmic/universe amd64 daemon amd64 0.6.4-1build1 [99.5 kB]
Fetched 99.5 kB in 0s (287 kB/s)
Selecting previously unselected package daemon.
(Reading database ... 22628 files and directories currently installed.)
Preparing to unpack .../daemon_0.6.4-1build1_amd64.deb ...
Unpacking daemon (0.6.4-1build1) ...
Processing triggers for man-db (2.8.4-2) ...
Setting up daemon (0.6.4-1build1) ...
root@grafana:~# /etc/init.d/prometheus-node-exporter stop
 * Stopping Prometheus exporter for machine metrics prometheus-node-exporter                                                  [ OK ]
root@grafana:~# /etc/init.d/prometheus-node-exporter start
 * Starting Prometheus exporter for machine metrics prometheus-node-exporter                                                  [ OK ]
root@grafana:~#

After that daemon can be started:

root@grafana:~# /etc/init.d/prometheus-node-exporter start
Starting Prometheus exporter for machine metrics prometheus-node-exporter                                                  [ OK ]
root@grafana:~#

Oracle Coherence – description of federation member states

States of Oracle Coherence federation are not very well described in the documentation. So I thought it would be good to ask at the source. Thanks for Patrick F for all these explanations!

Stopped states – Federation is not federating data to the destination, nor is it keeping a backlog of changes to send to the destination.  A “start” operation must be performed to start federating data again:
  • STOPPED – A stop operation was issued, or federation was set to start in the stopped state. Stopped states – Federation is not federating data to the destination, nor is it keeping a backlog of changes to send to the destination.  A “start” operation must be performed to start federating data again:
  • ERROR – An error occurred from which federation was unable to continue federating data
Paused state – Federation is not federating data, but is keeping a (growing) backlog of changes to be sent once a “start” operation is issued:
  • PAUSED – A pause operation was issued, or federation was set to start in the paused state.
Normal states – Federation is federating data:
  • INITIAL – default startup state.  A Coherence node will stay in this state until there is data to be federated
  • IDLE – federation is active and connected to the destination.  There is no data currently to send
  • READY – federation is transitioning out of CONNECTING, or YIELDING, or BACKLOG_NORMAL and will go to SENDING
  • SENDING – federation has data to send
  • CONNECTING – federation is connecting to the destination cluster
  • CONNECT_WAIT – federation is disconnected and will make a new connect attempt.  There may be a delay before making the next attempt depending on the circumstances under which federation was disconnected
  • YIELDING – federation has data to send, but is pausing briefly, likely due to a BACKLOG_EXCESSIVE event
  • DISCONNECTED – federation was disconnected from the remote destination.  If there is no data to send, federation may stay in this state. NB: DISCONNECTED is a “normal” state.  It means that a member of the cluster lost its federation connection to the remote cluster (probably because the remote member it was connected to was shut down).  If the member in the DISCONNECTED state has no data to send, it will stay in the DISCONNECTED state – basically the same as being IDLE.  Once there is some data for this member to send it will issue a new connection.
The following states do not exist – although they may have in 12.2.1.0.0 (federation should not be used with this version):
  • BACKLOG_EXCESSIVE 
  • BACKLOG_NORMAL

Proxmox: enp0s31f6: Detected Hardware Unit Hang

Since a few weeks, my Proxmox lab has started having issues with on-board network. The adapter enters in “hang” state, the logs are full of recurring errors like below:

[89276.274556] e1000e 0000:00:1f.6 enp0s31f6: Reset adapter unexpectedly
[89276.306147] vmbr0: port 1(enp0s31f6) entered disabled state
[89280.269563] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[89280.269626] vmbr0: port 1(enp0s31f6) entered blocking state
[89280.269631] vmbr0: port 1(enp0s31f6) entered forwarding state
[89282.226613] e1000e 0000:00:1f.6 enp0s31f6: Detected Hardware Unit Hang:
                 TDH                  <0>
                 TDT                  <1>
                 next_to_use          <1>
                 next_to_clean        <0>
               buffer_info[next_to_clean]:
                 time_stamp           <10153702e>
                 next_to_watch        <0>
                 jiffies              <101537150>
                 next_to_watch.status <0>
               MAC Status             <80083>
               PHY Status             <c8db>
               PHY 1000BASE-T Status  <a39b>
               PHY Extended Status    <ffff>
               PCI Status             <10>

Initially I was suspecting a hardware issue but after replacing the motherboard, the problem still persisted.

Next, I’ve found this thread in Proxmox forum. Workaround which is suggested there seems to be working, below how I’ve implemented it:

One time fix

root@wieloryb-pve:/etc/rc.d/init.d# /sbin/ethtool -K enp0s31f6 tx off rx off
Cannot get device udp-fragmentation-offload settings: Operation not supported
Cannot get device udp-fragmentation-offload settings: Operation not supported
Actual changes:
rx-checksumming: off
tx-checksumming: off
    tx-checksum-ip-generic: off
tcp-segmentation-offload: off
    tx-tcp-segmentation: off [requested on]
    tx-tcp6-segmentation: off [requested on]

Preserve the change across reboots

root@wieloryb-pve:~# cat /etc/network/if-up.d/ethtool2
#!/bin/sh

/sbin/ethtool -K enp0s31f6 tx off rx off

root@wieloryb-pve:~# chmod 755 /etc/network/if-up.d/ethtool2

Reboot and verify

root@wieloryb-pve:/etc#  shutdown -r now

root@wieloryb-pve:/etc/rc.d/init.d# /sbin/ethtool -k enp0s31f6
Features for enp0s31f6:
Cannot get device udp-fragmentation-offload settings: Operation not supported
rx-checksumming: off                   <--------- SHOULD BE OFF, HERE AND A FEW OTHER PLACES
tx-checksumming: off

 

Prevent Mac hidden files being written to a shared storage

MacOS tends to trash mounted disks with number of useless hidden files (e.g. ._* or .DS_Store ). There were number of recipes in the Internet, some of them were working at some of time but are not anymore.

Below notes from my fights (at a moment of writing, on MacOS High Sierra 10.13.4 (17E199).

Removing the files when they are created

I used to apply a find . -name … -print0 | xargs rm -0  combo but found that there is an built-in command to deal with these files:

$ dot_clean .

From the man:

NAME
dot_clean -- Merge ._* files with corresponding native files.

SYNOPSIS
dot_clean [-fmnsv] [--keep=[mostrecent|dotbar|native]] [dir ...]

DESCRIPTION
For each dir, dot_clean recursively merges all ._* files with their corresponding native files according to the rules specified with the given arguments. By default, if there is an attribute on the native file that is also present in the ._ file, the most recent attribute will be used.

Preventing creating the files

On the MacOS side

NOTE: None of these solutions worked for me!

$ defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool true
$ killall Finder

As I wrote, this didn’t work at all to me.

On the SMB server side

It actually turned the only working solution for me. Just add to the share configuration section a veto file parameter, e.g.:

$ sudo vi /etc/samba/smb.conf
[jhartman]
 path = /home/jhartman
.....
 veto files = /._*/.DS_Store/

And restart:

$ sudo service smbd restart

Note, after applying this change you will not be able anymore to use dot_clean  command from your Mac as any requests towards these files (including deletion) will be silently ignored by your SMB server hence files will remain untouched!

Oracle 12c 32-bit client and direct GOT relocation R_386_GOT32 against `lxecerr’

Edit 12/09/2018

As Ivan commented below:

with binutils release 27.28 it seems to be solved.

But see the workaround below in case it’s not…


When trying to install Oracle 12c 32-bit Client (12.1 or 12.2) on Oracle Linux 7.3 or 7.5, it throws error during linking static libraries:

INFO: Start output from spawned process:
INFO: ----------------------------------
INFO:

INFO: /u01/app/oracle/product/12.2.0_client32/bin/genclntsh

INFO: /bin/ld: /u01/app/oracle/product/12.2.0_client32/lib/libnls12.a(lxecg2e.o): direct GOT relocation R_386_GOT32 against `lxecerr' without base register can not be used when making a shared object
/bin/ld: final link failed: Bad value

INFO: collect2: error: ld returned 1 exit status

INFO: genclntsh: Failed to link libclntshcore.so.12.1

INFO: make: *** [client_sharedlib] Error 1

 

Not very specific, I found only one reference in Oracle KM Doc ID 2246237.1.

Rootcause and resolution in there was:

REASON

SLES 12 SP2 ships with binutils 2.26 which breaks compatibility for certain shared library links.

SOLUTION
01) Update binutils package to version binutils-2.26.1-9.15.1 or later.

But in fact, I do have in my server binutils-2.27-27.base.el7.x86_64 .

Workaround is to downgrade the binutils:

jhartman@sms.jhartman.pl:/home/jhartman$ sudo yum downgrade binutils*
Loaded plugins: ulninfo
Resolving Dependencies
--> Running transaction check
---> Package binutils.x86_64 0:2.25.1-32.base.el7_4.2 will be a downgrade
---> Package binutils.x86_64 0:2.27-27.base.el7 will be erased
---> Package binutils-devel.x86_64 0:2.25.1-32.base.el7_4.2 will be a downgrade
---> Package binutils-devel.x86_64 0:2.27-27.base.el7 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================================================================================================================================================================
 Package                                                            Arch                                                       Version                                                                     Repository                                                      Size
================================================================================================================================================================================================================================================================================
Downgrading:
 binutils                                                           x86_64                                                     2.25.1-32.base.el7_4.2                                                      ol7_latest                                                     5.4 M
 binutils-devel                                                     x86_64                                                     2.25.1-32.base.el7_4.2                                                      ol7_latest                                                     845 k

Transaction Summary

After this, the libs can be compiled and linked:

oracle$ export ORACLE_HOME=/u01/app/oracle/product/12.2.0_client32
oracle$ /u01/app/oracle/product/12.2.0_client32/bin/genclntsh
oracle$

I do believe that after finishing the installation, binutils can be upgraded again to the latest version (but not forget about this problem in case of installing the RSU).

PS: I’ve raised SR 3-17356002141 : Error when installing Oracle 32-bit client (direct GOT relocation R_386_GOT32 against `ipp_zcalloc’)