2013-07-19

MAC address swap with two network cards

1. Problem

1.1 When network setting is normal, found that MAC address was swapped.

eth0 –> 00:25:90:70:51:f1
eth1 –> 00:25:90:70:51:f0

[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:25:90:70:51:f1 brd ff:ff:ff:ff:ff:ff
    inet 172.16.110.230/24 brd 172.16.110.255 scope global eth0
    inet6 fe80::225:90ff:fe70:51f1/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:25:90:70:51:f0 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::225:90ff:fe70:51f0/64 scope link 
       valid_lft forever preferred_lft forever

1.2 But when made a bind to the two interface, it was correct.

[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 00:25:90:70:51:f0 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 00:25:90:70:51:f1 brd ff:ff:ff:ff:ff:ff
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 00:25:90:70:51:f1 brd ff:ff:ff:ff:ff:ff
    inet 172.16.110.230/24 brd 172.16.110.255 scope global bond0
    inet6 fe80::225:90ff:fe70:51f1/64 scope link 
       valid_lft forever preferred_lft forever

What had done when making the bind?

Use command dmesg, we found that if there were words like “udev: renamed XXX to XXX”,
the MAC addree was correct matched to the interface. If not it was error.

before changed:

[root@localhost ~]# dmesg | grep "PCI Express"
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
e1000e 0000:00:19.0: eth0: (PCI Express:2.5GT/s:Width x1) 00:25:90:70:51:f1
e1000e 0000:04:00.0: eth1: (PCI Express:2.5GT/s:Width x1) 00:25:90:70:51:f0

[root@localhost ~]# dmesg | grep udev
udev: starting version 147
udev: starting version 147

[root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
BOOTPROTO="dhcp"
HWADDR="00:25:90:70:51:F1"
NM_CONTROLLED="yes"
ONBOOT="no"
TYPE="Ethernet"
UUID="151c6203-2172-41b1-bd58-9bf03a2731e7"

[root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE="eth1"
BOOTPROTO="dhcp"
HWADDR="00:25:90:70:51:F0"
NM_CONTROLLED="yes"
ONBOOT="no"
TYPE="Ethernet"
UUID="a677dbc6-144a-455b-bcd3-6d8439560516"

[root@localhost ~]# cat /etc/udev/rules.d/70-persistent-net.rules 
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x8086:0x10d3 (e1000e) (custom name provided by external tool)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:70:51:f0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x8086:0x1502 (e1000e) (custom name provided by external tool)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:70:51:f1", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

after changed:

[root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
BOOTPROTO="dhcp"
HWADDR="00:25:90:70:51:F0"
NM_CONTROLLED="yes"
ONBOOT="no"
TYPE="Ethernet"
UUID="a677dbc6-144a-455b-bcd3-6d8439560516"
[root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE="eth1"
BOOTPROTO="dhcp"
HWADDR="00:25:90:70:51:F1"
NM_CONTROLLED="yes"
ONBOOT="no"
TYPE="Ethernet"
UUID="151c6203-2172-41b1-bd58-9bf03a2731e7"

then reboot system.

[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:25:90:70:51:f1 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::225:90ff:fe70:51f1/64 scope link 
       valid_lft forever preferred_lft forever
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:25:90:70:51:f0 brd ff:ff:ff:ff:ff:ff
    inet 172.16.110.230/24 scope global eth0
    inet6 fe80::225:90ff:fe70:51f0/64 scope link 
       valid_lft forever preferred_lft forever

[root@localhost ~]# dmesg | grep udev
udev: starting version 147
udev: starting version 147
udev: renamed network interface eth0 to rename2
udev: renamed network interface eth1 to eth0
udev: renamed network interface rename2 to eth1

[root@localhost ~]# cat /etc/udev/rules.d/70-persistent-net.rules 
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x8086:0x10d3 (e1000e) (custom name provided by external tool)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:70:51:f0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x8086:0x1502 (e1000e) (custom name provided by external tool)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:70:51:f1", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x8086:0x1502 (e1000e) (custom name provided by external tool)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:70:51:f1", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x8086:0x10d3 (e1000e) (custom name provided by external tool)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:25:90:70:51:f0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

[root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
BOOTPROTO="dhcp"
HWADDR="00:25:90:70:51:F0"
NM_CONTROLLED="yes"
ONBOOT="no"
TYPE="Ethernet"
UUID="a677dbc6-144a-455b-bcd3-6d8439560516"
[root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE="eth1"
BOOTPROTO="dhcp"
HWADDR="00:25:90:70:51:F1"
NM_CONTROLLED="yes"
ONBOOT="no"
TYPE="Ethernet"
UUID="151c6203-2172-41b1-bd58-9bf03a2731e7"
[root@localhost ~]#

1.3 make a bind to the network, MAC is correct also.

[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 00:25:90:70:51:f1 brd ff:ff:ff:ff:ff:ff
3: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 00:25:90:70:51:f0 brd ff:ff:ff:ff:ff:ff
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 00:25:90:70:51:f0 brd ff:ff:ff:ff:ff:ff
    inet 172.16.110.230/24 brd 172.16.110.255 scope global bond0
    inet6 fe80::225:90ff:fe70:51f0/64 scope link 
       valid_lft forever preferred_lft forever
[root@localhost ~]#

delete the bind, still ok.

1.4 How for group ip?

group set to 172.16.110.233
eth0 set to 172.16.110.230
eth1 set to 172.16.110.231
from below we found the MAC was ok as hoped.

[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:25:90:70:51:f1 brd ff:ff:ff:ff:ff:ff
    inet 172.16.110.231/24 scope global eth1
    inet 172.16.110.233/24 scope global secondary eth1
    inet6 fe80::225:90ff:fe70:51f1/64 scope link 
       valid_lft forever preferred_lft forever
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:25:90:70:51:f0 brd ff:ff:ff:ff:ff:ff
    inet 172.16.110.230/24 scope global eth0
    inet 172.16.110.233/24 scope global secondary eth0
    inet6 fe80::225:90ff:fe70:51f0/64 scope link 
       valid_lft forever preferred_lft forever

2. The steps to modify MAC of eth* look like as below

OS boot
udev start
e1000e start, get eth* and MAC
check /etc/sysconfig/network-scripts/ifcfg-XXX, call shell script
/lib/udev/write_net_rules to generate rule file to /etc/udev/rules.d/70-persistent-net.rules
if MAC was not match with what got from e1000e (view udev source, you wil find udev)
udev rename eth* if there have “add” action in rule file /etc/udev/rules.d/70-persistent-net.rules

2.1 Check /lib/udev/write_net_rules

36 RULES_FILE='/etc/udev/rules.d/70-persistent-net.rules'
37 
38 . /lib/udev/rule_generator.functions
39 
40 find_all_ifcfg() {
41     local links=$1
42     local __sed_discard_ignored_files='/\(~\|\.bak\|\.orig\|\.rpmnew\|\.rpmorig\|\.rpmsave\)$/d'
43 
44     files=$(echo /etc/sysconfig/network-scripts/ifcfg-* \
45         | LC_ALL=C sed -e "$__sed_discard_ignored_files")
46     for i in $files; do
47         (
48             . $i
49             [ -n "$HWADDR" ] && [ "${links%%[ \[\]0-9]*}" = "${DEVICE%%[ \[\]0-9]*}" ] && echo $DEVICE
50         )
51     done
52 }

70 write_rule() {
71         local match="$1"
72         local name="$2"
73         local comment="$3"
74 
75         {
76         if [ "$PRINT_HEADER" ]; then
77                 PRINT_HEADER=
78                 echo "# This file was automatically generated by the $0"
79                 echo "# program, run by the persistent-net-generator.rules rules file."
80                 echo "#"
81                 echo "# You can modify it, as long as you keep each rule on a single"
82                 echo "# line, and change only the value of the NAME= key."
83         fi
84 
85         echo ""
86         [ "$comment" ] && echo "# $comment"
87         echo "SUBSYSTEM==\"net\", ACTION==\"add\"$match, NAME=\"$name\""
88         } >> $RULES_FILE
89 }

153 write_rule "$match" "$INTERFACE" "$COMMENT"

2.1 Check udev source

search the string we found in dmesg:

[dennis@localhost udev]$ pwd
/home/dennis/Downloads/udev-174/udev
[dennis@localhost udev]$ grep -r -i --include "*.c" "renamed" .
./udev-rules.c:    /* handle device, renamed by external tool, returning new path */
./udev-event.c:    fprintf(f, "<30>udevd[%u]: renamed network interface %s to %s\n",
./udev-event.c:                info(event->udev, "renamed netif to '%s'\n", event->name);
[dennis@localhost udev]$ gvim udev-event.c 

static void rename_netif_kernel_log(struct ifreq ifr)
{
    int klog;
    FILE *f;

    klog = open("/dev/kmsg", O_WRONLY);
    if (klog < 0)
        return;

    f = fdopen(klog, "w");
    if (f == NULL) {
        close(klog);
        return;
    }

    fprintf(f, "<30>udevd[%u]: renamed network interface %s to %s\n",
        getpid(), ifr.ifr_name, ifr.ifr_newname);
    fclose(f);
}

Because udev write log to /dev/kmsg, so we can got output like below:

[root@localhost ~]# dmesg | grep udev
udev: starting version 147
udev: starting version 147
udev: renamed network interface eth0 to rename2
udev: renamed network interface eth1 to eth0
udev: renamed network interface rename2 to eth1

Continue tracking, found that rename_netif_kernel_log was called only by function:

static int rename_netif(struct udev_event *event)

And who call rename_netif, the answer is only function:

int udev_event_execute_rules(struct udev_event *event, struct udev_rules *rules, const sigset_t *sigmask)

From code segment from this function, we see:

....
/* rename a new network interface, if needed */
if (udev_device_get_ifindex(dev) > 0 && strcmp(udev_device_get_action(dev), "add") == 0 &&
    event->name != NULL && strcmp(event->name, udev_device_get_sysname(dev)) != 0) {
    char syspath[UTIL_PATH_SIZE];
    char *pos;

    err = rename_netif(event);
....

from the name of function udev_event_execute_rules and cod segment, we can say,
udev rename ethXXX by the rules, where should have action like as “add”.

2.2 Check e1000e source, found nothing

2.3 Check dmesg source, found nothing

2.4 Check system-config-network source, found nothing

3. How to fix

Use this shell scripts to check MAC

#!/bin/sh

# check MAC address, if not match the correct interface, fix it.
file_eth0=/etc/sysconfig/network-scripts/ifcfg-eth0
file_eth1=/etc/sysconfig/network-scripts/ifcfg-eth1

if [ -e $file_eth0 ] && [ -e $file_eth1 ]; then 
    # get MAC and UUID
    MAC0=$(grep HWADDR $file_eth0 | awk -F'"' '{print $(NF-1)}')
    MAC1=$(grep HWADDR $file_eth1 | awk -F'"' '{print $(NF-1)}')
    UUID0=$(grep UUID  $file_eth0 | awk -F'"' '{print $(NF-1)}')
    UUID1=$(grep UUID  $file_eth1 | awk -F'"' '{print $(NF-1)}')

    if [ "$MAC0" != "" ] && [ "$MAC1" != "" ]; then
        # compare MAC
        compare_flag=0
        compare_flag=`echo $MAC0 $MAC1 | awk '$1>$2 {print 1}'`
        if [ "$compare_flag" == "1" ]; then
            # swap MAC and UUID
            sed -i 's/'$MAC0'/'$MAC1'/g' $file_eth0
            sed -i 's/'$MAC1'/'$MAC0'/g' $file_eth1
            if [ "$UUID0" != "" ] && [ "$UUID1" != "" ]; then
                sed -i 's/'$UUID0'/'$UUID1'/g' $file_eth0
                sed -i 's/'$UUID1'/'$UUID0'/g' $file_eth1
            else
                sed -i '/UUID/d' $file_eth0
                [ "$UUID1" != "" ] && echo "UUID=\"$UUID1\"" >> $file_eth0
                sed -i '/UUID/d' $file_eth1
                [ "$UUID0" != "" ] && echo "UUID=\"$UUID0\"" >> $file_eth1
            fi
        fi
    fi
fi

4. Question

/etc/sysconfig/network-scripts/ifcfg-XXX were generated by who? system-config-network?

Matrix207's Blog