storage

iSCSI Enterprise Target

iSCSI Enterprise Target

Introduce

SCSI     : small computer system interface
iSCSI    : Internet SCSI
Initiator:
Target   :

component of protocol

  1. Communication

    • connect
    • session
      • login
      • full feature
    • pdu

  2. Naming and addressing

    • iqn.iSCSI
    • eui.IEEEEUI264

  3. Error Handle

    • misclassification
    • Recovery mechanism

  4. Security

    • Authentication
      • CHAP

Code flow

  1. Login phase
  2. Data execute of text phase
  3. logout pahse
    • Report LUNS(0xa0)->
    • Inquiry(0x12) ->
    • Read Capacity(10) (0x25) ->
    • Read(10)(0x28) ->
    • Mode Sense(6)(0x1a)

Source code

  • source:

  • build:

  • patch

    tar xvf iscsitarget-1.4.20.tar.gz
cd iscsitarget-1.4.20
patch -p1 < ../iscsitarget-1.4.20_patch/compat-2.6.32.patch
patch -p1 < ../iscsitarget-1.4.20_patch/iscsitarget-xxx-1.4.20_for_centos_6_2.patch
patch -p1 < ../iscsitarget-1.4.20_patch/iscsi_conn_close_raid_failed.patch
patch -p1 < ../iscsitarget-1.4.20_patch/iscsi_load_balance_bug_fix.patch
patch -p1 < ../iscsitarget-1.4.20_patch/iscsi_flow_control.patch
patch -p1 < ../iscsitarget-1.4.20_patch/iscsi_disable_sync_cache.patch

Analysis

Kernel module

iscsi_trgt.ko

most of the work were done in kernel, such as data send and recv,
and the user mode do session create, author and login.

sub-module list as below:

  • Files
    • iotype.c
    • nthread.c
      • listen iSCSI request from iSCSI Initiator
      • receive iSCSI request
      • initialize iSCSI Command structure and LUN parameters
      • allocate data cache (struct tio)
      • handle r2t data
    • wthread.c
      • communicate with nthread
      • handle iSCSI command from nthread
      • command can be divided into two types: related , not related
      • execute callback function
    • response
      • dispatch by wthread

User module

ietd daemon, manager tool for IET, function description:

  • configuration, such as port, uid, gid and so on.
  • read config info from /etc/ietd.conf, and send to kernel module

iSCSI mod

  • blockio
    translate network data to struct tio, then handle by
    blockio_make_request from block_io.c, submit_bio to
    common layer after generate bio
  • fileio
    still use VFS, and do system call read/write to operate data.
  • functional description

How IET add lun?

加入volume有两种方式:

  1. 通过配置文件ietd.conf
  2. 通过ietdadm命令，使用--op new --tid=[id] --lun=[lun] --params Path=[path]参数

第一种方式的流程如下: (用户态)
plain_init() -> __plain_lunit_create -> ki->lunit_create -> iscsi_lunit_create -> ioctl(ctrl_fd, ADD_VOLUME, &info)

第二种方式的流程如下: (用户态)
ietadm_request_exec -> cops->lunit_add() -> plain_lunit_create() -> __plain_lunit_create 后面同上

核态部分相同:

  • config.c:add_volume -> volume.c:volume_add -> volume.c:parse_volume_params
  • volume.c:parse_volume_params 函数检查发现是opt_iomode类型，

Type, IOMode and Sector Setting

Run by the ietadm command as below:

  • ietadm --op new --tid=%d --lun=%d --params Path=/dev/%s/%s,IOMode=%s,Sector=%s,Type=%s
    • IOMode: wb or wt or ro
    • Sector: 512 or 4096
    • Type: fileio or blockio

Function calling:

  • ietadm.c: main() -> lunit_handle() -> ietd_request() -> ietd_connect(),ietd_request_send(),ietd_response_recv()
    • ietd_connect() -> socket(AF_LOCAL, SOCK_STREAM, 0), connect()
    • ietd_request_send() -> write(fd, req, sizeof(*req))
    • ietd_request_recv() -> readv(), read()
  • ietd.c: main() -> message.c: ietadm_request_listen() ->
    • ietadm_request_listen() -> socket(AF_LOCAL, SOCK_STREAM, 0), bind(), listen()

Code analysis:

  • ietadm --op new --tid= --lun= --params
    • 调用 ietadm.c:lunit_handle函数 ->ietd_request函数 ->ietd_request_send函数
    • ietd.c的event_loop函数监听到有消息，执行message.c:ietadm_request_handle函数
    • message.c:ietadm_request_handle函数 -> ietadm_request_exec函数
      类型为C_LUNIT_NEW, 执行cops->lunit_add
    • plain.c:plain_lunit_create ->__plain_lunit_create -> ki->lunit_create
    • ctldev.c:iscsi_lunit_create -> ioctl(ctrl_fd, ADD_VOLUME, &info)
    • config.c:add_volume -> volume.c:volume_add -> volume.c:parse_volume_params
    • volume.c:parse_volume_params 函数检查发现是opt_iomode类型，
      如果是ro类型执行SetLUReadonly(volume);
      如果是wb类型执行SetLUWCache(volume);
      如果是wt类型则类型不可用
    • SetLUReadonly和SetLUWCache都在iscsi.h中定义，都只是修改struct iet_volume结构体中的flags的值。
    • 另外块大小blocksize保存在结构体struct iet_volume的blk_shift中,且块大小必须是
      2的幂次方，大于等于512，小于4096. 注意保存的是2的幂的次数，即如果是1024，则blk_shift是10
    • 如果fileio,进入file-io.c:fileio_attach函数, 调用parse_fileio_params, 这里会忽略好些参数的值.
      之用path参数会有处理。
    • 如果blockio,进入block-io.c:blockio_attach函数, 调用parse_blockio_params, 这里会忽略好些参数的值.
      之用path参数会有处理。
    • file-io.c 和 block-io.c要好好看看，这个逻辑卷到底怎么创建的.

Summary:

  • 总结:
    • Type: 如果是fileio那么读写操作通过vfs请求完成，如果是blockio则读写操作直接构造bio请求
    • IOMode: 如果是wb,
    • Sector

Other iSCSI implement

Reference

others

software agent

Concepts

M.Minsky曾经在1994年就指出，“Agent是一些具有特别技能的个体”。针对计算机系统，
Agent是指“当使用者向机器说明完成某些任务，而无需了解机器自身是如何工作，即将其处理为黑箱时，就称其为Agent。”
目前学术界对其尚无一个公认的定义，国内多将其译为智能代理。

Application

Reference

program

use moosefs

Master server installation

  • Installation

    groupadd mfs
useradd -g mfs mfs
cd /usr/src
tar -zxvf mfs-1.6.15.tar.gz
cd mfs-1.6.15
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var/lib 
--with-default-user=mfs --with-default-group=mfs --disable-mfschunkserver 
--disable-mfsmount
make
make install

  • Configuration

  1. Copy config

    cd /etc
cp mfsmaster.cfg.dist mfsmaster.cfg
cp mfsmetalogger.cfg.dist mfsmetalogger.cfg
cp mfsexports.cfg.dist mfsexports.cfg

# modify mfsexports.cfg 
# 192.168.2.0/24 / rw,alldirs,maproot=0

  2. Copy meta data file

    cd /var/lib/mfs
cp metadata.mfs.empty metadata.mfs

  3. Config host name

    echo 192.168.1.1 mfsmaster>> /etc/hosts

  • Run server

    /usr/sbin/mfsmaster start

# Run CGI monitor server, and view info under <http://192.168.1.1:9425>
/usr/sbin/mfscgiserv

Install Backup Server (metalogger)

Metalogger installation is very similar to that of the master server.
We issue the following commands:

groupadd mfs
useradd -g mfs mfs
cd /usr/src
tar -zxvf mfs-1.6.15.tar.gz
cd mfs-1.6.15
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var/lib 
--with-default-user=mfs --with-default-group=mfs --disable-mfschunkserver 
--disable-mfsmount
make
make install

cd /etc
cp mfsmetalogger.cfg.dist mfsmetalogger.cfg

Similarly, in /etc/hosts we add:

192.168.1.1 mfsmaster

Now we are ready to start the backup server process :

/usr/sbin/mfsmetalogger start

In a production environment you should set up automatic start of mfsmetalogger.

Chunk servers installation

We issue the following commands on the machines which are to be chunks servers:

groupadd mfs
useradd -g mfs mfs
cd /usr/src
tar -zxvf mfs-1.6.15.tar.gz
cd mfs-1.6.15
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var/lib 
--with-default-user=mfs --with-default-group=mfs --disable-mfsmaster
make
make install

Now we similarly prepare configuration files of the chunk server:

cd /etc/
cp mfschunkserver.cfg.dist mfschunkserver.cfg
cp mfshdd.cfg.dist mfshdd.cfg

use /mnt/mfschunks1 and /mnt/mfschunks2 locations , so we add these two
lines to mfshdd.cfg file:

/mnt/mfschunks1
/mnt/mfschunks2

Create a .lock file to make user mfs has rights to write the mounted partitions.

chown -R mfs:mfs /mnt/mfschunks1
chown -R mfs:mfs /mnt/mfschunks2

Similarly we add the following line in /etc/hosts file:

192.168.1.1 mfsmaster

Now we are ready to start the chunk server:

/usr/sbin/mfschunkserver start

Users’ computers installation

Install FUSE

cd /usr/src
tar -zxvf fuse-2.8.3.tar.gz
cd fuse-2.8.3
./configure
make
make install

In order to install mfsmount package we do the following steps:

tar -zxvf mfs-1.6.15.tar.gz
cd mfs-1.6.15
./configure --prefix=/usr --sysconfdir=/etc
--localstatedir=/var/lib --with-default-user=mfs
--with-default-group=mfs --disable-mfsmaster
--disable-mfschunkserver
make
make install

In a /etc/hosts file we add this line:

192.168.1.1 mfsmaster

Let’s assume that we will mount the system in a /mnt/mfs folder on a client’s
machine. We issue the following commands:

mkdir -p /mnt/mfs
/usr/bin/mfsmount /mnt/mfs -H mfsmaster

Now after issuing the df -h | grep mfs command we should get information
similar to this:

/storage/mfschunks/mfschunks1
    2.0G 69M 1.9G 4% /mnt/mfschunks1
/storage/mfschunks/mfschunks2
    2.0G 69M 1.9G 4% /mnt/mfschunks2
mfs#mfsmaster:9421 3.2G 0 3.2G 0% /mnt/mfs

Installing MooseFS on one server

  • Install FUSE
  • Install Moosefs

Basic MooseFS Usage

The number of copies for the folder is set with the mfssetgoal –r command: 

mfssetgoal -r 1 /mnt/mfs/folder1
/mnt/mfs/folder1:
inodes with goal changed: 0
inodes with goal not changed: 1
inodes with permission denied: 0
mfssetgoal -r 2 /mnt/mfs/folder2
/mnt/mfs/folder2:
inodes with goal changed: 0
inodes with goal not changed: 1
inodes with permission denied: 0

The command mfscheckfile is used for checking in how many copies the given
file is stored.

Reference

program

sms develop

Introduce

  1. Different between RS232 and USB interface
    For RS232, use serial operation directory, not need driver.
    For USB, should intall driver.

  2. OS support
    Both windows and Linux will be supported, if the device support RS232, and
    most of it support AT command, customer develop application with AT mostly.

Installation

  1. Minicom

Ref:

  1. Gnokii

Issue

minicom, not response for input
Serial port setup –>Hardware Flow Contorl : NO
Local on/off

http://blog.sina.com.cn/s/blog_5d0e8d0d01015svy.html

AT

  • get SIM ID

    at+ccid

  • get Chip Model

    ati3

    at+cgmr

  • get signal strength
    normal : 16~31,
    SMS err: <16 or="">31(display 99)

    at+csq

  • get baud rate

    at+ipr?

  • set baud rate (after set, need to re-connect)

    at+ipr=115200

  • save configuration

    at&w

  • set SMS format

    at+cmgf=1

  • send SMS, success if return “+CMGS: “

    at+cmgs=137**

    Hi, this is a SMS test

  • read SMS, 4 is the position of SMS

    at+cmgr=4

  • get number of SMS center

    at+csca?

  • calling

    atd137**;

  • hand up

    ath

  • check state of SMS module

    at+cfun?

  • reboot SMS module

    at+cfun=0
    OK
    at+cfun=1
    OK

  • get IMEI code

    at+wmsn or at+cgsn

  • Full view

    at+ccid
+CCID: "898600p1011130234927"

OK
ati3
657e09gg.Q24PL002 1961548 103107 17:56

OK
at+cgmr
657e09gg.Q24PL002 1961548 103107 17:56

OK
at+csq
+CSQ: 25,0

OK
at+ipr?
+IPR: 115200

OK
at+ipr=9600
OK

at
OK
at&w
OK

at+csq
+CSQ: 25,99

OK
at+cmgf=1
OK
at+cmgs=13707070909
> Hi, this is SMS test.
+CMGS: 201

OK

Ref http://www.sendsms.cn/download/js1.pdf

Features

  1. auto detect device by baut-rate
  2. simple interface
    1). send message
    2). recv message

FAQ

  1. Fail to find device
  2. Fail to send message
    there are many reasons to cause this, list the main reasons as below:
    1). device error
    1). baut-rate error
    1). signal too weak

Reference

source

moosefs source code analysis

mfs-1.5.9

  1. Files

    main.c —> Init,
    init.c —> defined a RunTab, use for mainloop
    changelog.c changelog_init change log
    random.c rndinit random generator
    matocsserv.c matocsserv_init communication with chunkserver
    matocuserv.c matocuserv_init communication with customer
    filesystem.c fs_init file system manager
    stats.c stats_init statistics module

    [dennis@localhost mfs-1.5.9]$ ls
    aclocal.m4 config.sub COPYING INSTALL Makefile.in
    config.guess configure depcomp install-sh mfschunkserver
    config.h.in configure.ac doc Makefile.am MFSCommunication.h
    mfsdata mfsmount README mfsmaster missing mfsmetarestore
    NEWS

  2. TODO

master server

  1. Files

    [dennis@localhost mfs-1.5.9]$ ls mfsmaster/
    cfg.c changelog.h filesystem.h lempelziv.h Makefile.am
    matocsserv.h random.c sockets.h cfg.h chunks.c
    datapack.h init.h main.c Makefile.in matocuserv.c
    changelog.c chunks.h filesystem.c lempelziv.c main.h
    matocsserv.c matocuserv.h sockets.c stats.h random.h stats.c

chunk server

  1. TODO

client

  1. TODO

Reference

program

use protobuf

checkout protobuf

svn checkout http://protobuf.googlecode.com/svn/trunk protobuf

compile, install

[dennis@localhost protobuf]$ cd protobuf
[dennis@localhost protobuf]$ sh autogen.sh 
[dennis@localhost protobuf]$ ./configure
[dennis@localhost protobuf]$ make check 
[dennis@localhost protobuf]$ make install

[dennis@localhost protobuf]$ protoc --version
libprotoc 2.5.1

write sample code

[dennis@localhost protobuf]$ mkdir ../google-code-sample
[dennis@localhost protobuf]$ cd ../google-code-sample
[dennis@localhost google-code-sample]$ vim addressbook.proto
[dennis@localhost google-code-sample]$ vim pb_write.cc
[dennis@localhost google-code-sample]$ vim pb_read.cc
[dennis@localhost google-code-sample]$ vim Makefile

cat addressbook.proto

package tutorial;

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    required string number = 1;
    optional PhoneType type = 2 [default = HOME];
  }

  repeated PhoneNumber phone = 4;
}

message AddressBook {
  repeated Person person = 1;
}

cat pb_write.cc

#include <iostream>
#include <fstream>
#include <string>
#include "addressbook.pb.h"
using namespace std;

// This function fills in a Person message based on user input.
void PromptForAddress(tutorial::Person* person) {
  cout << "Enter person ID number: ";
  int id;
  cin >> id;
  person->set_id(id);
  cin.ignore(256, '\n');

  cout << "Enter name: ";
  getline(cin, *person->mutable_name());

  cout << "Enter email address (blank for none): ";
  string email;
  getline(cin, email);
  if (!email.empty()) {
    person->set_email(email);
  }

  while (true) {
    cout << "Enter a phone number (or leave blank to finish): ";
    string number;
    getline(cin, number);
    if (number.empty()) {
      break;
    }

    tutorial::Person::PhoneNumber* phone_number = person->add_phone();
    phone_number->set_number(number);

    cout << "Is this a mobile, home, or work phone? ";
    string type;
    getline(cin, type);
    if (type == "mobile") {
      phone_number->set_type(tutorial::Person::MOBILE);
    } else if (type == "home") {
      phone_number->set_type(tutorial::Person::HOME);
    } else if (type == "work") {
      phone_number->set_type(tutorial::Person::WORK);
    } else {
      cout << "Unknown phone type.  Using default." << endl;
    }
  }
}

// Main function:  Reads the entire address book from a file,
//   adds one person based on user input, then writes it back out to the same
//   file.
int main(int argc, char* argv[]) {
  // Verify that the version of the library that we linked against is
  // compatible with the version of the headers we compiled against.
  GOOGLE_PROTOBUF_VERIFY_VERSION;

  if (argc != 2) {
    cerr << "Usage:  " << argv[0] << " ADDRESS_BOOK_FILE" << endl;
    return -1;
  }

  tutorial::AddressBook address_book;

  {
    // Read the existing address book.
    fstream input(argv[1], ios::in | ios::binary);
    if (!input) {
      cout << argv[1] << ": File not found.  Creating a new file." << endl;
    } else if (!address_book.ParseFromIstream(&input)) {
      cerr << "Failed to parse address book." << endl;
      return -1;
    }
  }

  // Add an address.
  PromptForAddress(address_book.add_person());

  {
    // Write the new address book back to disk.
    fstream output(argv[1], ios::out | ios::trunc | ios::binary);
    if (!address_book.SerializeToOstream(&output)) {
      cerr << "Failed to write address book." << endl;
      return -1;
    }
  }

  // Optional:  Delete all global objects allocated by libprotobuf.
  google::protobuf::ShutdownProtobufLibrary();

  return 0;
}

cat pb_read.cc

#include <iostream>
#include <fstream>
#include <string>
#include "addressbook.pb.h"
using namespace std;

// Iterates though all people in the AddressBook and prints info about them.
void ListPeople(const tutorial::AddressBook& address_book) {
  for (int i = 0; i < address_book.person_size(); i++) {
    const tutorial::Person& person = address_book.person(i);

    cout << "Person ID: " << person.id() << endl;
    cout << "  Name: " << person.name() << endl;
    if (person.has_email()) {
      cout << "  E-mail address: " << person.email() << endl;
    }

    for (int j = 0; j < person.phone_size(); j++) {
      const tutorial::Person::PhoneNumber& phone_number = person.phone(j);

      switch (phone_number.type()) {
        case tutorial::Person::MOBILE:
          cout << "  Mobile phone #: ";
          break;
        case tutorial::Person::HOME:
          cout << "  Home phone #: ";
          break;
        case tutorial::Person::WORK:
          cout << "  Work phone #: ";
          break;
      }
      cout << phone_number.number() << endl;
    }
  }
}

// Main function:  Reads the entire address book from a file and prints all
//   the information inside.
int main(int argc, char* argv[]) {
  // Verify that the version of the library that we linked against is
  // compatible with the version of the headers we compiled against.
  GOOGLE_PROTOBUF_VERIFY_VERSION;

  if (argc != 2) {
    cerr << "Usage:  " << argv[0] << " ADDRESS_BOOK_FILE" << endl;
    return -1;
  }

  tutorial::AddressBook address_book;

  {
    // Read the existing address book.
    fstream input(argv[1], ios::in | ios::binary);
    if (!address_book.ParseFromIstream(&input)) {
      cerr << "Failed to parse address book." << endl;
      return -1;
    }
  }

  ListPeople(address_book);

  // Optional:  Delete all global objects allocated by libprotobuf.
  google::protobuf::ShutdownProtobufLibrary();

  return 0;
}

cat Makefile

CC     = g++
FLAGS  = -Wall
PRO_F  = addressbook
PRO_SRC= $(PRO_F).pb.cc
LIBS   = -lprotobuf
TARGET = pb_write pb_read

all:$(TARGET) 

pb_write: proto_file 
    $(CC) $(FLAGS) $(LIBS) -o $@ pb_write.cc $(PRO_SRC)

pb_read: proto_file 
    $(CC) $(FLAGS) $(LIBS) -o $@ pb_read.cc $(PRO_SRC)

proto_file: $(PRO_F).proto
    protoc -I=./ --cpp_out=./ $(PRO_F).proto

.PHONEY:clean 
clean:
    rm -f $(TARGET) $(OBJS) $(PRO_F).pb.h $(PRO_F).pb.cc

Files:

[dennis@localhost google-code-sample]$ ls
addressbook.proto  Makefile  pb_read.cc  pb_write.cc

Compile and run:

[dennis@localhost google-code-sample]$ make
protoc -I=./ --cpp_out=./ addressbook.proto
g++ -Wall -lprotobuf -o pb_write pb_write.cc addressbook.pb.cc
g++ -Wall -lprotobuf -o pb_read pb_read.cc addressbook.pb.cc
[dennis@localhost google-code-sample]$ ls
addressbook.pb.cc  addressbook.pb.h  addressbook.proto  Makefile  pb_read  pb_read.cc  pb_write  pb_write.cc
[dennis@localhost google-code-sample]$ ./pb_write dennis
dennis: File not found.  Creating a new file.
Enter person ID number: 13
Enter name: dennis
Enter email address (blank for none): dennis.cpp@gmail.com
Enter a phone number (or leave blank to finish): 14159265358
Is this a mobile, home, or work phone? mobile
Enter a phone number (or leave blank to finish): 
[dennis@localhost google-code-sample]$ ./pb_write dennis
Enter person ID number: 19
Enter name: tina
Enter email address (blank for none): tina.mm@gmail.com
Enter a phone number (or leave blank to finish): 21714169628
Is this a mobile, home, or work phone? home 
Enter a phone number (or leave blank to finish): 
[dennis@localhost google-code-sample]$ ./pb_read dennis
Person ID: 13
  Name: dennis
  E-mail address: dennis.cpp@gmail.com
  Mobile phone #: 14159265358
Person ID: 19
  Name: tina
  E-mail address: tina.mm@gmail.com
  Home phone #: 21714169628
[dennis@localhost google-code-sample]$ ls
addressbook.pb.cc  addressbook.pb.h  addressbook.proto  dennis  Makefile  pb_read  pb_read.cc  pb_write  pb_write.cc

Problem:

solution of error while loading shared libraries,
look here
for details.

Reference:

program

solution of error while loading shared libraries

After I finish compile and install protobuf on my machine, then I flow the tutorial
to compile the sample code, but after that I can not run it, the error message is
as below: 

[dennis@localhost google-code-sample]$ vim addressbook.proto 
[dennis@localhost google-code-sample]$ ls
addressbook.proto
[dennis@localhost google-code-sample]$ protoc -I=./ --cpp_out=./ ./addressbook.proto 
[dennis@localhost google-code-sample]$ ls
addressbook.pb.cc  addressbook.pb.h  addressbook.proto
[dennis@localhost google-code-sample]$ vim pb_sample.cc
[dennis@localhost google-code-sample]$ ls
addressbook.pb.cc  addressbook.pb.h  addressbook.proto pb_sample.cc
[dennis@localhost google-code-sample]$ c++ addressbook.pb.cc pb_sample.cc -o pb_sample -lprotobuf
[dennis@localhost google-code-sample]$ ls
addressbook.pb.cc  addressbook.pb.h  addressbook.proto  pb_sample  pb_sample.cc
[dennis@localhost google-code-sample]$ ./pb_sample 
./pb_sample: error while loading shared libraries: libprotobuf.so.9: cannot open shared object file: No such file or directory

But I find the library is exist on /usr/local/lib

[dennis@localhost protobuf]$ ls /usr/local/lib/libproto* -l
-rw-r--r--. 1 root root 19747400 Jun  7 11:49 /usr/local/lib/libprotobuf.a
-rwxr-xr-x. 1 root root      968 Jun  7 11:49 /usr/local/lib/libprotobuf.la
-rw-r--r--. 1 root root  2065712 Jun  7 11:49 /usr/local/lib/libprotobuf-lite.a
-rwxr-xr-x. 1 root root     1003 Jun  7 11:49 /usr/local/lib/libprotobuf-lite.la
lrwxrwxrwx. 1 root root       25 Jun  7 11:49 /usr/local/lib/libprotobuf-lite.so -> libprotobuf-lite.so.9.0.0
lrwxrwxrwx. 1 root root       25 Jun  7 11:49 /usr/local/lib/libprotobuf-lite.so.9 -> libprotobuf-lite.so.9.0.0
-rwxr-xr-x. 1 root root   982744 Jun  7 11:49 /usr/local/lib/libprotobuf-lite.so.9.0.0
lrwxrwxrwx. 1 root root       20 Jun  7 11:49 /usr/local/lib/libprotobuf.so -> libprotobuf.so.9.0.0
lrwxrwxrwx. 1 root root       20 Jun  7 11:49 /usr/local/lib/libprotobuf.so.9 -> libprotobuf.so.9.0.0
-rwxr-xr-x. 1 root root  8343446 Jun  7 11:49 /usr/local/lib/libprotobuf.so.9.0.0
-rw-r--r--. 1 root root 23573484 Jun  7 11:49 /usr/local/lib/libprotoc.a
-rwxr-xr-x. 1 root root      984 Jun  7 11:49 /usr/local/lib/libprotoc.la
lrwxrwxrwx. 1 root root       18 Jun  7 11:49 /usr/local/lib/libprotoc.so -> libprotoc.so.9.0.0
lrwxrwxrwx. 1 root root       18 Jun  7 11:49 /usr/local/lib/libprotoc.so.9 -> libprotoc.so.9.0.0
-rwxr-xr-x. 1 root root  8689345 Jun  7 11:49 /usr/local/lib/libprotoc.so.9.0.0

How the sytem know Where the libraries is location? 

[dennis@localhost google-code-sample]$ ls /etc/ld.so.c* -l
-rw-r--r--. 1 root root 141410 Jun  7 12:49 /etc/ld.so.cache
-rw-r--r--. 1 root root     43 Jun  7 12:48 /etc/ld.so.conf

/etc/ld.so.conf.d:
total 48
-rw-r--r--. 1 root root 17 Sep  8  2012 atlas-x86_64.conf
-r--r--r--. 1 root root 63 Dec  3  2013 kernel-3.11.10-100.fc18.x86_64.conf
-r--r--r--. 1 root root 63 Nov  4  2013 kernel-3.11.7-100.fc18.x86_64.conf
-r--r--r--. 1 root root 63 Nov 21  2013 kernel-3.11.9-100.fc18.x86_64.conf
-rw-r--r--. 1 root root 17 May  3  2013 libiscsi-x86_64.conf
-rw-r--r--. 1 root root 16 May 29  2013 llvm-x86_64.conf
-rw-r--r--. 1 root root 17 Dec 10 23:50 mysql-x86_64.conf
-rw-r--r--. 1 root root 22 Jul 22  2012 qt-x86_64.conf
-rw-r--r--. 1 root root 24 Feb 26  2013 tracker-x86_64.conf
-rw-r--r--. 1 root root 15 Nov 25  2013 wine-32.conf
-rw-r--r--. 1 root root 17 Nov 25  2013 wine-64.conf
-rw-r--r--. 1 root root 21 Dec 10 03:28 xulrunner-64.conf

[dennis@localhost google-code-sample]$ cat /etc/ld.so.conf
include ld.so.conf.d/*.conf

Solution, add the library directory to config file, then update 

[dennis@localhost google-code-sample]$ su -c 'echo "/usr/local/lib" >> /etc/ld.so.conf'
Password: 
[dennis@localhost google-code-sample]$ cat /etc/ld.so.conf
include ld.so.conf.d/*.conf
/usr/local/lib
[dennis@localhost google-code-sample]$ su -c 'ldconfig'
Password:

right now, you will see that the timestamp of cache file is new

[dennis@localhost google-code-sample]$ ls /etc/ld.so.cache -l
-rw-r--r--. 1 root root 141410 Jun  7 13:07 /etc/ld.so.cache

Reference:

error while loading shared libraries: xxx.so.x错误的原因和解决办法

program

Open source need to master

Google open source

  • breakpad
    Crash reporting

  • gflags
    Commandline flags module for C++

  • glog
    Logging library for C++

  • gperftools
    Fast, multi-threaded malloc() and nifty performance analysis tools

  • gtest
    Google C++ Testing Framework

  • leveldb
    A fast and lightweight key/value database library by Google.

  • protobuf
    Protocol Buffers - Google’s data interchange format

Other

  • libevent
    an event notification library

  • libnl
    The libnl suite is a collection of libraries providing APIs to netlink protocol based Linux kernel interfaces.

  • libunwind
    The primary goal of this project is to define a portable and efficient C programming interface (API) to determine the call-chain of a program

  • mono

  • snappy
    A fast compressor/decompressor

  • fuse
    With FUSE it is possible to implement a fully functional filesystem in a userspace program.
    http://sourceforge.net/projects/fuse/files/fuse-2.X/

  • sshfs-fuse
    This is a filesystem client based on the SSH File Transfer Protocol.
    Since most SSH servers already support this protocol it is very easy to set up: i.e.
    on the server side there’s nothing to do.
    On the client side mounting the filesystem is as easy as logging into the server with ssh.
    http://sourceforge.net/projects/fuse/files/sshfs-fuse/2.5/

  • zookeeper
    Apache ZooKeeper is an effort to develop and maintain an open-source server which enables highly reliable distributed coordination.

metarates.tar.gz

program

Software debugging on windows

User mode and kernel mode debugger

Basic Debug methods

  • Run with IDE directory, and use breakpoint and single step to check the problem
  • Using OutputDebugString, and capture the message with debugview
  • Write debugging information to a log file

Advanced method

  • dump file
  • windbg

Memory leak detect tools

Reference