Skip to content

Google 是如何定义“开放”这个概念

2010年不期而至的第一场冬雪,打乱了原本出门看展览吃大餐的计划,宅在家里断断续续读完《Google将带来什么?》这本书,发现自己对 Google 核心理念的理解仍不是很到位。

这本书从内容上来看基本是个博客文章合集,在吹捧了一下 Google 的商业模式和众所周知的成功后,开始把成功经验往各行各业套用,预测在应用类似的开放、合作的理念后,会给行业带来哪些新变化。内容本身也还靠谱,有些思路也算有道理,但不成体系也不够严谨。

麻烦的是这种博客文章的形式,看完基本得不出太多结论反而引出一堆问题,刚好Google 官方博客前段放出一篇文章,负责产品管理的副总裁 Jonathan Rosenberg 的 The meaning of open 一文,解答了我的部分疑惑。

文中提到,Google 相信开放系统是双赢的,并将带来创新、价值、客户自由选择的权利,以及更有活力、可盈利和竞争性的商业生态系统。

At Google we believe that open systems win. They lead to more innovation, value, and freedom of choice for consumers, and a vibrant, profitable, and competitive ecosystem for businesses.

如果是其他某些公司有人这么说,我可能觉得里面虚伪做作的成分比较多,毕竟开放的代价是触及现有既得利益者,国内某些公司反反复复的“开放”就是最好写照。但对 Google 这样一个商业和社区的宠儿来说,我相信他们是真心坚守这个信念,因为他们实际上是这个开放商业生态系统的最大受益者。从 Google 打通了他的流量向广告变现能力的任督二脉开始,他实际上已经将开放作为推动产业和自身发展的原动力。

There are two components to our definition of open: open technology and open information. Open technology includes open source, meaning we release and actively support code that helps grow the Internet, and open standards, meaning we adhere to accepted standards and, if none exist, work to create standards that improve the entire Internet (and not just benefit Google). Open information means that when we have information about users we use it to provide something that is valuable to them, we are transparent about what information we have about them, and we give them ultimate control over their information. These are the things we should be doing. In many cases we aren’t there, but I hope that with this note we can start working to close the gap between reality and aspiration.

Google 把 open 定义为两个层面:开放技术(open technology)和开放信息(open information)。开放技术包括开源软件(Open Source)和开放标准(Open Standard),通过自身发布开源软件和鼓励员工和社区对开源软件的支持,推动Internet的发展;通过采用和创立开放标准,来改善整个Internet的环境。开放信息则被用来向用户提供更多价值,并在确保信息透明度的情况下,将控制权交给用户。

从开放系统中获得双赢的思想,实际上是和传统MBA教育相违背的,因为那些教科书都是针对封闭系统进行分析和设计的。Jonathan 在文中分析了开放系统的优劣,无非是开放则对市场需求响应速度快,封闭则利润空间大可控性强等等老生常谈。他提到了水果商在 iPod 和 iPhone 上的成功,也举了人类基因组工程和 GNU C 编译器的例子。

In an open system, a competitive advantage doesn’t derive from locking in customers, but rather from understanding the fast-moving system better than anyone else and using that knowledge to generate better, more innovative products. The successful company in an open system is both a fast innovator and a thought leader; the brand value of thought leadership attracts customers and then fast innovation keeps them.

最为关键的是他提到的如何将 open 理念变现的能力保障,也就是失去封闭系统里被锁定客户的竞争优势后,如何为对快速变化的系统持续提供更多更好的创新产品。快速创新者(fast innovator)、思想领导者(thought leader),没错,就是这两个已经被各种管理书籍说烂了的理念。通过思想领导者的品牌/口碑价值,吸引客户并通过快速创新“锁定”他们。这实际上是 Google 能够持续快速成长的关键因素,并不是什么简简单单的开源软件或者弄个 lab 头衔能够模仿的,而是通过快速而持续的创新能力,这个其他企业很难模拟的撒手锏,获得了开放系统中“锁定”客户的方法

所有的组织(organization)从成立开始,就在不断完善和演进之中。在这个自我进化的过程中,与热力学第二定律“封闭系统中熵永不减少”类似,组织规模越大时间越长,其结构和功能就会越趋于稳定,换句话说就是越来越僵硬。因而对 Google 的竞争者来说,在体制内想达到同等的创新能力,无论是成本还是效果上都很难保障,微软每年投入上百亿加几千人折腾的 Live!和现在的 Bing 就是最好写照。

在这个大前提下,Google 对开放的理解和策略制定,基本就水到渠成了。最根本的还是 Hal Varian 在其 Information Rules 一书中提到的:

Reward = (Total value added to the industry) * (Our share of industry value)

无论是 10% 的产业价值增加,还是 10% 的 Google 所占市场份额增加,对 Google 自身的推动作用都是同等的。Google 通过构建自身的外部动力,获得了推动产业发展同时带来自身发展的可能性。用比较直白的话说,internet用户使用网络越多,他接触和使用 Google 产品就会越多,相应 Google 获得的价值就会越大。这实际上是在通过创新能力锁定客户的同时,通过变现方法不同给竞争对手带来的第二道门槛。

以 Google 正在大举进入的手机领域为例。Google 通过开放的 Android 系统,以思想领导者的定位吸引厂商加入,然后通过各种免费创新应用吸引最终用户。这些“免费”和“开放”的应用,通过免费的 Gmail、GPS 和地图服务等传统厂商无法直接抗衡的方式,吸引并锁定目标客户,进而增加这个产业的价值和 Google 的份额。对传统厂商来说,他们缺乏持续创新的能力和动机,也缺乏在 GPS 和地图服务等领域进行对抗时,将用户数和流量变现的造血能力。「比免费更便宜」的商业模式一文中有更为详细的评述。

接下来就是如何将这些 open 的理念付诸实际,Jonathan 详细解释了开放技术和开放信息层面的理解。

开放标准和开源软件,是加快 Internet 发展的促进因素,因而 Google 有足够动力去推动他们,因为用的人越多他得到的越多。就好比 IBM 和 Intel 推动 Linux 和开源软件发展,归根结底是落实到卖自己的产品、服务和 CPU 上,你用的越多我卖的越好,资本家永远不是慈善家。这点前面讨论较多就不再罗嗦。

开放信息则较为敏感,因为这触及到了上述提及到的 Google 的另一个核心能力,将免费用户数和流量变现的强大能力。而支撑这一能力的根本,在于 Google 通过搜索引擎和各种免费服务,达到的对网络和用户的深刻理解。无论是之前的 AdSense 和 AdWords,还是发布不久的 Interest-Based Advertising,都需要能精确对网页内容和访问用户的行为记录,进行分析和匹配后选择合适广告投放来改善效果,最终变成维系 Google 千亿市值的涓涓细流。

另一方面,这些敏感的用户信息也是最容易让人对 Google 产生诟病的阿喀琉斯之踵,因而与开放技术不同这里有的只是对用户的解释。通过使用这些信息改善用户体验以增加价值,通过增加信息使用透明度降低用户反感,并通过将控制权交给用户来造成用户可以控制这些信息的假象。不过就如《Google将带来什么?》书中提到的,这种把控制权移交给用户的尝试,已经比其他行业正在做的好太多。而这些举措本身也从另一个角度改善了 Google 的 open 的定位和形象。

于是乎,Google 举着 open 这个大旗,通过引领变革维系品牌价值,以持续创新锁定用户,在促进产业发展带动自身,进而以开放信息洗白利用用户信息改善变现能力的本质,构建了一个贯彻整个行业的用户和流量变现大循环,将只能在某个环节困兽犹斗的传统企业踢出局,获得商业和社区层面的双丰收。好在以其 Don’t be Evil 的理念下,Google 相当长时间里也可以做到与其用户的双赢,只要上述的外部动力不变 Google 基本不会去得罪最终用户,这也是为什么他在各个领域严守道德底线,并不是因为企业有多么高尚,而是因为这维系着他大循环中一个最根本条件,也就是最终用户对他的信任。

你信任 Google 么?

Tagged , , ,

Google Public DNS 国内访问的性能

dnsperf 简单测试了一下 Google Public DNS 中 8.8.8.8 的响应时间
发送 22422 个请求,完成率 97.81%,平均延迟 0.54s,47%在0.2s内成功返回,平均每秒查询30个左右。
测试样本用的是 dnsperf 自带的一套国际域名查询记录,并发最多同时20个查询请求。回头有空再做个针对国内域名的对比测试。
$ ./dnsperf -p 53  -s 8.8.8.8 -d queryfile-example-3million -H 50 -T 10
Statistics:
  Parse input file:     once
  Ended due to:         interruption
  Queries sent:         22422 queries
  Queries completed:    21910 queries
  Queries lost:         492 queries
  Queries interrupted:  20 queries
  Avg request size:     40 bytes
  Avg response size:    80 bytes
  Percentage completed:  97.81%
  Percentage lost:        2.19%
  Started at:           Sun Dec  6 15:07:54 2009
  Finished at:          Sun Dec  6 15:19:58 2009
  Ran for:              724.357586 seconds
  Queries per second:   30.247492 qps
Latency: Min: 0.044573 s; Max: 5.036942 s; Avg: 0.546176 s; StdDev: 0.739284
Response latency distribution (total 21910 responses):
Latency   Success  Fail |
<    0.200s 10352   320 |##########################################################-
<    0.400s  1851   569 |##########—
<    0.600s  2012  1083 |###########——
<    0.800s  1058   759 |#####—-
<    1.000s   257   540 |#—
<    1.200s   142   320 |-
<    1.400s    68   163 |
<    1.600s    40    95 |
<    1.800s    33   503 |–
<    2.000s    61   526 |–
<    2.200s    29   171 |
<    2.400s    20   160 |
<    2.600s    11    65 |
<    2.800s    10    58 |
<    3.000s     1    53 |
<    3.200s     1   239 |-
<    3.400s     8    56 |
<    3.600s     2    69 |
<    3.800s     0    37 |
<    4.000s     5    28 |
<    4.200s     3    17 |
<    4.400s     3    11 |
<    4.600s     0    11 |
<    4.800s     2    63 |
<    5.000s     2    16 |
<    5.200s     0     7 |
<    5.400s     0     0 |
<    5.600s     0     0 |
<    5.800s     0     0 |
<    6.000s     0     0 |
<    6.200s     0     0 |
<    6.400s     0     0 |
<    6.600s     0     0 |
<    6.800s     0     0 |
<    7.000s     0     0 |
<    7.200s     0     0 |
<    7.400s     0     0 |
<    7.600s     0     0 |
<    7.800s     0     0 |
<    8.000s     0     0 |
<    8.200s     0     0 |
<    8.400s     0     0 |
<    8.600s     0     0 |
<    8.800s     0     0 |
<    9.000s     0     0 |
<    9.200s     0     0 |
<    9.400s     0     0 |
<    9.600s     0     0 |
<    9.800s     0     0 |
<   10.000s     0     0 |
>=  10.000s     0     0 |
Legend:
##### = success responses (RCODE was NOERROR or NXDOMAIN)
—– = failure responses (any other RCODE)

Change the default zoom mode in Google Chromium

As a WebKit based browser, Google Chrome use the text zoom mode as default zoom mode. It means, if you choose the Zoom -> Larger in the page context menu, or press the Ctrl + ‘+’ accelerator, all the text font size in the page will be increased by 20%. On the other hand, if you do the same thing in FireFox, the whole page will be zoomed instead the text font size.

In my opinion, I feel the page zoom mode more comfortable, although I don’t think any of those mode is better than the other. At least in my 24′ LCD with 1080p, I need the page zoom width enough to fill the right space.

So, I just modified Chrome source code, add two alternative accelerators for page zoom mode; I also add a command line switch to set the default zoom mode when starting.

The changes are straight, because WebKit had provided the build-in support for page zoom mode, and Chrome also had an internal switch for zoom mode.

Following steps are the changes base on latest SVN revision.

  1. Add two new command definition in the \app\chrome_dll_resource.h file, which define the alternative zoom command itself.

    #define IDC_ZOOM_VIEW_PLUS 38004
    #define IDC_ZOOM_VIEW_MINUS 38005

    And add new accelerator like Ctrl + Alt + ‘+’/’-‘ in the IDR_MAINFRAME table (app\chrom_dll.rc)

    VK_ADD,              IDC_ZOOM_VIEW_PLUS, VIRTKEY, CONTROL, ALT
    VK_SUBTRACT,         IDC_ZOOM_VIEW_MINUS, VIRTKEY, CONTROL, ALT

  2. Handle the commands in the Browser::ExecuteCommand function (\browser\browser.cc), and add a parameter for ZoomIn/ZoomOut functions which is the zoom mode

    // Zoom
    case IDC_ZOOM_PLUS: ZoomIn(true); break;
    case IDC_ZOOM_NORMAL: ZoomReset(); break;
    case IDC_ZOOM_MINUS: ZoomOut(true); break;
    case IDC_ZOOM_VIEW_PLUS: ZoomIn(false); break;
    case IDC_ZOOM_VIEW_MINUS: ZoomOut(false); break;

  3. Add two new zoom mode in PageZoom (\common\page_zoom.h), which will be used for page zoom; and refactor ZoomIn/ZoomOut functions (\browser\browser.cc) for new modes.

    enum Function {
    VIEW_SMALLER = -2,
    SMALLER = -1,
    STANDARD = 0,
    LARGER = 1,
    VIEW_LARGER = 2, };

    void Browser::ZoomIn(bool isTextOnly) {
    UserMetrics::RecordAction(L”ZoomPlus”, profile_);
    GetSelectedTabContents()->AsWebContents()->render_view_host()->Zoom( isTextOnly ? PageZoom::LARGER : PageZoom::VIEW_LARGER);}

    void Browser::ZoomOut(bool isTextOnly) {
    UserMetrics::RecordAction(L”ZoomMinus”, profile_);
    GetSelectedTabContents()->AsWebContents()->render_view_host()->Zoom( isTextOnly ? PageZoom::SMALLER : PageZoom::VIEW_SMALLER);}

    Then, the RenderViewHost::Zoom function will wrap the zoom mode as a ViewMsg_Zoom message, and send it to the render process.

  4. Modify the render handler, which redirect ViewMsg_Zoom message to OnZoom function (\renderer\render_view.cc)

    IPC_MESSAGE_HANDLER(ViewMsg_Zoom, OnZoom)

    The OnZoom function has a hard code kZoomIsTextOnly constant. So, we just change its value base on the command line parameter; and set two kind of zoom mode base on its default setting.

    void RenderView::OnZoom(int function) {
    static const bool zoomIsTextOnly = CommandLine().HasSwitch(switches::kZoomIsTextOnly);

    switch (function) {
    case PageZoom::VIEW_SMALLER:
    webview()->ZoomOut(!zoomIsTextOnly); break;
    case PageZoom::SMALLER: webview()->ZoomOut(zoomIsTextOnly); break;
    case PageZoom::STANDARD: webview()->ResetZoom(); break;
    case PageZoom::LARGER: webview()->ZoomIn(zoomIsTextOnly); break;
    case PageZoom::VIEW_LARGER: webview()->ZoomIn(!zoomIsTextOnly); break;
    default: NOTREACHED(); }}

  5. Besides the command workflow, there are some bookkeeping works.

    We need update the command status base on the current stage in the Browser::UpdateCommandsForTabState function (browser\browser.cc)

    controller_.UpdateCommandEnabled(IDC_ZOOM_VIEW_PLUS, is_web_contents);
    controller_.UpdateCommandEnabled(IDC_ZOOM_VIEW_MINUS, is_web_contents);

  6. To change the default zoom mode, I added a new switch for command line in \common\chrome_switches.h/.cc

    // Turns on text only mode zoom supportconst wchar_t
    kZoomIsTextOnly[] = L”zoom-is-text-only”;

    and add it into the switch_names array in RenderProcessHost::Init function (browser\render_process_host.cc). So, the browser process will propagate the switch to the child render process.

    static const wchar_t* const switch_names[] = {
    //…
    switches::kZoomIsTextOnly, };

Besides, I found a similar feature request has been submited to the chrome issues tracking. So, I will add some comment as patch for this issue. Wish Google could merge it into next offical release :)

Use 3rd party font render engine in Google Chromium

    As you know, there are some 3rd party font engines, such as GDI++, which could hook Windows font related APIs, and render font more smoother. 
    In my opinion, that engine seems better than 
ClearType or other build-in Windows font engine. 

    For Chromium, it use multi-processes architecture, a standalone renderer process will render the page in sandbox. It means that we should know which process need be hooked for font smoother. 




    So I made some slight changes in Chromium to support GDI++ 

    1. a new command line parameter “–use-gdipp=“, which give a setting file for GDI++ 

    I add a new key name kUseGdiPP (chrome_switches.h|.cc), which could be used to parse command line parameters.

const wchar_t kUseGdiPP[] = L”use-gdipp”;

    And also add it in switch_names array (render_process_host.cc), that control the main browser process propagate which switches to renderer child process.

    2. the initialize and finalize code in RendererMain function, which load and free the font engine 

    After the renderer process started, it call RendererMain function (renderer_main.cc) to initilize process. So I add the font engine related code just after HandleRendererErrorTestParameters calling. Because that function could help us debugging the children process.

  bool use_gdipp = parsed_command_line.HasSwitch(switches::kUseGdiPP);

  if (use_gdipp)
  {
    std::wstring setting_file_name = 
      parsed_command_line.GetSwitchValue(switches::kUseGdiPP);

    if (!LoadFontEngine(setting_file_name.c_str()))
    {
      LOG(WARNING) << "Fail to load GDI++ font engine.";

      use_gdipp = false;
    }
    else
    {
      DLOG(INFO) << "Loaded GDI++ font engine.";
    }
  }

    We can use “–renderer-startup-dialog” parameter to popup a message box before any other initialize code. And we could attach debugger to renderer process for troubeshooting. Besides, this parameter must be used with “–no-sandbox” parameter. Please check Debugging Chromium for more detail.

     On the other hand, we also free the font engine before uninitialize COM with CoUninitialize function.

  if (use_gdipp)
  {
    FreeFontEngine();
  }
    3. a customized GDI++ library which will be linked as static library in chrome.dll library 

    To compile the GDI++ library, you should download the latest FreeType version GDI++, with FreeType and Detours.
    And create a static library project in chrome solution, which should add all the source files in GDI++\src folder except run.cpp.

    Than, you should do some dirty works to make it compilable, remove some function definition and includes.
    Two funtions LoadFontEngine and FreeFontEngine should be added in hook.cpp as wrapper, and a new header file should be created to declare those functions.

bool LoadFontEngine(LPCTSTR lpszFile)
{
  hook_initinternal();

  CCriticalSectionLock::Init();

  if (!g_TLInfo.ProcessInit()) 
  {
    return false;
  }
  
  CGdippSettings* pSettings = CGdippSettings::CreateInstance();
    
  if (!pSettings || !pSettings->LoadAppSettings(lpszFile)) 
  {
    CGdippSettings::DestroyInstance();
    
    return false;
  }
  
  if (!FontLInit()) 
  {
    return false;
  }
  
  g_pFTEngine = new FreeTypeFontEngine;

  if (!g_pFTEngine) 
  {
    return false;
  }

  if (!InterlockedExchange(&g_bHookEnabled, TRUE))
  {
    hook_init();
  }  

  return true;
}

void FreeFontEngine()
{
  if (InterlockedExchange(&g_bHookEnabled, FALSE)) 
  {
    hook_term();
  }

  if (g_pFTEngine) delete g_pFTEngine;
  
  FontLFree();

  CGdippSettings::DestroyInstance();

  g_TLInfo.ProcessTerm();

  CCriticalSectionLock::Term();
}

After that, you could get a local build with GDI++, it depend on freetype6.dll, zlib1.dll and detoured.dll as cost. And you could customize your setting file for GDI++, and run your local chrome with it. For more detail, you could read its offical document (Japanese), guess it by name or google it. :)


Virtualization Detection

As you know, virtualization is changing traditional IT infrastructure.
VMware ESX server, the market leading hypervisor, has been accepted by customers of all sizes, including all of the Fortune 100; Microsoft Windows 2008 also integrate Hyper-V as a core server role; several open source projects also focus on it, such as Xen (Intel), KVM (Linux) and Virtual Box (Sun) etc.

There are a lot of research papers and online discussion about Virtualization Detection

The 1st option is using vender specific public interface or backdoor, which could provide rich information about host OS and virtualization.

Microsoft public a Hypervisor Functional Specification for its Hyper-V implementation, which includes definition about not only feature and interface discovery but also Hyper-Call based management interfaces.
We could directly use one simple CPUID instruction to detect the existent of Hyper-V.

bool CHyperV::Exists(void)
{
int cpuid[4];

__cpuid(cpuid, 1);

return cpuid[2] & (1 <<>Further more, vender ID, neutral interface name, version and supported features could be fetched through similar method.

VMware also has similar private interface through a magic I/O port, which includes not only implementation information but also runtime information, such as processor speed, mouse cursor position and clipboard etc.
We could use its magic I/O port to detect the existent of VMware.

bool CVMware::Exists(void)
{
bool exists = true;

__try
{

__asm
{
push edx
push ecx
push ebx

mov eax, ‘VMXh’
mov ebx, 0 // any value but not the MAGIC VALUE
mov ecx, 10 // get VMWare version

mov edx, ‘VX’ // port number

in eax, dx // read port

// on return EAX returns the VERSION
cmp ebx, ‘VMXh’ // is it a reply from VMWare?

setz [exists] // set return value

pop ebx
pop ecx
pop edx
}
}

__except(EXCEPTION_EXECUTE_HANDLER)
{
exists = false;
}
return exists;
}

For more completely commands list, you could check the following URL

VMware Backdoor I/O Port
http://chitchat.at.infoseek.co.jp/vmware/backdoor.html

VMware also provide an open source project, Open Virtual Machine Tools, which includes more detail usage of its backdoor.

Besides, Microsoft Virtual PC/Server, bochs, qemu has similar interface, you could check the following URLs for more detail.

How to detect Virtual PC or VMware from your program
http://www.codegurus.be/codegurus/Programming/virtualpc&vmware_en.htm

Attacks on Virtual Machine Emulators
http://www.symantec.com/avcenter/reference/Virtual_Machine_Threats.pdf

The 2nd option is checking VM implementation mechanism, which could detect some unknown VM vender in theory.

The mostly famous method is checking IDT register from Joanna’s Red Pill.

Red Pill… or how to detect VMM using (almost) one CPU instruction
http://invisiblethings.org/papers/redpill.html
The heart of this code is actually the SIDT instruction (encoded
as 0F010D[addr]), which stores the contents of the interrupt descriptor table register (IDTR) in the destination operand, which is actually a memory location. What is special and interesting about SIDT instruction is that, it can be executed in non privileged mode (ring3) but it returns the contents of the sensitive register, used internally by operating system.

Because there is only one IDTR register, but there are at least two OS running concurrently (i.e. the host and the guest OS), VMM needs to relocate the guest’s IDTR in a safe place, so that it will not conflict with a host’s one. Unfortunately, VMM cannot know if (and when) the process running in guest OS executes SIDT instruction, since it is not privileged (and it doesn’t generate exception). Thus the process gets the relocated address of IDT table. It was observed that on VMWare, the relocated address of IDT is at address 0xffXXXXXX, whereas on Virtual PC it is 0xe8XXXXXX. This was tested on VMWare Workstation 4 and Virtual PC 2004, both running on Windows XP host OS.

This method only use one SIDT instruction to detect the existent of VM, because a lot of x86 software VM implementation choose to change IDT for guest OS.

bool CUnknown::CheckIDT(void)
{
unsigned char idt[6];

_asm
{
sidt idt

}
return idt[5] > 0xd0;
}

Unfortunately, this method could be spoofed by VM implementation

SubVirt: Implementing malware with virtual machines
http://www.eecs.umich.edu/~pmchen/papers/king06.pdf
The redpill [39] virtual-machine detection technique detects the
presence of a VMM by using the sidt instruction. The sidt instruction reads the address of the processor’s interrupt descriptor table. This address is different for an operating system running directly above hardware than for an operating system running above a VMM. VMMs emulate the sidt instruction when it is called from kernel-mode, but for performance reasons this instruction is not virtualized when called from user-mode. Thus, user-mode code can execute the
sidt instruction to detect the presence of a VMM. To defeat this detection technique, we use virtual-machine introspection to emulate the sidt instruction when it is called by the redpill application.

And it also doesn’t work on some hardware accelerated VM implementation and multi-core CPU system.

Introducing Blue Pill
http://theinvisiblethings.blogspot.com/2006/06/introducing-blue-pill.html
SubVirt was implemented on x86 hardware, which doesn’t allow to achieve 100% virtualization, because there are number of sensitive instructions, which are not privileged, like the famous SIDT/SGDT/SLDT. This allows for trivial detection of the virtual mode – see e.g. my little Red Pill program. This however, doesn’t apply to Blue Pill, as it relies on AMD SVM
technology.

Detecting the Presence of Virtual Machines
Using the Local Data Table
http://www.offensivecomputing.net/files/active/0/vm.pdf
The SIDT mechanism as implemented by Tobias Klein [1] and
separately by Joanna Rutkowska [2] is a method for detecting the presence of a virtual machine environment. While the test is by no means thorough, it is an effective test for the presence of an emulated CPU environment on a single-processor machine. There are various problems with the implementation, however. If a multi-core CPU is used, the interrupt descriptor table can change significantly when the process is run on different cores. Furthermore if two or more physical processors are present the same implementation issues apply.

To improve the issues of SIDT mechanism, a lot of attempts focus on similar register, such as GDT, LDT or TS register.

Detecting the Presence of Virtual Machines Using the Local Data Table
http://www.offensivecomputing.net/files/active/0/vm.pdf

Methods for Virtual Machine Detection
http://www.s21sec.com/descargas/vmware-eng.pdf

bool CUnknown::CheckGDT(void)
{
unsigned char gdt[6];
_asm
{
sgdt gdt
}
return gdt[5] > 0xd0;
}

bool CUnknown::CheckLDT(void)
{
unsigned char ldt[2];
_asm
{
sldt ldt
}
return ldt[0] != 0 && ldt[1] != 0;
}

bool CUnknown::CheckTR(void)
{
unsigned char tr[2];

_asm
{
str tr

}
return tr[0] == 0 && tr[1] == 0x40;
}

The major problem of these kinds of detection methods is the researcher hard to test compatibility on every VM vender.

After I made a simple test, it seems we hard to trust any of them, but we could made a hyper detection method base on more test results.

The 3rd option is using VM specific defect, which mostly tight coupling with VM version.

For example, eEye researcher published a detection method base on wrong far execution transfers behaviors.

Another VMware Detection
http://eeyeresearch.typepad.com/blog/2006/09/another_vmware_.html

So, the real world is not perfect, we hard to use one hammer to solve all the problems. We should carefully define our purpose and scope of virtualization detection, and choose a hyper detection solution for major focus market.

New challenge for malware detection: Virtualization Based rootkit

Several days ago, Invisible Things Lab release a new open source project which named Blue Pill, the first battle ready hardware virtualization based rootkits. Even the code is not sophisticated in this version; I also believe its impact is profound significance. It is a starting gun for new trend of rootkits and malware, which will promote the battle field from OS in-house to VM level.


Just two years ago, virtualization can only be implemented by software emulation, base on interpreter, binary translation etc. We got some software solution, includes VMware Workstation, Microsoft Virtual PC and Virtual Box etc. But they are hard to ensure enough performance and compatibility.

But as the dual core and x64 become common, hardware virtualization solution become the mainstream, includes Xen, VMware ESX Server, Microsoft Longhorn etc. These solutions base on CPU level support includes Intel VT and AMD Pacifica (AMD-V), which introduce a new isolated level beside x86’s ring 0-3.

A mini OS kernel will run in hypervisor mode (VMM), which manage multi guest OS in normal mode (VM). VMM can monitor the status of VM, and take over some operation in VM, such as IO, privileged instruction etc.

This is the common workflow as the designer expected.

But on the other hand, the world is not perfect. Some malicious guys also can use those features to bypass traditional security solutions.

First, some white hat guys from University of Michigan and MSR release a paper SubVirt: implementing malware with virtual machines in 2006. They discuss the possibility for a new type of malware, named virtual-machine based rootkit (VMBR), which installs a virtual-machine monitor underneath an existing operating system and hoists the original operating system into a virtual machine.

Second, Joanna Rutkowska presented at the Black Hat Briefings 2006, for hers hardware virtualization based rootkits, named Blue Pill. This implementation base on CPU support, don’t need any binary translation, and very hard to detect from VM in-house.

Besides, Dino Dai Zovi from matasano also presented at Black Hat USA 2006 about hardware virtualization rootkits, with their implementation base on Xen 3.0.

With my experience, a new concept from idea to malware need one or two year. Now, one year has gone, source is available, and the hardware support will be more and more popular. Everything for this new type of malware is almost ready, only two actors are still missing, a hardware virtualization based rootkit from real world, and detection and clean security solution.

Wireshark Dissector Plugin for Look’n’Stop

From a developer viewpoint, Look’n’Stop is a great personal firewall. Even their design may not very clearly for the normal user, but if you have enough background knowledge, it can be a powerful analyzer for the security threats.

After a packet be allowed or blocked by rule, Look’n’Stop record it to log and provide a dialog for detail information. But these information not enough for me, so I decide to write a plugin to got more :)
Fortunately, they provide some plugin API for log display and rule editor. Through those interfaces, I can pop up my dissector dialog to display the protocol tree of packet.
To avoid reinvent the wheel, I choose Wireshark as background dissector. Because Wireshark, or more well know name – ethereal, is the best open source network protocol analyzer, and is the standard in many industries.
Even Wireshark has encapsulate all dissector in a library, its interface not clearly and stable, so I decide use its terminal-based edition – tshark, as the major dissector, because it can read packet from stdin, and dump the protocol tree to stdou as XML format.
So the major data and control flow includes:
  1. Look’n’Stop pass the packet data to our plugin through its API
  2. plugin fork a tshark process for dissect
  3. plugin dump the packet as libpcap format to tshark’s stdin
  4. tshark dissect the packet to protocol tree and output the XML to stdout
  5. plugin fetch the XML output and parse it with expat
  6. plugin popup a tree-based dialog and render the protocol tree
  7. popup dialog provide more feature, for example, save the packet as libpcap format

Combine those steps, we got a new dissector plugin :)

You can select a field in protocol tree, and the corresponding data bytes will be highlight in the bottom editor. If you want to save the packet for more analyzer, just right-click the windows title, and choose “Save As” in system menu. it support save packet as libpcap, xml and text format.
But before this, you should download and install wireshark first, and configure the installation path in plugin, such as

If you input a valid path, plugin will fetch the version and copyright from tshark, and save it to registry to reuse in future.
For the china user, I integrate some location information of IP address. You should download the latest IP database from cz88.net, and configure the QQWry setting page, such as
After you choose “Use QQWry …”, a location information will be appended after some IP field in protocol tree.
If you have interest about this, please download the prebuild binary or compire it by youself.
That’s all, if you have any advices or want to improve it by yourself, please contact me directly :)
Get Adobe Flash player