sFlow.org Peter Phaal http://www.sFlow.org/ InMon Corp. info@sflow.org Robert Jordan ImageMovers Digital July 2010 sFlow Host Structures Copyright Notice Copyright (C) sFlow.org (2010). All Rights Reserved. Abstract This memo describes sFlow version 5 structures for exporting host related data. Table of Contents 1. Overview ...................................................... 1 2. Reference Model ............................................... 2 3. sFlow Datagram Extensions ..................................... 3 4. References .................................................... 12 5. Author's Addresses ............................................ 13 1. Overview This document describes additional structures that allow an sFlow agent to export information about host resources. Current trends toward virtualization and convergence tightly link networking and system performance and blur the line between the network and the servers. For example, virtualization places switching and routing functions on servers. Monitoring the network in this environment requires that the servers are also monitored. Similarly anyone concerned with application and server performance must now also be concerned about network performance since the application features (such as virtual machine migration) can significantly affect and be affected by network performance. Currently, performance monitoring of servers and applications is FINAL sFlow.org [Page 1] FINAL sFlow Host Structures July 2010 highly fragmented. Each server vendor, operating system vendor and application developer creates agents and software for performance monitoring, none of which interoperate. Monitoring performance of a server might require the same (or similar but incompatible metrics) be monitored more than once since each management application requires its own agent. In addition, there may be other agents monitoring different hardware and software elements within the server. Host sFlow extends sFlow's scalable measurement protocol to include server performance monitoring. An interoperable standard for exporting host performance metrics breaks the linkage between agents and performance analysis applications, providing a standard set of metrics that can be collected once and shared among different performance analysis tools. Extending sFlow to monitor servers unifies network and system performance monitoring, delivering the real-time, integrated view of performance needed to manage converged services. sFlow version 5 is an extensible protocol that allows the addition of new data structures without impacting existing collectors. This document does not change the sFlow version 5 protocol [1], it simply defines additional, optional, data structures that a host device can use to report on host resources. 2. Reference Model The figure below shows the basic elements of a host. In this abstract model a host consists of one or more physical machines (PM), each of which may contain zero or more virtual machines (VM) and/or application services (AS). +--------------------------------------+ | +------------------------------+ | | | PM 1 +------+ +------+ | | | | | AS 1 | ... | AS J | | | | | +------+ +------+ | | | | +------+ +------+ | | | | | VM 1 | ... | VM K | | | | | +------+ +------+ | | | +------------------------------+ | | . | | . | | . | | +------------------------------+ | | | PM I | | | +------------------------------+ | +--------------------------------------+ FINAL sFlow.org [Page 2] FINAL sFlow Host Structures July 2010 The term application service is used to refer to applications providing networked services. Examples would include: a web server implementing the HTTP protocol [2], a storage array supporting the NFS protocol [3], a distributed cache implementing the memcached protocol [4], or a distributed computing facility such as Hadoop [5]. A stand-alone server would just have a single physical machine. A blade server may have a large number of physical machines (one per blade). Physical machines are indicated as PM 1 .. PM I. Application services are indicated as AS 1 .. AS J. Virtual machines are indicated as VM 1 .. VM K. Each physical machine, application service and virtual machine is monitored by at least one sFlow data source. In this model there is a strict containment hierarchy, each physical machine contains application services and/or virtual machines. Each virtual machine may in turn contain application services. The objective of Host sFlow is to export statistics relating to the PM, AS and VM entities using a unified data model that permits correlation between the host statistics and network statistics provided by sFlow agents residing in physical and virtual switches. The sFlow version 5 standard requires the implementation of Packet Flow Sampling and Counter Sampling within the packet forwarding function of a device. In order to be compliant, a host must implement sFlow within any physical or virtual switching functions implemented within the host. However, in the case where no switching function is performed by the host, either because it does not contain virtual machines or because the inter-VM switching function has been offloaded to the adjacent switch (see Edge Virtual Bridging [6]) then the host is not required to perform the sFlow Packet Flow Sampling function. The combination of sFlow from the adjacent switch and sFlow performance metrics from the host provides a complete picture of network and server performance. 3. sFlow Datagram Extensions The SNMP Entity-MIB [7] can be used to describe the physical and logical containment hierarchy of host resources. Physical machines (PMs) can be modeled as physical entities, an already supported sFlow data source type. Application services (ASs) and virtual machines (VMs) can be modeled as logical entities. Extending sFlow support for logical entities provides a means of exporting data related to FINAL sFlow.org [Page 3] FINAL sFlow Host Structures July 2010 application services and virtual machines. The sFlow MIB [1] identifies data sources by SNMP OID, so the only change needed is a comment indicating that a logical entity is a valid data source type: - entLogicalEntry. An SFlowDataSource of this form refers to a logical entity within the agent and is called a 'logical-entity-based' dataSource. In addition, a mapping for logical entity data sources in the sFlow datagram needs to be specified: 3 = entLogicalEntry These changes are backward compatible with existing sFlow agents. An sFlow collector must be able to ignore and skip over the MIB entries and data structures related to the logical data source type. However, since there is very little functional overlap between Host sFlow and existing switch based sFlow, sending Host sFlow to a collector that does not support the standard should be avoided. As Host sFlow becomes more common, it is likely that many sFlow analyzers will be extended to support the new structures in order to provide integrated network and system monitoring functionality. SNMP[8] is a standard management protocol for network equipment and sFlow monitoring of switches is often augmented by additional information obtained by SNMP (e.g. ifName, ifStack etc.). However, SNMP is much less frequently used in host monitoring. It is important that the Host sFlow structures define an internally consistent model of the host without depending on SNMP for critical information. However, when an SNMP agent is present on the host, the sFlow logical and physical entity data sources and their containment hierarchy must be consistent with data exported via SNMP. The host_parent structure is used to describe the containment hierarchy between application services, virtual machines and physical machines. The host_adapter structure provides the link between host performance statistics and sFlow implemented in network equipment. Identifying the MAC addresses associated with a physical or virtual network adapter allows traffic generated by that adapter to be identified on the network. Physical machine data sources must export counter_sample structures FINAL sFlow.org [Page 4] FINAL sFlow Host Structures July 2010 containing host performance statistics (host_cpu, host_memory, host_disk_io, host_net_io). Each counter_sample must also include host_adapter and host_descr structures. If the physical machine is hosting virtual machines then network connectivity to the virtual machines is typically provided by a virtual switching function. The virtual switch function connects one or more of the physical server's physical network adapters with virtual network adapters associated with each virtual machine. The virtual switch must export sFlow counter_sample and flow_sample structures, providing exactly the data that would be available from a physical switch. The data sources defined for a switch are typically ifIndex data sources and in the case of a virtual switch would represent each of the physical and virtual network adapters associated with the switch. Again, if an SNMP agent is running on the server, the ifIndex numbers exported via sFlow must correspond to the ifIndex numbers exported via SNMP. A data source associated with each of the virtual machines must export counter_sample structures containing performance metrics for that virtual machine (virt_cpu, virt_memory, virt_disk_io, virt_net_io). Each counter_sample must also include host_adapter, host_parent and host_descr structures. Installing an sFlow agent on each virtual machine is not required. Implementing sFlow as part of the hypervisor functionality provides access to the virtual switch and performance counters for all the virtual machines. However, if an sFlow agent is running on a virtual machine, it must export its own "physical server" statistics. Application services may export both counter_sample structures and flow_sample structures. When sampling at the application level, transactions do not have a simple correspondence with network packets. A single packet may contain more than one application level transaction, or an application level transaction may span many packets. Application level sampling operates on the stream of application transactions, sampling a random selection of completed transactions and reporting on their attributes. For example, a data source monitoring a web server randomly samples 1-in-N HTTP requests, capturing information about each completed request and exports the information as a flow_sample structure. In addition, the data source maintains counters of request totals, types and errors and export these counters periodically using a counter_sample structure. The socket extended_flow structures provide the link between application level flow_sample structures and traffic measurements made using sFlow in network devices. Each application flow_sample FINAL sFlow.org [Page 5] FINAL sFlow Host Structures July 2010 must include a socket structure describing the socket on which the sampled transaction was received. This document creates a framework for exporting application service metrics. However, it does not specify application service counter_sample or flow_sample structures. Each application service will have its own metrics and these will be defined in separate documents. The goal of sFlow monitoring is to provide a consistent, network wide view of performance. In order to achieve this goal with system performance monitoring a common set of metrics needs to be defined that can be collected in multi-vendor, multi-OS environments. The Ganglia Monitoring System [9] has defined a basic set of system performance metrics and developed a portable library, libmetrics, that allows these metrics to be obtained on a wide variety of platforms. Making use of Ganglia's metric definitions to specify the sFlow performance counter structures simplifies the implementation of a Host sFlow agent, building on a mature and widely accepted set of metrics and tools. While the sFlow structure definitions are designed to permit an sFlow performance monitoring system to yield results that are compatible with Ganglia measurements, there are significant architectural differences between the two technologies that require some explanation. The Ganglia agent periodically polls systems counters and computes rates. For example packet counters are polled and packet rates are exported (e.g. delta packets / delta time). In the sFlow architecture the sFlow agent exports raw counter values and the sFlow analyzer computes deltas and rates. Just as Ganglia's libmetrics library defines a set of portable metrics for physical server performance monitoring, the libvert library [10] defines a standard set of metrics for virtual server performance monitoring. Defining sFlow counter structures based on the libvert library simplifies the implementation of a Host sFlow agent, making use of a standard library that has been implemented on a wide variety of host operating systems and hypervisors. Exporting libvert metrics ensures consistency between sFlow performance monitoring and other tools making use of libvert. The Host sFlow agent [11] is an open source implementation of the Host sFlow specification that demonstrates the relationship between Host sFlow structures. A single physical entity data source periodically exports an sFlow counters_sample containing a FINAL sFlow.org [Page 6] FINAL sFlow Host Structures July 2010 host_descr, host_adapters, host_cpu, host_memory, host_disk_io and host_net_io structure. If virtual machines are present, the physical entity data source includes a virt_node structure in its set of counters. In addition, a logical entity data source corresponding to each virtual machine periodically exports an sFlow counters_sample containing a host_descr, host_adapters, host_parent, virt_cpu, virt_memory, virt_disk_io and virt_net_io structure. The following sFlow structures are defined to export performance and dependency information related to physical machines, virtual machines and applications: /* Data structures for exporting Host statistics relating to logical and physical entities */ /* The machine_type enumeration may be expanded over time. Applications receiving sFlow must be prepared to receive host_descr structures with unknown machine_type values. The authoritative list of machine types will be maintained at www.sflow.org */ enum machine_type { unknown = 0, other = 1, x86 = 2, x86_64 = 3, ia64 = 4, sparc = 5, alpha = 6, powerpc = 7, m68k = 8, mips = 9, arm = 10, hppa = 11, s390 = 12 } /* The os_name enumeration may be expanded over time. Applications receiving sFlow must be prepared to receive host_descr structures with unknown machine_type values. The authoritative list of machine types will be maintained at www.sflow.org */ enum os_name { unknown = 0, other = 1, FINAL sFlow.org [Page 7] FINAL sFlow Host Structures July 2010 linux = 2, windows = 3, darwin = 4, hpux = 5, aix = 6, dragonfly = 7, freebsd = 8, netbsd = 9, openbsd = 10, osf = 11, solaris = 12 } /* Physical or virtual host description */ /* opaque = counter_data; enterprise = 0; format = 2000 */ struct host_descr { string hostname<64>; /* hostname, empty if unknown */ opaque uuid<16>; /* 16 bytes binary UUID, empty if unknown */ machine_type machine_type; /* the processor family */ os_name os_name; /* Operating system */ string os_release<32>; /* e.g. 2.6.9-42.ELsmp,xp-sp3, empty if unknown */ } /* Physical or virtual network adapter NIC/vNIC */ struct host_adapter { unsigned int ifIndex; /* ifIndex associated with adapter Must match ifIndex of vSwitch port if vSwitch is exporting sFlow 0 = unknown */ mac mac_address<>; /* Adapter MAC address(es) */ } /* Set of adapters associated with entity. A physical server will identify the physical network adapters associated with it and a virtual server will identify its virtual adapters. */ /* opaque = counter_data; enterprise = 0; format = 2001 */ struct host_adapters { adapter adapters<>; /* adapter(s) associated with entity */ } /* Define containment hierarchy between logical and physical entities. Only a single, strict containment tree is permitted, each entity must be contained within a single parent, but a parent may contain more than one child. The host_parent record is used by the child to identify its parent. Physical entities form the roots of the tree and do not send host_parent structures. */ FINAL sFlow.org [Page 8] FINAL sFlow Host Structures July 2010 /* opaque = counter_data; enterprise = 0; format = 2002 */ struct host_parent { unsigned int container_type; /* sFlowDataSource type */ unsigned int container_index; /* sFlowDataSource index */ } /* Extended socket information, Must be filled in for all application transactions associated with a network socket Omit if transaction associated with non-network IPC */ /* IPv4 Socket */ /* opaque = flow_data; enterprise = 0; format = 2100 */ struct extended_socket_ipv4 { unsigned int protocol; /* IP Protocol type (for example, TCP = 6, UDP = 17) */ ip_v4 local_ip; /* local IP address */ ip_v4 remote_ip; /* remote IP address */ unsigned int local_port; /* TCP/UDP local port number or equivalent */ unsigned int remote_port; /* TCP/UDP remote port number of equivalent */ } /* IPv6 Socket */ /* opaque = flow_data; enterprise = 0; format = 2101 */ struct extended_socket_ipv6 { unsigned int protocol; /* IP Protocol type (for example, TCP = 6, UDP = 17) */ ip_v6 local_ip; /* local IP address */ ip_v6 remote_ip; /* remote IP address */ unsigned int local_port; /* TCP/UDP local port number or equivalent */ unsigned int remote_port; /* TCP/UDP remote port number of equivalent */ } /* Physical server performance metrics */ /* Physical Server CPU */ /* opaque = counter_data; enterprise = 0; format = 2003 */ struct host_cpu { float load_one; /* 1 minute load avg., -1.0 = unknown */ float load_five; /* 5 minute load avg., -1.0 = unknown */ float load_fifteen; /* 15 minute load avg., -1.0 = unknown */ unsigned int proc_run; /* total number of running processes */ unsigned int proc_total; /* total number of processes */ unsigned int cpu_num; /* number of CPUs */ unsigned int cpu_speed; /* speed in MHz of CPU */ FINAL sFlow.org [Page 9] FINAL sFlow Host Structures July 2010 unsigned int uptime; /* seconds since last reboot */ unsigned int cpu_user; /* user time (ms) */ unsigned int cpu_nice; /* nice time (ms) */ unsigned int cpu_system; /* system time (ms) */ unsigned int cpu_idle; /* idle time (ms) */ unsigned int cpu_wio; /* time waiting for I/O to complete (ms) */ unsigned int cpu_intr; /* time servicing interrupts (ms) */ unsigned int cpu_sintr; /* time servicing soft interrupts (ms) */ unsigned int interrupts; /* interrupt count */ unsigned int contexts; /* context switch count */ } /* Physical Server Memory */ /* opaque = counter_data; enterprise = 0; format = 2004 */ struct host_memory { unsigned hyper mem_total; /* total bytes */ unsigned hyper mem_free; /* free bytes */ unsigned hyper mem_shared; /* shared bytes */ unsigned hyper mem_buffers; /* buffers bytes */ unsigned hyper mem_cached; /* cached bytes */ unsigned hyper swap_total; /* swap total bytes */ unsigned hyper swap_free; /* swap free bytes */ unsigned int page_in; /* page in count */ unsigned int page_out; /* page out count */ unsigned int swap_in; /* swap in count */ unsigned int swap_out; /* swap out count */ } /* Physical Server Disk I/O */ /* opaque = counter_data; enterprise = 0; format = 2005 */ struct host_disk_io { unsigned hyper disk_total; /* total disk size in bytes */ unsigned hyper disk_free; /* total disk free in bytes */ percentage part_max_used; /* utilization of most utilized partition */ unsigned int reads; /* reads issued */ unsigned hyper bytes_read; /* bytes read */ unsigned int read_time; /* read time (ms) */ unsigned int writes; /* writes completed */ unsigned hyper bytes_written; /* bytes written */ unsigned int write_time; /* write time (ms) */ } /* Physical Server Network I/O */ /* opaque = counter_data; enterprise = 0; format = 2006 */ struct host_net_io { FINAL sFlow.org [Page 10] FINAL sFlow Host Structures July 2010 unsigned hyper bytes_in; /* total bytes in */ unsigned int pkts_in; /* total packets in */ unsigned int errs_in; /* total errors in */ unsigned int drops_in; /* total drops in */ unsigned hyper bytes_out; /* total bytes out */ unsigned int packets_out; /* total packets out */ unsigned int errs_out; /* total errors out */ unsigned int drops_out; /* total drops out */ } /* Hypervisor and virtual machine performance metrics */ /* Virtual Node Statistics */ /* See libvirt, struct virtNodeInfo */ /* opaque = counter_data; enterprise = 0; format = 2100 */ struct virt_node { unsigned int mhz; /* expected CPU frequency */ unsigned int cpus; /* the number of active CPUs */ unsigned hyper memory; /* memory size in bytes */ unsigned hyper memory_free; /* unassigned memory in bytes */ unsigned int num_domains; /* number of active domains */ } /* Virtual Domain CPU statistics */ /* See libvirt, struct virtDomainInfo */ /* opaque = counter_data; enterprise = 0; format = 2101 */ struct virt_cpu { unsigned int state; /* virtDomainState */ unsigned int cpuTime; /* the CPU time used (ms) */ unsigned int nrVirtCpu; /* number of virtual CPUs for the domain */ } /* Virtual Domain Memory statistics */ /* See libvirt, struct virtDomainInfo */ /* opaque = counter_data; enterprise = 0; format = 2102 */ struct virt_memory { unsigned hyper memory; /* memory in bytes used by domain */ unsigned hyper maxMemory; /* memory in bytes allowed */ } /* Virtual Domain Disk statistics */ /* See libvirt, struct virtDomainBlockInfo */ /* See libvirt, struct virtDomainBlockStatsStruct */ FINAL sFlow.org [Page 11] FINAL sFlow Host Structures July 2010 /* opaque = counter_data; enterprise = 0; format = 2103 */ struct virt_disk_io { unsigned hyper capacity; /* logical size in bytes */ unsigned hyper allocation; /* current allocation in bytes */ unsigned hyper available; /* remaining free bytes */ unsigned int rd_req; /* number of read requests */ unsigend hyper rd_bytes; /* number of read bytes */ unsigned int wr_req; /* number of write requests */ unsigned hyper wr_bytes; /* number of written bytes */ unsigned int errs; /* read/write errors */ } /* Virtual Domain Network statistics */ /* See libvirt, struct virtDomainInterfaceStatsStruct */ /* opaque = counter_data; enterprise = 0; format = 2104 */ struct virt_net_io { unsigned hyper rx_bytes; /* total bytes received */ unsigned int rx_packets; /* total packets received */ unsigned int rx_errs; /* total receive errors */ unsigned int rx_drop; /* total receive drops */ unsigned hyper tx_bytes; /* total bytes transmitted */ unsigned int tx_packets; /* total packets transmitted */ unsigned int tx_errs; /* total transmit errors */ unsigned int tx_drop; /* total transmit drops */ } 4. References [1] Phaal, P. and Lavine, M., "sFlow Version 5", http://www.sflow.org/sflow_version_5.txt, July 2006 [2] IETF, "RFC 2616: Hypertext Transfer Protocol -- HTTP/1.1", June 1999 [3] IETF, "RFC 1813: NFS Version 3 Protocol Specification", June 1995 [4] "Memcached", http://memcached.org/ [5] "Hadoop", http://hadoop.apache.org/ [6] IEEE, "802.1Qbg - Edge Virtual Bridging" [7] IETF, "RFC 2737: Entity MIB (Version 2)", December 1999 FINAL sFlow.org [Page 12] FINAL sFlow Host Structures July 2010 [8] Case, J., Fedor, M., Schoffstall, M., and J. Davin, "Simple Network Management Protocol", STD 15, RFC 1157, May 1990. [9] "Ganglia Monitoring System", http://ganglia.sourceforge.net/ [10] "libvirt: virtualization API", http://libvirt.org/ [11] "Host sFlow", http://host-sflow.sourceforge.net/ 5. Author's Address Peter Phaal InMon Corp. 580 California Street, 5th Floor San Francisco, CA 94104 Phone: (415) 283-3263 EMail: peter.phaal@inmon.com Robert Jordan ImageMovers Digital 9 Hamilton Landing Novato, CA 94949 Phone: (415) 475-5800 EMail: rjordan@imagemoversdigital.com FINAL sFlow.org [Page 13]