iii. Packet initialization

iv. Thread scheduling and context switching

v. Waiting for acknowledgements:

iii. Packet initialization

iv. Thread scheduling and context switching

v. Waiting for acknowledgements:

Communication primitives

• Some kernels designed for distributed systems have provided communication primitives tailored to the types of invocation. 
• Example: 
  • Amoeba provides doOperation, getRequest and sendReply as primitives. 
  • Amoeba, the V system and Chorus provide group communication primitives. 
  • Middleware provides RMI over UNIX’s connected (TCP) sockets, then a client must make two communication system calls (socket write and read) for each remote invocation. 
  • Over Amoeba, it require only a single call to doOperation. 
• Despite the widespread use of TCP and UDP sockets provided by common kernels, research continues to be carried out into lower-cost communication primitives in experimental kernels. 

Protocols and openness

• One of the main requirements of the operating system is to provide standard protocols that enable internetworking between middleware implementations on different platforms. 
• By contrast, the designers of the Mach 3.0 and Chorus kernels (as well as L4) decided to leave the choice of networking protocols entirely open. 
  • These kernels provide message passing between local processes only, and leave network protocol processing to a server that runs on top of the kernel.
• Protocols are normally arranged in a stack of layers. 
• Many operating systems allow new layers to be integrated statically.

• By contrast, dynamic protocol composition is a technique whereby a protocol stack can be composed on the fly to meet the requirements of a particular application, and to utilize whichever physical layers are available given the platform’s current connectivity.
• Example: a web browser running on a notebook computer should be able to take advantage of a wide area wireless link while the user is on the road, and then a faster Ethernet connection when the user is back in the office. 
• Support for protocol composition appeared in the design of the UNIX Streams facility [Ritchie 1984], in Horus [van Renesse et al. 1995] and in the x-kernel [Hutchinson and Peterson 1991], construction of configurable transport protocol CTP on top of the Cactus system [Bridges et al. 2007]. 

Invocation performance

• Invocation performance is a critical factor in distributed system design. 
• The more designers separate functionality between address spaces, the more remote invocations are required. 
• Invocation costs
• Calling a conventional procedure or invoking a conventional method, making a system call, sending a message, remote procedure calling and remote method invocation are all examples of invocation mechanisms. 
• Each mechanism causes code to be executed outside the scope of the calling procedure or object. 
• Each involves, in general, the communication of arguments to this code and the return of data values to the caller. 

Figure : Invocations between address spaces

• Invocation over the network
  • A null RPC (and similarly, a null RMI) is defined as an RPC without parameters that executes a null procedure and returns no values. 
  • Its execution involves an exchange of messages carrying some system data but no user data. 
  • Much of the observed RPC delay is accounted by the actions of the operating system kernel and user-level RPC runtime code and not from the total network transfer time. 
  • Null invocation costs are important because they measure a fixed overhead, the latency. 

Figure : RPC delay against parameter size

• RPC throughput is also a concern when data has to be transferred in bulk. 
• Steps in an RPC
• A client stub marshals the call arguments into a message, sends the request message and receives and unmarshals the reply. 
• At the server, the worker thread calls the appropriate server stub. 
• The server stub unmarshals the request message, calls the designated procedure and marshals and send the reply. 
• Main components accounting for remote invocation delay, beside network transmission times:
  • Marshalling → Marshalling and unmarshaling, which involve copying and converting data, create a significant overhead as the amount of data grows. 
  • Data copying
    • Even after marshalling, message data is copied several times:
      • Across the user-kernel boundary, between the client or server address space and kernel buffers;
      • Across each protocol layer
      • Between the network interface and kernel buffers
    • Transfer between network interface and main memory are usually handled by direct memory access (DMA).
  • Packet initialization
    • Involves initializing protocol headers and trailers, including checksums. 
  • Thread scheduling and context switching
    • Several system calls are made during an RPC, as stubs invoke the kernel’s communication operations
    • One or more server threads is scheduled.
    • If the operating system employs a separate network manager process, then each Send involves a context switch to one of its threads. 
• Waiting for acknowledgements:
• The choice of RPC protocol may influence delay, particularly when large amounts of data are sent. 

• Memory sharing
  • Bershad et al. [1990] report a study that showed that, in the installation examined, most cross-address-space invocation took place within a computer and not, as might be expected in a client-server installation, between computer. 
  • Bershad et al. developed a more efficient invocation mechanism for the case of two processes on the same machine called lightweight RPC (LRPC). 
  • It would be more efficient to use shared memory regions for client-server communication, with a different (private) region between the server and each of its local clients. 
  • Such as region contains one or more A stacks. 
  • Instead of RPC parameters being copied between the kernel and user address spaces involved, the client and server are able to pass arguments and return values directly via an A stack. 
  • The same stack is used by the client and server stubs. 
  • In LRPC, arguments are copied once: when they are marshalled onto the A stack. 
  • In an equivalent RPC, they are copied four times: 
    • from the client stub’s stack onto a message; 
    • from the message to a kernel buffer, 
    • from the kernel buffer to a server message, and 
    • from the message to the server stub’s stack. 
  • Instead of setting up one or more threads, which then listen on ports for invocation requests, the server exports a set of procedures that it is prepared to have called. 
  • Threads in local processes may enter the server’s execution environment as long as they start by calling one of the server’s exported procedures. 
    
    

    Figure : A lightweight remote procedure call