End-To-End Arguments in System Design April 2, 2009

The three most important contributions are as follows:

1. Low level functionality is about performance tradeoffs verses correctness. If correctness is attempted at a low level, some checks will have to be redundant due to the check being necessary at the end points.
2. Functionality at a low level forces everyone to pay the cost. If there are correctness checks built in at a low level, every user has to have the check, regardless if they need it or not.
3. Don’t put the intelligence in the network, put it in the nodes.

The most glaring problem with the paper:

The end-to-end argument does not take into consideration network management. Sometimes it is advantageous to have intelligence put into the network. For example, lets say that an ISP, or other administrative domain such as a corporation, wants to prevent its users from using a third party service. The ISP can not modify the third party to prevent them from providing the service, instead it must use the network to filter incoming/outgoing packets for that service.

Future research implications:

Nodes become more complex, and therefore more expensive. The end-to-end argument favors a predominantly packet based network, as opposed to a circuit switching network. In some ways, this also makes it easier to deploy new modifications to network applications. It is much easier to provide an update that users must download, verses trying to convince ISPs to modify their network for your application.

End-To-End Arguments in system Design – Review April 2, 2009

This paper tries to put forward end-to-end argument in system design. It stresses on the fact that there is a tension between placing the functionalities in the lower level of the system versus providing them in the application layer. The paper illustrates this tussle using communication subsystem and a file transfer application as an example which tries to send a file from one computer to another on a network using the communication subsystem and the issues involved with the reliable transfer, both at the application level and the communication system level. One important aspect mentioned in this context is that keeping the functionality in the application keeps the cost of all the reliable transfer checks to the application layer itself and the lower level system still remains same to all the applications. But, the flipside to the approach is that there can be a performance hit when the transactions scale for the application (in this case, it is increasing file size) as there can be many more retries of the same process and it may blow the performance. To counter that, some reliable transfer mechanism can be provided in the communication subsystem itself. But, this also can lead to two drawbacks which are important highlights of the paper. One of the drawbacks is that all the applications have to pay for the functionality if even they do not need it. The other drawback is that pushing the functionality to the lower levels hides a lot of application specific information and the job efficiency suffers. The other important aspect of the paper is that the argument of end-to-end system design is actually specific to every application and therefore, there has to synergy in what goes under the layers and what is exposed in the application. One of the drawbacks of the paper is the system design where target may not be of importance and the host only need to needs to broadcast the information to the targets. Here, reliability may not be an issue and the end-to-end argument may not be of much significance. The pointer provided provided for future work is to keep the end-to-end arguments in mind while pushing the functionalities in the communication subsystem layers.

End-to-end Arguments in System Design April 2, 2009

In this paper, the authors argue that no amount of cleverness, redundancy, error detection, etc. at a low level of the system can obviate the need to provide end-to-end protections at the application level. The crux of their argument is that the potential sources of data corruption and loss include sources which are fundamentally uncorrectable at the low level. As an example, no amount of network protections can prevent an intermediate host from corrupting data as it passes it from network link to network link. In addition to being insufficient to prevent all errors, low-level protections can actually add overhead which can cause unnecessary performance degradation. In the best case, the low-level functions can serve as a performance optimization to reduce the amount of time before an error is detected, but cannot provide all of the protection necessary.

In my opinion, the biggest problem with this paper is that it ignores the programming overhead imposed at the application level as a result of having to implement everything in an end-to-end manner. While it is true that the application level will always have to provide some kind of error checking and data verification, there is no need to make every program perform some of the tasks which can be performed at a low level. In addition to the programming overhead, there is also a potential performance overhead. If functions are implemented at the hardware level, it is possible to design specialized hardware which can provide the functionality much faster than equivalent application-level software could.

Future research in this field needs to be domain-specific, since the mechanisms for packet-switched networks likely overlap little with those for copying data between two hard disks in a system. As mentioned in the paper, the key for determinining the system’s position on the low-level/end-to-end spectrum is a careful analysis of which functions can be more efficiently and effectively implemented at a low level.

Fundamentally, the concept of this paper is sound. There are virtually no applications in which there does not need to be some end-to-end checking regardless of the underlying implementation. The key is striking a compromise between efficiency, modularity, and effectiveness.

End-to-end Arguments in System Design April 2, 2009

Three important facts brought out by the paper

A lot of communication sub-systems tend to provide several functions aimed at making the system more reliable, secure, error proof, etc. The paper takes a look at the big picture, including both the communication sub-system and the upper application layer, and brings out the fact that a lot of these functions would provide a more complete coverage if present in the upper, application layer.

The paper also mentions how providing the same functionality in both, the communication sub-system and the application layer would only introduce more redundancy into the system without actually helping performance or coverage to a great extend.

In addition, it talks of the trade-offs and the purpose achieved by providing these functions at a lower or upper layer. It takes a look at six major functions relevant to a networked, distributed system: reliability, delivery guarantee, secure data transmission, duplicate message suppression, in order delivery and transaction management. It discusses the major benefit and functionality achieved by providing each of these features at a communication sub-system level or the upper application layer or both. This provides some rather interesting insight into how several of these features perform redundant work given others are already in place and only result in unnecessary wastage of bandwidth and compute time.

Facts it overlooks or doesn’t provide

It presents no concrete experimental results. It provides a lot of reasoning which makes sense. However, there are a lot more trade-offs between providing a functionally in the lower or upper layer and additional aspects to the ones mentioned than that touched upon by the paper. For example, it does not talk of the overhead/time wasted in letting an error get to the application layer before it is detected. It does mention how large files might benefit with per packet error checking and how this can also be accomplished by a “careful file transfer” application. However, it makes no mention of the time taken and computation required to perform this error checking in the application layer as compared to the lower, communication layers. It is possible that performing checks in the lower layers takes up half as much time and if majority of the errors occur in the lower layers, a lot of performance benefit is to be got by providing efficient error checking in the lower layers itself. In the case of end-to-end reliability checking, there is also the overhead of sending a checksum from the client machine to the server machine as part of the last step to compare the original checksum with the final checksum. If the communication sub-system were to detect this error a priori, this overhead can be averted. A clearer and more concrete picture of all these trade-offs working at the same time would be got if actual experimental results were presented and a comparisons made between providing functions at the lower and upper layers.

Future Work

One of the extensions that can be made to the idea presented by this paper is to provide quantitative measures that take into consideration the network type, traffic, protocol etc to determine if a functionality should be provided in the upper or lower layer. A further extension could be to make this a dynamic decision, wherein, depending on the number of packet errors or the QOS provided by the network, a functionality might be realized in the lower or upper layers. For example, if there were more than one or two packet errors in a single file, error checking/reliability could be provided in the lower layer to prevent spending time in receiving the entire file before declaring it to contain an error.

End-to-end Arguments in System Design April 2, 2009

(i) the three most important things the paper says

The main important aspect of the paper is describing the end-to-end argument as  a design principle. Placing functionality at the lower layers of abstraction may be redundant, incomplete, of little value, or increase cost, compared to providing the functionality at a higher layer or level of abstraction.

Then the paper goes into detail about specific examples such as the file transfer application and makes the point that “it would be too simplistic to conclude that the lower levels should play no part in obtaining reliability”. This leads to talking about performance issues with implementing both functionality at a lower level and a higher level and “using performance to justify placing functions in a low-level subsystem must be done carefully”. It must be designed carefully because “the same or better performance can be achieved at a higher level” due to reduced overhead and application specific knowledge and increased knowledge that the lower layers will not have. But also in some cases, having that functionality at a lower layer will be more efficient if there is low overhead and it is a common subsystem to many applications.

Finally, the paper makes the conclusion that “Thus the end-to-end argument is not an absolute rule, but rather a guideline that helps in application and protocol design analysis; one must use some care to identify the end points to which the argument should be applied.”

(ii) the most glaring problem with the paper

The paper acknowledges that there is still a need for both end-to-end and low level designs depending on the application and future needs. A glaring problem is identifying the end in end-to-end design. This can be specific to the application, properties of the application, any number of other properties and variables. One interesting example the paper mentions is RISC processor architecture as an example of end-to-end system design.

(iii) the future research directions of the work.

There are many future research directions taking this end-to-end argument and applying it to design of operating systems, cryptographic systems, distributed systems, next generation network protocols, real life situations, etc…