The Twitter Network Layer (TNL) is a framework for interfacing with the Apple provided
NSURLSession
stack that provides additional levels of control and insight over networking requests,
provides simple configurability and minimizes the cognitive load necessary to maintain a robust and
wide-reaching networking system.
The Twitter Network Layer sits on top of the connection/session layer provided by the Apple NSURL framework. Those frameworks are build on top of the HTTP/1.1 and HTTP/2. The layer chart appears like this:
/--------------------------------------\
| |
| User Layer |
| The actual user (Layer 8) |
| |
|--------------------------------------|
| |
| Application Layer |
| MVC, MVVM, etc (Layer 7e) |
| |
|--------------------------------------|
| |
| Concrete Operation/App Layer | <------ Operations, Requests &
| TNL (Layer 7d) | Responses built on TNL
| |
|--------------------------------------|
| |
| Abstract Operation/App Layer |
| TNL (Layer 7c) | <------ TNL
| |
|--------------------------------------|
| |
| Connection/App Layer |
| NSURL Stack (Layer 7b) |
| |
|--------------------------------------|
| |
| Protocol/App Layer |
| HTTP/1.1 & HTTP/2 (Layer 7a) |
| |
|--------------------------------------|
| |
| Presentation Layer |
| Encryption & Serialization (Layer 6) |
| |
|--------------------------------------|
| |
| Session Layer |
| A Feature of TCP (Layer 5) |
| |
|--------------------------------------|
| |
| Transport Layer |
| TCP (Layer 4) |
| |
|--------------------------------------|
| |
| Routing/Network Layer |
| IP (Layer 3) |
| |
|--------------------------------------|
| |
| Data Link Layer |
| IEEE 802.X (Layer 2) |
| |
|--------------------------------------|
| |
| Physical Layer |
| Ethernet (Layer 1) |
| |
\--------------------------------------/
Twitter Network Layer provides a framework on top of Apple's NSURLSession
framework with
numerous benefits. Here are some of the features provided by TNL:
- All the features of
NSURLSession
, simplified where appropriate NSOperation
based request operations (forNSOperation
features)- Strong separation of roles in the framework's objects
- Immutable/mutable pairings of requests (
TNLHTTPRequest
) and configurations (TNLRequestConfiguration
) - Encapsulated immutable responses (
TNLResponse
) - Prioritization of requests
- Selectable response body consumption modes (
NSData
storage, callback chunking or saving to file) - Request hydration (enables polymorphic requests and dynamically populated requests)
- Dynamic retrying with retry policies
- More events (request operation state transitions, progress, network state/condition updates, etc)
- Modular delegates for separation of roles and increased reusability and easy overriding
The high level concept of how to use TNL is rather straightforward:
- Set up any reuseable settings (by doing any combination of the following):
- Build shared accessors to resuable
TNLRequestConfiguration
instances - Implement a
TNLRequestDelegate
(if added functionality is desired beyond just handling the result) - Configure a
TNLRequestConfiguration
for reuse - Configure the
TNLGlobalConfiguration
- Build shared accessors to resuable
- Set up any reusable
TNLRequestOperationQueue
objects once (ex: one for API requests, one for image requests, etc.)[TNLRequestOperationQueue initWithIdentifier:]
- Generate and enqueue any desired
TNLRequestOperation
with the following objects:TNLRequest
conforming object (includingTNLHTTPRequest
concrete class andNSURLRequest
)TNLRequestConfiguration
(optional)TNLRequestDelegate
(optional)
- Handle the events appropriately via the callbacks, particularly the completion callback that provides the
TNLResponse
- Delegate callbacks will go to the appropriate sub-protocol in the
TNLRequestOperation
'sTNLRequestDelegate
- Delegate callbacks will go to the appropriate sub-protocol in the
Twitter Network Layer documentation starts with this README.md
and progresses through the APIs via their documentation.
The core objects of a service based architecture are request, response and operation/task/action (referred to as an operation from here on). The request encapsulates data to send and is not actionable; the response encapsulates the data received and is not actionable; and the operation is the object that delivers the request and retrieves the response and is the only actionable object in the set of core objects.
This high level concept translates directly into a network architecture as we will have requests that encapsulate the data of an HTTP request which are Headers and a Body, responses that encapsulate the data of an HTTP response which are Headers and a Body, and last the operation that executes delivering the request and retrieving the response.
- request
- encapsulates data to send
- immutability provides stability
- not actionable, just data
TNLRequest
is the protocol for requests in TNLTNLHTTPRequest
andNSURLRequest
are concrete classes (both are immutable/mutable pairs)
- response
- encapsulates data received
- immutability provides stability
- not actionable, just data
TNLResponse
is the object for responses in TNL (composite object that includes anNSHTTPURLResponse
)
- operation
- the executing object
- delivers the request
- retrieves the response
- actionable (e.g. starting, canceling, priotiziation, modifying dependencies)
TNLRequestOperation
is the operation in TNL (subclassesNSOperation
) and is backed byNSURLSessionTask
In addition to a service architecture having requests, operations and responses; support objects are often present that aid in the management of the executing operations, configuration of their behavior and delegation of decisions or events.
The configuration object encapsulates how an operation behaves. It will have no impact on what is sent in the operation (that's the request), but can modify how it is sent. For instance, the configuration can indicate a maximum duration that the operation can take before it should fail.
The delegate object provides the extensibility of on demand decision making when prudent as well as the delivery of events as the operation executes.
The manager object coordinates the execution of multiple operations within a logical grouping.
- configuration
- encapsulation of behavior settings
TNLRequestConfiguration
is the config in TNL (applied per operation)NSURLSessionConfiguration
is the config in NSURL stack (applied per manager)
- delegate
- provides extensibility
- has callbacks for on demand decisions
- has callbacks for events as they occur
TNLRequestDelegate
is the delegate in TNL (provided per operation)NSURLSessionDelegate
is the delegate in NSURL stack (provided per manager)
- manager
- coordinator of multiple operations
- permits grouping of operations
TNLRequestOperationQueue
is the manager object in TNLNSURLSession
is the manager object in NSURL stack
Note: You can already see there is a fundamental architecture difference between NSURLSession
networking
and Twitter Network Layer. The configuration and delegate per operation approach in TNL
is much more scalable when dealing with dozens or hundreds of unique configurations and/or delegates
given a plethora of requests and their needs. Coupling the configuration and/or delegate to the
reusable manager object(s) is unwieldy and can lead to mistakes w.r.t. correct configuration and event
on a per request basis.
TNL uses the TNLRequest
as the interface for all network requests. In practice, the protocol
is used in one of 3 ways:
- Concrete
TNLHTTPRequest
- Configuring a concrete
TNLHTTPRequest
object (orTNLMutableHTTPRequest
)
NSURLRequest
NSURLRequest
explicitely conforms toTNLRequest
protocol via a category in TNL making it supported as request object.- However, since TNL rigidly segregates configuration from request, only the request properties on
NSURLRequest
are observed and the configuration properties ofNSURLRequest
are ignored.
- Implementing a custom
TNLRequest
- TNL supports having anything that conforms to
TNLRequest
as an original request for an operation. - Makes it simple for an object that encapsulates the minimal amount of information necessary to take the place as the original request.
- You could have a
APPRetrieveBlobRequest
that has 1 property, the identifier for the "Blob" call blobIdentifier. - That object doesn't need to have any methods that actually represent anything related to an HTTP request and that's ok. However, in order for the operation to send the original request, it needs to be able to be treated as an HTTP request, which is to say it must conform to
TNLRequest
. This can be done in 2 ways:- have the object implement
TNLRequest
and have its methods that populate the values by the relevant properties (in our example, the blob identifier) - have the delegate implement the request hydration callback to convert the opaque request into a well formed
TNLRequest
ready for HTTP transport.
- have the object implement
- See Custom TNLRequest examples
- You could have a
When it comes to making the choice, it can boil down to convenience vs simplicity of reuse. If you have a one shot request that has no reuse, options 1 and 2 will suffice. If you have a request that can be reused throughout the code base, option 3 clearly offers the cleanest interface. By having the caller only need to know the class of the request and the relevant values for populating the request, any concern over the HTTP structure is completely eliminated.
TNLHTTPRequest:
NSString *URLString = [NSString stringWithFormat:@"http:https://api.myapp.com/blob/%tu", blobIdentifier];
NSURL *URL = [NSURL URLWithString:URLString];
TNLHTTPRequest *request = [TNLHTTPRequest GETRequestWithURL:URL
HTTPHeaderFields:@{@"User-Agent": [MYAPPDELEGATE userAgentString]}];
TNLMutableHTTPRequest:
NSString *URLString = [NSString stringWithFormat:@"http:https://api.myapp.com/blob/%tu", blobIdentifier];
NSURL *URL = [NSURL URLWithString:URLString];
TNLMutableHTTPRequest *mRequest = [[TNLMutableHTTPRequest alloc] init];
mRequest.HTTPMethodValue = TNLHTTPMethodValueGET;
mRequest.URL = URL;
[mRequest setValue:[MYAPPDELEGATE userAgentString] forHTTPHeaderField:@"User-Agent"];
NSString *URLString = [NSString stringWithFormat:@"http:https://api.myapp.com/blob/%tu", blobIdentifier];
NSURL *URL = [NSURL URLWithString:URLString];
NSMutableURLRequest *mRequest = [[NSMutableURLRequest alloc] init];
mRequest.HTTPMethod = @"GET";
mRequest.URL = URL;
[mRequest setValue:[MYAPPDELEGATE userAgentString] forHTTPHeaderField:@"User-Agent"];
1) Request Hydration
APPRetrieveBlobRequest *request = [[APPRetrieveBlobRequest alloc] initWithBlobIdentifier:blobIdentifier];
// ... elsewhere ...
- (void)tnl_requestOperation:(TNLRequestOperation *)op
hydrateRequest:(APPRetrieveBlobRequest *)request // we know the type
completion:(TNLRequestHydrateCompletionBlock)complete
{
NSString *URLString = [NSString stringWithFormat:@"http:https://api.myapp.com/blob/%tu", blobRequest.blobIdentifier];
NSURL *URL = [NSURL URLWithString:URLString];
TNLHTTPRequest *newReq = [TNLHTTPRequest GETRequestWithURL:URL
HTTPHeaderFields:@{@"User-Agent": [MYAPPDELEGATE userAgentString]}];
complete(newReq);
}
2) Request with HTTP support
APPRetrieveBlobRequest *request = [[APPRetrieveBlobRequest alloc] initWithBlobIdentifier:blobIdentifier];
// ... elsewhere ...
@implementation APPRetrieveBlobRequest
- (NSURL *)URL
{
NSString *URLString = [NSString stringWithFormat:@"http:https://api.myapp.com/blob/%tu", self.blobIdentifier];
return [NSURL URLWithString:URLString];
}
- (NSDictionary *)allHTTPHeaderFields
{
return @{@"User-Agent":[MYAPPDELEGATE userAgentString]};
}
// POINT OF IMPROVEMENT:
// utilize polymorphism and have an APPBaseRequest class that implements
// the "allHTTPHeaderFields" so that all subclasses (including APPRetrieveBlobRequest)
// will inherit the desired defaults.
// This can apply to a wide range of HTTP related TNLHTTPRequest properties
// or even composition of subcomponents that are aggregated to a single property.
// For example: the host of the URL (api.myapp.com) could be provided
// as a property on APPBaseRequest that permits subclasses to override the host, and then
// the construction of the `URL` uses composition of variation properites that the subclasses
// can provide.
@end
When an operation completes, a TNLResponse is populated and provided to the completion block
or completion callback (depending on if you use a delegate or not). The TNLResponse has all
the information necessary to understand how the operation completed, as well as what information
was retrieve. Additionally, with response polymorphism, the response can be extended to
provide better contextual information regarding the result, such as parsing the response body as
JSON or converting the response body into a UIImage
.
The way you deal with a TNLResponse should be systematic and straighforward:
- deal with any errors on the response
- TNLResponse has an operationError property but custom subclasses could expose other errors too.
- Subclass response objects that have extra errors should consider having an
anyError
property for quick access to any error in the response.
- deal with the status code of the response
- It is important to know that a 404 is not an operation error so it won't be set as an error.
- It is actually the status of the successful operation and needs to be handled accordingly.
- For designs that want to treat HTTP Status codes that are not success as errors, they should expose an HTTP error on their response subclass(es).
- deal with the response payload
- This could be the response HTTP headers, the response body (as
NSData
or a file on disk), etc - response subclasses should consider deserializing their response's body into a model object and exposing it as a property for concrete interactions.
- This could be the response HTTP headers, the response body (as
One benefit to using response polymorphism is the ability to handle the response and populate
the hydrated response with the information that's pertinent to the caller.
For example: if your network operation yields JSON, and all you care about is if that JSON came
through or not, at hydration time you could check for any error conditions then parse out the JSON
and if everything is good have a property on the custom TNLResponse
subclasss that holds the
NSDictionary
result property (or nil
if anything along the way prevented success).
Things you can inspect on a response by default:
- the operation error (if one occurred)
- the original request
- the response info
- this object encapsulates the information of the HTTP response including:
- the source of the response (local cache or network load)
- the response body (as
data
ortemporarySavedFile
if the operation was configured to maintain the data) - the final
NSURLRequest
that loaded the response - the final
NSURLResponse
object
- it also provides convenience accessors
- the response's HTTP status code
- the final
NSURL
- all the HTTP header fields
- this object encapsulates the information of the HTTP response including:
- the response metrics
- detailed metric information such as execution timings, durations, bytes sent/received, attempt counts, etc.
- this is the detail that TNL exposes for every request/operation/response that really empowers programmers to maximize impact with their networking.
Twitter Network Layer provides a highly robust API for building network operations with a great deal of flexibility and extensibility. However, there are often occasions when you just need to execute an operation and need things to be as simple as possible. Twitter Network Layer provides all the convenience necessary for getting what needs to be done as simply as possible.
NSString *URLString = [NSURL URLWithString:@"http:https://api.myapp.com/settings"];
NSURLRequest *request = [NSURLRequest requestWithURL:URLString];
[[TNLRequestOperationQueue defaultOperationQueue] enqueueRequest:request
completion:^(TNLRequestOperation *op, TNLResponse *response) {
NSDictionary *json = nil;
if (!response.operationError && response.info.statusCode == 200) {
json = [NSJSONSerialization JSONObjectWithData:response.info.data options:0 error:NULL];
}
if (json) {
[self _myapp_didCompleteSettingsRequestWithJSON:json];
} else {
[self _myapp_didFailSettingsRequest];
}
}];
Often the vanila configuration for an operation will suffice, however it is common to need particular behaviors in order to get specific use cases to work. Let's take, as an example, firing network operation when a specific metric is hit. In this case, we don't care about storing the response body and we also want to avoid having a cache that could get in the way.
NSURL *URL = [NSURL URLWithString:@"http:https://api.myapp.com/recordMetric?hit=true"];
TNLHTTPRequest *request = [TNLHTTPRequest GETRequestWithURL:URL HTTPHeaderFields:nil];
TNLMutableRequestConfiguration *config = [TNLMutableRequestConfiguration defaultConfiguration];
config.responseDataConsumptionMode = TNLResponseDataConsumptionModeNone;
config.URLCache = nil; // Note: 'URLCache' is now 'nil' by default in TNL, but the illustration still works
TNLRequestOperation *op = [TNLRequestOperation operationWithRequest:request
configuration:config
completion:^(TNLRequestOperation *o, TNLResponse *response) {
assert(response.info.source != TNLResponseSourceLocalCache);
const BOOL success = response.info.statusCode == 202;
[self didSendMetric:success];
}];
[[TNLRequestOperationQueue defaultOperationQueue] enqueueOperation:op];
Now, sometimes, you may want to have the same defaults for certain kinds of operations. That can easily be accomplished with a category or some other shared accessor.
@interface TNLRequestConfiguration (APPAdditions)
+ (instancetype)configurationForMetricsFiring;
@end
@implementation TNLRequestConfiguration (APPAdditions)
+ (instancetype)configurationForMetricsFiring
{
static TNLRequestConfiguration* sConfig;
static dispatch_once_t onceToken;
dispatch_once(&onceToken, ^{
TNLMutableRequestConfiguration *mConfig = [TNLMutableRequestConfiguration defaultConfiguration];
mConfig.URLCache = nil; // Note: 'URLCache' is now 'nil' by default in TNL, but the illustration still works
mConfig.responseDataConsumptionMode = TNLResponseDataConsumptionModeNone;
sConfig = [mConfig copy];
});
return sConfig;
}
@end
@implementation TNLMutableRequestConfiguration (APPAdditions)
+ (instancetype)configurationForMetricsFiring
{
return [[TNLRequestConfiguration configurationForMetricsFiring] mutableCopy];
}
@end
Twitter Network Layer was designed from the ground up with REST APIs in mind. From simple APIs to complex API layers that require a complicated system for managing all operations, TNL provides the foundation needed.
As a pattern for creating concrete API operations, one of the first places to extend TNL for
your API layer is by concretely building API requests and responses. For requests, you
implement a TNLRequest
for every request your API provides with properties that configure each
request appropriately. Those requests should be subclassing a base request that does the busy
work of pulling together the generic properties that the subclasses can override to construct the
HTTP properties of the request. Each subclassed request then overrides only what is
necessary to form the valid HTTP request. For things that are context or time sensitive, such as
request signing, request hydration should be used to fully saturate the custom API request at
the time the request is going to sent (vs at the time it was enqueued).
Following from custom API requests are custom API responses. At a minimum, it makes sense to
have an API response that subclasses TNLResponse
. To provide an even simpler interface to
callers, you can implement a response per request. For response hydration, you merely
extract whatever contextually relevant information is valuable for an API response and set those
properties on you custom subclass of TNLResponse
(such as API error, JSON result, etc).
If the API layer is advanced enough, it may warrant further encapsulation with a managing object which is often referred to as an API client. The API client would manage the queuing of requests, the delegate implementation for operations (including hydration for requests and subclassing responses so they hydrate too), the vending of operations, authentication/signing of requests, high level retry plus timeout behavior, custom configurations and oberving responses for custom handling.
With an API client architecture, the entire HTTP structure is encapsulated and callers can deal with things just the objects they care about. No converting or validating. No configuring. The power of TNL is completely utilized by API client freeing the caller of any burden.
APISendMessageRequest *request = [[APISendMessageRequest alloc] init];
request.sender = self.user;
request.receiver = otherUser;
request.message = message;
self.sendOp = [[APIClient sharedInstance] enqueueRequest:request
completion:^(TNLRequestOperation *op, APIResponse *response) {
[weakSelf messageSendDidComplete:op withResponse:(id)response];
}];
// ... elsewhere ...
- (void)messageSendDidComplete:(TNLRequestOperation *)op withResponse:(APISendMessageResponse *)response
{
assert(self.sendOp == op);
self.sendOp = nil;
if (!sendMessageResponse.wasCancelled) {
if (sendMessageResponse.succeeded) {
[self updateUIForCompletedMessageSendWithId:sendMessageResponse.messageId];
} else {
[self updateUIForFailedMessageSendWithUserErrorTitle:sendMessageResponse.errorTitle
errorMessage:sendMessageResponse.errorMessage];
}
}
}
// Insight:
// Presumably, APISendMessageResponse would subclass a base response like APIBaseResponse.
// Following that presumption, it would make sense that APIBaseResponse would expose
// wasCancelled, succeeded, errorTitle and errorMessage while APISendMessageResponse would
// expose messageId (since that is part of the response payload that is specific to the request).
// It would likely make sense that if the API used JSON response bodies,
// the base response would also expose a "result" property (NSDictionary) and
// APISendMessageResponse's implementation for messageId is just:
// return self.result[@"newMessageId"];
Twitter Network Layer includes a target for building a macOS tool called tnlcli
. You can build this tool
run it from Terminal from you Mac, similar to cURL or other networking command line utilities.
Usage: tnlcli [options] url
Example: tnlcli --request-method HEAD --response-header-mode file,print --response-header-file response_headers.json https://google.com
Argument Options:
-----------------
--request-config-file <filepath> TNLRequestConfiguration as a json file
--request-headers-file <filepath> json file of key-value-pairs for using as headers
--request-body-file <filepath> file for the HTTP body
--request-header "Field: Value" A header to provide with the request (will override the header if also in the request header file). Can provide multiple headers.
--request-config "config: value" A config setting for the TNLRequestConfiguration of the request (will override the config if also in the request config file). Can provide multiple configs.
--request-method <method> HTTP Method from Section 9 in HTTP/1.1 spec (RFC 2616), such as GET, POST, HEAD, etc
--response-body-mode <mode> "file" or "print" or a combo using commas
--response-body-file <filepath> file for the response body to save to (requires "file" for --response-body-mode
--response-headers-mode <mode> "file" or "print" or a combo using commas
--response-headers-file <filepath> file for the response headers to save to (as json)
--dump-cert-chain-directory <dir> directory for the certification chain to be dumped to (as DER files)
--verbose Will print verbose information and force the --response-body-mode and --responde-headers-mode to have "print".
--version Will print ther version information.
Copyright 2014-2020 Twitter, Inc.
Licensed under the Apache License, Version 2.0: https://www.apache.org/licenses/LICENSE-2.0
Please report sensitive security issues via Twitter's bug-bounty program (https://hackerone.com/twitter) rather than GitHub.