Achieving S3 Read-After-Update Consistency

Originally posted on the Hatch Blog

The team at Hatch spun up the Labour Exchange in a few days re-purposing our tech to help stood down workers find employment during the covid-19 crisis.

In order to get the system up and running in such a short time frame we decided to use S3 as a flat-file data store to maintain our serverless batch job states and caches. After a very cursory search to satisfy ourselves S3 would guarantee read-after-write consistency, we flew on.

From the documentation

Amazon S3 provides read-after-write consistency for PUTS of new objects in your S3 bucket in all Regions with one caveat. The caveat is that if you make a HEAD or GET request to a key name before the object is created, then create the object shortly after that, a subsequent GET might not return the object due to eventual consistency.

Unfortunately, we missed the bolded section and the part further down the page that explicitly states our use case:

A process replaces an existing object and immediately tries to read it. Until the change is fully propagated, Amazon S3 might return the previous data.

As these processes run on a schedule (not as part of a user facing API) we could afford to spend some extra calls to S3 to roll our own read-after-update consistency. Knowing that S3 guarantees read-after-firstWrite, we can write a new file for every change, read the latest file and make sure we cleanup.

So every time we write a file we:

  • Append a timestamp to the filename
  • Remove older files

When we read a file we:

  • List all files with the key prefix (S3 guarantees listing files will be ordered by ascending UTF-8 binary order)
  • Get the newest file in the list

import { writeS3Obj, getS3FileAsObj, listObjects, deleteS3Files } from "./S3";
/**
* S3 does not provide read-after-update consistency.
* It does provide read-after-firstWrite consistency (as long as no GET has been requested)
* We write a new file every time it changes, and we read the latest file.
* S3 guarantees list of files are sorted in ascending UTF-8 Binary Order
*
*/
const cleanUp = async (key: string) => {
const response = await listObjects({
MaxKeys: 1000,
Bucket: process.env.BUCKET_NAME!,
Prefix: key,
});
const keys = response.Contents?.map((c) => c.Key!) || [];
await deleteS3Files(keys.slice(0, keys.length 1));
};
export const writeServiceState = async (key: string, state: any) => {
await writeS3Obj(`${key}.${Date.now()}`, state);
await cleanUp(key);
};
export const getServiceState = async (key: string, defaultVal: T): Promise => {
const response = await listObjects({
MaxKeys: 1000,
Bucket: process.env.BUCKET_NAME!,
Prefix: key,
});
if (!response.Contents || response.Contents.length === 0) {
console.log("No state file for key " + key);
return defaultVal;
}
return getS3FileAsObj(response.Contents[response.Contents.length 1].Key!);
};
view raw s3.ts hosted with ❤ by GitHub

Logstash and IIS

Note: If you are also using Kibana as your front end, you will need to add a MimeType of “application/json” for the extension .json to IIS.

We are pushing all of our logs into Elasticsearch using Logstash. IIS was the most painful part of the process so I am writing up a few gotchas for Logstash 1.3.3 and IIS in general.

The process is relatively straight forward on paper:

  1. Logstash monitors the IIS log and pushes new entries into the pipeline
  2. Use a grok filter to split out the fields in the IIS log line (more on this below)
  3. Push the result into Elasticsearch

Firstly there is a bug in the Logstash file input on windows (doesn’t handle files named the same in different directories) which results in partial entries being read. To remedy this you need to get IIS to generate a single log file per server (default is per website). Once that is done we can read the IIS logs with this config

input {
file {
type => "iis"
path => "C:/inetpub/logs/LogFiles/W3SVC/*.log"
}
}

view raw
Logstash IIS Input
hosted with ❤ by GitHub

Once we have IIS log lines pumping through the veins of Logstash, we need to break down the line into its component fields. To do this we use the Logstash Grok filter. In IIS the default logging is W3C but you are able to select the fields you want outputed. The following config works for the default fields and [bytes sent] so we can see bandwidth usuage. The Heroku Grok Debugger is a lifesaver for debugging the Grok string (paste an entry from your log into it and then paste you GROK pattern in)

filter{
grok {
match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:iisSite} %{IPORHOST:site} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clienthost} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:bytes:int} %{NUMBER:timetaken:int}"]
}
}

view raw
IIS Logstash Grok
hosted with ❤ by GitHub

Below is the complete IIS configuration for logstash. There are a few other filters we use to enrich the event sent to logstash as well as a conditional to remove IIS log comments.

input {
file {
type => "iis"
path => "C:/inetpub/logs/LogFiles/W3SVC/*.log"
}
}
filter {
#ignore log comments
if [message] =~ "^#" {
drop {}
}
grok {
match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:iisSite} %{IPORHOST:site} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clienthost} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:bytes:int} %{NUMBER:timetaken:int}"]
}
#Set the Event Timesteamp from the log
date {
match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]
timezone => "Etc/UCT"
}
ruby{ code => "event['kilobytes'] = event['bytes'] / 1024.0" }
#https://logstash.jira.com/browse/LOGSTASH-1354
#geoip{
# source => "clienthost"
# add_tag => [ "geoip" ]
#}
useragent {
source=> "useragent"
prefix=> "browser"
}
mutate {
remove_field => [ "log_timestamp"]
}
}
output {
elasticsearch {
host => "127.0.0.1"
}
}

view raw
IIS Logstash
hosted with ❤ by GitHub

Centralising Logs with Logstash and Kibana

Image

We have recently centralised our logs (IIS, CRM, our application of about 5 components) into Elasticsearch on Windows Server using Logstash as the data transformation pipeline (over RabbitMQ) and Kibana as the UI.   It allows us to see all our logs in one place (and if needed in a single timeline), developers can access live logs in a way that they can easily slice and dice the information with out requiring server access. And the front end (pictured) Kibana, is damn sexy! Its dead easy as well. All in all it took about a day to setup.

Architecture

Log Producers

All servers that produce file logs have Logstash installed as a service. Logstash monitors the log file and puts new entries onto a local RabbitMQ exchange . There are much lighter weight shippers out there, however they write directly to Elasticsearch. We wanted something a little more fault tolerant.

Log producers which we control (i.e. our custom components) write directly to RabbitMQ. We use NLog and a modified version (I’ll post more about that later) of the NLog.RabbitMq Target to write our log messages directly (async) to the local RabbitMQ exchange.

Log Server

Our centralized log server has Elasticsearch (the datastore) and Kibana (the UI) running. It also has another logstash agent that reads the messages off RabbitMQ, transforms them into more interesting events (extracting fields for search, GeoLocating IP addresses etc), and then dumps them into Elasticsearch.

Making Ajax play with Passive ADFS 2.1 (and 2.0) – Reactive Authentication

The first post, described the issue of using ADFS and Ajax to create SSO between a WebApp and a WebAPI. This solution looks at the changing the WebAPI to return 401 if the request is not authorized and then using an iFrame to authenticate the user for subsequent calls.

The last solution, pre-authorized on the first AJAX call per page load, which adds some overhead. This was because JSONP has no means of returning status codes (this is not entirely true, you can return a 200 and then have the real response inside a payload, but that is beyond this article). This solution makes use of normal AJAX calls and 401 responses to perform authorization only when it is required.

Caveats

  • This uses normal AJAX calls, so it requires CORS to be enabled on the WebAPI server for cross-domain requests. (See this guide)
  • IE8 & 9 do not support the passing of cookies with cross domain requests and therefore this method will not work as described. However, it should be possible to pass the token in the body of the AJAX request (use POST and HTTPS to maintain security) and write a customized AuthenticationModule to read the token and provide it to the WSFederatedAuthenticationModule. (This is outside the scope of this solution however)

Solution

By default, the WSFederationAuthenticationModule redirects the user to ADFS if the user is not currently authenticated (there is no valid session cookie). This can be changed with the following code

FederatedAuthentication.WSFederationAuthenticationModule.AuthorizationFailed += (sender, e) =>
{
    if (Context.Request.RequestContext.HttpContext.Request.IsAjaxRequest())
    {
        e.RedirectToIdentityProvider = false;
    }
};

By adding this code to ApplicationStart, or a HttpModule, we can make the WebAPI return a HttpStatus of 401 every time authentication is required (during an AJAX request). We then handle this response in our javascript.

The following Gist shows some javascript that handles the 401 response and then uses the idea of authenticating in a iFrame from the last solution, before retrying the AJAX call. The second attempt should now have the needed session cookies to authorize and succeed.

[gist https://gist.github.com/thejuan/4e535a0c468fa47fd9cc]

Making Ajax play with Passive ADFS 2.1 (and 2.0) – JSONP & Pre-Authentication

The first post, described the issue of using ADFS and Ajax to create SSO between a WebApp and a WebAPI.
This solution looks at using JSONP and pre-authentication to achieve SSO between sites on different domains.

Solution Overview

We add a html page (or handler) to the WebAPI solution.
Whenever we make a call to the WebAPI we first load the html page in an iFrame, this iFrame call handles all the ADFS redirects and sets the session cookies for the WebAPI.
These session cookies are then sent (automatically) with the next JSONP call to the server.

Caveats

  • Like all the solutions, this expects that the user has authenticated with ADFS via the WebAPP. When the iFrame hits the WebAPI pre-auth html page and the request gets redirected to ADFS if the user already has a session (that is compatible with the WebAPI relying party) a token will be issued without further authentication.

//Requires Jquery 1.9+
var hasPreAuthenticated = false;
var webAPIHtmlPage = "http://webapi.somedomain/preauth.html"
function preauthenticate() {
//ADFS breaks Ajax requests, so we pre-authenticate the first call using an iFRAME and "authentication" page to get the cookies set
return $.Deferred(function (d) {
if (hasPreAuthenticated) {
console.log("Already pre-authenticated, skipping");
d.resolve();
return;
}
//Potentially could make this into a little popup layer
//that shows we are authenticating, and allows for re-authentication if needed
var iFrame = $("<iframe></iframe>");
iFrame.hide();
iFrame.appendTo("body");
iFrame.attr('src', webAPIHtmlPage);
iFrame.load(function () {
hasPreAuthenticated = true;
iFrame.remove();
d.resolve();
});
});
};
function makeCall(){
return authenticate().then(function () {
var options = //JSONP ajaxOptions
return $.ajax(options)
});
}
view raw gistfile1.js hosted with ❤ by GitHub

Making Ajax play with Passive ADFS 2.1 (and 2.0) – Piggy-Backing

The first post, described the issue of using ADFS and Ajax to create SSO between a WebApp and a WebAPI.
This solution looks at the easiest solution, Piggy-Backing.

The central idea with Piggy-Backing is that the WebApp authenticates in the usual redirecty ADFS way and has the session cookies set.
The WebAPI then uses the same session cookie, thus not needing to ever authenticate with ADFS directly.

Caveats

  • The two applications must be able to share cookies (same root domain)
  • If the WebAPI attempts to authenticate with ADFS it will error; it will error as the AJAX calls will break as per the problem description, it will also break because the redirect url after authentication will be to the WebApp not the WebAPI.

Setting up this solution is easy. Just configure the WebApp as you normally would for ADFS and then use the same config for the WebApi project i.e. set the realm to be the same as the WebApp realm.
You will also need to set the CookieHandler section of web.config to match.

Sub-Domains

If you are using subdomains webapp.contoso.com and webapi.contoso.com then your cookiehandler will look this in both applications web.config

 <cookieHandler requireSsl="true" domain=".contoso.com" /> 

Different Ports or Virtual Directories

If you are using virtual directories or differing ports such as contoso.com/webapp and contoso.com:8000/webapi then your cookiehandler will look this in both applications web.config

 <cookieHandler requireSsl="true" path="/" /> 

A note on Web Farms and Microsoft Dynamics Crm 2011

If you are using load-balancing and/or MS CRM see this article. You need to implement step #3 as well to enable piggy-backing. The session cookie encryption method is changed to be more farm friendly.

Making Ajax play with Passive ADFS 2.1 (and 2.0) – The Problem

The Problem

ADFS is Microsoft’s Federated Identity Service, but if you are reading this you probably know that.
You may also know that the way it does passive authentication doesn’t work well with Ajax calls.

Below is a fiddle that shows the steps involved in authenticating a request for an ASPX page (that page belongs to the awesome Communica) protected by ADFS. This process happens the first time a request is made to an application, after that the authentication information is stored in a cookie (by default named FedAuth if you are using WIF)

Image

Request 37 responds with Http Status code 200 (ok); the response is a payload of secret information that is submitted to the return url (your application) via javascript that submits the returned form (circled above).

This is fine when we are in a browser as the browser runs the javascript happily and submits the form. Everything is dandy. Unfortunately in an Ajax world the first Http Status of 200 is presumed to be the response and the call is never completed as intended.

This is only a problem if your application is entirely an API. If it is a mixed API/Web Application your user will be authenticated when they load the application. Any subsquent call by your application to API endpoints will already have a session and will not perform the ritual outlined above.

However, if like me, you have a 100% API based application and you are attempting to enable SSO with other applications, then there is some work to be done.

In the next posts I’ll look at 3 possible solutions:

Refactoring Definition

Refactoring can only be called refactoring if the *same
tests* can be used to exercise *different implementations* of the
*same behavior*.

Really liked this definition of Refactoring, as apposed to Refuctoring, from the DDD/Cqrs Group
I
t speaks to the scope of what a Unit Test should test… and how!