Cloud Programming in F#

Introduction

In the second part of this series, we're going to be looking a cloud programming on F#. This is a big area, so we're going to split it into several broad sections and briefly cover some examples use cases for each of them. It's important to keep in mind that cloud platforms such as Microsoft Azure or Amazon Web Services provide excellent opportunities to rapidly develop systems without having to worry about many of the infrastructural issues that we might be used to, at the cost of giving up some level of control over managing those systems directly. As we'll see, that might be desirable, or not! We'll cover four main areas:

  • Hosting - a high-level overview of the different ways to host services in the cloud.
  • Storage - the ability to store and retrieve data from a central data store at low cost in a flexible manner.
  • Compute - running computational tasks - be they simple functions that run on a daily basis to full blown web applications or distributed systems.
  • Messaging - an illustration of some of the different messaging systems and applications of them that we can use.

Along the way, we'll related these to F# technologies and features, and illustrate how we can map and abstract these services directly into F#.

Hosting

With the advent of the cloud, new ways of deploying and hosting applications have developed, each of which offer trade-offs with regards to costs, performance, flexibility and control.

We can compare and constrast these options in the following table:

Model

Examples

Benefits

Costs

On Premise

Server rack, desktop machines etc.

Total control

Hardware, upgrades / repairs, installation

IAAS

Virtual Machines farm,

Quickly spin up new instances etc.

OS management, application installation and services

PAAS

Azure App Service, Azure Functions, SQL Azure, Elastic Beanstalk etc.

Quickly and easily deploy code and instances, simple model

No access to OS

SAAS

Azure AD, Office 365

Great fit for off-the-shelf products in a managed environment

Constrained model not always suitable for application development

Most likely the "sweet spot" for application developers is a platform service (PAAS) model of some sort. In this model, we're provided with a simplified model for deploying and managing our application code - meaning that we can more readily focus on developing business value, whilst avoiding the costs of OS installations, hardware delivery and costly upgrades etc. There's normally no up-front cost for such services; instead, a consumption-based model is normally adopted, based on various factors such as scale and features.

In the rest of this post, we'll show some examples of platform services on the Microsoft Azure cloud that enable us to quickly and easily create powerful applications that are both flexible and can be readily scaled to meet demand.

Storage

We'll start by looking into some of the managed storage services offered by Azure; there are many others, from big data stores to cache services. All of them have an F# story - here are some of the more popular ones.

All of them can be created in less than 60 seconds (when was the last time you created a SQL Server and Database that quickly!) and offer a wide variety of pricing models to suit your requirements.

Azure Storage

Azure Storage provides a quick and cheap way to easily store large amounts of data centrally. Azure Storage comes with several features, including storing blobs (essentially a secure, simple, scalable file system in the cloud) and tables (a simple and cheap way to store tabular database in a scalable manner).

In addition to the standard .NET SDK for Azure Storage, F# also has two custom APIs - the Azure Storage Type Provider and the F# Azure Storage library. Both offer significant advantages over the standard SDK, including increased type safety by making use of F# features such as discriminated unions and async workflows.

Here's an example of using the F# Storage Type Provider to create a strongly-typed query against a live table without the need to first create any POCOs, whilst at the same time guaranteeing that we won't generate a query with the Table service that could fail at runtime.

1: 
2: 
3: 
4: 
Azure.Tables.employee.Query()
     .``Where Years Working Is``.``Greater Than``(14)
     .``Where Name Is``.``Equal To``("Fred")
     .``Where Is Manager Is``.True()

When coupled with the REPL and scripting capabilities of F#, this provides an extremely compelling experience.

SQL Azure

SQL Azure is a managed service that provides the ability to create, manage and scale a Microsoft SQL-compatible database and server entirely in the cloud. As a managed service, it abstracts you from many concerns that you normally have such as log files etc., and provides simple management features such as vertical scaling on demand, point-in-time restore and encryption at-rest.

In addition to the normal .NET libraries for working with SQL such as Entity Framework and Dapper, F# has a number of custom libraries that take full advantage of the language to provide quick and easy access to SQL. These include the SQL Provider and the SQLClient type providers.

The following example illustrates creating a strongly-typed query that is compile-time checked against SQL using the SQL Client type provider.

Cosmos DB

Cosmos DB is a multi-model database service, offering document, graph and tabular database stores. It's extremely scalable, offering virtually no upper limits whilst offering seamless replication across the globe. Lastly, it's flexible, featuring a number of different consistency models to suit your application needs. F#'s query expressions work seamlessly with DocumentDb's LINQ support, allowing you to create relatively strongly-typed queries:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
let getQueryExpression name = async {
    let! collection = getCollection "LpLibrary" "Lps"
    let query = query {
        for d in client.CreateDocumentQuery<Lp>(collection.DocumentsLink) do 
        where (d.Name = name)
        select d }
    
    match query |> Seq.toList with
    | [] -> return None
    | document :: _ -> return Some document
}

Observe how we use F#'s pattern matching over the query result and map it into a native F# option value.

Compute

Azure has several compute models for hosting and running our application code; each have different strengths and weaknesses; let's review a few of them here.

Virtual Machines

The most obvious option for hosting and running F# applications is via a Virtual Machine hosted in Azure. This is a very safe bet, providing an easy migration path from on-premises models; Azure has a wide variety of virtual machine models, ranging from cheap-and-cheerful single-core VMs to high powered mutli-core machines with 400GB of RAM and SSDs. You can choose from a multitude of OSes and pre-build images, or supply your own. As mentioned earlier though, VMs don't necessarily take advantage of Azure; you can scale them up and down, and manage VNets etc. easily enough.

You might need a VM if you need complete control over deployment and management of your application - but as a software developer, you might not want to focus on those aspects of deployment and management and focus on the coding aspect.

App Service

The Azure App Service is a flexible service that hosts applications with an IIS front-end thrown in for you. It's a perfect fit for ASP .NET applications (including Giraffe), and has excellent support for working with standalone F# applications that respond to HTTP traffic, such as Suave or Freya. App Service has excellent options for the developer - there are multiple options for deployment, turn-key authentication, reporting, scale out and scale up, security and alerting (and more!). In effect, it provides an excellent way for you to host, monitor and manage HTTP-enabled applications. The App Service is also capable of running non-HTTP facing applications - even ones as complex and powerful as the full MBrace runtime!

Functions

We've covered Azure Functions in a recent blog post, but in a nutshell Functions offers you the ability to run individual functions in a decoupled and reactive manner. Unlike the App Service, the pricing model is based on the number of function calls as opposed to uptime - so it's a perfect fit for batch processes or simple functions that operate periodically as a result of external events. F# functions work seamlessly with Azure Functions and are a great fit for deploying your code to (note that there is also support for the same reactive model outside of Azure Functions through the Azure WebJobs SDK).

Service Fabric

Service Fabric is a powerful and scalable compute service that provides the ability to host replicated services in a cluster. Service Fabric also offers a number of in-built services, such as cluster messaging and stateful data replication across nodes - and a natural fit for things like actor models or micro services. Service Fabric works out of the box with F#, including custom services and features including the built-in actor model.

Messaging

We often need a way to coordinate multiple processes together - be they reactive applications processing thousands of messages, or batch processes that need to be run once a day, ever day. Especially important in a distributed environment, we also need some guarantees that our messages are delivered reliably - and that they've been correctly processed! Azure offers a number of messaging systems, of which we'll discuss two here.

Storage Queues

Storage Queues are a simple, secure and low-cost way to work with distributed messages in a reliable manner that live inside Azure Storage Accounts. A message is posted onto the queue, and a receiver polls the queue endpoint to receive the message before processing it (or utilise the WebJobs SDK to seamlessly tie in an F# function to a queue). In this way, you can set up distributed systems with a minimum of effort. Consider using F#'s discriminated unions and pattern matching to handle multiple commands on a single queue; here's a simple example that shows a queue handler for messages controlling a batch process orchestrator:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21: 
22: 
23: 
24: 
open System
open open Newtonsoft.Json

type Command = ReportProgress of jobId:Guid | StartJob | CancelJob of jobId:Guid

let handleMessage message =
    match message with
    | ReportProgress jobId -> () // handle reporting...
    | StartJob -> () // start a new job
    | CancelJob jobId -> () // cancel the job specified

let jobQueue = Azure.Queues.``job-queue``

// Put a StartJob message on a queue, then dequeue it.
async {
    do! StartJob |> JsonConvert.SerializeObject |> jobQueue.Enqueue

    //...later on

    let! message = Azure.Queues.``job-queue``.Dequeue()
    
    match message with
    | None -> printf "Nothing on the queue!"
    | Some message -> JsonConvert.DeserializeObject<Command>(m) |> handleMessage }

Since JSON.Net natively supports F# discriminated unions, the serialization and deserialization occurs seamlessly in the background. Also observe how the F# Storage Type Provider returns an Option value, since we have no way to know if there is actually a message on the queue or not. F# forces us to do the right thing and safely deal with both Some and None cases here.

Service Bus

Azure Service Bus is a dedicated messaging system that provides powerful and flexible messaging capabilities such as pub-sub (with multiple subscribers) as well as session support, which allows modelling actor systems reliably and easily. This can be bound to F#'s native Mailbox Processor which provides a native abstraction over the queue. Indeed, there's an existing open-source implementation of this known as CloudAgent. Here's an example of setting up a CloudAgent listener:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
let connection = ServiceBusConnection "servicebusconnectionstringgoeshere"

// A DTO
type Person = { Name : string; Age : int }

// A function which creates an Agent on demand.
let createASimpleAgent agentId =
    MailboxProcessor.Start(fun inbox -> async {
        while true do
            let! message = inbox.Receive()
            printfn "%s is %d years old." message.Name message.Age })

// Create a worker cloud connection
let cloudConnection = WorkerCloudConnection(connection, Queue "myMessageQueue")

// Start listening! A local pool of agents will be created that will receive messages.
ConnectionFactory.StartListening(cloudConnection, createASimpleAgent >> BasicCloudAgent)

and now the associated client:

1: 
2: 
3: 
4: 
5: 
6: 
let sendToMyMessageQueue = ConnectionFactory.SendToWorkerPool cloudConnection

// These messages will be processed in parallel across the worker pool.
sendToMyMessageQueue { Name = "Isaac"; Age = 34 }
sendToMyMessageQueue { Name = "Michael"; Age = 32 }
sendToMyMessageQueue { Name = "Sam"; Age = 27 }

If you need more a powerful queue model than Storage Queues, Service Bus provides the flexibility you'll need, whilst F#'s language capabilities means that you can easily find natural abstractions over Azure primitives that fit your domain.

Why F# on Azure?

We've looked at a number of different services in Azure, from hosting and computations to messaging and data services. Many of the reasons for using F# in general - such as correctness, rapid development and prototyping, and powerful data access - are amplified when using Azure and AWS. For example, the ability to use the REPL to interact with live data services in the cloud can drastically reduce the time required to perform many tasks, whilst F#'s ability to create reliable and consistent code that can easily operate across multiple threads is a natural fit for cloud-hosted solutions. Also, remember that all Azure services that support .NET implicitly support F# through F#'s excellent C# interop, whilst a number of services also have bespoke F# APIs that provide an even more seamless experience.

Cloud providers such as Microsoft Azure offers a growing number of valuable platform services that can be cheaply and quickly employed and then utilised. These services enable rapid application development and deployment at low cost, with minimal upfront costs. When coupled with F#'s powerful type system, scripting capabilities and impressive collection of libraries, Azure becomes an even more compelling package.

val getQueryExpression : name:'a -> Async<'b option> (requires equality)

Full name: safecloud.getQueryExpression
val name : 'a (requires equality)
val async : AsyncBuilder

Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.async
val collection : obj
val query : System.Linq.IQueryable<'b>
val d : 'b
custom operation: where (bool)

Calls Linq.QueryBuilder.Where
custom operation: select ('Result)

Calls Linq.QueryBuilder.Select
module Seq

from Microsoft.FSharp.Collections
val toList : source:seq<'T> -> 'T list

Full name: Microsoft.FSharp.Collections.Seq.toList
union case Option.None: Option<'T>
val document : 'b
union case Option.Some: Value: 'T -> Option<'T>
namespace System
type Command =
  | ReportProgress of jobId: Guid
  | StartJob
  | CancelJob of jobId: Guid

Full name: safecloud.Command
union case Command.ReportProgress: jobId: Guid -> Command
Multiple items
type Guid =
  struct
    new : b:byte[] -> Guid + 4 overloads
    member CompareTo : value:obj -> int + 1 overload
    member Equals : o:obj -> bool + 1 overload
    member GetHashCode : unit -> int
    member ToByteArray : unit -> byte[]
    member ToString : unit -> string + 2 overloads
    static val Empty : Guid
    static member NewGuid : unit -> Guid
    static member Parse : input:string -> Guid
    static member ParseExact : input:string * format:string -> Guid
    ...
  end

Full name: System.Guid

--------------------
Guid()
Guid(b: byte []) : unit
Guid(g: string) : unit
Guid(a: int, b: int16, c: int16, d: byte []) : unit
Guid(a: uint32, b: uint16, c: uint16, d: byte, e: byte, f: byte, g: byte, h: byte, i: byte, j: byte, k: byte) : unit
Guid(a: int, b: int16, c: int16, d: byte, e: byte, f: byte, g: byte, h: byte, i: byte, j: byte, k: byte) : unit
union case Command.StartJob: Command
union case Command.CancelJob: jobId: Guid -> Command
val handleMessage : message:Command -> unit

Full name: safecloud.handleMessage
val message : Command
val jobId : Guid
val jobQueue : obj

Full name: safecloud.jobQueue
val message : obj option
val printf : format:Printf.TextWriterFormat<'T> -> 'T

Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.printf
val message : obj
val connection : obj

Full name: safecloud.connection
type Person =
  {Name: string;
   Age: int;}

Full name: safecloud.Person
Person.Name: string
Multiple items
val string : value:'T -> string

Full name: Microsoft.FSharp.Core.Operators.string

--------------------
type string = String

Full name: Microsoft.FSharp.Core.string
Person.Age: int
Multiple items
val int : value:'T -> int (requires member op_Explicit)

Full name: Microsoft.FSharp.Core.Operators.int

--------------------
type int = int32

Full name: Microsoft.FSharp.Core.int

--------------------
type int<'Measure> = int

Full name: Microsoft.FSharp.Core.int<_>
val createASimpleAgent : agentId:'a -> MailboxProcessor<Person>

Full name: safecloud.createASimpleAgent
val agentId : 'a
Multiple items
type MailboxProcessor<'Msg> =
  interface IDisposable
  new : body:(MailboxProcessor<'Msg> -> Async<unit>) * ?cancellationToken:CancellationToken -> MailboxProcessor<'Msg>
  member Post : message:'Msg -> unit
  member PostAndAsyncReply : buildMessage:(AsyncReplyChannel<'Reply> -> 'Msg) * ?timeout:int -> Async<'Reply>
  member PostAndReply : buildMessage:(AsyncReplyChannel<'Reply> -> 'Msg) * ?timeout:int -> 'Reply
  member PostAndTryAsyncReply : buildMessage:(AsyncReplyChannel<'Reply> -> 'Msg) * ?timeout:int -> Async<'Reply option>
  member Receive : ?timeout:int -> Async<'Msg>
  member Scan : scanner:('Msg -> Async<'T> option) * ?timeout:int -> Async<'T>
  member Start : unit -> unit
  member TryPostAndReply : buildMessage:(AsyncReplyChannel<'Reply> -> 'Msg) * ?timeout:int -> 'Reply option
  ...

Full name: Microsoft.FSharp.Control.MailboxProcessor<_>

--------------------
new : body:(MailboxProcessor<'Msg> -> Async<unit>) * ?cancellationToken:Threading.CancellationToken -> MailboxProcessor<'Msg>
static member MailboxProcessor.Start : body:(MailboxProcessor<'Msg> -> Async<unit>) * ?cancellationToken:Threading.CancellationToken -> MailboxProcessor<'Msg>
val inbox : MailboxProcessor<Person>
val message : Person
member MailboxProcessor.Receive : ?timeout:int -> Async<'Msg>
val printfn : format:Printf.TextWriterFormat<'T> -> 'T

Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.printfn
val cloudConnection : obj

Full name: safecloud.cloudConnection
val sendToMyMessageQueue : (Person -> obj)

Full name: safecloud.sendToMyMessageQueue