In the last decade, the options for cloud services have expanded exponentially, leading to a vast array of product offerings and vendors. While the variety and complexity of services offered by cloud providers such as Amazon Web Services (AWS) presents advantages to big teams with economies of scale and scope, these same factors may be unnecessary or even burdensome to smaller teams that seek a more focused and streamlined experience.
Cloud providers can be divided up into two categories: Layer 1 cloud providers and Layer 2 cloud providers. A Layer 1 cloud provider such as Amazon Web Services owns server hardware and sells compute infrastructure as a commodity. A Layer 2 cloud provider purchases compute infrastructure from a Layer 1 provider and builds a high quality industry-specific experience on top of that compute infrastructure.
Escape Technology's Sherpa is a Layer 2 cloud provider for the creative industries, providing an easy to use Platform-as-a-Service for creatives. While Sherpa is in some ways a competitor to large cloud providers for customers who wish to deploy components of the creative pipeline in the cloud, it also builds on the likes of AWS cloud infrastructure and acts as a “middleman” between the cloud provider and the end user. The key bit for a Layer 2 cloud provider is that it brings modular components to a specific industry, rather than a huge toolbox of nuts and bolts; Sherpa is built with the know-how and years of experience of Escape Technology and specifically focuses on the creative industry.
No longer just a buzzword
In the late 2000s, the rise of cloud computing service offerings such as Amazon’s EC2 heralded a shift in how components of a studio's pipeline were deployed and managed over the web. Formerly, so-called “bare metal” servers were built in on-premise data centres that would act as the host for a company’s pipeline. Cloud providers such as AWS offered “virtualization” services, whereby server hardware was partitioned by a hypervisor into virtual machines. Virtualization abstracted away the underlying hardware management; subsequently, cloud providers built infrastructure services to interact with and manage VMs. This gave rise to Infrastructure-as-aService (IaaS) offerings, which provide high-level APIs used to dereference various low-level details of underlying network infrastructure like physical computing resources, location, data partitioning, scaling, security, backup, etc. Cloud providers that focus on the management of hardware up through the IaaS level are called “Layer 1 Cloud Providers”. Layer 1 cloud providers require significant scale and scope to operate effectively due to the complex operational challenges of managing hardware and server operations at a granular level.
Above is a diagram of a burst render architecture deployed on AWS. As studios grow in scale, the number and complexity of the services used can increase dramatically. The advantages of precise configuration and complex architectures are significant for companies with large operations teams. However, growth in the scope of services creates challenges of its own...
“Everything that companies like AWS offer as the wide disparate services that they provide means that you really get a giant kit of solutions to problems that you don’t even know you have sometimes.” (Mark Turner, Heroku)
Standing on the shoulders of giants...
Sherpa is a platform-as-a-service (PaaS) provider that focuses on providing a streamlined experience for deploying and managing anything from a burst render setup to a studio that is born in the cloud. Sherpa allows you to choose which Layer 1 cloud provider to use for the
components of your pipeline, and seeks to act as a layer of abstraction between users and the underlying IaaS providers - managing several of the underlying infrastructural tasks that a studio may face when running pipeline components in the cloud. Essentially, Sherpa is taking on the role of an operations team, doing a lot of things behind the scenes for you. Streamlining the user experience does not translate to simplifying the underlying infrastructure; the task of adapting layer 1 technologies to a layer 2 interface is a difficult engineering task and building a Layer 2 cloud provider is a very different challenge than building a Layer 1 cloud provider. A Layer 1 provider must focus on low level problems such as hardware infrastructure and virtualization. This does not leave much time for focusing on the end user experience. A Layer 1 cloud provider must be able to serve every type of potential software customer. A Layer 2 provider can provide a streamlined experience.
What is the level of complexity you want to get involved with?
While the big studios may find competitive value in the adoption of the newest, shiniest cloud technologies, the operational burden of extra complexity may be unnecessary for organizations without the need or, especially, resources to adopt a highly configured approach. Many creative companies we work with have fewer than 100 employees and I guess that most of those companies can't devote all employees to installing a render manager.
Sherpa comes at an additional marginal cost over what one would pay to use AWS alone. Despite the business and technological challenges of operating in the gap between customers and Layer 1 cloud providers, Sherpa offers a sustainable value proposition to creative teams and companies who seek to reduce overhead, pass off complexity, and allocate resources most effectively towards their core business.
Disclaimer: I stumbled across this great write-up on Heroku, a Layer 2 Cloud provider for (web) developers, by Danny Seymour on Software Engineering Daily, which has been used as a solid base for this post.