TL;DR: The list basically boils down to using strong unique passwords, enabling multi-factor authentication (MFA) at third party providers and moving as many things as possible behind a virtual private network (VPN).

AWS

Run all your infrastructure (EC2, RDS, ElastiCache etc) within VPC.

• Most N-tier web architectures should be provisioned entirely within VPC and should not be publicly addressable.

• Do not enable or use the root Access Key, instead use IAM and follow the best practices.

• Enable multi-factor authentication on the root and all IAM accounts with AWS console access.

Google Apps

A breach of an administrator account could allow an attacker to reset passwords and wreak havoc.

• Enable 2-Step verification for administrator accounts.

• Monitor and enforce strong user passwords.

• Be diligent in suspending or deleting unused accounts.

GitHub

Access to your codebase could allow an attacker to gain insights on how to target attacks on your application.

• Ask every team member to enable 2FA.

• Audit user access; only assign users to repositories they need access to.

Domain Registrar

A breach at your domain registrar account could allow an attacker to hijack your domain name or DNS.

• Use strong passwords and enable MFA.

• Use a registrar lock.

Internal Tools

This includes all employee/customer support tools, continuous integration, staging/demo environments etc. All of these should be placed behind a VPN. OpenVPN is fairly easy to setup.

• Use RBAC for admin/backoffice tools. Only give people the permissions they need to perform their job functions.

• Maintain an audit/activity log of actions taken by admin accounts.

• Enforce strong password requirements for these applications and implement multi-factor authentication where feasible.

• Employees that need VPN access should get their own unique set of credentials.

Your Application

Learn some basic web security.

• Don’t put in version control anything that’s meant to be secret: passwords, API keys etc.

• Use bcrypt to hash user passwords.

• Encrypt any user supplied OAuth tokens, API keys etc.

• Audit your external code dependencies; be mindful of the fact a lot of RubyGems, NPM packages etc aren’t written with YOUR security needs in mind.

Third Party Application Services

Make sure sensitive information is being scrubbed before being sent from your application to third party tools such as the following:

• Application monitoring services - New Relic, Librato, Datadog etc.

• Application logging services - Loggly, Papertrail, Logentries etc.

• Error monitoring services - Airbrake, Bugsnag, Honeybadger etc.

As your startup grows, so too will your security requirements.

Limitations

The official docs note that Sentinel is a work in progress. It’s important to understand Sentinel’s operating parameters and its limitations as a high availability (HA) solution for your particular environment. For that, I defer you to these excellent posts:

• Call me maybe: Redis
• Reply to Aphyr attack to Sentinel
• Asynchronous replication with failover
• Reply to Aphyr, part 2

Setup

A prerequisite for using Sentinel for automatic failover is to have a master-slave replication setup. Both replication and Sentinel are easy to setup and well documented in the official docs.

Once you have everything setup, you then need to configure your Redis clients (web apps, queues etc) to connect to the Sentinel instances instead of directly to a master. The Sentinel instances will then report back to the clients on the appropriate address to use to connect to an available master.

Client Library Support

Redis Sentinel support in the various clients libraries seems limited at this time. Some client libraries have extensions that provide Sentinel support. You could implement Sentinel support in your desired client library by following the Sentinel Client Guidelines.

Listed below are examples for configuring clients when using Ruby & Node.js. The examples assume a YAML config similar to this:

redis:
  master_name: master-name
  sentinels:
    - host: sentinel1.example.com
      port: 26380
    - host: sentinel1.example.com
      port: 26381
    - host: sentinel2.example.com
      port: 26380
    - host: sentinel2.example.com
      port: 26381

master_name should match the name you use in the sentinel.conf and the host & port entries should match the hosts and ports that you run Sentinels on.

Ruby

At the time of writing, the redis-rb client doesn’t yet support Sentinel. redis-sentinel extends redis-rb to provide the necessary support and acts as a drop-in replacement anywhere a redis-rb client is expected.

# Ruby Examples

# Redis.new works as expected with the added benefit of clients being informed
# when a new master is elected
redis_config = YAML.load_file('/path/to/redis_config.yml')['redis']
redis = Redis.new(redis_config)
redis.info

# Configure Sidekiq in Ruby on Rails (config/initializers/sidekiq.rb)
require 'redis/namespace'

redis_config = YAML.load_file('/path/to/redis_config.yml')['redis']
size = options[:size] || (Sidekiq.server? ? (Sidekiq.options[:concurrency] + 2) : 5)
namespace = 'sidekiq'
timeout = 1

Sidekiq.configure_server do |config|
  redis = ConnectionPool.new(:timeout => timeout, :size => size) do
    Redis::Namespace.new(namespace, :redis => Redis.new(redis_config))
  end

  config.redis = redis
end

Sidekiq.configure_client do |config|
  redis = ConnectionPool.new(:timeout => timeout, :size => size) do
    Redis::Namespace.new(namespace, :redis => Redis.new(redis_config))
  end

  config.redis = redis
end

Node.js

At the time of writing, the node_redis client doesn’t yet support Sentinel. node-redis-sentinel extends node_redis to provide the necessary support and acts as a drop-in replacement anywhere a node_redis client is expected.

# CoffeeScript Examples

yaml = require('js-yaml') # https://github.com/nodeca/js-yaml
redis = require('redis') # https://github.com/mranney/node_redis
sentinel = require('redis-sentinel') # https://github.com/ortoo/node-redis-sentinel

redis_config = require('/path/to/redis_config.yml')
masterName = redis_config.master_name
sentinels = redis_config.sentinels
opts = {} # standard node_redis client options

redisClient = sentinel.createClient(sentinels, masterName, opts)

Pow is a simple Rack server that also contains a DNS server capable of port proxying.

These two tools combined allow for easy management of a SOA based local development environment.

Imagine a primary Ruby on Rails application that depends on a couple of other services (Rails, Rack, Sinatra, Faye, Node.js etc) to be fully functional in development. These other services/apps could have their own Gemfile and process dependencies (worker, clock etc) that are also managed locally by Foreman.

Service Oriented Foreman by Matthew Kocher describes an approach that uses Foreman to manage these inter-app processes. Here’s a more complete example of that approach:

# primary_app/Procfile
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec sidekiq -C config/sidekiq.yml
service1: script/run_app service1_app

# primary_app/.foreman
port: 8000

#!/usr/bin/env ruby

# primary_app/script/run_app
require 'bundler'

app_dir = "../#{ARGV[0]}"
cmd = %Q{bash -lc "cd #{app_dir} && bundle exec foreman start"}
Bundler.with_clean_env do
  system({'NOEXEC' => 'skip'}, cmd)
end

# service1_app/Procfile
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb

# service1_app/.foreman
port: 8001

Assuming the Service1 App has the relevant .foreman file then typing foreman start in the Primary App terminal will also start all the associated services specified in the primary Procfile.

In order to easily keep the associated services up-to-date (which may be developed by other teams), it’s nifty to include a helper script in the Primary App that helps make this easy. Here’s an example of what that may look like:

#!/usr/bin/env ruby

# primary_app/script/update_apps
require 'yaml'

{ 'service1_app'      => {},
  'service2_app'      => {:bundle => true, :pow => true},
  'service3_app'      => {:bundle => true, :pow => true, :npm => true}
}.each do |app, opts|

  app_dir = File.expand_path("../#{app}")
  bundle = opts[:bundle]
  npm = opts[:npm]
  pow = opts[:pow]

  bundle_install = bundle ? '&& bundle install' : ''
  npm_install = npm ? '&& npm install' : ''

  puts "=== Updating App: #{app}"
  if File.directory?(app_dir)
    update_apps_cmd =
      %Q{bash -lc "cd #{app_dir} && git pull --rebase #{bundle_install} #{npm_install}"}
  else
    update_apps_cmd = %Q{bash -lc "cd #{File.expand_path('../')} &&
      git clone git@github.com:myorg/#{app}.git &&
      cd #{app_dir} #{bundle_install} #{npm_install}"}
  end
  system(update_apps_cmd)

  if File.exists?("#{app_dir}/script/copy_configs")
    puts "=== Copying example configs for #{app}"
    system(%Q{bash -lc "cd #{app_dir} && script/copy_configs"})
  end

  if pow
    pow_app_config = File.expand_path("~/.pow/#{app}")
    unless File.exists?(pow_app_config)
      puts "=== Creating Pow port proxy config for #{app}"

      port = YAML::load_file("#{app_dir}/.foreman")['port']
      File.open(pow_app_config, 'w') { |f| f.write("#{port}\n") }
    end
  end

end

This setup has been working well enough with just a few services in development and using Webmock to stub out requests in the test environment. Going forward, each service could also provide a private library/gem that mocks its interface out in the test environment.

There are other approaches to managing services in development that are worth evaluating like service stubs and more comprehensively using Vagrant as alluded to in this talk.

I’ve been looking to level up my Chef skills, specifically on the latest best practices. Opscode seems to have an aversion to promulgating current standards and leaving it up to the community to come up with whatever works best for them.

These blog posts and presentations shed light on some good current practices:

• Beginning Chef Antipatterns (video & slides)

• The Berkshelf Way (video & slides)

• How to Write Reusable Chef Cookbooks, Gangnam Style

• Chef: Patterns and Anti-Patterns for Cookbooks, Environments, Roles

• Cookbook Refactoring & Abstracting Logic to RubyGems (video & slides)

• Getting More Chefs in the Kitchen – Developing Infrastructure as a Team (video & slides)

Here’s a summary of some of the patterns, practices & terminology that I have come across that weren’t prevalent back when I first started working with Chef.

Single Repo Per Cookbook

A single-chef-repo with all the cookbooks used to be the norm. There are still benefits to this all in one repo approach - everything is centralized in a single view and it’s easy to search across the cookbooks.

The single-repo-per-cookbook approach on the other hand makes it easier to keep up-to-date with upstream community cookbooks, to open source cookbooks and to a certain extent it just makes more sense for cookbooks to be versioned and managed independently.

A common convention seems to be to have a separate GitHub Org account to host these individual cookbooks, e.g. github.com/opscode-cookbooks. I’m more in favour of just sticking with a primary Github Org account and naming the repos in the form <name>-cookbook.

Data bags, roles, environments and such should remain in a private chef-repo.

Cookbook Bundlers

Berkshelf and Librarian-Chef are cookbook bundlers (in the spirit of Bundler) but that are philosophically very different from each other in how they work.

I think the rule of thumb when deciding on which to adopt depends on how you prefer to view/manage your cookbook infrastructure:

If you’re using the single-repo-per-cookbook approach then Berkshelf is likely a great fit. Make sure to checkout the nifty built-in Vagrant integration that makes the local build test/inspect cycle really easy.

If you’re using the single-chef-repo approach then Librarian-Chef could improve your workflow, just move your existing privately maintained cookbooks into a site-cookbooks directory within your chef-repo. Berkshelf could still work with this approach but it’s definitely not the recommended way.

Cookbook Types

There are a bunch of terms being bandied around to describe the type/flavor of a cookbook. Since there’s no official description for these terms, I thought I’d take a stab at trying to restate some of the descriptions I’ve come across:

Application Cookbook - just what you would imagine it to be, something like a mycompany-app cookbook that installs and configures a complete application (e.g. a web app). It could depend on all other types of cookbooks. Postgresql & Nginx should be considered Application cookbooks too! These cookbooks should have their versions locked at the Environment level.

Library Cookbook - contains Definitions, LWRPs, Libraries that are used by other cookbooks. They may or may not include recipes. The database cookbook is a good example of a Library cookbook. These cookbooks shouldn’t be directly assigned to nodes. Cookbooks that depend on Library cookbooks should lock their required Library versions in their metadata.rb.

Wrapper Cookbook - a more specific type of Application cookbook that depends directly on a single other Application cookbook and potentially Library cookbooks. For example, a hypothetical bitium-phantomjs cookbook could wrap around the phantomjs community cookbook and contain attribute overrides and recipes that orchestrates phantomjs to our company’s specific needs.

Roles Cookbook

I’ve been using roles extensively to maintain run lists. Roles aren’t versioned unlike cookbooks so there’s always the chance a bad change could end up in a production environment by accident.

I’m planning on switching to using a roles cookbook that contains recipes like base.rb & web_server.rb that use include_recipe to define a run list. I’ll still use the native roles but they would just have a call to the corresponding role recipe within its run list. Here’s an example:

# roles-cookbook/metadata.rb
name    'roles'
version '0.1.0'

recipe 'base', 'Base Server Role'
recipe 'web_server', 'Web Server Role'

depends 'hostname'
depends 'nginx'
depends 'ntp'

# roles-cookbook/recipes/base.rb
include_recipe 'hostname::default'
include_recipe 'ntp::default'

# roles-cookbook/recipes/web_server.rb
include_recipe 'roles::base'
include_recipe 'nginx::default'

# roles/base.rb
name 'base'
description 'Base role for all servers'

run_list(
  'recipe[roles::base]'
)

# roles/web-server.rb
name 'web-server'
description 'Web server role'

run_list(
  'recipe[roles::web_server]'
)

Remember to bump up the version of the roles cookbook when making changes and freeze it when uploading to the Chef Server.

The roles cookbook can now be versioned in an environment file:

# environments/production.rb
name 'production'
description 'Production Environment'

cookbook_versions({
  'roles' => '= 0.1.0'
  # ... and other cookbooks
})

You could probably do away completely with the native roles and just use the roles' recipes directly in a node’s run list. I’m going to keep them for now but using them only as described in the examples above.

Private Recipes

This is just a simple convention to use an underscore to prefix a recipe name to mark it as private, e.g. recipes/_common_setup.rb. Some guidelines for when to make a recipe private:

• meant for logical code separation

• included only in recipes from the same cookbook

• not to be included directly in the run list of a node

Cookbook Testing

The testing landscape for Chef has definitely gone through a revolution. There’s now a good selection of tools for every category of tests.

Syntax Checking

• Built-in knife cookbook test.

Linting

• Foodcritic. See also the additional rules by Etsy and CustomInk.

Unit Tests

• ChefSpec. Built on top of RSpec. It performs assertions without actually converging a node so it should be fast. See also Fauxhai for mocking out Ohai attributes in specs.

Integration Tests

• Minitest Chef Handler. Built on top of minitest and works with Vagrant. It does its assertions post convergence, i.e after a Vagrant up/provision run.

• Test Kitchen. Cross platform testing with Vagrant. The 1.0 version (a complete rewrite) is still alpha level and lacking documentation. It uses Vagrant to run Minitest Chef Handler across various platforms, similar to how Travis CI runs tests across different language runtimes. See this talk announcing the 1.0 version.

• Vagrant. Plain old manual testing using the Chef Solo Provisioner.

You can use Strainer to manage your various tests for a cookbook, similar to how Foreman manages application processes.

See here & here for more in-depth coverage on testing.

Cookbook Build System / Pipeline

This just means having an automated system for building (testing, versioning & deploying) your cookbooks.

You could use a CI system like Jenkins with a flow similar to the following:

• Make changes to a cookbook locally.

• Push changes to a remote repo.

• A Jenkins Job runs the cookbook’s tests (possibly using Strainer).

• If all the tests pass, use Knife Spork to bump up the cookbook version, upload & freeze the cookbook on the Chef Server and potentially promote the cookbook into another environment by updating the environment’s cookbook version constraint.

• Potentially invoke a chef-client run on the relevant nodes in a specific environment.

Special mention here for Ridley, a well written Ruby based Chef API client. It’s a useful tool for building out a custom workflow or supporting tools outside of knife. I’ve been using it in Capistrano recipes to help with application deployments and will likely use it in the build system.

Update (June 19): Seth Vargo’s Test Driven Infrastructure with Chef presentation is a good how-to for getting started with a simple build system.