My Background

Final year B.Eng (Software) candidate

Summer Software Imaging Assistant

Summer Software Packager

Intern in Release Engineering (RelEng)

Mentored by Rail Aliiev

Release Engineering in a Nutshell

"Release Engineering is to software engineering, as manufacturing is to an industrial process."
- Boris Debic, Google Inc.

public class HelloWorld{
	public static void main(String[] args){
		System.out.println("Hello, world!");
	}
}
						

• Creating the source-to-product pipeline
  • Building
  • Signing
  • Packaging
  • Updating
  • Testing...
• Across multiple OS...
• Across multiple platforms...
• Supporting multiple product versions...

Previous Release Process

• Old process has two set of builds: CI and Release
• Release builds started manually with modified build configs
• These builds do not go through CI testing
• Still used by Fennec and Thunderbird

Promotion

• 'Promote' passing CI builds to users
• Eliminates builds, reducing time-to-ship
• Better visibility, tasks report to Treeherder
• Shipped product is exactly what has been tested

Pulse-Notify

Purpose

• Re-implement notifications for new process
• When things go wrong, make sure people know!
• Collect all log data to a single location

Technical Details

• Asynchronous Python 3.5
• Built on top of Taskcluster, Mozilla Pulse
• Uses dependency injection with drop-in functionality
• Deployed in Heroku

Listen for Release events on Pulse

• Declare a queue to consume messages from
• Routing key "route.index.releases.v1.#" catches all events related to releases
• Bind the exchanges to the queue with the releases key
  • exchange/taskcluster-queue/v1/artifact-created
  • exchange/taskcluster-queue/v1/task-completed
  • exchange/taskcluster-queue/v1/task-failed
  • exchange/taskcluster-queue/v1/task-exception
• Create consumer object, bind dispatch method

A wild message appears!

1. Consumer calls dispatch method
2. Task status is pulled from exchange name
3. Task is retrieved from taskcluster
4. Collect notification info from task definition
5. Notify based on this configuration

{
  "provisionerId": "aws-provisioner-v1",
  "workerType": "opt-linux64",
  "schedulerId": "task-graph-scheduler",
  "taskGroupId": "HZjhv3tTShGzLEZypjvoXA",
  "dependencies": [
    "yA3WvU7GQ8abiADJLYTChw"
  ],
  "requires": "all-completed",
  "routes": [
    "tc-treeherder-stage.jamun.73e28b00b4dd8a3dc4d17da05bebd9e0cfab159a",
    "index.releases.v1.jamun.6b77678f82113030e95a052c80c3afc5cdd8148c.firefox.45_2_5esr.build2.partials_beetmover.45_0_2esr.macosx64.6",
    "index.releases.v1.jamun.latest.firefox.latest.partials_beetmover.45_0_2esr.macosx64.6"
  ],
  "priority": "high",
  "retries": 5,
  "created": "2016-07-07T16:58:33.871Z",
  "deadline": "2016-07-11T16:58:33.871Z",
  "expires": "3016-07-07T16:58:33.939Z",
  "scopes": [],
  "payload": {
    "maxRunTime": 7200,
    "encryptedEnv": [
      "wcBMA1yCZyapHLXMAQgAkF1KATmcidCdu5qLSCjD8ydiQ7lMyNdFYWDqYXPiIxeIRjVdbPznnYHuU9O+JuqybBjU6ZGSEcJ7qexgwlrVlaDQKh/gJbNXELA1lkaX86koiMjI2lAc+pqDEKyUNp3t8ze5jpwhyYj0H498seof8DVOygLC37mV6pQzSCnoRDny5jwthPb0wGrxl7HqXr9qFixGcGvtjUzbdJ2ZaE/X2K/gzkxSMIVOKkUKgnRbvnZXq7FGGbtBuOXqEdCIGhF43IDwbENmTCBFy/Z66EQtmPfQAyH218b+XXnP90/B5jagVGjkpNJ8zwrrJNUouK2M/b7aRdlTNbgGmKl2tEdMvtLACAFrlLTJtu4Ch16WuxIZ5IQ/16McbYBVA+p2mbMlJIxKKbiWf8Z1pPjg867eGYcXr3mCnv4STj/As0PSIj856wo9H3eMHDPRwcMP7xgcbo7YJBe8JZLUV6Th3rAIDy8bcQFYWF5TFTqeqJSMMq/WShbJfGDuG4wbTSRyUiQ13GCVoDl3uPDTA0Z8ZyupGhew6SwCuERfB8zd6Kvk6fqq8z/bCvPeaBndSSe2TeOpjgE88Pi4Ofe4WaHxuNd5s2PVr/+WLmUs0xqs",
      "wcBMA1yCZyapHLXMAQgAmzB0h5CxoSWSehZ0oCMGpaDE9qbh1Imz2xesUZvCK2me/JYJpxtsHGwEBDd0cBFVrgtwhhfAlI2LEUAZjyq5SbgA5ffEFD/XhmbOXyw7R3GGL2SRrnlZ5k4opL2HhN9EvvjHFZ6lL5o79CL1KsFW2vJDBql1jxKSAeNNv6n9dvwmIK+9QCvLJ8PuyQm0EO/Qm146f6pGXzCIbEZf3qZhxoSja1d1rnEwrD3rB0aU0YyWMnAoNllaO6dYmpF5Rp3CLzCmL+APZ7IahCYu9QDdwmD6BQvH8surRJ88C+bUZTEqu1L6kjKnIs4/q70s/jjMXGdpLHv4AURTdMge5uVazdLAIQGUYkPYQSnf4/PXM0e83M/F6miOJ71CXXokXrpJMV4sXHwxdPR7c6h+ctg6qNDYPs+cgEAF8wknEJkIhlrdi5KOoylOz/WmBbjaF1xv60PGQw9MsoICpJQq91A7LiK66CH3LPUlQgFI9hgcdSYiisJpnJmRPnFxp41KPr78UW8Y4O5suDP1HaB/5qPatNoUXYcA2J4KvSMF0rIMFkSlVJPmTqSZpdCWwquiqwKdlJ8DvZKLhnI+kag2Mf5zKRTS1Ikc7oSjnT/rgRHOD5DhiCwJuv0a6Iglu/Dwe5pYTRYurg=="
    ],
    "image": {
      "path": "public/image.tar",
      "type": "task-image",
      "taskId": "cVtZ8mnLR_ug38pq6YNlqQ"
    },
    "command": [
      "/bin/bash",
      "-c",
      "wget -O mozharness.tar.bz2 https://hg.mozilla.org/projects/jamun/archive/6b77678f82113030e95a052c80c3afc5cdd8148c.tar.bz2/testing/mozharness && mkdir mozharness && tar xvfj mozharness.tar.bz2 -C mozharness --strip-components 3 && cd mozharness && python scripts/release/beet_mover.py --template configs/beetmover/partials.yml.tmpl --platform mac --version 45.2.5esr --partial-version 45.0.2esr --artifact-subdir env --locale id --locale is --locale it --locale ja-JP-mac --locale kk --locale km --locale kn --locale ko --locale lij  --taskid l4YeP37JROavj20wsNUedw --build-num build2 --bucket net-mozaws-stage-delivery-firefox --no-refresh-antivirus\n"
    ],
    "env": {
      "DUMMY_ENV_FOR_ENCRYPT": "fake"
    }
  },
  "metadata": {
    "owner": "release@mozilla.com",
    "source": "https://github.com/mozilla/releasetasks",
    "name": "[beetmover] firefox jamun macosx64 locales partials candidates 6/10",
    "description": "moves partial artifacts for locale based builds to candidates dir"
  },
  "tags": {},
  "extra": {
    "build_props": {
      "product": "firefox",
      "build_number": 2,
      "platform": "macosx64",
      "partials": "45.0.2esrbuild2",
      "version": "45.2.5esr",
      "branch": "jamun",
      "mozharness_changeset": "6b77678f82113030e95a052c80c3afc5cdd8148c",
      "locales": [
        "id",
        "is",
        "it",
        "ja-JP-mac",
        "kk",
        "km",
        "kn",
        "ko",
        "lij"
      ],
      "revision": "6b77678f82113030e95a052c80c3afc5cdd8148c"
    },
    "notifications": {
      "task-failed": {
        "message": "Uh-oh! [beetmover] firefox jamun macosx64 locales partials candidates 6/10 failed.",
        "ids": [
          "releng"
        ],
        "subject": "[beetmover] firefox jamun macosx64 locales partials candidates 6/10 failed"
      },
      "task-completed": {
        "message": "[beetmover] firefox jamun macosx64 locales partials candidates 6/10 has completed successfully! Yay!",
        "ids": [
          "releng"
        ],
        "subject": "[beetmover] firefox jamun macosx64 locales partials candidates 6/10 completed"
      },
      "task-exception": {
        "message": "Uh-oh! [beetmover] firefox jamun macosx64 locales partials candidates 6/10 resulted in an exception.",
        "ids": [
          "releng"
        ],
        "subject": "[beetmover] firefox jamun macosx64 locales partials candidates 6/10 exception"
      }
    },
    "signing": {
      "signature": "eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE0Njc5MTA3MjIsInZlcnNpb24iOiIxIiwidGFza0lkIjoieUEzV3ZVN0dROGFiaUFESkxZVENodyIsImV4cCI6MTQ2ODI1NjMyMn0.Yn-QBiItHxLN_P6UIOn_ZfdCTVENU8T8JQndxVmJ46f4MGMemQ5OzeSMg0H3um7y9ze6OcjpTlggT4XN17whpySWWo5A6XrSm8IiOx2wkVeYuh82N5hr7fDcdOTI4Kg5ZV-K5GFo3hY-5nOO85k4Dj_PDs6t72Tyl2bDqlpJQsMw9YR2udgqQ_DrLxD-sIuAtVBO87vjxoox7dMFYvJvJ0ZhaAIANohwlVS1qIG6GgQjDDXSusDutB_xkJPZwCzergFWEV6UJ0u5NB2yxvFC1ABLt5Fa7wJgp5dFKK9natLN8EpTP27tm8i2wF1psIlyOtD2MQ0v2qBtSjupv8QdaA"
    },
    "treeherder": {
      "machine": {
        "platform": "osx-10-10"
      },
      "symbol": "l10n-6-45.0.2esr",
      "groupSymbol": "BM",
      "collection": {
        "opt": true
      },
      "build": {
        "platform": "osx-10-10"
      }
    },
    "treeherderEnv": [
      "staging"
    ]
  }
}
						

Plugins

Plugins

• Defines a single notification
• A file with class Plugin, inherited from BasePlugin
• Override a method notify that is called by the consumer

Initializing Objects

services_list = os.environ['PN_SERVICES'].split(':')

#  notifiers contains objects representing a plugin's method of notification
self.notifiers = {}
for service in services_list:
    try:
        self.notifiers[service] = import_module('pulsenotify.plugins.' + service).Plugin()
    except ImportError:
        log.exception('No plugin named %s, initialization failed', service)
						

ses

• Sends emails via Amazon SES
• Uses Jinja2 template

log_collect

• Collects and stores release logs
• A single Amazon S3 bucket for easy access
• URL builder built in to BasePlugin

IRC

• Pushes notifications to IRC channels
• Colour coded by task status
  • completed
  • failed
  • artifacts
  • exception

IRC

Integrating Into the Existing Pipeline

Releasetasks

• Library for creating release related task graphs
• Loads from a YAML config
• Generates JSON task definitions with Jinja2 template engine

Notification Macro

{% macro task_notifications(taskname, failed=None, exception=None, artifact=None, completed=None) %}
{% include "notifications.yml.tmpl" %}
{% endmacro %}
						

{{ task_notifications(taskname="[beetmover] {} {} {} en_US completes candidates".format(product, branch, platform), completed=["releasetasks"], failed=["releasetasks"], exception=["releasetasks"]) | indent(12) }}
						

IDs

Too much repetition

• A lot of repeated data within the task
• Calls to the macro were messy and long
• Needed cleaner way to default

IDs

• Pre-configured notification info
• Assigned a name
• Service will overwrite data to notify based on this config
• Notifications without ID would still work

ID YAML File

id1:
    plugins:
        - irc
        - ses
    channels:
        - "#chan1"
    emails:
        - example@mozilla.com
id2:
    plugins:
        - log_collect
						

Example Config from Task

"task-failed": {
	"message": "The task failed! Uh-oh...",
	"plugins": [
	  "irc",
	  "smtp",
	],
	"nicks": [
	  "sheehan",
	  "rail",
	],
	"emails": [
	  "someone@mozilla.com",
	],
	"ids": [
	    "id1",
	    "id2",
	]
}
						

Config for id1

"task-failed": {
    "message": "The task failed! Uh-oh...",
    "plugins": [
      "irc",
      "ses",
    ],
    "nicks": [
      "sheehan",
      "rail",
    ],
    "emails": [
      "example@mozilla.com",
    ],
    "channels": [
      "#chan1"
    ],
    "ids": [
        "id1",
        "id2",
    ]
}
						

Config for id2

"task-failed": {
    "message": "The task failed! Uh-oh...",
    "plugins": [
      "log_collect",
    ],
    "nicks": [
      "sheehan",
      "rail",
    ],
    "emails": [
      "example@mozilla.com",
    ],
    "channels": [
      "#chan1"
    ],
    "ids": [
        "id1",
        "id2",
    ]
}
						

New Python 3.5 Syntax!

merged_dict = dict1.copy()
merged_dict.update(dict2)
merged_dict.update(dict3)
						

merged_dict = {**dict1, **dict2, **dict3}
						

Takeaways

Met some cool people!