Vexor at HighLoad++ 2017

Alexandr Kirillov reported about how to build a cluster to calculate thousands of high-CPU / high-MEM tasks at one of the biggest Russian IT conferences
12 December   3520

The HighLoad++ is professional conference for developers of high-load systems is the key professional event for everyone involved in the creation of large, heavily-frequented, and complex projects.

Main purpose of the event is to exchange knowledge and experience among leading developers of high-performance systems, which support millions of users simultaneously.

Agenda consists of all crucial web development aspects, such as:

  • Large scale architectures,
  • databases and storage systems,
  • system administration,
  • load testing,
  • project maintenance, etc.

This year the conference program will be dazzled with current trends: IoTBlockchainNeural networksArtificial Intelligence, as well as Architecture & Front-end performance.

The 11th HighLoad++ conference took place on the 7th and 8th of November 2017. 

  • 66% of participants work in large companies (of 30+ employees), 
  • 60% earn above the market, 
  • 55% hold leadership positions and have subordinates. 
  • 9% of conference visitors work as technical directors,
  • 12% work as heads of technical departments, and 29% work as lead developers and team leads.

Alexandr Kirillov, CTO at Evrone, had a speech at HighLoad++ 2017 "How to build a cluster to calculate thousands of high-CPU / high-MEM tasks and not go broke"

Alexandr Kirillov at HighLoad++ 2017
Alexandr Kirillov at HighLoad++ 2017
Alexandr Kirillov at HighLoad++ 2017
Alexandr Kirillov at HighLoad++ 2017
Alexandr Kirillov at HighLoad++ 2017
Alexandr Kirillov at HighLoad++ 2017

Our project is a cloud-based CI-service, where users run tests of developed projects.
This year the system of auto purchase of our project purchased 37218 machines (Amazon Instances). This allowed us to process 189,488 "tasks" (test runs) of our customers.

Tests are always resource-intensive tasks with the maximum consumption of processor capacities and memory. We can not predict how many parallel computations and at what point in time it will be. Before us was the task of building the architecture of the system, which can very quickly increase, as well as rapidly reduce the power of the cluster.

All this was complicated by the fact that the resource-intensive calculations did not allow us to use the classic tools AWS or GoogleComputeEngine. We decided to write our own system of automatic scaling, taking into account the requirements of our subject area.

Alexandr Kirillov
CTO, Evrone

At his report, Alexandr told about how they designed and built the architecture of the service, which is the system of automatic procurement of machines.

Additionally, he told more about the main architectural blocks of projects that solve similar problems.

How to set up CI for an Elixir Project?

A step-by-step tutorial on how to set up CI for an Elixir Project
03 August   436
Continuous Integration

The practice of merging all developer working copies to a shared mainline several times a day

Continuous Integration (CI) is the practice of software development, which consists in merging working copies into a common main development branch several times a day and performing frequent automated project assemblies to identify and solve integration problems as quickly as possible. In the usual project, where developers work independently on different parts of the system, the integration stage is final. It can unpredictably delay the completion of work. The transition to continuous integration can reduce the complexity of integration and make it more predictable due to the earliest detection and elimination of errors and contradictions.

CI is an integral part of modern software development process. There are big amount of CI solutions on the market, even cloud-based product is available - Vexor.

If you need to set a CI for an Elixir project, Semaphore can be a solution. With Semaphore, you can easily automate the process of software testing and delivery in the cloud. It is designed to be easy to use and engineered for high performance. Docker support also included. 

In this tutorial, you will learn how to set up continuous integration for an Elixir project using Semaphore. The example project you will be working on is Firestorm, an open-source forum engine, with an Elixir Phoenix backend, and an Elm frontend. It has step-by-step instructions with screenshots and code examples.