Facebook to Open Source Code of Linux Kernel Tools

The tools are designed to solve some problems with managing large streams of data
01 November 2018   390

Facebook has published code for six components of the Linux kernel, designed to solve some problems with managing large streams of data. These include the Btrfs file system, a system for measuring the state of a memory device and a CPU, as well as a mechanism for structuring workload and resource allocation.

More about tools:

  • BPF is a flexible and efficient mechanism for executing code in the Linux kernel. It allows you to provide for different code different behavior of the kernel. Facebook exemplifies the ability to filter network packets, as well as running certain processes in the sandbox.
  • Btrfs is a file system that works on the principle of copying when writing. It focuses on advanced feature implementation, fault tolerance, recovery, and simple administration. The system is designed to manage large storage subsystems, supports the creation of snapshots and online defragmentation.
  • Netconsd is the “daemon” of the Netconsole module for sending kernel log messages. It  presents them in a structured form, which makes it easier for engineers to find errors and problems.
  • Cgroup2 is a mechanism for grouping and structuring workloads, as well as for controlling the amount of system resources associated with each group. In addition, it helps isolate workloads, prioritize and tune resource allocation.
  • PSI (Pressure Stall Information) - a system for measuring the load on the main resources. It provides data on memory, I / O, and CPU devices. These load metrics, coupled with other tools that work in the kernel or user space, allow you to fix the lack of resources on time.
  • Oomd is a user-space tool for forcing the process to terminate in the event of an “Out of memory” error. It takes into account its context and priority among other processes running in the system. Oomd also allows you to customize the automatic response of the system to situations with low memory.

All open source Facebook projects, including the Linux components presented, can be viewed on the relevant site.

In May 2018, the company published the Katran library, a software network load balancer in which the BPF mechanism described above plays a leading role.

Facebook to Release PyTorch 1.0

This release added support for large cloud platforms, a C ++ interface, a set of JIT compilers
10 December 2018   102

Facebook has released a stable version of the library for machine learning PyTorch 1.0. This iteration added support for large cloud platforms, a C ++ interface, a set of JIT compilers, and various improvements.

The stable version received a set of JIT compilers that eliminate the dependence of the code on the Python interpreter. The model code is transformed into Torch Script - a superstructure over Python. Keeping the opportunity to work with the model in the Python environment, the user can download it to other projects not related to this language. So, the PyTorch developers state that the code processed in this way can be used in the C ++ API.

The torch.distributed package and the torch.nn.parallel.DistributedDataParallel module are completely redesigned. torch.distributed now has better performance and works asynchronously with the Gloo, NCCL and MPI libraries.

The developers added a C ++ wrapper to PyTorch 1.0. It contains analogs of Python interface components, such astorch.nn,torch.optim, torch.data. According to the creators, the new interface should provide high performance for C ++ applications. True, the C ++ API is still experimental, but it can be used in projects now.

To improve the efficiency of working with PyTorch 1.0, a Torch Hub repository has been created, which stores pre-trained models of neural networks. You can publish your own development using the hubconf.py file, after which the model will be available for download by any user via the torch.hub.load API.

Support for C extensions and the module torch.utils.trainer were removed from the library.

Facebook released the preliminary version of PyTorch 1.0 at the beginning of October 2018, and in two months the developers brought the framework to a stable state.