Facebook to Open Source Code of Linux Kernel Tools

The tools are designed to solve some problems with managing large streams of data
01 November 2018   1509

Facebook has published code for six components of the Linux kernel, designed to solve some problems with managing large streams of data. These include the Btrfs file system, a system for measuring the state of a memory device and a CPU, as well as a mechanism for structuring workload and resource allocation.

More about tools:

  • BPF is a flexible and efficient mechanism for executing code in the Linux kernel. It allows you to provide for different code different behavior of the kernel. Facebook exemplifies the ability to filter network packets, as well as running certain processes in the sandbox.
  • Btrfs is a file system that works on the principle of copying when writing. It focuses on advanced feature implementation, fault tolerance, recovery, and simple administration. The system is designed to manage large storage subsystems, supports the creation of snapshots and online defragmentation.
  • Netconsd is the “daemon” of the Netconsole module for sending kernel log messages. It  presents them in a structured form, which makes it easier for engineers to find errors and problems.
  • Cgroup2 is a mechanism for grouping and structuring workloads, as well as for controlling the amount of system resources associated with each group. In addition, it helps isolate workloads, prioritize and tune resource allocation.
  • PSI (Pressure Stall Information) - a system for measuring the load on the main resources. It provides data on memory, I / O, and CPU devices. These load metrics, coupled with other tools that work in the kernel or user space, allow you to fix the lack of resources on time.
  • Oomd is a user-space tool for forcing the process to terminate in the event of an “Out of memory” error. It takes into account its context and priority among other processes running in the system. Oomd also allows you to customize the automatic response of the system to situations with low memory.

All open source Facebook projects, including the Linux components presented, can be viewed on the relevant site.

In May 2018, the company published the Katran library, a software network load balancer in which the BPF mechanism described above plays a leading role.

LLVM 10.0.0 to be Released

New version of the popular development toolkit brings, among other things, support for the C++ Concepts
26 March 2020   955

After six months of development, the release of the LLVM 10.0 project, a GCC-compatible toolkit (compilers, optimizers, and code generators), compiling programs into an intermediate bitcode of RISC-like virtual instructions (a low-level virtual machine with a multi-level optimization system), is presented. The generated pseudo-code can be converted using the JIT compiler into machine instructions directly at the time of program execution.

Among the new features of LLVM 10.0, there are support for C ++ Concepts (C ++ Concepts), termination of the launch of Clang in the form of a separate process, support for CFG checks (control flow guard) for Windows, and support for new CPU features.

The main innovations of LLVM 10.0:

  • New interprocedural optimizations and analyzers have been added to the Attributor framework. The prediction of the state of 19 different attributes, including 12 attributes of 12 LLVM IR and 7 abstract attributes such as liveness, is provided.
  • New built-in compiler matrix mathematical functions (Intrinsics) have been added, which, when compiled, are replaced by effective vector instructions.
  • Numerous improvements to the backends for the X86, AArch64, ARM, SystemZ, MIPS, AMDGPU, and PowerPC architectures. Added support for Cortex-A65, Cortex-A65AE, Neoverse E1 and Neoverse N1 CPUs. For ARMv8.1-M, ​​the code generation process has been optimized (for example, support for loops with minimal overhead has appeared) and support for auto-vectorization using the MVE extension has been added. Improved support for CPU MIPS Octeon. PowerPC includes vectorization of mathematical routines using the MASSV (Mathematical Acceleration SubSystem) library, improved code generation, and optimized memory access from loops. For x86, the processing of vector types v2i32, v4i16, v2i16, v8i8, v4i8 and v2i8 has been changed.
  • Improved code generator for WebAssembly. Added support for TLS (Thread-Local Storage) and atomic.fence instructions. Significantly expanded support for SIMD. WebAssembly object files added the ability to use function signatures with multiple values.
  • When processing cycles, the MemorySSA analyzer is used to determine the dependencies between different memory operations. MemorySSA can reduce compilation and execution time, or can be used instead of AliasSetTracker without sacrificing performance.
  • The LLDB debugger has significantly improved support for the DWARF v5 format. Improved build support with MinGW and added the initial ability to debug Windows executable files for ARM and ARM64 architectures. Added descriptions of options offered when autocompleting input by pressing tabs.
  • Enhanced LLD Linker Features. Improved support for the ELF format, including full compatibility of glob templates with the GNU linker, added support for the compressed debug sections ".zdebug", added the PT_GNU_PROPERTY property to determine the .note.gnu.property section (can be used in future Linux kernels), implemented modes "-z noseparate-code", "-z separate-code" and "-z separate-loadable-segments". Improved support for MinGW and WebAssembly.

Get more at the release notes.