When you run queries in ClickHouse, you might notice that the profiler often shows the LZ_decompress_fast function near the top. What is going on? This question had us wondering how to choose the best compression algorithm.

ClickHouse stores data in compressed form. When running queries, ClickHouse tries to do as little as possible, in order to conserve CPU resources. In many cases, all the potentially time-consuming computations are already well optimized, plus the user wrote a well thought-out query. Then all that's left to do is to perform decompression.



So why does LZ4 decompression becomes a bottleneck? LZ4 seems like an extremely light algorithm: the data decompression rate is usually from 1 to 3 GB/s per processor core, depending on the data. This is much faster than the typical disk subsystem. Moreover, we use all available CPU cores, and decompression scales linearly across all physical cores.

JavaScript developers, I am working on an exciting opensource project pursuing two goals:

1. Learning best practices in JavaScript/NodeJS
2. Helping developers and myself to develop and launch MVPs to validate ideas quickly.

As developers, we have tons of ideas and would be awesome to have a simple tool to scaffold a secure project quickly, add a couple of forms, some project specific logic, and here you go — deploy and test your idea.

If you were asked to draw a picture of several people in ski gear, standing in the snow, chances are you’d start with an outline of three or four people reasonably positioned in the center of the canvas, then sketch in the skis under their feet. Though it was not specified, you might decide to add a backpack to each of the skiers to jibe with expectations of what skiers would be sporting. Finally, you’d carefully fill in the details, perhaps painting their clothes blue, scarves pink, all against a white background, rendering these people more realistic and ensuring that their surroundings match the description. Finally, to make the scene more vivid, you might even sketch in some brown stones protruding through the snow to suggest that these skiers are in the mountains.

Now there’s a bot that can do all that.

Since its completion more than 675 years ago, the medieval cathedral of Notre-Dame has captivated millions of people with its incomparable beauty. From its legendary stained glass rose window to its towering spire, it’s widely regarded as one of the most stunning examples of medieval architecture in history.

Hi everyone!

I like Tox and respect the participants of this project and their work. In an effort to help Tox developers and users, I looked into the code and noticed potential problems that could lead to a false sense of security. Since I originally published this article in 2016 (in Russian), many improvements have been made to Tox, and I lead a team that re-wrote secure Tox software from scratch using the Rust programming language (check out Tox-rs). I DO recommend using tox in 2019. Let's take a look what actually made us rewrite Tox in Rust.

Original article of 2016

There is an unhealthy tendency to overestimate the security of E2E systems only on the basis that they are E2E. I will present objective facts supplemented with my own comments for you to draw your own conclusions.

Spoiler: The Tox developers agree with my points and my source code pull request was accepted.

This week I spent coding my very first public pet-app based on Telegram chat bot which acts as a Bitcoin wallet and allows to send and receive tips between Telegram users and other so-called “Lightning Apps”. I assume that you are familiar with Bitcoin & Telegram in general, i’ll try to post short and without deep jump into details. More resources about Bitcoin can be found here and Telegram is simply an instant messenger that allows you to create your custom apps (chat-bots) using their platform.

What are the key points of such app?

• Allows to rate other users ideas and answers with real value instead of
  ‘virtual likes’. This brings online conversation to completely new level
• Real example of working micro-payment app which can act with other entities
  over internet using open protocol
• All the modules are open-source projects and can be easy re-used and adjusted
  for your own project. App does not relay on third-party commercial services.
  Even it falls under e-commerce field, which is currently almost closed, the app
  is based on open solutions.

What are the use-cases?

something like this…

It's been two years since we last checked the code of the LLVM project with PVS-Studio, so let's see if PVS-Studio is still the leader among tools for detecting bugs and security weaknesses. We'll do that by scanning the LLVM 8.0.0 release for new bugs.

You must have already guessed from the title that today's article will be focusing on bugs in software source code. But not only that. If you are not only interested in C++ and in reading about bugs in other developers' code but also dig unusual video games and wonder what «roguelikes» are and how you play them, then welcome to read on!

Free Wireguard VPN service on AWS

The reasoning

The increase of Internet censorship by authoritarian regimes expands the blockage of useful internet resources making impossible the use of the WEB and in essence violates the fundamental right to freedom of opinion and expression enshrined in the Universal Declaration of Human Rights.

Article 19
Everyone has the right to freedom of opinion and expression; this right includes freedom to hold opinions without interference and to seek, receive and impart information and ideas through any media and regardless of frontiers.

The following is the detailed 6 steps instruction for non-IT people to deploy free* VPN service upon Wireguard technology in Amazon Web Services (AWS) cloud infrastructure, using a 12 months free account, on an Instance (virtual machine) run by Ubuntu Server 18.04 LTS.

I tried to make this walkthrough as friendly as possible to people far from IT. The only thing required is assiduity in repeating the steps described below.

Java developers have access to a number of useful tools that help to write high-quality code such as the powerful IDE IntelliJ IDEA, free analyzers SpotBugs, PMD, and the like. The developers working on CUBA Platform have already been using all of these, and this review will show how the project can benefit even more from the use of the static code analyzer PVS-Studio.

This is the second post in our series of articles about the results of checking open-source software working with the RDP protocol. Today we are going to take a look at the rdesktop client and xrdp server.

 Intro

It is a well-known fact to many users of the Windows version of VirtualBox (from now on, VB; not to be confused with Visual Basic) that starting with 4.3.14 the developers added the so-called «hardening» designed to prevent malicious injections into VB. Although the intentions were good, the implementation happened to cause numerous conflicts with totally legitimate products such as antiviruses, cryptographic modules and even some updates of the Windows itself, and when such a conflict occurs VB simply stops working. Users have to wait for at least a month till the new VB version is released with the proper exclusions added. Worst case is, the conflicting application or update has to be uninstalled, or VB itself has to be downgraded to the version 4.3.12 which was the latest one without hardening. Numerous requests to add a user-controlled exclusion list, or an option to disable hardening, are all left unanswered. The only reply from developers sounds like «if you don't want it build it from source code yourself». Well, looks like we'll have to.

Although the build instructions are described on the official project Wiki, they are incomplete and somewhat outdated, while the build procedure often fails with vague error messages. So when, in the end, I got it working I thought it was worth documenting in full details in a separate article. This instruction is being updated from time to time, and at the moment it is adapted to building VB version 6.1.18. However, if you need information on building earlier versions of VB or auxiliary libraries you can always get it from the history of changes.

Wireshark Foundation released the final stable-version of the popular network traffic analyzer — Wireshark 3.0.0. The new release fixes several bugs, it is now possible to analyze the new protocols, apart from that the driver on Npcap WinPcap is replaced. Here is where quoting of the announcement ends and our note about bugs in the project starts off. The projects authors definitely haven't done their best in fixing bugs before the release.

Let's collect hotfixes right now to give a motive in doing a new release :).

Introduction

Wireshark is a well-known tool to capture and analyze network traffic. The program works with the vast majority of known protocols, has intuitive and logical graphical interface, an all-powerful system of filters. Wireshark is cross-platform, works in such OSs, as: Windows, Linux, macOS, Solaris, FreeBSD, NetBSD and many others.

To do the source code analysis, we used PVS-Studio static code analyzer. To analyze the source code, first we needed to compile the project in an OS. The choice was wide not only due to the cross platform nature of the project, but also because of that of the analyzer. I chose macOS for the analysis. You can also run the analyzer under Windows and Linux.

The PVS-Studio team has been keeping the blog about the checks of open-source projects by the same-name static code analyzer for many years. To date, more than 300 projects have been checked, the base of errors contains more than 12000 cases. Initially the analyzer was implemented for checking C and C++ code, support of C# was added later. Therefore, from all checked projects the majority (> 80%) accounts for C and C++. Quite recently Java was added to the list of supported languages, which means that there is now a whole new open world for PVS-Studio, so it's time to complement the base with errors from Java projects.

The Java world is vast and varied, so one doesn't even know where to look first when choosing a project to test the new analyzer. Ultimately, the choice fell on the full-text search and analytical engine Elasticsearch. It is quite a successful project, and it's even especially pleasant to find errors in significant projects. So, what defects did PVS-Studio for Java manage to detect? Further talk will be right about the results of the check.

A characteristic feature of modern programming is the use of the global network as a source of reference information, in particular, a source of patterns for solving unknown or little-known problems for a specific programmer. Such an approach saves a lot of time and often gives quite qualitative results. However, the solutions laid out in the network although usually correct, do not always take into account all the subtleties of solving a problem, which leads to the appearance in the source code of sections that usually work correctly, but under not quite standard circumstances become sources of unpleasant surprises.

Consider the topic of using sequence files in the Linux kernel, such files are considered to be the most convenient mechanism for printing from kernel mode. But in practice, using them correctly is much more difficult than you would think.

A lot of materials on this topic are available online. The best is the source code of the kernel itself which has quite detailed comments. The problem with this source of information is its volume. If you do not know exactly what to look for, it is better if you only have limited time, not to try at all. For me, when I became interested in the topic, Google provided several seemingly excellent sources of information relating to my search: the famous book The Linux Kernel Module Programming Guide and a series of articles by Rob Day. These sources are not new, but very solid.

It has been three months since 2018 had ended. For many, it has just flew by, but for us, PVS-Studio developers, it was quite an eventful year. We were working up a sweat, fearlessly competing for spreading the word about static analysis and were searching for errors in open source projects, written in C, C++, C#, and Java languages. In this article, we gathered the top 10 most interesting of them right for you!

What's new in CUBA 7

Three years ago we announced the second publicly available major version of the framework. CUBA 6 was the game-changing version — the licensing was turned from proprietary to Apache 2.0. Those days we couldn't even guess where it was going to bring the framework in long term. CUBA community started to grow exponentially, so we have learned a lot of possible (and sometimes impossible) ways of how developers use the framework. Now we are happy to announce CUBA 7, which, we hope, will make development more coherent and joyful for all community members from those just starting their journey in CUBA and Java to skilled enterprise developers and Java experts.

For Russian speaking posted translated version here.

This is how Alex Viscreanu’s talk on Moscow Python Conf++ named. Now it's two weeks till before the conference, but of course, I've already heard what Alex will speak about. Find below some spoilers and talk preparing backstage: what kind of an open source Zoo developed in Kiwi, how it tests Python code and what’s the difference between The Zoo and for example mypy.

— Tell us a bit about Kiwi, yourself and what is your work within a company?

Kiwi.com is an online travel agency based in Czech Republic. We aim to make travelling as simple and accessible as possible. The company was founded in 2012 as Skypicker, and since then it has become one of the five biggest online sellers of airline tickets in Europe. It was renamed to Kiwi.com in 2016.

The special feature that we, at Kiwi.com, offer is the virtual interlining, which allows us to connect flights from companies that don’t usually cooperate together, and we are covering the possible connection issues caused by delayed flights.

Some of the numbers that we manage at Kiwi.com include 90 000 000+ daily searches, 25 000 seats sold daily, and a total of 15 000 000 000+ flight combinations available.

Here we are, continuing to explore the code of calculators! Today we are going to take a look at the project called SpeedCrunch, the second most popular free calculator.

Introduction

SpeedCrunch is a high-precision scientific calculator featuring a fast, keyboard-driven user interface. It is free and open-source software, licensed under the GPL and running on Windows, Linux, and macOS.

The source code is available on BitBucket. I was somewhat disappointed by the build documentation, which could be more detailed. It says that you need «Qt 5.2 or later» to build the project, but it actually required a few specific packages, which wasn't easy to figure out from the CMake log. By the way, it is considered a good practice nowadays to include a Dockerfile into the project to make it easier for the user to set up the development environment.

FreeRDP is an open-source implementation of the Remote Desktop Protocol (RDP), a proprietary protocol by Microsoft. The project supports multiple platforms, including Windows, Linux, macOS, and even iOS and Android. We chose it to be the first project analyzed with the static code analyzer PVS-Studio for a series of articles about the checks of RDP-clients.