The Veriff Hackathon Winners - Canary Releases

Veriff's first ever Hackathon took place in December 2020, and you can read all about the online event elsewhere on our blog. Here though, we speak to Siim Tiilen and Kateryna Porshnieva who were part of the winning team, that built 'Canary Releases'. We ask them all about the idea itself, how they used their 48 hours, and how their presentation won over the judges.

Can you please explain what is the idea behind canary releases? How does it work?

Siim: Canary releases is a software testing technique used to reduce the risk of introducing a new software version into production. Instead, it gradually rolls out the change to a small subgroup of users, before introducing it to the entire platform/infrastructure.

Kateryna: This technique was inspired from the fact that canaries were used in coal mines to alert miners when toxic gases reached dangerous levels, therefore saving a lot of lives. 

In software development the same can be achieved by rolling out a change gradually. It works by showing a new feature to the subset of users first and then gradually increasing the number of users that are exposed to the new feature. This way we can experiment and innovate in a safe manner.

In Veriff, we have an internal tool which we call the Deploy Manager. It helps us manage all deployments across all the infrastructure, it has testing capabilities and is the tool that all engineers in Veriff use daily and it saves everyone tons of time. 

How long has the idea of canary releases been around in Veriff?

S: The theory itself has been around as long as I can remember, but it's always been something that would be nice to have.

K: Yes, the idea was definitely around for a long time, but since Veriff has very complex infrastructure with around 100 services, making it work would require a significant effort, so the idea sat in the backlog for a long time. 

How far ahead did you plan to deliver it in this hackathon?

S: The plan was to deliver a working solution at the end of the hackathon and this was achieved. 

K: We initially wanted to have a working solution for at least one use case by the end of the hackathon, but we got carried away and made 5 different options and even automatic configuration.

Who pitched it, and how was it preparing that pitch?

S: I made the initial pitch, which was quite ad hoc, with a promise to deliver a working solution at the end of the hackathon if a team could be formed. 

K: I've been helping Siim out with the design of our Deploy Manager and at some point we briefly discussed the possibility of adding a canary releases feature to deploy manager and created mock-ups of it. After that, Siim researched the possibilities and we decided to join the hackathon with this idea. Siim did the initial pitch and formed the team.

How did you approach the intense timeframe - did you have a structure, or you just dived in and started development?

S: I did some research before the hackathon, so had quite a solid plan around what needed to be implemented.  We more or less formed two mini teams inside our hackathon team, one for backend development (me, Timofei and Dmitri) and the other for frontend (Kateryna and Annalia) - after that we more or less just started producing code as fast as possible to get things done.

K: We started by defining the our MVP requirements, deciding that we wanted to make it work for one project with one configuration option. 

We then split the work among ourselves. Siim was focused on backend work, Timofei was researching infrastructure capabilities, Dmitry was doing research on different configuration support and Annalia and I were focused on making a nice and pretty UI.

In the middle of the second day we got one project with one configuration option working and from there we got carried away with adding more and more features. We tested with different projects, conducted interviews with other engineers to understand use cases better and fixed all the problems we could find. 

It was definitely a team effort and having such an amazing team of dedicated people with different backgrounds was a huge factor in making the idea work. What the guys did was magic, they managed to do something which had been estimated to take months of development work in one day - for all of Veriff's services.

Kalev mentioned in his interview that the judges were won over by your presentation - how did you put it together?

S: We had a solution done at the end of the first full day, so we put most of the second day into the final presentation - Kateryna did a really awesome job there, we were all surprised in the end.

K: Our project was the only engineering-focused project in the hackathon and we understood that we need to do a very good job on the pitch to make everyone understand the impact of our work.

We decided to go with the real-world analogy of developing Covid vaccines and how they are tested, and then mapping this to Veriff and how our software is developed. We did a live demo, showed all the features, demonstrated how it works for all the different types of products in Veriff and explained different potential uses.

The whiteboard above shows the team's pitch structure

During the second day we were really working on the presentation - polishing slides, drafting ideas and rehearsing the pitch.

And it really worked! Honestly, we did not expect to win, but we were super happy and genuinely surprised when Janer came to congratulate us on taking first place!

In practical terms, what is the major benefit of canary releases?

S: As a practical example, a few days after it became possible to use canary releases it was already used by one of our development teams to release one major refactoring in a safe way.

K: We want innovation to be the heartbeat of Veriff, and canary releases allow us to experiment and innovate in a safe way. 

In my own team we have already used canary releases multiple times to ship features gradually. Also, we created a group of users within our product who receive new features first. In this way we can get early feedback and iterate without affecting all of the users at once.

In cloud services, such as ML and automation, canary releases allow us to ship new features bit by bit, therefore reducing the risk of incidents happening and making our product more stable.

Finally - there’ll likely be another hackathon sooner rather than later, got any ideas already?

S: I’m a total hackathon junkie with around 20 in my belly, I can’t wait for a new one. I have a few ideas, but I'm perhaps more interested to see ideas from others next time to instead help implement some of those. 

K: Next time I'd like to participate with more of a user facing project for a change. When the time comes, I'm sure there will be a lot of great ideas where I could apply my skills.

Interested in a career with Veriff? Learn more here.