The secret sauce for a testable, maintainable and documented ETL

Data warehouses are typically assembled from a variety of data sources with different formats and purposes. As such, ETL (Extract, Transform and Load) is a key process to bring all the data together into a standard, homogeneous environment. The ETL process is composed of:

  • Extraction – data is extracted from its sources
  • Transformation – data is transformed for storage in the proper format or structure, for querying and analysis
  • Loading – it is loaded into the final target (i.e. database)

Managing big ETL process might be a challenge because of:

  • overall and ever-growing complexity
  • transformation dependency management hell (when final transformations depend on other transformations)
  • inaccurate, outdated or missing ETL process documentation
  • a lack of regression tests that can prevent unexpected errors after a framework upgrade or unintentional changes in seemingly unrelated areas

and many more. There is a simple solution: unit/integration tests.

Continue reading →

Transforming Your Organisation to Support Multiple Platforms

New platforms have and will keep bubbling up - be it smartphones, tablets, smartwatches or embedded devices. Both startups and established organisations encounter a need to enter and support them. We went through different stages of transformation to support multiple platforms. This story can be useful in understanding how to best do that.

Continue reading →

Frequent Releases & Major Changes

At Vinted, we’re big fans of continuous deployment. Our code is deployed hundreds of times per day into production. While offering a lot of advantages, continuous deployment also introduces its own set of challenges. Release of major changes, like big features or considerable refactorings, is one of them. How to release a sizeable improvement without stopping deploying continuously? Mindaugas Mozūras answered that in his talk during DevOps Pro Vilnius conference.

Check out the slides and the video:

Takeaways from PolyConf 2016

The PolyConf conference took place in the end of June. It unites polyglot programmers focusing on new as well as functional programming languages.

This is my overview of its most interesting ideas.

Continue reading →

Sublime Text plugin development basics

Sublime Text is amazing editor. It shines at its speed, file search and multiple cursors. If some functionality is missing, you can easily extend it via plugins.

I have written TestRSpec plugin for running RSpec specs. Give it a try and provide me feedback via GitHub issues.

Continue reading →

Takeaways from React Europe 2016

The React Europe conference took place in the beginning of June. It unites people who shape or are interested in the future of web as well as mobile applications.

Let me share the most interesting ideas from the conference.

Continue reading →

Takeaways from Devops Pro Vilnius

On the 26th of May, some of us took a day off from our everyday work and went to learn new things at Devops Pro Vilnius. Here are some things we found interesting.

Continue reading →

Engineering for Engineering's Sake

LOGIN took place in Vilnius at the start of this month, on May 5-6th. It’s the biggest tech community gathering in Baltic States, attracting 5000 visitors annually. During its Startup Fair, our own Mindaugas Mozūras gave a talk about Engineering for Engineering’s Sake. Check out the slides and the video:

Internship over. Now what?

Last Friday marked the end of my journey as an iOS developer intern at Vinted.

It all started with my future mentor Mindaugas teaching an introductory iOS course at Vilnius University. Developing iOS apps has been a dream of mine, which is why I signed up for the additional course. My interest in Vinted was sparked by Mindaugas, who taught his class using real examples from the app they were building. One day, mid-semester, I approached Mindaugas with a question, ‘Maybe you are looking for a guy who wants to become an iOS dev?’

Continue reading →

Spring Cleaning: Replacing Resque with Sidekiq

At Vinted, we have Guilds. No, not the World of Warcraft type - if this was based on computer games, I’d push, oh, so hard for us to have factions (go figure). The Spotify type. At Vinted, a Guild unifies people from different product teams who own and develop a certain area - backend, iOS, analysts, design - to name a few.

Every other thursday we have a Guild day. That’s a whole day dedicated to Guild efforts. People in a Guild conceive their own initiatives, form teams, make a backlog of tasks and work away.

Quite predictably, the iOS Guild will be working on something new and cool, like Vinted for Apple TV, the design Guild might be working on something they call “posh feed”, while we in the backend Guild work mostly on making our life easier via automation of reoccurring tasks and give engineering love to parts of our backend system that no team can reasonably own. Say, the background jobs.

We rely on background jobs for a lot of things - from sending CRM emails and push notifications, to processing transactions. In all our portals combined, we process hundreds of jobs per second under normal circumstances and thousands if we have an intensive CRM campaign going on.

It’s a critical piece of our infrastructure. I was quite surprised when on my first weeks at Vinted I took at look at our Resque admin panel and asked my fellow backenders:

Continue reading →

Life Embetterment With Hundreds of Deployments a Day

What happens when your code reaches the end-user just once in a while?

Imagine. It’s a late Friday evening. A developer (still on his 3 month trial period) just finished manually deploying changes to production. He’s alone in the office, with everyone gone for the weekend. Checks the home page. Everything seems fine. Opens up Hacker News, skims a couple of articles and gets ready to head home. But before this developer goes out the door, “All Along the Watchtower” starts playing. That’s his phone’s ringtone and it’s the COO calling.

Continue reading →

Save bandwidth by turning off TCP timestamps

We care very much about all performance aspects, even networking. If you do too, our System Engineer Donatas Abraitis wrote a post that explains how to save bandwidth by turning off TCP timestamps. Find it on High Scalability.

Vinted at Big Data Strategy 2015 conference

Big Data Strategy 2015 conference took place in Vilnius on October 5th, 2015. Vinted engineers gave two talks about our Hadoop -based data warehouse, metric computation, and lessons learnt while implementing it.

Continue reading →

Vinted in 30 Days

It was the last day of my internship as a front-end developer. Or, more like two days after the last day of my internship. The office in Žirmūnai street was in a festivity mode. At Vinted, celebrating colleagues’ birthdays is as typical as having a cup of coffee in the morning. Such simple yet profound traditions makes everyone feel valuable here at one of the most successful Lithuanian startups.

Continue reading →

One First Less

The post was written by Elena Lapinskaitė after her internship at Vinted.

When you’re 18, there are so many ‘firsts’ to experience. Your first driving ticket, your first holiday abroad on your own (and getting lost in Warsaw in the middle of the night with a dead phone, fun), your first morning not remembering what happened the night before, your first diagnosis of arthritis and first dentures. And your first proper work experience.

Continue reading →

Failover with Redis Sentinel

At Vinted, we use a data structures server Redis for many things including Resque, news feed, application, etc. We are not able to restart or upgrade Redis instances without having zero downtime. High availability is critical for us. Therefore, we decided to try database services like Redis Sentinel or Redis Cluster.

Continue reading →

Monitoring growing memcached

The engineers at Vinted began the year working on the improvement of the caching layer. We managed to significantly improve the performance of our Rails application and substantially reduce the number of database queries. But this additional load to our caching layer brought up some new challenges.

Continue reading →

Boldly refactoring complex code

At Vinted, new code is deployed to production machines all the time with zero downtime for members. Deployment changesets are small. So we can move fast and fearlessly.

To keep the speed of development from decreasing we have an extensive test suite for easy refactoring.

But in cases where it’s difficult to reproduce a complete dataset from a production environment there are other approaches to testing changes safely.

Continue reading →

Vinted Architecture on High Scalability

We wrote a guest-post about Vinted Architecture on High Scalability. If you’re interested in our technology stack and workflow, it’s worth checking out.

Proper Git

Git is one of the most wonderful tools developers can use to make their lives better. As Linus himself put it best: “If you actually like using CVS, you shouldn’t be here. You should be in some mental institution.”

However, Git is often underused and abused with badly written messages, junk commits and weird branching. You should not treat Git, or any other SCM tool for that matter, as a way to “Save” your code, just like you would save your progress in a game.

Continue reading →

Hello World

These are the voyages of code tailors that help create Vinted. Our mission: to make second-hand the first choice.

It’s a big undertaking. We learned a lot already and expect to boldly discover even more. Here we’ll be writing about problems we tackle, tools we use and things we learn.

Welcome to the Vinted engineering blog!