Collaborate on Uber Bill Format for Finance with Ease Using airSlate SignNow

Uber bill format for Finance

How can software handle billions of payments without missing a single transaction? Today we're going to talk about the Uber engineers solution to. Billions and billions of payments. and easily explain how it all works. So let's go. Now we all know how Uber works. Click a button that pays adjusted money for a leaf from one place to another. Wait for your driver. Get in the car and arrive at your destination. Leave and then Uber pays the money the driver is owed. Pretty simple right? add 100 million users. Billions of rides and hundreds of billions worth of money being passed around later. And then our easy solution is no longer an easy solution. As Uber grew and grew in popularity, they would evolve their tech stack, going from a monolith application aka just one big application. They deployed to a microservice. One which is like thousands of applications all over the place. They'd switch to Postgres and MySQL. And just build technology everywhere around it. However, if there's one thing that you just can't mess up in your text hack, it's what's dealing with money. their system was messing up big time. first it was just beyond unstable. The system was fragmented and multiple different areas and services transactions were really, really slow. then reconciling transactions was just an absolute nightmare. Payment discrepancies and just major delays were everywhere. Imagine being an Uber driver, driving and giving rise for two days straight. Then a cyberattack happens and then all your payments that you've been expecting for the last few days, it's just gone. Nowhere to be seen, nowhere collected. So they rebuilt their payment system from scratch. First, they switched their system to a job order system. The job in this case represents the trip that you go on. Or an Uber Eats order. This job system allowed flexibility in having multiple orders assigned to the same job. So, for example, asking to stop at McDonald's before getting home. We've all been there. It's it's okay to admit it. Inside these orders are entries, which represents money moving from a payer to app. The sum always zero. No creating or destroying money. So if I paid $20 for an Uber ride, it would come up in two different items. trip fare, which is $18, and the service fee, which is $2. This then goes into the Uber drivers account and then Uber's account. Trip fare 18. Service fee $2. This essentially is how double entry accounting works. Credit and debit. I've covered so many different types of databases on my channel in the past, but none have specifically been designed to tailor towards the need of money or finance. So like, how would you even do that in the first place? A ledger. A line by line table of your transactions. Don't ask the crypto people what a ledger means. What makes ledger still popular to this day is that it's the only way to truly know where you're financially. If my bank account starts at $100 and ends at $1,000, but the purchases and depositors don't end in $1,000, well, some suss stuff is happening in my bank account. doing two seconds of research on this, I can see that, you know, Postgres or MySQL is clearly capable of doing something like this. So why isn't it suitable for finance? Well, first, most databases aren't immutable. if I wanted to change my name on any website, once I change it on the form, the database goes to the existing entry and then updates it, removing the previous value. This is because it is mutable. if I were to try this on an immutable database, it would have to create a brand new row with the new information and duplicate existing information. Then in the UI, I would have to grab the latest version of this. So obviously this is just not good and so wasteful. If you had a database of transactions and then they all equal to zero, and then some dingus deleted a row out of that database. you forever screwed up the integrity of your books. This transaction is just gone forever. So an immutable system would force an append only system, meaning that if a mistake happens, the correction is appended at the bottom of the database. Rather than deleting the record completely. For example, if I were to document a transaction, I could buy lunch for $10. That's -$10 from my account. However, the deli accidentally charged me $15. So am I. Can't you see -$15? Well, rather than just deleting the records and trying again, the deli has to refund me $5, which then turns into a credit on my account. You fix things by adding or subtracting to the end, not by replacing. popular open source options exist like IMDb, but at scale. Uber made the classic programing decision to make their own version from scratch, so they created ledger store. ledger store as an application layer that sits above existing technologies like NoSQL databases. It specifically adds three core concepts. Ceiling closes a time range of changes. This means that transactions that happened last month, you can look at and validate that they are correct, making them now read only. Uber specifically mentioned that they have 30 minute spans of seals. Manifest are files generated and the sealing process. This is a security measure to show that only ledger store can access seal transactions in the past. Just remember though, transactions are happening every second, so they use Apache Kafka, which helps the streaming of this data in real time to calculate and revisions. As for when, sometimes the data needs to be corrected in an already closed time range. Revision is like a table that shows the corrections that have happened while also making the previous additions Queryable. And leisure store. It was great. but Uber had a really, really big problem. You see, Leisure Store was built to be an opinionated about its underlying technology. So when they moved to their new internal database, they had to migrate over 250 billion records. And so they only had two options process all of this 300 terabyte of data in one go and then just dump it into their new database, or break down historical data into parts and process them individually and create checkpoints throughout. The issue with doing number one is that with such a big data, at any point that it goes wrong, you have to restart the entire process. Think about installing a software for a long time and then it just fails. On the last step. So they went with option two and here's how it worked. A worker boots up and reads the checkpoint of where the process needs to be started from. it then goes to cold storage where the data lives and processes this data and moves it to the new database, and then updates the checkpoint to say that the partition it filled out is successful so it doesn't get repeated. And it then stores this in a metadata database so that other simultaneous workers don't work on the same ones. And this process only allowed that data to take a couple of weeks and saved on a lot of money. have our historical data taken care of on a separate database. But how do we put this over live onto one of the busiest applications of all time? This is when Uber introduced the Shadow Rider. The Shadow Rider sits in between two databases, the current one and the one they wanted to migrate over to. The Shadow Rider pushes data to the main database as usual, synchronously, everything executed in order to keep things fast. The Shadow Rider pushes data to the new database asynchronously, meaning our system isn't dependent on if it saves right. The second. but here's another issue. What happens if the items fail? I'm. because it's asynchronous. We don't know if it's going to work or not work. To prevent this, they deployed a spot repair job to look at any issues and fix them on the spot. They also created a dual read and merge system. This reads both databases at the same time and compares the two requests. If the primary database had five records and the secondary had three, it would merge the objects together and say that to the secondary database and return to the user. This put the new database into production gradually to see how it handles under the pressure of a production system, but eventually that leap of faith was needed to be done. So they finally decided to commit to the new databases and made the switch. dock store being the primary and then Dynamo being the secondary. Gradually, less requests were being read from Dynamo and Doc Store continue to grow and grow. Oh, and thank God for that. But why all of this work? Just to change databases. When we talk about the ledger, we can think of it like a piece of paper. With each row dedicated to a new entry. I mean, makes sense for like 50 transactions. But what about trillions of transactions over billions of ledgers? Indexes. Now imagine this scenario. You walk into a movie store and you want a copy of the movie jaws. The cashier says, one moment and one by one, make sure that each movie in a gigantic pile of movies is jaws. I mean, you would be there forever. It's just stupid. An index in a database would be categorizing each movie by the first letter or by genre. So if I ask for jaws, you already know to look at aisle J, But in this case 26 I think 26 indexes are just not going to cut it. And this is why Uber has trillions of indexes that they have managed in their database systems. And there's different types. A strongly consistent indexes used to handle credit card authorization. When you use Uber, your credit card has a hold on it until the end of your trip. If the database isn't able to look up the whole fast, It risks charging you again and continuing this hold. Yikes. This is done with an intricate write and read process. First, ledger store preps the indexes that will be inserting a database record. Once it's prepped, it writes the record and if it's successful, will commit the indexes needed. They found that their indexes get stuck in the intent phase. So to counteract this, if they read that there's a record in the database, they commit the index. Or if it doesn't it rolls back the index. A lot goes into making this extremely reliable. eventual consistent indexes. As for parts of the application that doesn't require extreme precision. So these could be places like previous rides greens where waiting a little bit is fine as long as you get the end result. Think about a ledger for a second. One of the biggest key columns you could probably think of, if not the most important one, is date and time. Time range indexes does exactly that. is able to index within a certain time frame of ledger entries. When working with data, this is a great way to offload storage that is really old. In this case you can move data from hot storage. Think like OneDrive. You know easily accessible files all the way to cold storage, which is just, you know, really cheap storage that takes a while to get your files. So we now know that these types of indexes exist. Great. But how do we manage 2 trillion of them? After the index is created, it back fills itself with historical data. This is done in batches to save on data download speeds then is validated for completeness. This is done by using a checksum comparison on the data and then decommission this index so that the new index can get started and move along. the idea that so much technology goes into what we developers or even users take for granted is just absolutely insane. It just takes another level of engineering in order to figure all this stuff out. There's a lot of cloud services that exist that specialize in transitioning your database system over from one cloud provider or on premise to another cloud or on premise database. But at the scale of Uber, you have to figure out your own solutions, because what works for you isn't something that's going to work for everybody. Unless like you're left or something. A big shout out to the three engineers from the Uber engineering team that have written the articles that I've used to reference for this video. literally without that, I wouldn't be able to make this video. But also, I just think it's so cool that they can be vulnerable about the different processes they've tried failed and just, you know, done. What's the biggest transition you ever had to make? Let me know in the comments below and don't forget to subscribe to the channel! Peace out. Coders.