eSign Musical Ticket Made Easy

Esign musical ticket

all right welcome to today's video this is a system design video about e-commerce and ecommerce hosting platforms like Shopify let's go welcome to today's video this is a system design video about platforms ecommerce platforms like Shopify and other similar offerings I'm not affiliated with any of them but I thought it's an interesting challenge to take a look and try to figure out by myself how something like Shopify would work internally now why do we make a video about Shopify and similar platforms and not any other e-commerce website the reason is simple ecommerce websites that can be extremely complex and it's an interesting system design challenge but a platform like Shopify allows the customer to create their own web shops web stores including the front end and the back end and usually is extremely simple and extremely fast so without any technical knowledge you can go there and you can create your own store now to me at least this sounds like an interesting challenge because in the background there has to be a lot of automation and a lot of automated management of your resources your shop your front and your back and your api's and your databases you know all your inventory all your products and everything so let's take a look at how Shopify and similar platforms could be built okay big disclaimer here two disclaimers actually first of all I have no idea how something like Shopify is actually built and in reality it's gonna be a lot more complex than all the things that we're gonna talk about here today this is one second of all this is a system design video but I cannot guarantee that every system design interview will go along as this video will so we're going to talk about a couple of interesting topics we're not gonna cover every topic we're gonna miss a lot of important and interesting topics but as in every systems I mean to you you're never gonna cover everything there's always going to be focus areas and it's important that you cover those and that you listen to your interviewer and follow along where he or she steers the interview so let's go now this Shopify topic is so complex and this domain is so big that we need to focus on a couple of very important bits and pieces we can't you know cover every topic that Shopify has so we need to at least gather a couple of keywords from those keywords will drive and deep dive into the two to three key features and key elements so if you if you start with those keywords and just you know write down what comes to your mind when you think about platform like like Shopify and as I said this is step one so impart the key words that come to my mind when I think about a platform like Shopify definitely customers products and orders and we also have things like payment methods and so far everything looks like a very standard ecommerce website now the more interesting part comes along with Shopify will have customer product orders payment methods but you'll have your own sort of data storage so whatever Shopify owns to run its own business and then you have the whole other perspective where every Shopify customer who sets up a web store has its own customers products and orders and everything so we have to make this distinction between Shopify the companies I'll say and the shop owners so let's put it honest the shop owners here okay so we briefly touched on the distinction between the shop owners and the Shopify customers so if I go to Shopify or a platform like that and I want to create my online shop to me at least as somebody who has a background technology I would want to for I would want for my shop to be as isolated as possible from every other online shop this is important to me but this could also be a very important business feature because it's you know required by law so I could imagine that there are companies which we want to host with Shopify but they were gonna host their online shop managed by Shopify on their own premise on their own machine or at least they were gonna want to pick where exactly this shop is gonna be hosted you know only the you were only in North America and and for sure I don't want to host my products and all of my data in a database which is shared with other shop owners because imagine that something goes wrong and your products or your customer data shows up in a different store so a big topic here is isolation okay now when I think of something like Shopify but also you know think of Squarespace and similar hosting platforms where you can go and create your own websites and I think of plugins also WordPress for example and this is an this is an area a domain where it's extremely important to build more and more feature and allow your customers to plug in more functionality into their product so plugins is an important point and then lastly let's talk about the functionality of maybe having your own front-end so your own website where you just talk to the Shopify back-end so potentially you're a web designer or a front-end developer you're comfortable hosting your own front-end you have a very specific design in mind but you don't want to deal with the storage layer you don't want to deal with all the administration maybe you're not comfortable dealing with payment methods and payment management so to say so then you would want to use something like Shopify but only a back-end part of it right so and we have to talk about the storefront so a storefront I mean the web sites are say the front end versus the back end of the store so the inventory products and on the data and definitely when we talk about the back end and these two components and maybe switching out one component which is not hosted by Shopify we need to talk about api's okay we talked about a lot of features here a lot of key words that come to mind when you think about a platform like this but we can't take from all of it you could come up with even more right you could talk about analytics you could talk well you know authentication and login with maybe third-party authentication mechanisms but we can't cover everything so we need to figure out what makes a platform like Shopify important what makes it good what makes it unique right so let's focus on three key areas and let me draw some boundaries here second step the three key features and what I would say what's the most unique selling point right the unique selling point is you go there as a customer no technical background no idea about programming front-end development HTML CSS whatever and it allows you to create the store in a few minutes you know maybe you know fine tune it for a few hours but they do all of it so the three things that come to mind here is an automated storefront creation then they would also create and manage the back end for you so back end and ultimately they create it they host it and they wired up altogether so there's the network there's the domain management and all setting up the DNS setting up the proxies protecting it against against DDoS attacks and so on so let's just call it hosting and networking okay now we have the keywords we have two key features now we know what to focus on now let's jump into a very high level system design redraw you know the mayor the very basic components the high-level components you need to start at a very high level because if you dive right into one of the areas so if you dive right in let's say the backend architecture you're gonna miss how everything is wired together so you want to start at the very high level and you know what I usually do is I start at the customer level so the the visitor of the website server say I'm interested in creating a store I have no idea how to do it so I find this website and I visit Shopify right so so let's have a browser here and this person here visits the Shopify website now they will see a lot of static content of course they will see you know all the offerings that Shopify has then they will create an account or something like this and let's jump into this the creation of the store right so there has to be some sort of a component which orchestrates the creation of such a store and in reality this is going to be a lot of smaller components playing together but as I said we need to start at a high level and then we'll zoom into it so let's call this the store Orchestrator okay what are we doing with the store Orchestrator we talked about automated store creation okay so yes this Don Orchestrator will play a big role in this automated creation and we talked about hosting networking so this thing imagine it like the brain of Shopify when you as a customer go there you want to create your own store this knows every step on how to get there what to create and what order to create it so the big question will be how do we implement this automation and how do we achieve this isolation that we talked about what comes to my mind in such a case is containerization or at least automatic creation of virtual machines so what we could do is we could host each storefront each back-end each database with you know the customers the products and the orders in a virtual machine which is containerized so there is a doctor container which has an exact recipe so to say of what to create and how to link it together so there is something like a template template of a single store and every customer that goes to the website and creates one of the 1:1 account with a store the store Orchestrator will instantiate one of those containers and will create all the resources that are needed so how could this look like this here we'll have a big layer here and imagine this is this is not like a physical machine this is just a logical diagram so to say this is the layer which which owns all the components that are owned by each customer that hosts all of them so there is going to be an environment for customer one we'll call it a c1 and a very separate environment for customer c2 so each of those environments can have you know let's say if it's a double yes you can have easy two instances but have its own set of ap is may be configured by API gateway it will have its own databases like you know RDS databases and it will come so to say isolated by nature right because there are all their own instances may be there they're only the last account so to say they will not be connected to each other the only thing that they have in common is that they talk so to say to a store Orchestrator now when the store Orchestrator wants to create a new environment for a new customer so let's call it CN customer new then there's going to be a template but this is the T this is the template that contains all the resources so say the ec2 machines the databases the gateways whatever the proxy settings everything and the store Orchestrator talks directly sort of say with this with this template here this copy as I said a doctor container which gets instantiated for this new customer ok I'm sure you can already think of a couple of advantages and disadvantages of this design but let's follow along here and let's zoom into this template here right so let's zoom into this component let's draw it a little bit bigger so this is this is the template for a brand new Shopify shop right so this template is like imagine like a recipe so the store mr. an Orchestrator kicks it off and then it you know it just executes step by step first that could be create an ec2 or whatever virtual machine service a instance with certain properties you know certain cpu certain memory and so to say so well let's make boxes instead of numbers you see two and and I need some more space here because we can't forget about the API is of course so let's make this box down here okay so that one could be created ec2 instance right and step two and three we could deal about and we could deal with with a data storage with a layer of you know creating your your relationships between customers products orders may even payment methods although with payment methods you could hold off for now and you could outsource it so to say to a third party and you could only accept let's say PayPal if you want so you don't need to deal with you know very high security and critical data sort of thing so let's say step two of this template is to create a database with multiple tables and it's going to be the same for every Shopp right because this is like a skeleton every shop will need at least in the beginning the same kind of storage layers so there's going to be a table which is customers there's going to be a table which is the product and the oversimplifying of course another table which is for orders okay as I said very important is this distinction between the store front and the back end API so you could think of this template only defining the back end if you want right we could this way so the last thing that this template needs to do is to define an API layer this could literally be an API gateway definition no build with cloud without front I mean confirmation so you'll have apic-em those implies will act as a gateway to interact with your store so it would allow you to create customers put in an order retrieve the order history of a customer all these things all these things and the nice part is these can be public right so you can use your own storefront and if you're a bigger company or if you have a storefront already you're not interested in this storefront creation that Shopify gives you you can talk about you can talk with your own store through these set of api's now of course we have to go back and also look at the Shopify infrastructure that they need for their own customers right so this is a template this is a you know customer to customer one they have their own complete infrastructure and but of course Shopify itself will also need a storage layer for example and another layer here which serves as a as a controller for its own business logic for example so there could be let's just call it a control layer and there could be a storage layer which has the Shopify customers and maybe their subscription because this is I think at least a subscription-based system so you can have a subscription table in here for example and there could also be things like themes and more front-end related things so and those have to be stored somewhere as well right so there's going to be markup or code for four themes and these kind of things this is all going to be in own Shopify storage layer with the control layer and there could be a control application that manages this side thing the store Orchestrator will leave it separate as its own module so to say on purpose because it's very specialized specializes in creating a new store based on a certain template now let's take a step back and see what we already achieved and we talked about automated store creation so the store Orchestrator plays the biggest role here and this thing will create our back-end so point to it will create the backend of the shop and there's nothing you can also just create the front end with this Orchestrator right and we'll talk about this in a minute and what we haven't talked about so far is this whole thing networking part we talked a little bit about hosting of course because these resources here are going to be created in the cloud somewhere you know it could be hosted by Shopify burg you know potentially could be hosted by as you were ec2 or something I mean AWS or something like this so we talked about the hosting part but let's dive into a bit more detail let me erase this this diagram here so we get a bit more space before we dive into more detail let's talk about the advantages and disadvantages of this design here where every shop is its own box its own component its own container maybe and compared to a system where you know you have a few databases with a few tables there is like one big one big customer table and every shop owner would just own a few goals in this table right so compared to to the Senate we just draw there's a lot less isolation so one of the big advantages is this isolation so let's take the plaster so isolation you have your own database your own application layer your own virtual machines and potentially you could even host those components on your own premise right so big companies which is which is interesting that kind of stuff they could use the same templates that Shopify offers and shop if I could still manage it but they would set it up on on their machines so as isolation is a big plus your on-premise will put it here as well it's a big plus because if you would just you know create a few tables where all the Shopify customers would share tables with orders and and sort of saying and customers you couldn't just extract one of those customers and then put it somewhere else so this is a this design gives you a certain flexibility what this also gives you is a system which is not dependent on a single point of failure right so if all the Shopify customers and their online shops would share the same storage layers maybe the same infrastructure components you have a single point of failure it Shopify's down so to say then you know everything is down and in your shop so we'll see that in a minute but let's talk about resiliency so here are the advantages isolation so privacy we have the flexibility of an on-premise solution if you want to and we have a resiliency because everything is its own component has no disadvantages when other components other shops would have an outage but there are also a few big disadvantages because everything is hosted as its own box there's a certain maintenance overhead now let's make an example if you create a hundred or a thousand of those customers stores along this template and then you realize you would want to change something template maybe you want to change the database schema in a template and you want every customers store to follow this new schema you have to migrate every instance of these of your customers so now all your 1,000 stores with your 1,000 databases which after old schema need to migrate to need to be migrated to a new schema and that's of course that makes it a bit more inflexible and it creates a bigger maintenance overhead for Shopify for us the company managing and hosting office so let's call it maintenance and the other disadvantage I can think of is definitely cost because everything is its own box and it will has to be it has to be hosted by itself there is going to be a cost overhead compared to a system where you have a few tables and a few databases a few application layers or you can optimize and you know shrink down as much as you need with strict isolation like this you have to create more resources and you have to maintain more just little components which can add up to a big amount so not very scientific approach here but at least my my view on things tells me that this is going to be more costly good so advantages and disadvantages I'm sure you can find more of both but let's go ahead with the design and let me erase some of the stuff to get more space so what we still need to talk about a little bit more is the whole whole thing the whole orchestration and again I like to start at the customer and Alexis are at the beginning of this process and then go step by step through this creation of a store and then how everything is wired together now again let's take a color here and let's think about how everything comes together definitely will need our store Orchestrator again this is the brain again and meet a customer who now let's say triggers the creation of a vase store right so okay create there's a create command which which comes from this from this story this could be a REST API if you want of course after the customer has been authenticated after a customer has you know purchased a subscription and so on so there is a create command the orchestrator now kicks off its big machine and and starts working okay so we will start with what we just talked about the storage trailer will create this box with all the resources which our customers receive so this is going to be C 1 customer 1 so our first customer let's say there is going to be a computation layer or something or an application layer so this is what's an ec2 instance and there's going to be right so talked about this before this thing will as as we mentioned expose a set of api's and these api is of course they could be specific for for every shot so for example if you want to go into a lot of detail you wanna offer a lot of interesting feature as the company in Shopify you could allow the customer to customize this set of api's and to introduce you know additional authentication methods methods are introduced you know roles and permissions so I could imagine that you have one account with Shopify and your store has multiple internal accounts where only the store owner can now change the resources and maybe add products and something like that but multiple associate accounts can only use certain api's to read the shops stairs so they could you know look at the orders so they could pack it up and ship them so there's a there's a lot you can do with with a fan but let's just say roles and authentication here okay so now let's also assume that customer one is interested in the backend and the store from and the front-end and the customer does not want to you know work with HTML and CSS so they will use all the kind of content management systems that Shopify you know offers and this is a completely different topic of course that you can dive into how content management systems work we're not going to do that today but let's assume in the store Orchestrator can also create a template like a wordpress template so say of you know HTML CSS maybe JavaScript for you with a skeleton storefront let's draw this year so they start on this trailer when the create command comes in will create this is a one-time action here this whole setup down here all the resources here and also one-time action it will create static static content website which has some JavaScript in it this javascript is going to be wired up to exactly this set of API here so this could be a REST API and this is this normal direction and this front-end then through the JavaScript application which could be you know whatever you want angular react talk to these two this api's and now you can already see that this is going to be its own stack which is you know to a certain degree uncoupled from from Shopify so the creation now is mostly done right so we have a lot of code which creates the front and it creates that I can't now in here of course there can be a lot of a lot of detail it can dive into a lot of things like you know blogging mechanisms and so on and maybe payment methods as well so how you manage the payment methods in this er we use a separate third-party service for that but what we need to talk about now is how the traffic comes into the system and how it hits your front end and there can be you know you have a couple of options here but let's talk about one where Shopify has its own proxy layer which will take all the requests and forward it to your to your system here so I'm gonna take this blue color man so blue will be components that are owned by by Shopify and let's draw a let's draw a big proxy layer here and what we haven't talked about so far is the DNS and domain level right so when you create one of those stores they will also create DNS settings and you know the cname management for you that means you choose and you always also pay for a top-level domain so let's say this Orchestrator also takes care of that and they go and also create a bit older whatever disk comm domains and up here there's the DNS so they will populate the DNS with this new domain and they will point it to this proxy layer for example this proxy mayor will take this request and they will know that once a customer or you as the owner you hit this IP address first you will hit the domain and then the IP address this proxy layer will exactly know how to forward it to your front end and of course you will notice that this proxy layer here is actually a component that is owned by Shopify so if this thing is not working this is a bit of a single point of failure but it also gives you a couple of bandages right so this proxy layer here can be used to protect all the shop front-ends and backends from things like you know DDoS attacks and so on you could also think of the proxy layer being created by the store Orchestrator as part of every customers shop right so you would have another green box up here which replaces this blue box but it's all see layer but again lead introduces more and more maintenance overhead because then every customer will have its own proxy but it's also capable so we talk to the customer one here right let's grab the different color this is going to be red for customer - so customer - let's say comes in here and creates a different kind of account customer who is only interested in the back can't hear so the starkest Raina will again create based on this template the very same skeleton of all the resources there's going to be an ec2 instance and the database and the set of api's let me draw it the same way our customer too has it's very own front end right so now wherever this is hosted because we have this set of API senior customer to can have its own whatever web application front-end can be dynamic and aesthetic doesn't have to be owned by or hosted by Shopify at all and it's thinkable that this customer here would not use the proxy that is owned by Shopify so they could talk directly with these api's here I know it's also thankful that the Shopify will decide to always go through his proxy layer because maybe this API is here are these eight guys here are in turned about protector that the proxy will do the termination of everything so multiple options here but gives you a very nice flexibility of you know having this template create the backend and uncoupled from the content layer because the problem layer can change a lot right so let's say the store Orchestrator our store management system will then just inject different HTML and different CSS based on all the thieves that Shopify offers so you wish the customer you go to this admin admin UI and you would want to change a few so a completely new set of HTML and CSS is gonna be injected into this front-end here that could be hosted like an s3 bucket for example but the backend with all the components is not going to be touched at all the time stay exactly the same to all the products and customer tables and all the application layers are going to be exactly the same oh yeah just to be consistent here this should be another one-time action this should be like this but the creation here would be one time right so this Orchestrator creates this once we covered a lot of stuff let's just give him everything here so we talked about customer products and orders payment methods you could you could build this into into this API into your own application of course you could also let's say have a secure storage so let's say which is owned by Shopify and they own internally all the payment methods and all the payment management for for all the shop owners because maybe you don't want to put that into every single resource or environment here but you could also just rely on third-party like you know pay Paul and I don't know Amazon pay and these kind of things so we talked about the distinction between Ashanti Phi customers and the shop owners and their customers we talked a lot about isolation we haven't talked about plugins yet and plugins is an interesting topic because it can mean a lot of things but but you could interpret it this way that because anything is its own little world its own little environment a plugin could just be a another lambda running in here on other ec2 instance running here which offers a different set of API set some more functionality right so let's say a regular shop of customer one would only offer a product database with products you can buy you know in bulk or in a single product so to say but there could be a subscription plugin right so there's going to be a a application inside of the store Orchestrator which will add this plugin to the customer environment let's say it's a different ec2 host which may be notice to us maybe it's something like you know a workflow engine which will allow customer one to offer subscriptions for their customers and I'll take off and I'll think of Amazon Prime or think of patreon these kind of things where they can sell these subscriptions through their their online front-end to the online store and the subscription the recurring payments are triggered in this little shop here so plugins with this architecture because everything is a cloud native and and isolate from each other can be implemented really nicely and it's gonna be of course built for every customers needs right so this customer Sewall will play pay additional fee for those plugins and customer to maybe pay even less because customer who is not using the front attraction to the Shopify offers and as a very last topic let's talk about the operational aspects of this so we have a lot of creation you know a lot of steps which go into the creation of all of this but Shopify as a company needs to make sure that this is all maintainable and it's running smoothly right so one thing that you need in a system of this size for sure are a lot of metrics and automated alarms and you know health checks so to say so you want to know if every single one of your customer stores is alive can be reached and works as intended so for sure part of every template here part of every shop and there's going to be a component which you know in an ideal world is completely separate from the store components which is only doing the health checks right there hc' component this could be a lambda for example let me take the red one here this could also be something you know like a cron job running on a sir on a certain schedule but but let's say this is a lambda which runs every every few seconds or something like this this thing here will report the healthiness of this whole environment of customer 1 to a Shopify system so there's going to be a module down here which is our our monitor and alarm component and this will also offer an API so this thing here other recurring basis every few seconds that's every 5 seconds we'll send a heartbeat to this monitoring component here and this thing can be again part of the template will be what exactly that is this component will check so this could be are all the API is reachable now it could ping the API it could measure the latency of each of each API of course the api's themselves will also have latency metrics on them right so how long does it take to make an order you know how long does it take to retrieve the order history of a customer these kind of things so this component will know about their own little health checks but also the metrics in the latency and the exception count you know maybe even the logs and it could go to this monitoring portal maybe the logs you want to say you want to keep them separate so that the customers can retrieve their own logs isolating their own system it could also be some legal requirement and but at least the health checks would run into a component that is owned by we fight so of course the same would go here for for this shop for customer to and whatever happens you know data corruption in a database maybe down here this monitoring portal would immediately notice that something is wrong in customer to shop so an alarm would go off and one of the source a level one or level two support engineers from Shopify would see this alarm and would then decide to look into customers to customer whose instance and to its and to customer to environment and and do something about it right maybe restart one of those machines or figure out where the corruption happened these kind of things now of course the same the same goes here if this component that sends the the health checks this thing here fails this monitoring could go both ways right so this monitoring module knows how many customer instances are out there and how many customer instances this certain module is so to say observing in monitoring and if it doesn't hear from let's say customer twos instance in a minute or five minutes because maybe this this component fails then this monitoring module would alarm again and say hey I can't hear from customer twos environment something is wrong and again an engineer or maybe an automated fault that would take place okay we talked about a lot of things again this has nothing to do probably how a Shopify is set up with reality this is one way of doing you know something like this at least in my mind think about it now talk about it with good friends and colleagues it's always interesting to play through those kind of exam in your own mind made with other people I think it's very interesting because if you go one step further you could even do the ultimate inception here and you could eat your own dog food so to say you could set up your own Shopify based on the Shopify infrastructure to build right so you could imagine that Shopify uses all its own infrastructure to create its own store to sell those Shopify shops not sure if they're actually doing that but it's a it's an interesting idea okay this is today's video I hope you liked it let me know how you would create and how you would design a Shopify platform and really interested in your thoughts also if you have any kind of feedback let me know in the comment section below if you want to see more of these videos please subscribe give me a thumbs up my name is Ramona Lopez and this is success in tech [Music] you [Music]