Effortlessly Manage Your Dropbox Invoice for R&D with airSlate SignNow

Dropbox invoice for R&D

hello everyone today we are going to discuss the  design of a cloud-based file storage service like   dropbox or google drive so if an interviewer asks  you to design a file storage service whether it   is google drive or microsoft onedrive or dropbox  the design will almost be same and of course the   design is always based on the requirements  that you will discuss with the interviewer   during the interview so let's first discuss those  requirements before going into the requirement   collection phase you need to understand one thing  about the file storage service system design this   interview question is a very tricky question there  are so many different things which an interviewer   can ask you during this interview question if you  haven't watched my previous mock system design   interview video or designing dropbox service you  should watch the video the link to the video you   can find here and you will understand that this  interview question is very tricky an interviewer   can actually go in much depth if he wants to if  he asks this question and that is why you need to   give due importance to this interview question  and you should prepare well for this question   and unfortunately the all the other online sources  that are available right now they don't go in much   detail in the design of a dropbox service they  just discuss the design at a very high level   and that's it but that is not enough especially  if you are targeting l5 or l6 levels in facebook   google etc so let's first discuss the functional  requirement the very first functional requirement   is the user needs an account in order to  use the file sharing service so an account   and what it also means is that we can have two  types of users we can have free users and we can   have premium users where we can assume that the  free users are using our service in a limited   capacity for example they might have a  limited storage available as compared to   the storage space that is available for  the premium users similarly the network   bandwidth that we provide to the free users  might be way less than the network bandwidth   that we provide to the premium users the second  requirement is that the user can have multiple   devices and now what he can do is that he can  actually specify a root folder in any device   and whatever files and folders he create in that  root folder that is get replicated to all the   other user devices right now we are assuming that  the maximum file size that we are going to support   is 1gb another very important requirement that  the interviewer can ask you to design is that   a user should be able to share files and  folders with other users so file folder   sharing so by default the original owner of  the files and folders he has the desired access   but when a user is sharing file or folders with  other users he can either provide them read that   access or he can also just provide them read  only access so the original owner of a file   or the folder has read that access to it and he  can also the one who can actually share the files   and fold up with other users a user can only share  a file or folder which he owns a user cannot share   a file or folder with other users if he is not the  original owner of that file or folder so the owner   has full access and what full access means he can  read write the files he can delete the files he   can share the files while the other users can have  just the desired or read only so other users can   have either read write or just read only access to  the files and folders the other requirement is the   system should support storing files and folders  up to a certain limit only and once a user has   reached his file storage limit the system should  not allow any other rights and the requirement is   the system should allow offline creation of files  or offline update or deletion of files and folders   so what it means that while a device is offline  the user can go and either modify a file or delete   a file or even create a new file and when that  device gets connected with the dropbox service   then all those changes get replicated to all the  other devices for the user there can be some other   extended functional requirements for example  the system should allow multiple versions of a   file and the user should be able to recover the  file to a previous version if he wants however   this requirement is a very tricky requirement  because it imposes several design constraints and   questions as follow the very first question that  comes to mind is that how the versioning should be   performed whether it is from an update to another  update or it's a daily snapshot of all the updates   another question is how it will affect the files  and folders which are shared among various users   if a user with read write access decided to revert  some changes of a file how those changes will be   replicated to the devices of all the other  users also the third question is how it will   affect the storage capacity of a user whether the  total space consumed by each version or snapshot   is counted towards the total space consumed by  the user or not and what should be done once a   user has reached its limit in that case should we  start dropping older versions or should we just   stop creating new versions etc so you should  clarify these type of requirements and constraints   with the interviewer during the interview and  then based on what you decide with the interviewer   you need to design the system ingly  another extended requirement is the system   should be able to keep track of all the analytics  related to the storage and network consumption   also one of the exchange requirement is which  actually comes into picture when you start   allowing shading of files and folders among users  that what will happen if two users try to update   a same file and if a conflict arises because of  that how you will resolve that conflict another   important requirement is data security the risk  of data read during transmission can be mitigated   through encryption technology encryption in  transit protects data as it is being transmitted   to and from the device to the cloud or vice versa  encryption at rest protects data that is stored   in the cloud service another requirement is to  provide such facility so that a user can actually   able to go and search his files and folders so  these are the different functional requirements   now we are going to discuss some non-functional  requirements of the system i think by now   if you guys have seen all my previous videos you  must already know now which are the important   non-social requirements that we need to discuss  so the very first non-functional requirement is   the system needs to be highly available and fault  tolerant the second non-functional requirement   is the system should be highly scalable and it  should scale with increasing load or increasing   users and data the third requirement is the  file synchronization should use minimal network   bandwidth the fourth non-functional requirement is  that the filed transfer should happen with minimal   latency these two non-social requirements about  the file simulation requiring minimal bandwidth   and minimal latency are very important because  due to these two requirements we will decide to   upload every file by first dividing it into small  segments or chunks this will enable us to only   upload or download modified chunks of a file and  also if uploading or downloading of a chunk failed   we only need to retry the upload or download of  that particular chunk instead of retrying the   upload or download of the complete file the fifth  non-functional requirement is the dropbox service   guarantee acid requirements for the files that are  stored with the dropbox service acid stands for   atomicity consistency isolation and durability  let's suppose in one of your device a file was   changed from one version to another and those  changes get replicated to all the other devices   so now if you go to the second device in the  second device the file which was changed in   the in some in the first device is now getting  all the changes which will move it from the   first version to the second version and it's  totally possible that in order to move from   the first version to the second version there  might be multiple chunks that got modified and   so they all those chunks get uploaded separately  and then on the second device they gets downloaded   separately but now the issue is that if you just  apply one chunk at a time it's totally possible   that the user will see the file on that device  in a transient state where he will see the first   chunk get applied then the second change of  blood and so on however it will break our   atomicity requirement the atomicity means either  all or nothing so the user should not see all the   transient changes while we are moving a file  from one version to another version so how we   will achieve this atomicity requirement  in the dropbox service is that in our   client-side dropbox application we will first  apply all the changes to our temporary file and   that file could be stored in a temporary folder  or maybe in the same folder with a different   name and that file could be hidden from the user  and once all the changes are applied to the file   then we can actually use an atomic file operation  like link or the name to switch the temporary file   with the original file atomically however if you  check all the online sources none of them actually   discuss about this thing the fee in the asset  requirement stands for consistency consistency   in the acid properties means that data moves  from one correct state to another correct state   with no possibility that the readers could view  different values that don't make sense together   for example if a user has deleted a file in one  device then other users or devices should not see   that file getting partially deleted first this is  an inconsistent state that would cause errors if   someone tried to redirect the partial file this  is different from the replication consistency   which we are going to discuss later where an  update happen in a device but takes some time   to actually get replicated to other devices i  in acid properties stand for isolation it means   transactions that are happening concurrently will  not affect each other in case of dropbox service   it means two things first of all any updates to  different files are totally isolated from each   other they do not affect each other second is  related to update of a single file in multiple   devices if a file get updated in multiple devices  at the same time then one of them will have to   wait for other right to complete first however  this is really hard to enforce a two distributed   system consider updates that are happening to  a single file on two different devices and both   of those devices are actually offline so you see  if we have one requirement where we say that we   would allow updates on files offline then  having this isolation requirement where   we actually try to make sure that updates don't  happen to the same file at the same time on the   device actually impose a lot of constraints on  our system because now our system has to handle   this case also where updates can  happen to a file and both these   on two different devices and those devices can be  offline and now when those two devices gets online   then those updates will flow to each other and it  could cause conflicts and then we need some way   to resolve those conflicts which is a very hard  problem now d in acid properties is durability   it means that once a change has been made and  uploaded then it will not be lost for example if   a file has been created to a device and synced to  the remote file storage then that file should not   be lost this does not imply that another operation  later on will not modify or delete the file it   just means that the changes are available to the  next operation to work with as necessary it also   implies that any lights that happen to a dropbox  service should be replicated to multiple data   centers even across continents to make sure that  any large-scale disaster for example earthquake   or flooding of fire that can wipe out the entire  data center does not cause data loss here i'd like   to add one more thing if you check other online  sources you will see that they discussed that   using a relational database actually provide this  asset requirements and if you are using no sql   database then of course then you have to provide  these asset requirements programmatically but this   is where these other resources are wrong just  using additional database will not provide the   asset requirements because we need to understand  what the asset means in case of the box service   if the a in the atomicity means that when a file  change from one version to another version on a   device the user should not see any transient state  then just having additional database in the back   end will not be enough so this is i like to point  out here that don't just assume and don't mention   this to the interviewer that is okay if you just  go with the relational database it is going to   fulfill the asset requirements because that will  be wrong and then the interviewer will ask you   what does the asset requirement means and the same  thing you can see in the previous mock interview   where the candidate went discuss the asset  requirements i specifically asked him what does   the asset means in case of dropbox service now  the sixth requirement is replication consistency   the replication consistency here is different  than the consistency in the acid requirement   what it means here is that when an update happen  on the device it will take some time to that gets   replicated to the file storage service and also to  other devices and the replication consistency will   be eventual we cannot have a strong replication  consistency here due to the nature of the   problem we are allowing offline updates to a file  so it means a user can actually update some file   on a device which is offline and then only when  the device gets online then the changes will get   replicated to the fastest service and from there  it could go to to other devices only when those   devices are online as well so the replication  consistency will be eventual in case of dropbox   service and please do not confuse the depletion  consistency with the consistency here in the   asset requirements by the way if you want more  details on the requirements and the design of   the box service you should check out my course the  link is in the description below now let's discuss   some apis that our service will expose so the  very first api is upload file it takes user file metadata and file content the file content  could be a stream of different chunks for the file   the upload file will return a file id now there  can be other apis like update file metadata it   takes our as input file id and then file  metadata and what could be the file metadata   is like uh file creation time for updation time  file size etc etc another api could be update file it takes user as input file id change  metadata and change segments or modified segments   changed segments and just change segments is  the input stream for all the different chunks or   segments of the file that are changed similarly we  can have an api to delete the file which will be   read file user and file id a user can also  list files so there could be a list file api   this files api which takes user root folder  page size and page another api is share   file or folder share file or folder which takes  user file or folder id another user id and   set of permissions and permissions can be redirect  or read only similarly there can be an api for   stop sharing stop share and it can take a user  and a name space id so by the way this   name space id is the id that is returned by the  share file or folder function the name space the   concept i will discuss later and of course a  user can always list all the shared namespaces   so list shared namespaces so user as  input and this returns all the shared namespaces   for a user now let's discuss what a  dropbox namespace means in order to explain   what a dropbox namespace is i'm going to consider  an example suppose we have two users user a   and user b the user a has this root folder where  it has a file one file four and then it has a sub   folder folder one which has file two and file  three the user b also has some file let's say   file 11 and then some folder folder 12 and then  it which has file 12 in it here this collection   of files and folders will be called as the user a  root or home name space similarly this collection   of files and folders is called user b home name  space what is the name space a name space is   nothing but just a collection of files and folders  with certain permission access now what happens is   that this user a shared this file for with user  b let's suppose this file 4 gets shared here   i'm just giving it a name file 4 bar so at that  time what happens is that this file for when the   user is shared with user b it moved from the home  name space to a shared namespace and this file 4   bar here in user b is a proxy namespace that gets  created when the user is shared the file here so   we have three types of namespaces we have number  one home name space number two shared namespace   and number three proxy namespace so this  was essentially home namespace when a user a   shared a file with user b we create a shared  namespace for user a and a proxy name space for   user b and this proxy name space actually points  to the shared namespace you will find all the   details about this in the chapter on the dropbox  service design in my course now let's discuss the   overall high level design of dropbox service so  on my right side we have a client application and   the client application itself comprises of five  different components we have a component called   watcher the watcher component in the client  application is actually looking for all the   file folder changes that are happening under  the root folder that the user has specified to   be shared with all the other devices and with the  dropbox service now we have a chunker the chunker   is a component that is responsible for breaking  a file into multiple chunks and it all it is also   responsible for calculating the cryptographic hash  for each chunk so the cryptographic hash function   that we can use could be sha-256 or sha  512 then we have this file indexer the file   indexer actually stores an index of all the files  which are stored within that root folder that the   user has specified with dropbox service and the  internal database is a file database stored in the   client app where the indexer stores all the metal  data information about the file and all all its   different chunks the dropbox client application  can use any lightweight database like sqlite   or berkeley database etc for internal db then the  synchronous component in the client application is   the one which is responsible for syncing all the  changes that happen on the device to the remote   dropbox service or vice versa you can find more  details about these components and their design   in my course but there are some questions i  would ask you here the very first question   is that we already decided okay we are going to  divide every file into small chunks or segments   and the two reasons and and the reason why  we are doing that is that because we have two   non-functional requirements that actually cause  us to actually upload or download a file in the   form of chunks the question is what would be the  suitable chunk size should we use four kilobyte   eight kilobyte 64 kilobyte 1 mb 4 mb 64 mb what  do you think what would be the most suitable chunk   size and how to calculate an optimal chunk size  for the file so this was the client application   and if you see the client application is  communicating with the dropbox service   over the internet and now these are different  microservices within the dropbox service we   have a gateway service we have a synchronization  service we have file and folder metadata service   we have a user and devices we have notification we  have object storage service we have block service   or chunk service and we have billing service i'm  not going to details of the design of the gateway   service if you see my previous videos on designing  uber you will find it is there the user and   devices actually store information about the user  and all the devices that the user has the file and   folder metadata service actually store all the  information all the metadata information about   a file and also it also stole the information  about if a file is shared with multiple users   then that sharing information is also stored in  the file metadata service the synchronization   service is basically a service which is the main  component which is talking with the synchronizer   here in the client application any changes that  happens on a device the synchronizer informs those   changes to synchronization service and then from  there synchronizing service the changes are stored   in the block service and object service service  and then the file editor service for the file so   whenever a new file gets created the information  is stored here in the file metadata service   similarly when a file gets shared with another  user then the information gets added here in the   file folder metadata service and similarly other  user devices if they are running and they are   online the synchronizer component in the client  app communicates with the synchronization service   to receive all the changes that has been done on  different files under the user name space or the   shared namespace which the device is subscribed  to so and the synchronizer then talks with the   synchronization service to get all those updates  if you check out the online sources i think none   of them has discussed what would be the internal  design of these services what are the different   database schemas used by these services you can  find that information in more detail in my course   now if you go and check my course you will  find that there are multiple ways to store   file metadata information and i have  discussed all those different approaches   and i have also discussed all the pros and cons of  those approaches and this is very very important   if you are applying for an l5 l6 or senior  principal level engineer job what the interviewer   is looking at is different trade-offs the interval  doesn't ask you to just design a system and just   give you one design approach and that's it he  actually loves if you provide multiple approaches   and you can discuss the pros and cons of different  approaches and then based on that you decide which   approach is better so for example there are  multiple ways to design the file metadata   database schema i'm just i'm not going to address  right now and either you can check it in my course   or if you want you can google it about different  approaches to store the file metadata information   the first approach is adjacency list model  using the parent reference the second approach   is adjacency list model with the children list the  third approach is materialized path and the fourth   approach is nested sets model you can get details  of those models either in my course or if you want   you can google it and see what it looks like but  of course in my course i'm also defining all these   schemas based on different approaches now there  are some questions that are this like if you are   using one approach to design the file metadata  then how the file sharing will actually happen   in case of that approach as compared to if you  use a different approach to design the metadata   for example if a user want to stop sharing files  or folders with other users then what needs to   be done and what are the changes that needs to be  done in the in the metadata service in that case   and now as far as the block service is concerned  this is a block or chunk service this actually   store the metadata information about each chunk of  course the chunks itself are stored in the object   storage but we actually store the chunk metadata  information in the block service or the chunk   service this blog service is also responsible  for generating different version information for   different files and in the course you will find  the details about two approaches that we can use   to generate versions or snapshots of each file  in the video i haven't discussed the design of   the synchronization service retail but let me tell  you that it comprises of some app servers and of   course some distributed queues for example kafka  and some agents that are listening on those queues   and so what happens now whenever a user create  a file the synchronizer actually informs the   synchronization service or actually the app  servers here about file creation the request goes   to one of the app server the app actually goes  into the file manager service to create uh the   entry for that file with the append status pending  after that all the file chunks are uploaded and   whenever chunk gets uploaded to object storage  the metadata information for your chunk is also   get added here in the blog service and then  once all the chunks are uploaded we actually   go and change the status of pending to upload  it in the file metadata for the file similarly   when a file gets updated it's the synchronizer  on the app uploads the chunks that are modified   those are written into the object storage and  then the synchronization service calls into the   blog service also insert the new entries for those  modified chunks similarly when a file gets deleted   the synchronizer and form the synchronization  service and then the synchronization service   actually go and mark the file excluded in the file  metadata and then of course we run a workflow in   the background that actually goes and delete all  the different chunks of that file from the blog   service and the object storage so the main point  i like to convey here in this video is that the   very first important thing is always come up with  the functional and non-functional requirements   once you have discussed the functional and  non-functional requirements with the interviewer   then you have to of course design the system  based on those requirements and it is always   better to design the system in terms of different  microservices you see i'm not putting any load   balancer here because right now these are just  logical diagrams for different micro services   what i would like to say is that do not try  to design a system as a monolithic service   in an interview always design the service in terms  of different micro services and you can go deep   into the design of these components individually  later on and that's why you are not showing load   balancer here right now in that design but of  course each service will have some app servers   behind some load balancer i hope that you must  have found some information very useful in this   video and if you find that video user please do  like the video and please do comment below as well   and please do subscribe to my channel there will  be more videos coming soon thank you and take care