Send Heterogenous Cc Number with airSlate SignNow

Send heterogenous cc number

hello everyone i'm imi from qinghai university i'm going to present our work a unified architecture for accelerating distributed dna and training in heterogeneous gpu and cpu clusters this is a joint work with biodance deep neural network has evolved rapidly in recent years we have seen numerous emerging dna models such as resnet firt and gp3 and they have broad fundamental improvements for applications such as computer vision and natural language processing on the other hand training these dna models is time consuming since they have large number of parameters that need to be trained for example to train a birth large model on one tesla v100 gpu the estimated time to converge is about 78 days so in practice we need distributed training to scale out a common way for distributed training is called data parallelism which means that each gpu carries a complete model and loads different data to train here is an example using two gpus for data parallelism first each gpu takes different input data and goes through the forward propagation to get the output then in the back propagation then calculate the loss function and propagate the gradients from the last layer to the first layer after that each gpu communicates with others to aggregate the gradients and use the new gradients to update their parameters and then go to the next iteration to repeat these processes and in this work we will cover the communication stage and the parameter update stage and discuss how to accelerate them in practice there are two architectures based on data parallelism all reduce mps for all reduce all workers are homogeneous and they use collective communication to exchange the gradients with each other for parameter server the architecture is a heterogeneous bipartite graph where the gpu workers and cpu servers connect with each other in the communication stage the gpu workers push their gradients to the cpu server and then pulls the latest parameters back unfortunately we find that existing solutions are insufficient this figure shows the performance of state of the art or reduced mps when training the vgg-16 with 32 gpus we find that even with the optimizations from by scheduler they are still far from optimal so our question is what are the problems of existing solutions problem one is the sub-optimal inter-machine communication we focus on dn training in heterogeneous clusters with gpus and cpus but all reduce nps fail to leverage the heterogeneous resources efficiently for example if we use all reduced for training it cannot leverage cpu machines because it is a homogeneous architecture as shown in this figure no matter how much the number of cpu machines change the or reduce plot is always flat then if we use ps it may create traffic hotspot if there are insufficient cpu machines and the application output is very low so we find that existing solutions fail to address the characteristics of heterogeneous clusters problem 2 is the sub-optimal intro machine communication in practice there are often multiple gpus in the gpu machine and the internal topology is also a network with different link bandwidths interestingly we observed that the next bandwidth is now close to the pcie bandwidth in recent years compared to the past when the nic is the only bottleneck now the pcie can also be the bottleneck but current intermachine solutions do not timely address this problem and they will cause pcie contention which prevents the nick from saturating its maximum bandwidth this motivates us to consider the intro machine topology carefully problem 3 is the cpu bottleneck this is a motivating example of the ps architecture when gpu workers send the gradients to the cpu server the server first aggregate the gradients and then update the parameters using the optimizer function as a typical setup in modern dln training clusters we use 100 gigabits gradient flow as the input and the cpu server uses the six channel ddr4 memory which is also used in media dgx2 then we can calculate the maximum number of memory assets to process the gradient flow is about 10 times but in fact many popular optimizers including ims pro and atom require memory access much more than 10 times this indicates that the cpu can be a bottleneck for running the optimizers for example if we run ims probe on cpus then the support is lower than the network rate so our question is how to address the cpu bottleneck to briefly summarize we have discussed three problems the initial machine and intro machine communication performance and the cpu bottleneck in this work we propose our solution called byps that can address all these three problems first it introduces an optimal intermachine communication strategy that is generic and can unify or reduce nps second it has intro machine optimization that can accelerate the communication inside gpu machines with diverse topology and finally it introduces a new abstraction called summation service that can aggregate the gradients on cpus and move the parameter update to gpus the summation service can address the cpu bottleneck efficiently next we will move on to the design and implementation we first introduce the design goal we focus on homo heterogeneous clusters with gpu and cpu machines in practice we have some interesting findings this is a three-month trace collective from an internal cluster of buy downs and we find that the average cpu utilization is only about 20 to 35 percent and there are about 20 to 45 gpu machines that only run non-distributed jobs meaning their bandwidth is unused so an up a new opportunity is that there are spare cpus and bandwidth in heterogeneous clusters and our design goal is to leverage any of these spare resources then we start from the inter machine communication as mentioned before ps only uses the links between gpu and cpu machines if there are insufficient cpu machines then the bandwidth of gpu machines is not fully utilized on the other hand all reduce only uses links between gpu machines so the cpu bandwidth is not used at all so the best strategy is to combine them together that can leverage the bandwidth of all machines and also utilize the cpu resources in this example we not only enable the connection between gpu and cp machines but also enable the connection between gpu machines like all reduce but since we combine these strategies together we need to determine how to partition the linked workload to solve this problem we use x and y to represent the amount of traffic for the two combined strategies respectively and after some modeling we've calculated optimal x and y as these two equations where n represent the number of gpu machines and k represent the number of cpu machines in theory this strategy can achieve minimal communication time and here we use an example to show how it performs this figure shows the communication time of three strategies including ps all reviews and the optimal one and we have three findings first if k is zero then the optimal value is equal to or reduce when case n the optimal time is equal to ps and when k is between zero and n it is better than all reduce mps so this strategy can unify psn or reduce and is optimal with any number of cpu machines next we move on to the intro machine communication we use this topology with a gpus as an example since it is widely used in this topology there are multiple links with different bandwidths and we find that this bottleneck is the link between the cpu and pcie switch our goal is to minimize the traffic on this link however current solutions such as mpi or nicole choose to perform or reduce for all these a gpus directly the traffic on the link at bottom line link will be 7 m divided by 4 according to all reduce algorithm where m represent the model size on each gpu this traffic volume is too large for this link our solution addresses this problem using a technique called cpu assist aggregation and it contains several steps first it lands four gpus under the same psi switch to perform a local reduced scatter operation so that each gpu will have a quarter of the aggregated gradients next each gpu copies its quarter to the host memory and now there are two copies of complete gradients on the host memory and each from an individual new model so we need to sum up these two copies using cpus we can see that the traffic on bottleneck link is now only equal to m since we have avoided the plus number gpu communication and in the end this cpu assist aggregation can outperform mbi or nicole by 24 in theory and we also summarize the design principle for this topology that we need to avoid the direct copy between gpus and the different pcie switches there are also more details in the paper such as the solution for every link-based machines the design principles for different topology and the optimality analysis and also the discussion about gpu dirac rdma and please refer to the paper for more details the third design point addressed the cpu bottleneck we have mentioned that using cpu to run optimizers is inefficient but since our design goal is to leverage the spare cpu resources we need a module that can run on cpu with high performance our solution is based on the observation that the optimizer function can actually be divided into two stages the gradient summation followed by the parameter update stage and while the latter is heavy for cpu we find the gradient summation is actually cpu friendly now here's a figure showing the submission throughput on cpu we use synthetic fp16 and fp32 tensors which are two common data types for deep learning we find that both of their summation throughput is much higher than the network bandwidth meaning the cpu summation is faster than the network with this finding let's rethink the function placement of dn training for ps it places the forward and backward propagation on gpus which is a common practice but puts the entire optimizer including the summation and update on cpus our abstraction called submission service is different while we do not change the forward and backward propagation we move the parameter update which is more computation intensive to gpus and keeps the much simpler summation in cpus this way the summation service abstraction can address the cpu bottleneck efficiently then let's put three pieces together and show the overall system architecture we have many machines and each on each gpu machines there is a module called communication service that can aggregate the gradients of the local gpus on each machine there is also a module called summation service that runs on cpus and can process the gradients from other gpu machines and all these modules interact with others using the network communication a mountain the communication service is responsible for intro machine optimization when aggregating the local gradients the summation service module can address the cpu bottleneck and the network communication uses the optimal inter-machine strategy to maximize the performance as for usage ips can support tensorflow pytorch and msnet it is also easy to use because it is compatible to most widely used apis including horrible and native apis for python and tensorflow we note that ips has been deployed in bidox for many tasks such as computer vision and natural language processing next we'll move on to the evaluation section we evaluate our system using popular dna models including resnet 15 vg16 ugadi gun transformer birth large and gpd2 the machines we use have av 100 gpus and a 100 gigabit nic the network is rocket v2 with full bisection bandwidth and the baseline we compare are horrible python ddp native ps of tensorflow and msnet and all of our experiments are performed on production clusters and all chosen models are representative of production workloads first we test the inter machine communication the figure on left shows the traffic micro benchmark on agp machines we can see that ips can achieve near optimal communication performance and the figure on right shows the end to end result on 664 gpus using two models including gpu 2 and the uk gun we see that with more cpu machines ips can achieve higher end-to-end performance next we evaluate the intro machine optimization the figure on the left shows the result on pcie-only machines and for this topology we have up to 30 percent gain the figure on rise shows the result of every link based machines and for this topology we have up to 80 gain finally we we evaluate the entries scalability with up to 256 gpus we use different cv and nlp models implemented in tensorflow msny and pytorch for each model we run the experiments using 8 to 256 gpus and the results show that ips has gain for all cases the more gpus we use the higher gain we will get in summary ips can outperform or reduce fps by up to 84 percent and 245 respectively we also analyze the breakdown of the performance gains we compare the performance with native ps using four gpu machines and two cpu machines we see that with intel machine optimization we have 66 gain and with intro machine optimization the gain is 22 percent more and with summation service we have 80 more gain next i would like to mention a few related work the first dimension is communication acceleration some previous work proposes gradient compression and scheduling to accelerate the dna training of communication these work are complementary to ips and actually we have integrated them as optional features in our system some researchers propose pipeline parallelism like pipe dream ips can benefit pipe dream in the data parallel stage there are also related work that propose hierarchical or reduce such as blue connect but essentially it is still all reduced and cannot leverage the heterogeneous resources another dimension is using new hardware or architecture for dn training for example there are many new ai chips such as tpu and habana and in fact ips design is generic and can also apply to these chips some researchers use new architecture including using infiniband switch asic to perform in-network or reduce or using p4 switch to perform in network ps and using a red scale dedicated servers with multiple nics to accelerate the communication but they require special redesign of the hardware or architecture and in our work we focus on using more generally available devices to conclude ips is a unified system for distributed dna training acceleration you optimize the intel machine and intro machine communication and address the cpu bottleneck with the simulation service abstraction it has been deployed in buy-downs for many training tasks including cv and nlp it is open it is also open source at github with that i'm happy to take any questions thank you