Fax Diverse Conditional with airSlate SignNow

Fax diverse conditional

hello everyone my name is turbo from the Chinese University of Hong Kong and that this work is about generating diverse and natural captions for image captioning and for those status in all the familiar with image captioning this page show how state of the art model works it will extract some features and then generate captions word by word instead of the our models are already capable of producing very good captions however there are problems to show that imagine a robot that can describe policies using state-of-the-art state-of-the-art captioning models now here is an example of it describes a snow board game the first player he is flying through the air while riding a snowboard and the second player he is flying through the air while riding a snowboard oh the third player he is flying through the air while riding a snowboard as you can see a test to produce identical captions for similar images which is dry and boring sometimes this may also lead to ambiguity and these problems are not surprising because the generation and evaluation for current image captioning models are based on engrams space which is only a small subset of the semantic space specifically when we train captioning models usually we use maximum likelihood estimation which computes some of log likelihoods of each word condition the input image and the race previous words and such objective were encouraged model 2 follows the detail warnings of chaining captions as a result for this sample image with conscious Academy sitting in English Inc the board train using mue will give much higher probability to the first caption than second one although the second one has all indifference relating words consequently with similar images are presented same caption is very likely to be produced again again following the same wording so how to solve this to solve that people know this that evaluation metrics such as blue and cider on the other hand think there's two captions are both good because they're mashing engrams with their own juices regardless of the positions of those engrams in the original sentences so inspired by this recent works proposed to directly or optimize on the evaluation matrix by it by reinforcement learning subset Lee the caption generation is regarded as a sequential action selection process and guided by the evaluation matrix as the reverse and when training the caption generator were sample captions and aim at maximizing the expected reward and by using the matrix as global guidance to avoid mimicking the detailed warnings of chaining captions however this is still not enough because they're mashing engrams with the ground shoes they will prefer captions that contains more frequent training and grams rather than the captions that are semantically similar but syntactically different then how to solve this to solve that we revealed goals of caption generation and evaluation and noticed generation is to generate captions there as human-like as possible and the evaluation is to pick out those bad machine generated captions in fact during a diverse real relationship and more importantly this relationship is in the semantic space it's exactly what we need so we proposed to use a condition again to join him deal with caption generation and evaluation supposedly our approach also uses reinforcement learning to avoid technical issues such as non differentiable sampling operations but instead of using the evaluation metrics as rewards we use a parametric evaluator that takes an image and the caption as inputs and outputs a score branching in 0 & 1 as an estimation for the caption quality and by using this evaluator as the guidance there are our caption generator is similar to the original structure but with this additional important eros and noise factor Z this by using this noise back to Z our caption generator can additionally generate more diverse captions because we can sample different Z to influence the caption generation process and by joint retraining the caption generator and the evaluator we can generate diverse captions and estimate their qualities to show that let's look has some results on the MS koko della said and here's some example details of our experiments in all of our experiments we used encode and decode a structure as our model and we use 1516 as the encoder and the airstream net as the decoder and before adverse training we were respectively preaching the generator and the evaluator has full results for similar images the motor channel using game can produce more distinctive captions and one can easily pick out original images by just looking at captions as a for single image we can sample different Z to get different captions cover covering different details of the original image and those captions are semantically similar but has less large virus in terms of syntax and as mission before the evaluation metrics such as blue and cider on our design for doing evaluation in terms of semantic qualities so we use user studies to compare different methods and users asked to set a better one from pairs of captions and compared to model channel using emoji the same model trained using again can win 61 percent of the time indicated produce better captions however when compared to human both can marry a far from competitive indicates they are still click clear gaps between machine generated captions and natural ones and interestingly if we compute the evaluation scores using the evaluation match evaluation matrix we will found that the matrix such as blue and cider prefers Emily over human by a large margin while I will learn evaluator and prefers human over both Emily energy and again specifically Gary's in the image with two captions where the second one is relatedly more informative however the classical matrix will give much higher scores to the first one as it contains more frequent training and grants except for our evaluator which gives relatively better scores and of course and model trained using games not perfect sometimes there'll be arrows in the generated captions and the main type of arrow is rowing details and here are some examples and our guess is that in order to get high score in terms of semantics qualities the caption generator is encouraged to include more details or taking the risk some of them may be incorrect until we conclude we joined to deal with caption generation and evaluation using a condition again and by that we can change the caption immortals in a semantic space and we can also provide a parametric estimation for caption qualities in terms of semantic relatedness and by focusing on the semantics rather than syntax we can generate diverse and natural captions thank you [Applause] thank you any questions alright I have a question so you mentioned using reinforcement learning in order to avoid some of the differentiability issues with your gun model so can you give us some insight why reinforcement learning would help you or is reinforcement learning the reinforcement learning method you used also not using sampling for you to compute expectations yeah yeah you know in implementation we charge sample captions to estimator expectations so this operation is not differentiable so we have to resolve this issue by the help of techniques such as reinforcement learning I see but in reinforcement learning in the reinforcement learning method you use you don't use something like reinforce so what type of method you use for example colorado's okay