Write Human Zip with airSlate SignNow

Hirsch zipf form

hey Vsauce Michael here about 6% of everything you say and read and write is the the is the most used word in the English language about one out of every 16 words we encounter on a daily basis is the the top 20 most common English words in order are the of and to a in is I that it for you was with on as have but be they that's a fun fact a piece of trivia but it's also more you see whether the most commonly used words are ranked across an entire language or in just one book or article almost every time a bizarre pattern emerges the second most used word will appear about half as often as the most used the third one third is often the fourth one fourth as often the fifth one fifth is often the sixth 1/6 as often and so on all the way down seriously for some reason the amount of times a word is used is just proportional to one over its rank word frequency and ranking on a log-log graph follow a nice straight line a power-law this phenomenon is called zips law and it doesn't only apply to english it also applies to other languages like well all of them even ancient languages we haven't been able to translate yet and here's the thing we have no idea why it's surprising that something as complex as reality should be conveyed by something as creative as language in such a predictable way how predictable we'll watch this according to word count dot org which ranks words as found in the british national corpus sauce is the five thousand five hundred and fifty fifth most common English word now here is a list of how many times every word on Wikipedia and in the entire Gutenberg corpus of tens of thousands of public domain books shows up the most used word the shows up about 181 million times knowing these two things we can estimate that the word sauce should appear about 30 thousand times on Wikipedia and Gutenberg combined and it pretty much does what gives the world is chaotic things are distributed in myriads of ways not just power laws and language is personal intentional idiosyncratic what about the world and ourselves could cause such complex activities and behaviors to follow such a basic rule we literally don't know more than a century of research has yet to close the case moreover Zips law doesn't just mysteriously describe word use it's also found in city populations solar flare intensities protein sequences and immune receptors the amount of traffic websites get earthquake magnitudes the number of times academic papers are cited last names the firing patterns of neural networks ingredients used in cookbooks the number of phone calls people receive the diameter of moon craters the number of people that die in wars the popularity of opening chess moves even the rate at which we forget there are plenty of theories about why language is zippy but no firm conclusions and this video doesn't contain a definite explanation either sorry I know that's a bummer since we appear to like knowing more than mystery but that said we also asked more than we answer so let's dive in to zips ramifications some related patterns some possible explanations and the depth of the mystery itself zip slaw was popularized by George Zipp a linguist at Harvard University it is a discrete form of the continuous Pareto distribution from which we get the Pareto principle because so many real-world processes behave this way the Pareto principle tells us that as a rule of thumb it's worth assuming that 20% of the causes are responsible for 80% of the outcome like in language where the most frequently used eighteen percent of words account for over 80% of word occurrences in 1896 vilfredo pareto showed that approximately 80% of the land in Italy was owned by just 20% of the population it is said that he later noticed in his garden 20% of his pea pods contained 80% of the Peas he and other researchers looked at other datasets and found that this 8020 imbalance comes up a lot in the world the richest 20% of humans have 80 2.7 percent of the world's income in the u.s. 20% of patients use 80 percent of healthcare resources in 2002 Microsoft reported that 80% of the errors and crashes in Windows and Office are caused by 20% of the bugs detected a common rule of thumb in the business world states that 20 percent of your customers are responsible for 80% of your profits and 80 percent of the complaints you receive will come from 20% of your customers a book titled the 8020 principle even says that in a home or office 20 percent of the carpet receives 80 percent of the where oh and as Woody Allen famously said 80 percent of success is just showing up the Pareto principle is everywhere which is good by focusing on just 20% of what's wrong you could often expect to solve 80% of the problems a variety of different unrelated factors cause this to be true from case to case but if we can get to the bottom of what causes some of them maybe we'll find that one or more of those mechanisms is responsible for zips law in language George Zipp himself thought languages interesting ranked frequency distribution was a consequence of the principle of least effort the tendency for life and things to follow the path of least resistance Ziff believed it drove much of human behavior and hypothesized that as language developed in our species speakers naturally preferred drawing from as few words as possible to get their thoughts out there it was easier but in order to understand what was being said listeners preferred larger vocabularies that gave more specificity so that they had to do less work the compromise between listening and speaking zip felt led to the current state of language a few words are used often and many many many words are used rarely recent papers have suggested that having a few short often used predictable words helps dissipate information load density on listeners spacing out important vocab so that the information rate is more constant this makes sense and much has been learned by applying the least effort principle to other behaviors but later researchers argued that for language the explanation was even more simple just a few years after zips seminal paper benoit mandelbrot showed that there may be nothing mysterious about zips law at all because even if you just randomly type on a keyboard you will produce words distributed according to zips law it's a pretty cool point and this is why it happens there are exponentially more different long words than short words for instance the English alphabet can be used to make 26 one letter words but 26 squared two letter words also in random typing whenever the spacebar is pressed a word terminates since there's always a certain chance that the spacebar will be pressed longer stretches of time before it happens are exponentially less likely than shorter ones the combination of these Exponential's is pretty zippy for example if all 26 letters and the spacebar are equally likely to be typed after a letter is typed and a word has begun the probability that the next input will be a space thus creating a one letter word is just one in 27 and sure enough if you randomly generate characters or hire a proverbial typing monkey about one out of every 27 or three point seven percent of the stuff between spaces will be single letters two letter words appear when after beginning a word any character but the space bar is hit a 26 and 27 chance and then the space bar a three letter word is the probability of a letter another letter and then a space if we divide by the number of unique words of each length there can be we get the frequency of occurrence expected for any particular word given its length for example the letter V will make up about point 1 42 percent of random typing the word Vsauce point zero zero zero zero zero zero zero zero nine nine three percent longer words are less likely but watch this let's spread these frequencies out according to the ranks they'd take up on a most often used list there are 26 possible 1 letter words so each of the top 26 ranked words are expected to occur about this often the next 676 ranks will be taken up by two letter words that show up about this often if we extend each frequency according to how many members it has we get zip subsequent researchers have detailed how changing up the initial conditions can smooth the steps out our mysterious distribution has been created out of nothing but the inevitabilities of math so maybe there is no mystery maybe words are just the result of humans randomly segmenting the observable world and the mental world into labels and zips law describes what naturally happens when you do that case closed and as always thanks for what wait a minute actual language is very different from random typing communication is deterministic to a certain extent utterances and topics arrive based on what was said before and the vocabulary we have to work with certainly isn't the result of purely random naming for example the monkey typing model can't explain why even the names of the elements the planets end the days of the week are used in language according to zips law sets like these are constrained by the natural world and they're not the result of us randomly segmenting the world into labels furthermore when given a list of novel words words they've never heard or used before like when prompted to write a story about alien creatures with strange names people will naturally tend to use the name of one alien twice as often as another three times as often as another zips law appears to be built into our brains perhaps there is something about the way thoughts and topics of discussion a band flow that contributes to zips law another way zippy and distributions occur is via processes that change according to how they've previously operated these are called preferential attachment processes they occur when something money views attention variation friends jobs anything really is given out according to how much is already possessed to go back to the carpet example if most people walk from the living room to the kitchen across a certain path furniture will be placed elsewhere making that path even more popular the more views a video or image or post has the more likely it is to get recommended automatically or make the news for having so many views both of which give it more views it's like a snowball rolling down a snowy hill the more snow it accumulates the bigger its surface area becomes for collecting more and the faster it grows there doesn't have to be a deliberate choice driving a preferential attachment process it can happen naturally try this take a bunch of paper clips and grab any two at random link them together and then throw them back in the pile now repeat over and over again if you grab paper clips that are already part of a chain link them anyway more often than not after a while you will have a distribution that looks as if Ian's a small number of chains contain a disproportionate amount of the total paper clip count this is simply because the longer a chain gets the greater proportion of the whole it contains which gives it a better chance of being picked up in the future and consequently made even longer the rich get richer the big get bigger the popular get popular it's just math perhaps languages Ziff mystery is if not caused by at least strengthened by preffer Rachele attachment once a word is used it's more likely to be used against soon critical points may play a role as well writing and conversation often stick to a topic until a critical point is reached and the subject is changed and the vocabulary shifts processes like these are known to result in power laws so in the end it seems tenable that all these mechanisms might collude to make Zips law the most natural way for language to be perhaps some of our vocabulary and grammar was developed randomly according to Mandelbrot's theory and the natural way conversation and discussion follow preferential attachment and criticality coupled with the principle of least effort when speaking and listening are all responsible for the relationship between word rank and frequency it's a shame that the answer isn't simpler but it's fascinating because of the consequences it has on what communication is made of roughly speaking and this is mind-blowing nearly half of any book conversation or article will be nothing but the same 50 to 100 words and nearly the other half will be words that appear in that selection only once that's not so surprising when you consider the fact that one word accounts for six percent of what we say the top 25 most used words make up about 1/3 of everything we say and the top 100 about half seriously I mean whether it's all the words in wet hot American summer or all the words in Plato's Complete Works or in the complete works of Edgar Allen Poe or the Bible itself only about 100 words are used for nearly half of everything written or said in Alice's Adventures in Wonderland 44 percent and in Tom Sawyer forty-nine point eight percent of the unique words used appear only once in the book a word that is used only once in a given selection of words is called a hey pax legume anon hey pax lag Amina are vitally important to understanding languages if a word has only been found once in the entire known collection of an ancient language it can be very difficult to figure out what it means now there is no corpus of everything ever said or written in English but there are very very large collections and it's fun to find hey pax lagaa Mona in them for instance and this probably won't be the case after I mention it but the word quiz Aish ously is in the Oxford English Dictionary but appears nowhere on Wikipedia or in the gutenberg corpus or in the british national corpus or the american national corpus but it does appear when searched in just one result on Google fittingly in a book titled elder speak that lists it as a rare word quiz Aish asleep by the way means in a mocking manner as in the parodist rattled off quiz Asia Slee hey Vsauce Michael here but who is Michael and how much does he weigh it's a little sad that quiz Asia Slee has been used so infrequently it's a fun word but that's the way things go in a zip ian's system some things get all the love some get little most of what you experience on a day to day basis is forgotten forgettable the dictionary of obscure sorrows as it often does has a word for this Oh Lika the awareness of how few days are memorable I've been alive for almost 11,000 days but I couldn't tell you something about each one of them I mean not even close most of what we do and see and think and say and hear and feel is forgotten at a rate quite similar to zip slaw which makes sense if a number of factors naturally selected for thinking and talking about the world with tools in a zip Ian way it makes sense we'd remember it that way to some things really well most things hardly at all but it bums me out sometimes because it means that so much is forgotten even things that at the time you thought you could never forget my locker number senior year it's combination the jokes I liked when I saw a comedian on stage the names of people I saw every day 10 years ago so many memories are gone when I look at all the books I've read and realize that I can't remember every detail from them it's a little disappointing I mean why even bother if the Pareto principle dictates that my zippy in mind will consciously remember pretty much only the titles and a few basic reactions years later Ralph Waldo Emerson makes me feel better he once said I cannot remember the books I've read any more than the meals I have eaten even so they have made me and as always thanks for watching Music