Add Byline Validated with airSlate SignNow

Add byline reply

all right well let's get started uh my name is ken archer and i'm the vice president of product at upwave i am joined here by miles younger who's a senior director of the global data practice at mighty hive part of s4 capital and we are here to talk about a super timely topic the impact of the idf idfa opt-in on measurement so as we're going through our presentation as you have questions feel free to enter them into the q a and miles and i will address them at the close the presentation should take about 15 to 20 minutes and then we'll do some uh q a so uh this webinar is for you you know you're in the right place if uh privacy safe tracking is a priority uh for you in 2021 and um all uh surveys indicate that um after covid uh this is the number one priority uh for advertisers uh this year and also if you're concerned uh with the drawbacks of person-level tracking and need to know viable alternatives so uh what are we talking about here so uh when it comes to uh measurement we are um obviously losing device level identifiers we're losing maids we're losing cookies and what has really pushed the envelope and forced the hand of measurement companies has been apple's announcement apple's announcement uh first that they are uh zeroing out idfas and changing idfa tracking to being uh opt-in um and uh while this is something that is um you know they've said will happen early this year uh what we and other measurement companies are seeing is this line chart you're seeing at the bottom is that the percent of ios devices with zeroed idfas is has already been increasing at an alarming extent right now 40 to 50 percent of ios traffic is not trackable um so but also apple has made it clear that email is not going to take the place of idfa so that we're here to talk about what are the options uh to device level tracking um first uh in terms of personal level tracking and what are those drawbacks and then we're given the the many drawbacks of personal level tracking uh we are gonna miles is going to talk about a really exciting development that um some innovative ad tech companies are working on around micro cohort level tracking so what are the limitations to personal level tracking at this point well there there's a lot of people uh see personal level tracking um as sort of the panacea uh for the loss of cookies and maids but it is really not when you dive into it um so the the two approaches that are being explored one is fingerprinting and you know the pro fingerprinting uh just to define this uh is um taking uh device characteristics specifically um the ip address plus information from the user agent that's part of the an http pixel request so ip address alone is not sufficient if you combine that with information from the user agent you can resolve an impression down to a specific device uh with a high level of accuracy um however this not only does this violate the terms of all operating system and browser platforms um these platforms uh have are are committed to winning a cat and mouse game over fingerprinting and making it technically impossible to do fingerprinting um there are still some measurement platforms that are doing fingerprinting generally when you hear them talking about some type of probabilistic identifier that uses machine learning and then there's just hand waving after that point that's a pretty good sign that they're doing fingerprinting and that is not a future proof solution but that the second is hashed emails so um there are a lot of companies that are focused on hashed emails and here we're not talking about hashed emails for targeting uh for for media buying we're talking about hashed emails as a solution to measurement um and uh what what we have found is that hashed emails actually are not a panacea for the loss of cookies and maze they are not a one-to-one replacement uh for the following reason one is they do not scale um so 90 or more of internet traffic is not logged in um that's so most of your programmatic buy uh is not going to be uh logged in but even a large portion of your direct buys um is inventory that is not logged in um also uh for users that are logged in most media companies don't have the technical ability to manage hashed email exposure data so you're really just talking about a small subset of inventory from the largest media companies that are able to share this kind of exposure data with third-party measurement companies um so it doesn't scale the second problem is apple has announced that hashed email sharing by ios mobile publishers will violate its privacy policy uh beginning early this year so uh we have talked to multiple major uh media companies um that are saying they are terrified of being kicked off the app store um and will not be sharing exposure data keyed against uh hashed emails with uh third-party measurement companies so this lack of scale this loss of ios the problem with that from a measurement perspective is that it makes incremental incrementality impossible because you have massively contaminated control groups at that point so yes you may still get some smaller subset of exposure data but without with once you've lost the ability to have a non-contaminated control group it's really difficult to do measurement um because you can't make any claims to causality so that's the basic um problem that we see with different solutions out there that claim to be uh one-to-one replacements of cookies and made so that what we need to do at this point um is really back up uh as an industry and say why was one-to-one tracking important in the first place um so when we started uh doing uh digital measurement why was this exciting uh from a measurement perspective what uh what options did this give us one-to-one tracking that we didn't have before well these graphics are meant to illustrate um why one-to-one tracking was important uh when you before digital uh advertising measurement was reliant on top-down uh media mixed data so basically looking at data by month by market and trying to understand which media driver was most correlated with with sales or or brand metrics um and the problem with this is that at this hot this higher level of data it's very hard to tease apart which media driver is uh driving sales or driving brand impact in a short period of time it basically it takes a longer period of time to collect more and more of this data before you can you can get any kind of a read on what media drivers are driving brand effectiveness uh the the st the stats word for this is multi this is the challenge of multi collinearity you have highly correlated uh media driver data at this uh top-down level from a but then if we have bottoms up media mixed data well then you don't have that problem so with the granular bottoms up data um of the type that's yielded by digital advertising it's easier to isolate and measure uh the causal drivers of ad effectiveness so this is uh really why one-to-one tracking is important um so the big point we want to make here is that the data set doesn't need to be person level to overcome multicollinearity and deliver the benefits of bottoms up measurement we really think this is going to be the linchpin to opening up new innovations and privacy safe measurement when when we realize what we're really trying to do here is overcome this challenge of multi-linearity do to do bottoms-up measurement so a small set of individuals can still yield enough variation in drivers for a measurement model to parse out the incremental impact of each driver in fact uh what we're starting to learn is perhaps this is what we were doing all along and just didn't know it and that's due to research that's coming out on device sharing where we're seeing that due to the large uh large amount of device sharing that happens within households we've act we actually haven't been doing one-to-one tracking all along we actually have been doing tracking um actually at the household level um so this opens up a lot of new opportunities uh for measurement between the top-down and traditional bottoms-up approach that miles will then talk through so miles with that i will turn it over to you cool cool thanks a lot ken um and thanks a lot for having me uh i appreciate you having me on um so yeah i'm gonna talk about cohorts specifically as a a potential solution uh to this kind of measurement conundrum that uh that we find ourselves in and you know i i'm gonna drop something into the chat this is actually um kind of a uh an add-on to a byline that ken and i published in in ad exchanger back in november december um so anyway i would i would recommend giving it a read because it's it's basically the the condensed written uh version of what i'm about to present um and ken since you're driving the slides you wouldn't mind skipping ahead one here so if we we start with this kind of goldilocks problem that uh ad tech and digital ads measurement um have had which is uh we've had you know these very powerful top-down measurement techniques including media mix modeling we've had very powerful bottoms up measurement techniques which would include you know multi-touch attribution log-level analytics and you know using both of these in combination you can get a lot of insight the problem is they kind of exist at extreme ends on a spectrum and and there hasn't really been a lot of focus on the middle ground and digital advertising especially has put a tremendous amount of focus on uh on bottoms up measurement um and you know both of these techniques have various pros and cons uh you know bottoms up you can measure within a channel very effectively with with techniques like mta uh something like media mix modeling measures across channels very effectively uh and then media mix modeling has a great view of like exogenous drivers so i don't know anything like you know the weather or interest rates consumer sentiment etc on the other hand bottoms up measurement gets none of those insights it's only getting digital touch points so anyway they've got lots of pros and cons but they exist at both ends of a spectrum and ken if you want to advance one slide um the problem that we now run into of having kind of invested so much at both ends of the spectrum both ends of this spectrum is that we're losing now this micro user level end of the spectrum due to things like you know third-party cookie blocking idfa removal you know double-click redacting log files differential privacy that doesn't reveal user level analytics and so forth we're basically starting to lose that that end of the spectrum there's more and more holes in it for media measurement um and so measuring at a cohort level presents this possible middle middle path where uh we don't necessarily need user level data um it's privacy friendly etc uh and it it also has certain pros and cons uh you know it could potentially matter measure both within and across channels because of the way that you put cohorts together in the way you kind of build a taxonomy around them they can definitely be cookie-less and as we're going to talk about you can you can definitely measure them deterministically at the cohort level so you're not necessarily losing deterministic methodologies and of course this is all this is all a balanced breakfast so cohorts have their cons too you know they're also not a holistic uh view they don't give a holistic view on exogenous drivers uh and and they're definitely gonna require some more statistical expertise because as we're gonna talk about it they get a little bit this is this is gonna be a new thing for the industry to have to adopt so um anyway ken if you want to go to the next slide we can start to talk about what what cohorts are so traditionally if you were to measure a group of users you would represent that group of users with a using bottoms up measurement using a list of user ids whether these are hashed emails cookie ids ip addresses fingerprints etc this is how you would represent them and if you want to just advance one again there's a little slide build here a cohort is just a grouping of those users and so it's this observable and distinguishable group where the the underlying individuals are otherwise basically undifferentiated or anonymous and so uh there's there's two other things i'll point out is one you're preserving user level anonymity so it's very privacy friendly and so um you can basically use it as a as this privacy-safe surrogate for measuring individuals and so within this cohort bubble here i i just as an example to help people understand sort of what a cohort would be versus what a collection of individuals is is let's just say that this cohort we're looking at is defined as 50 users saw ad placement x on business insider between 1 pm and 2 pm eastern on 26th of january 2021 we have no idea who those 50 people are but they do meet these distinguishable and observable characteristics um so ken if you want to go forward one more okay so let's just like dig in a little bit more on cohorts um um so they showed that they share these characteristics you know that it could be household zip plus four something like that we don't know exactly who's in them um so some characteristics may need to be modeled or estimated so this might be the number of individuals inside the cohort might be something like ad exposure probability you might need to estimate some of these characteristics um and then either you know within a cohort or across multiple cohorts or across multiple cohort schemes the number of member individuals could vary so you know you could have some cohorts with thousands of people in them you could have some cohorts possibly with you know dozens maybe down to the household level it can vary that's not part of the the definition uh and lastly it's just a very privacy friendly uh methodology um ken next next slide if you will now on to micro cohorts so this is what ken and i wrote about in ad exchanger and this is a pretty interesting area if you consider that a cohort could be anything up to tens hundreds of thousands of people potentially and there's only so much you could do with that granularity there's a lot of innovation going on right now with these so-called micro cohorts and they're basically just small cohorts so this could be say 50 individuals which happens to be the aggregation threshold in adds data hub but you get that small over the entirety of a campaign and a high degree of measurement granularity becomes possible because if you have a large enough number of small micro cohorts you get enough variance in your data set that you can start to use these deterministic bottoms up measurement approaches that frankly are quite familiar to the industry um next slide please ken all right so let's let's dig a little bit deeper on micro cohorts um so getting back to what ken was saying about contamination and bias because your cohorts aren't at the user level you don't have to worry as much about uh you know cookie or id coverage within the cohort because a lot of times if you're using user level data you literally don't know who isn't or in who is or is not in your data set whereas if you cohort them together properly using what you can observe you remove that contamination secondly you get enough variance in your data set you get enough micro cohorts you can measure causal drivers you can measure incrementality the same as you could measure using one-to-one methods and then there's just there's a lot of flexibility here you know how you define a cohort is not set in stone you know it's basically some recognizable observable consistent dimension that you can use to group users so this could be geo-based it could be utm parameters it could be based on ip address anything like that next slide please so just again to echo some stuff that ken said earlier it's just important to note that deterministic measurement as we see with micro cohorts it doesn't have to be at a one to one level you can basically kind of go one aggregation step up the spectrum to the micro cohort level and use many of the same techniques and frankly get just as powerful of of results so next slide please all right so um i i think this is my last slide just just to kind of wrap up um this look at cohorts i'm gonna these are a couple of examples just to to help the audience kind of get a better handle on what cohorts are or what cohorts could be in the future given that this is still relatively relatively new so let's think about different types i'm going to start in the lower right so that i can end on my my highlighted one um so the lower right we have probabilistic cohorts so you could even and this could be with first party data second party did a third party data you could define a cohort as some sort of ephemeral probability within a larger data set so you know a lot of like top-down modeling does this already where you can't say you know out of a thousand users we estimate 50 were exposed to x we will never know which 50 but we can use that probability as an input into into our model uh if we go up to the upper right we've got aggregate cohorts so you know these could be these are user level data but they're undifferentiated so for instance if you look at your site analytics and you only look at your utm data you know you have a thousand users coming in on a particular day for a particular campaign that could be a cohort but if you don't have any other dimensions those users are completely undifferentiated to you and that could be fine in certain circumstances and then also you know aggregate could be first party second party third party you have a lot of flexibility there so moving over to the upper left you can have first party deterministic cohorts so you know these are this is sort of like the most airtight cohort you could possibly have so these are cohorts you were building using your own deterministic first party data meaning you do actually know who's in the cohort but you need to cohort your users to perhaps make it fit into a model you're trying to run you might need to do it to normalize your first party cohorts against something that is on a second party or a third party basis you know netflix is is pretty famous for doing pretty sophisticated things with uh cohorting their own audience data so that they can build uh content preferences and and not have you know uh so many variants to deal with that their models just completely break down all right so last one is and this is the one with the star on it second party deterministic cohort so this is where there's a lot of interesting stuff happening so these are cohorts that are built from another data owner's first party user level data so like the underlying data is deterministic but you the consumer so whether you're a measurement vendor whether you're a brand whether you're an agency whether you're an ad tech company you don't have any access to that user level data but it does exist and so what you do have access to are cohorts often micro cohorts so a couple of examples of technologies that support this right now and these are only a few it's it's a growing list um so google's ads data hub is probably the most famous that you know like i mentioned they have an aggregation threshold of 50 users which would certainly uh fit into the category of micro cohort um amazon marketing cloud uh you know it's been in beta for a long time they just recently publicly announced it i'm sorry i don't know the aggregation threshold off the top of my head uh infosum i mean this is basically like you know build your own data clean room so this is you know democratizing what ads data hub or amazon marketing cloud do for those respective walled gardens infosum is enabling anybody in the industry to collaborate with data often at the cohort level because it is so effective at preserving privacy and preserving user anonymity and then lastly new star fabric there's been a lot of news around a new star fabric um you know you read some of their stuff and they mention an aggregation threshold of a hundred users uh which is you know likely driven by other technologies that they need to normalize against um um so anyway there's there's a lot of interesting stuff going on here with with micro cohorts i i would you know encourage you to read that ad exchanger piece um you know start following some of these technologies because it seems pretty clear that micro cohorts are a key direction that the industry and that measurement are are headed in uh and with that ken uh i will i will hand these back things out we've got some questions so i'm just kind of going to jump to the end um so that we can um we can feel them so there's a lot of interest in this topic um as is evidenced by this um registrations for this webinar uh so the first question that came in is um how do you do this in clean rooms um so people who have access to a clean room like what should they be looking for so with a clean room um you know and not necessarily every clean room is going to use this methodology but if you take like google's ads data hub which is by far the most heavily documented clean room out there i encourage you to go look at their documentation they have a lot of it um you know they're using uh what's known as differential privacy so they've got kind of a couple of things going on to make sure that you never see more than a cohort you're never going to get down to the user level with ads data that's the whole point of it is they're using differential privacy which is you know has some aggregation thresholds basically you're seeing data intersections and then they're adding a little bit of statistical noise basically to make sure that you could never possibly reverse engineer down to a user level by say running several different queries and then triangulating the results and so um what you're going to get out of that are cohorts at you know a minimum threshold of 50 users and so what you need to do is if you're running any measurement models etc you need to not think in terms of user level normalization you need to think of how can i normalize my data from my model at this threshold of 50 users or more you know what am i going to normalize against is it going to be geography is it going to be site is it going to be device type you know inside of ads data hub you have access to or google has access to a tremendous amount of user level data and so the question with ads data hub is really how can you the user get better at asking the right questions to get the right cohorts that you need because google almost certainly has the information you need they just can't expose it to you anymore uh in that at that user level um i don't know ken if you have anything to add well that actually is a great transition to the next question which is um uh cross channel measurement um is a very high priority doesn't this break cross-channel measurement so if you have different clean rooms you know for different media platforms for example and you can't normalize at the person level across media platforms instead at this micro cohort how can you do cross media uh measurement i've got some thoughts well yeah so i mean this basically gets back to that that sort of quad slide that i presented with these four different types of of measurement cohorts where you're going to have to start to mix and match those types right where you just aren't going to know at a user level you know a campaign that i ran on facebook and a campaign that i ran on amazon a campaign that i ran on google did i reach what was my you know unique reach what was my overlap you may not ever know that entirely but if you build proper cohorts and i mean maybe you're doing it at a basic like zip code level and you can then run some experiments that allow you to you know at least estimate your audience overlap across those those platforms and so you can start to blend these cohorts together um probabilistically um and if you're if you're normalizing over something that those three platforms do share which they share a lot if you think about media exposure they share geo they share time of day they share websites they share all sorts of things uh demographics um um there are all sorts of things you can share across uh at a cohort level anyway yeah it can i'm curious if you have any thoughts on that yeah so well obviously having like defining your micro cohorts the same way like at the zip 4 level or zip codes it's a spot group level across clean rooms across media platforms but also and this is this gets to your very interesting you know probabilistic uh micro cohort type um one of the things we have to keep in mind uh is that while we we lose a little bit of granularity we gain an enormous amount of data in these clean rooms that otherwise we don't have access to and a lot of people um overlook that that we aren't we're actually gaining a lot more than we're we're losing uh with this approach and so what you can do is train a propensity to expo to be exposed model in one platform using model features say age gender geography that are shared across your media platforms and then simply uh ingest that model into your other media platforms so that way you can you can actually share media exposure data across media platforms but at this probabilistic uh cohort level uh rather than at the personal level so again this requires more comfort with using statistics for this but it actually enables us to to get to the same result but in a privacy safe way another question is this approach doesn't feel transparent um if i have a log level data i know what the data source is um how can i trust this we could do a whole webinar on the trustworthiness of log-level data of course yeah um so yeah i'm just i'm trying to think of i mean ken i don't know if you have like a quick answer to that i can go on about that for though well something off the top of my head is that um i mean there is a lack of transparent there has been a lack of transparency in uh the models that are used for ad effectiveness measurement uh for years because the focus has entirely been on the data set and not the model this is a point that you've made miles is that the um that the difference between say a model that uses uh shapley regression versus a model that uses random force uh i mean that will though that will result in a significant changes in uh how you're handling uh your your data how you're measuring your media because you're in taking different approaches to the underlying problem of what in data science is called overfitting of a model it actually results in different answers and you know what what i just said sounds pretty opaque to most people you know but uh it's just sort of hand waved away so this really isn't adding any opacity that wasn't already there um you just have to have statisticians on your team uh who unders just like you have software engineers on your team uh who understand your organization's mission the the other thing that i'll add um just to kind of broaden the scope here is that you know yes there's definitely opacity issues with the cohort because you don't know exactly who's in it it's not entirely deterministic uh you can't deny that there's there yeah there's some opacity there but um there's two things is going back to the slide where we looked at sort of this you know macro micro and then cohort level the goldilocks uh scenario i mean it really is this balanced breakfast approach that you have to take so you know you can certainly um uh gut check what your cohort models are telling you with your macro models because your macro models are you know especially if you're measuring against things like a sales volume sales revenue store traffic things that can't be faked those are very good benchmarks and then if you can also down at the person level uh benchmark things or test things or or calibrate things against your own first party data which again shouldn't be lying to you because it's your own data um you can at least triangulate against the truth and i and identify any transparency issues or inaccuracy excuse me inaccuracies in in your micro cohort level outputs right right you're right we didn't get to that is validating these micro cohort based models against what personal level data you do have yeah well great well let's wrap it up uh with that we try to keep these at 30 minutes so we've gone a couple minutes over uh thanks everyone for attending um also check out um the quarterly up wave masters in measurement uh panel is coming up in a couple weeks on uh is short termism a problem in advertising uh so check that out at upwave.com and uh thanks so much for joining miles and i and miles thanks so much uh for uh joining up wave today at this webinar i really enjoyed hearing your insights and learning from thanks a lot for having me ken all right